# ADR: Comparative Benchmark Suite Design

## Context

Task 18.2 required creating a comparative benchmark suite to measure jetliner performance against other Avro reading libraries.

## Decisions

### 1. Standalone Script vs pytest-benchmark

**Decision**: Use a standalone Python script (`benches/python/compare.py`) instead of pytest-benchmark.

**Rationale**:
- Benchmarks are not tests - they shouldn't be mixed with the test suite
- pytest-benchmark adds overhead from pytest machinery
- Standalone script gives full control over output format and comparison tables
- Can run directly: `python benches/python/compare.py` or via `poe bench-compare`

### 2. Statistical Approach

**Decision**: Use manual timing with `time.perf_counter` and `statistics` module instead of pyperf.

**Rationale**:
- pyperf's `Runner` class takes over argument parsing, making it difficult to add custom CLI args
- For comparative benchmarks, we need custom args (--scenario, --readers, --output)
- Manual approach with warmup runs + multiple timed runs provides sufficient statistical rigor
- Reports mean, stdev, median, min, max for each benchmark

**Configuration**:
- Quick mode: 1 warmup, 3 timed runs
- Default: 2 warmups, 5 timed runs
- Full mode: 3 warmups, 10 timed runs

**Outlier Detection**: Initially implemented IQR-based outlier detection, but removed because:
- With 5 samples (default), outlier % is always multiples of 20% (1 outlier = 20%, 2 = 40%, etc.)
- IQR method needs larger sample sizes to be statistically meaningful
- Standard deviation already captures variability
- Showing "20% outliers" is misleading when it's just 1 out of 5 measurements

### 3. Libraries Compared

**Decision**: Compare 6 libraries (expanded from original 5):
1. jetliner (our library)
2. polars.read_avro (built-in)
3. polars-avro (third-party IO plugin)
4. fastavro (pure Python)
5. fastavro + pandas (common pattern)
6. apache-avro (official library, package name: `avro`)

**Rationale**: polars-avro is a direct competitor as another Polars IO plugin for Avro, making it highly relevant for comparison.

### 4. Benchmark Scenarios

**Decision**: Focus on large-scale scenarios (1M records) that represent real-world usage.

**Scenarios**:
- `large_simple`: 1M records, 5 cols - raw throughput
- `large_wide`: 1M records, 100 cols - column handling
- `large_complex`: 1M records with realistic complex structures:
  * Nested records (2 levels deep: address with embedded coordinates)
  * Arrays of records (contacts with type, value, primary fields)
  * Arrays of primitives (tags, variable length 0-5)
  * Maps (metadata, variable length 0-3, including empty)
  * Nullable fields (coordinates present in ~50% of records, score in ~67%)
- `projection`: 1M records, select 5/100 cols - projection pushdown

**Removed**: Small scenario (100 records) - too fast to measure accurately, dominated by timing noise, not representative of real usage.

### 5. File Location

**Decision**: Place benchmarks in `benches/python/` alongside Rust benchmarks in `benches/`.

**Rationale**:
- Groups all benchmarks together
- Keeps benchmarks out of pytest test discovery
- `benches/data/` already gitignored for benchmark data files

### 6. Dependencies

**Added**:
- `pyperf` - initially considered but not used due to CLI limitations
- `polars-avro` - competitor library for comparison
- `avro` - official Apache Avro library (not `apache-avro`)

**Removed**:
- `pytest-benchmark` - not needed with standalone approach

## Poe Tasks

- `poe bench-compare` - Run default benchmarks (2 warmups, 5 runs)
- `poe bench-compare-quick` - Quick benchmarks (1 warmup, 3 runs)
- `poe bench-compare-full` - Full benchmarks (3 warmups, 10 runs)
- `poe bench-data` - Generate benchmark data files

## Error Handling

**Decision**: By default, catch exceptions and display errors in results. Add `--fail-fast` flag to abort on first error.

**Rationale**:
- Different readers have different capabilities (e.g., polars.read_avro() doesn't support Avro maps)
- Graceful error handling allows comparing readers across diverse scenarios
- Errors displayed in results table with RED "ERROR" marker
- `--fail-fast` mode useful for CI/CD to catch unexpected failures
- More flexible than maintaining manual skip lists

## JSON Output

**Decision**: Add `--output` flag to save complete structured JSON report with metadata, configuration, and all results.

**Structure**:
```json
{
  "metadata": {
    "timestamp": "2026-01-11T...",
    "python_version": "3.12.1",
    "platform": "macOS-...",
    "processor": "arm",
    "machine": "arm64"
  },
  "configuration": {
    "warmup_runs": 2,
    "timed_runs": 5,
    "fail_fast": false
  },
  "scenarios": {
    "large_simple": {
      "description": "1M records, 5 cols - raw throughput",
      "file_name": "large_simple.avro",
      "file_size_bytes": 12345678,
      "columns": null
    }
  },
  "results": {
    "large_simple": {
      "jetliner_scan": {
        "mean_sec": 0.123,
        "stdev_sec": 0.005,
        "median_sec": 0.122,
        "min_sec": 0.118,
        "max_sec": 0.130,
        "outliers": 0,
        "outlier_pct": 0.0,
        "runs": 5,
        "times": [0.118, 0.122, 0.123, 0.125, 0.130]
      }
    }
  }
}
```

**Features**:
- Complete data including raw timing arrays for detailed analysis
- Metadata includes system info for reproducibility
- Scenario metadata includes file sizes
- Ready for plotting, analysis, and tracking over time

**Rationale**:
- Enables tracking performance over time
- Structured format for generating plots and visualizations
- Includes context needed for reproducibility
- Can be used for regression detection in CI/CD
- Complete data export - no artificial limitations

## Usage

```bash
# Quick comparison of all libraries on all scenarios
poe bench-compare --quick

# Full benchmark with JSON output (includes all data)
poe bench-compare --full --output results.json

# Specific scenario and readers
poe bench-compare --scenario large_simple --readers jetliner_scan polars

# Abort on first error (useful for CI)
poe bench-compare --fail-fast

# CI/CD: Run benchmarks and save results for tracking
poe bench-compare --full --output "results/bench-$(date +%Y%m%d).json"
```
