# Story 10.11: Benchmark Script Key Alignment

**Status:** Complete
**Priority:** Medium
**Depends on:** None

---

## Context / Background

The GPT-5.2 audit identified that `scripts/benchmarking.py` has inconsistent key references between the `benchmark()` function output and the `derive_verdicts()` function that consumes it.

**Current problem in `scripts/benchmarking.py:349-351`:**

```python
def derive_verdicts(results: dict[str, object]) -> None:
    throughput = results["throughput"]
    latency = results["latency_us"]
    memory = results["memory_peak_bytes"]

    speedup = float(throughput["speedup_factor"])      # KEY DOESN'T EXIST
    lat_red = float(latency["reduction_pct_avg"])      # KEY DOESN'T EXIST
    mem_red = float(memory["reduction_pct"])           # KEY DOESN'T EXIST
```

**Actual keys from `benchmark()` (lines 183-234):**

```python
results["throughput"] = {
    "speedup_factor_off": ...,   # Not "speedup_factor"
    "speedup_factor_on": ...,
    ...
}
results["latency_us"] = {
    "reduction_pct_avg_off": ...,  # Not "reduction_pct_avg"
    "reduction_pct_avg_on": ...,
    ...
}
results["memory_peak_bytes"] = {
    "reduction_pct_off": ...,  # Not "reduction_pct"
    "reduction_pct_on": ...,
    ...
}
```

**Impact:**
- Running `python scripts/benchmarking.py` raises KeyError
- README performance claims cannot be validated
- Undermines trust in benchmark methodology

---

## Scope (In / Out)

### In Scope

- Fix key references in `derive_verdicts()` to match actual output
- Decide which variant to use (_off or _on) for claims evaluation
- Add smoke test to CI to catch future regressions
- Verify script runs end-to-end

### Out of Scope

- Changing benchmark methodology
- Adding new performance claims
- Comprehensive benchmark CI (too slow)

---

## Acceptance Criteria

### AC1: Script Runs Without KeyError

**Description:** `python scripts/benchmarking.py --iterations 100 --latency-iterations 50` completes without errors.

**Validation:**
```bash
python scripts/benchmarking.py --iterations 100 --latency-iterations 50
echo $?  # Should be 0
```

### AC2: Verdicts Use Consistent Keys

**Description:** `derive_verdicts()` uses the `_on` variant (serialize_in_flush=True, the optimized path) for claim evaluation.

**Validation:**
```python
# In derive_verdicts:
speedup = float(throughput["speedup_factor_on"])
lat_red = float(latency["reduction_pct_avg_on"])
mem_red = float(memory["reduction_pct_on"])
```

### AC3: Evaluation Table References Correct Metrics

**Description:** The claims evaluation table shows metrics from the chosen variant.

**Validation:**
```python
# Claims should reference fapilog_on_logs_per_sec, not fapilog_logs_per_sec
f"fapilog {throughput['fapilog_on_logs_per_sec']:.1f} logs/s ..."
```

### AC4: CI Smoke Test Added

**Description:** CI runs a minimal benchmark to catch KeyErrors.

**Validation:**
```yaml
# In .github/workflows/ci.yml or similar
- name: Benchmark smoke test
  run: python scripts/benchmarking.py --iterations 100 --latency-iterations 50
```

---

## Implementation Notes

### File Changes

```
scripts/benchmarking.py (MODIFIED - fix key references)
.github/workflows/ci.yml (MODIFIED - add smoke test)
```

### Key Code Fix

```python
def derive_verdicts(results: dict[str, object]) -> None:
    throughput = results["throughput"]
    latency = results["latency_us"]
    memory = results["memory_peak_bytes"]

    # Use _on variant (optimized path with serialize_in_flush=True)
    speedup = float(throughput["speedup_factor_on"])
    lat_red = float(latency["reduction_pct_avg_on"])
    mem_red = float(memory["reduction_pct_on"])

    claims = [
        (
            "D001",
            "README.md:25",
            "50x throughput improvement over traditional logging",
            "High",
            "Remove unsubstantiated performance claims",
            speedup >= 50.0,
            f"fapilog {throughput['fapilog_on_logs_per_sec']:.1f} logs/s vs "
            f"stdlib {throughput['stdlib_logs_per_sec']:.1f} logs/s; "
            f"speedup {speedup:.2f}x",
        ),
        (
            "D002",
            "README.md:26",
            "90% latency reduction with async-first design",
            "High",
            "Remove unsubstantiated performance claims",
            lat_red >= 90.0,
            f"avg latency reduction {latency['reduction_pct_avg_on']:.2f}% "
            f"(median {latency['reduction_pct_median_on']:.2f}%, "
            f"p95 {latency['reduction_pct_p95_on']:.2f}%)",
        ),
        (
            "D003",
            "README.md:27",
            "80% memory reduction with zero-copy operations",
            "High",
            "Remove unsubstantiated performance claims",
            mem_red >= 80.0,
            f"peak memory reduction {memory['reduction_pct_on']:.2f}% "
            f"(stdlib {memory['stdlib']} B -> fapilog {memory['fapilog_on']} B)",
        ),
    ]
    # ... rest of function
```

---

## Tasks

### Phase 1: Fix Keys

- [ ] Update `speedup` to use `speedup_factor_on`
- [ ] Update `lat_red` to use `reduction_pct_avg_on`
- [ ] Update `mem_red` to use `reduction_pct_on`
- [ ] Update claim evidence strings to use correct keys

### Phase 2: Validation

- [ ] Run script locally end-to-end
- [ ] Verify output table is coherent
- [ ] Check JSON output structure

### Phase 3: CI Integration

- [ ] Add smoke test job to ci.yml
- [ ] Keep iterations low for speed (100/50)

### Phase 4: Documentation

- [ ] Update CHANGELOG

---

## Tests

### Manual Verification

```bash
# Full run
python scripts/benchmarking.py

# Quick smoke test
python scripts/benchmarking.py --iterations 100 --latency-iterations 50
```

### CI Smoke Test

- Job that runs minimal benchmark
- Fails on any Python exception

---

## Definition of Done

### Code Complete

- [ ] All key references fixed
- [ ] Script runs without errors
- [ ] Output is coherent

### Quality Assurance

- [ ] Local test passes
- [ ] CI smoke test added
- [ ] `ruff check scripts/` passes

### Documentation

- [ ] CHANGELOG updated

---

## Risks / Rollback

### Risks

1. **Risk:** Benchmark results differ from previous runs
   - **Mitigation:** This is expected; script was broken before

2. **Risk:** CI smoke test too slow
   - **Mitigation:** Use minimal iterations (100/50)

### Rollback Plan

If issues occur:
1. Revert script changes
2. Remove CI job

---

## Related Stories

- **Related:** README performance claims documentation

---

## Change Log

| Date | Change | Author |
|------|--------|--------|
| 2026-01-16 | Initial draft from GPT-5.2 audit | Claude |
