# Story 10.19: Performance Regression Detection in Nightly CI

**Status:** Ready
**Priority:** Medium
**Depends on:** None

---

## Context / Background

The project has benchmark tests (`tests/benchmark/test_perf_benchmarks.py`) that measure performance of critical operations:
- JSON serialization throughput
- Ring queue enqueue/dequeue
- Sink write latency
- File I/O performance

**Current State:**
- Benchmarks run on push to main but only **report numbers**
- No automated detection of performance regressions
- Only two hard-coded assertions exist:
  - `assert benchmark.stats.stats.mean < 0.0001` (pretty format < 100μs)
  - `assert pretty_mean < json_mean * multiplier` (relative comparison)

**The Problem:**
Traditional benchmark regression detection compares against historical baselines, but this fails in CI because:
- Shared runners have unpredictable performance (noisy neighbors)
- Different runner hardware across runs
- 20-50% variance is normal on cloud CI

**The Solution:**
Use **runner-invariant** regression detection techniques:
1. **Relative comparisons** - Compare operations against each other in the same run
2. **Scaling tests** - Verify O(n) stays O(n), not O(n²)
3. **Ratio-based assertions** - "Operation A should be ≤ Nx slower than Operation B"

These techniques catch real regressions (algorithmic changes, accidental sleeps) while ignoring runner noise.

---

## Scope (In / Out)

### In Scope

- Add scaling tests that verify algorithmic complexity
- Add relative performance comparisons between operations
- Integrate into nightly CI workflow
- Add clear failure messages explaining what regressed
- Document the approach for future contributors

### Out of Scope

- Historical baseline tracking (requires dedicated infrastructure)
- Absolute time thresholds (too runner-dependent)
- Real-time performance dashboards (separate tooling story)
- Micro-optimization of existing code (separate stories)

---

## Acceptance Criteria

### AC1: Scaling Tests Detect Algorithmic Regressions

**Description:** Tests verify that operations scale linearly (or as expected) with input size. Catches O(n²) regressions.

**Validation:**
```python
def test_serialization_scales_linearly():
    """Serializing 10x more data should take ~10x longer, not 100x."""
    small_payload = {"key": "x" * 100}
    large_payload = {"key": "x" * 1000}

    small_time = benchmark(lambda: serialize(small_payload), rounds=100)
    large_time = benchmark(lambda: serialize(large_payload), rounds=100)

    # 10x more data should be ≤15x slower (allowing for overhead)
    # If it's 100x slower, we have an O(n²) regression
    ratio = large_time / small_time
    assert ratio < 15, f"Scaling regression: 10x data took {ratio:.1f}x longer (expected ≤15x)"
```

### AC2: Relative Comparisons Between Operations

**Description:** Compare related operations against each other to catch regressions in one path.

**Validation:**
```python
def test_json_vs_pretty_format_ratio():
    """Pretty formatting should be ≤5x slower than JSON serialization."""
    payload = {"level": "INFO", "message": "test", "data": {"key": "value"}}

    json_time = measure(lambda: json_serialize(payload))
    pretty_time = measure(lambda: pretty_format(payload))

    ratio = pretty_time / json_time
    assert ratio < 5, f"Pretty format regression: {ratio:.1f}x slower than JSON (expected ≤5x)"
```

### AC3: Queue Operations Scaling Test

**Description:** Verify queue enqueue/dequeue scales linearly with batch size.

**Validation:**
```python
def test_queue_scales_linearly():
    """Processing 10x more items should take ~10x longer."""
    queue = NonBlockingRingQueue(capacity=100000)

    time_1k = measure_enqueue_dequeue(queue, count=1000)
    time_10k = measure_enqueue_dequeue(queue, count=10000)

    ratio = time_10k / time_1k
    assert ratio < 15, f"Queue scaling regression: 10x items took {ratio:.1f}x longer"
```

### AC4: Nightly CI Integration

**Description:** Performance regression tests run in nightly CI and fail the build on regression.

**Validation:**
- `.github/workflows/nightly.yml` includes performance regression job
- Job runs `pytest tests/benchmark/ -m scaling` (or similar marker)
- Failure blocks nightly but doesn't block PR merges
- Slack/email notification on regression (if configured)

### AC5: Clear Regression Messages

**Description:** When a regression is detected, the error message explains what regressed and by how much.

**Validation:**
```
FAILED test_serialization_scales_linearly
AssertionError: Scaling regression detected!
  - Operation: JSON serialization
  - Input scaling: 10x
  - Expected time scaling: ≤15x
  - Actual time scaling: 47x
  - This suggests O(n²) behavior was introduced
```

---

## Implementation Notes

### File Changes

```
tests/benchmark/test_perf_benchmarks.py (MODIFIED - add scaling tests)
tests/benchmark/test_scaling_regression.py (NEW - dedicated scaling tests)
tests/benchmark/conftest.py (NEW - shared benchmark fixtures)
.github/workflows/nightly.yml (MODIFIED - add perf regression job)
docs/contributing/performance-testing.md (NEW - document approach)
```

### Key Implementation Patterns

**Scaling Test Pattern:**
```python
import pytest
from time import perf_counter

def measure(fn, rounds=100):
    """Measure average execution time over multiple rounds."""
    start = perf_counter()
    for _ in range(rounds):
        fn()
    return (perf_counter() - start) / rounds

@pytest.mark.scaling
def test_operation_scales_linearly():
    small = measure(lambda: operation(size=100))
    large = measure(lambda: operation(size=1000))

    # 10x input should be ≤15x time (linear + overhead)
    ratio = large / small
    assert ratio < 15, f"Scaling regression: {ratio:.1f}x (expected ≤15x)"
```

**Relative Comparison Pattern:**
```python
@pytest.mark.scaling
def test_operation_a_vs_operation_b():
    """A should be no more than Nx slower than B."""
    time_a = measure(operation_a)
    time_b = measure(operation_b)

    ratio = time_a / time_b
    assert ratio < EXPECTED_RATIO, f"Regression: A is {ratio:.1f}x slower than B"
```

### Recommended Scaling Tests

| Operation | Small Input | Large Input | Max Ratio |
|-----------|-------------|-------------|-----------|
| JSON serialize | 100 bytes | 1000 bytes | 15x |
| Queue enqueue/dequeue | 1K items | 10K items | 15x |
| Envelope build | 1 field | 10 fields | 12x |
| Redactor apply | 1 field | 10 fields | 15x |
| File sink write | 100 events | 1000 events | 12x |

### Recommended Relative Comparisons

| Operation A | Operation B | Max Ratio |
|-------------|-------------|-----------|
| Pretty format | JSON serialize | 5x |
| Envelope build | Dict creation | 10x |
| Redacted serialize | Plain serialize | 3x |

---

## Tasks

### Phase 1: Core Scaling Tests

- [ ] Create `tests/benchmark/test_scaling_regression.py`
- [ ] Add scaling test for JSON serialization
- [ ] Add scaling test for queue operations
- [ ] Add scaling test for envelope building
- [ ] Add scaling test for file sink writes
- [ ] Add `@pytest.mark.scaling` marker to pyproject.toml

### Phase 2: Relative Comparisons

- [ ] Add pretty vs JSON comparison test
- [ ] Add redacted vs plain serialize comparison
- [ ] Add envelope build vs raw dict comparison
- [ ] Verify existing relative comparison in test_perf_benchmarks.py

### Phase 3: CI Integration

- [ ] Add performance regression job to nightly.yml
- [ ] Configure job to run only scaling-marked tests
- [ ] Add meaningful failure output formatting
- [ ] Test by intentionally introducing a regression

### Phase 4: Documentation

- [ ] Create docs/contributing/performance-testing.md
- [ ] Document how to add new scaling tests
- [ ] Document how to interpret failures
- [ ] Update CHANGELOG

---

## Tests

### Benchmark Tests (New)

- `tests/benchmark/test_scaling_regression.py`
  - `test_json_serialization_scales_linearly`
  - `test_queue_operations_scale_linearly`
  - `test_envelope_build_scales_linearly`
  - `test_file_sink_scales_linearly`
  - `test_redactor_scales_linearly`

### Relative Comparison Tests

- `tests/benchmark/test_perf_benchmarks.py` (existing, enhanced)
  - `test_pretty_vs_json_ratio` (enhance existing)
  - `test_redacted_vs_plain_ratio` (new)

---

## Definition of Done

### Code Complete

- [ ] All acceptance criteria implemented
- [ ] Scaling tests cover critical code paths
- [ ] Relative comparisons cover key operation pairs
- [ ] CI job configured and tested

### Quality Assurance

- [ ] All new tests pass locally
- [ ] Tests are deterministic (no flaky failures from timing)
- [ ] Ratios are tuned to avoid false positives
- [ ] Intentional regression is detected correctly
- [ ] `ruff check` passes
- [ ] No regression in existing tests

### Documentation

- [ ] Performance testing guide created
- [ ] CI workflow documented
- [ ] CHANGELOG updated

---

## Risks / Rollback

### Risks

1. **Risk:** False positives from ratio thresholds being too tight
   - **Mitigation:** Start with generous ratios (15x), tighten based on data

2. **Risk:** Tests still flaky due to GC pauses or other system noise
   - **Mitigation:** Use multiple rounds, warm-up iterations, and statistical measures

3. **Risk:** Scaling tests miss real regressions in constant factors
   - **Mitigation:** Combine with relative comparisons; constant factor regressions show up there

### Rollback Plan

If tests are too noisy:
1. Increase ratio thresholds
2. Add `continue-on-error: true` to CI job temporarily
3. Mark tests as `@pytest.mark.skip` while investigating

---

## Related Stories

- **Related:** Story 10.17 - Schema Contract Test Infrastructure (similar testing philosophy)
- **Enables:** Future stories for performance optimization can measure impact

---

## Change Log

| Date | Change | Author |
|------|--------|--------|
| 2026-01-19 | Initial draft | Claude |
