# Story Series 11.x: Test Suite Performance & CI Reliability

## Overview

This series addresses findings from the Test Suite Performance Review conducted in January 2026. The goal is to eliminate timing-dependent test failures, reduce CI flakiness, and improve overall test execution speed while maintaining test quality.

## Context / Background

The test suite has ~1,600 tests across 178 files with 90%+ coverage. However, several patterns cause intermittent CI failures on slower runners:

- **9 tests** marked `@pytest.mark.flaky` due to timing issues (highest impact)
- **4 test files** exceeding 1,000 lines, limiting parallelization
- **Hypothesis tests** overriding the global `max_examples` setting
- **Hard sleeps** (1.0s, 1.2s) in integration tests (lower impact — these tests are skipped in normal CI)

These issues result in non-deterministic CI failures that erode developer confidence and waste CI resources on retries.

## Guiding Principles

1. **Event-based over time-based** — Use `asyncio.Event`, `threading.Event`, or condition variables instead of fixed sleeps
2. **CI-aware timeouts** — Timeouts should scale based on environment (local vs CI)
3. **Deterministic tests** — A test should pass or fail consistently, never intermittently
4. **Parallelization-friendly** — Tests should be isolated and splittable for concurrent execution
5. **Fix, don't ignore** — Remove `@pytest.mark.flaky` by fixing root causes, not by perpetual quarantine

## Stories

### Quick Reference Table

| Story | Title | Priority | Effort | Status | Dependencies |
|-------|-------|----------|--------|--------|--------------|
| 11.0 | [Series Overview](./11.0-series-overview.md) | - | - | - | - |
| 11.1 | [Replace Hard Sleeps with Polling in Integration Tests](./11.1.replace-hard-sleeps-event-coordination.md) | Low | 0.5 day | Planned | None |
| 11.2 | [Fix Flaky Test Root Causes](./11.2.fix-flaky-test-root-causes.md) | Critical | 2-3 days | Complete | None |
| 11.3 | [Standardize Hypothesis Test Settings](./11.3.standardize-hypothesis-settings.md) | High | 0.5 day | Planned | None |
| 11.4 | [Add CI Timeout Multipliers](./11.4.ci-timeout-multipliers.md) | High | 1 day | Planned | None |
| 11.6 | [Split Large Monolithic Test Files](./11.6.split-large-test-files.md) | Medium | 2 days | Planned | None |
| 11.7 | [Optimize Disk I/O in Rotating File Tests](./11.7.optimize-disk-io-tests.md) | Low | 0.5 day | Planned | None |
| 11.8 | [Configure Loop Stall Detection Thresholds](./11.8.loop-stall-thresholds.md) | Low | 0.5 day | Planned | None |
| 11.9 | [Enable Parallel Test Execution with pytest-xdist](./11.9.pytest-xdist-parallel.md) | Medium | 1-2 days | Planned | 11.6 |

### Phase 1: Critical Fixes (11.2)

**Goal:** Eliminate the most impactful sources of CI flakiness

| Story | Title | Description | Status |
|-------|-------|-------------|--------|
| 11.2 | Fix Flaky Tests | Address root causes of 9 `@pytest.mark.flaky` tests | Complete |

**Success Criteria:**
- [x] All 9 flaky tests either fixed or skipped-in-CI

### Phase 2: Standardization (11.3 - 11.4)

**Goal:** Consistent timeout/timing behavior across environments

| Story | Title | Description | Status |
|-------|-------|-------------|--------|
| 11.3 | Hypothesis Settings | Remove per-test `@settings(max_examples=...)` overrides | Planned |
| 11.4 | CI Timeout Multipliers | Add `CI_TIMEOUT_MULTIPLIER` env var for slower runners | Planned |

**Success Criteria:**
- [ ] All Hypothesis tests respect `HYPOTHESIS_MAX_EXAMPLES` env var
- [ ] CI can scale timeouts via environment variable

### Phase 3: Performance Optimization (11.1, 11.6 - 11.9)

**Goal:** Reduce total CI execution time

| Story | Title | Description | Status |
|-------|-------|-------------|--------|
| 11.1 | Integration Test Polling | Replace hard sleeps with polling in integration tests (low impact - tests already skipped in normal CI) | Planned |
| 11.6 | Split Test Files | Break up 4 files with 1,000+ lines | Planned |
| 11.7 | Disk I/O Optimization | Use memory-backed storage for file tests | Planned |
| 11.8 | Loop Stall Config | Make stall detection thresholds configurable | Planned |
| 11.9 | pytest-xdist | Enable parallel test execution | Planned |

**Success Criteria:**
- [ ] No test file exceeds 1,000 lines
- [ ] CI test execution time reduced by 30%+
- [ ] Tests can run with `pytest -n auto`

## Dependency Graph

```
11.1 Hard Sleeps ─────────────────────────────────────► (independent)
11.2 Flaky Tests ─────────────────────────────────────► (independent)
11.3 Hypothesis ──────────────────────────────────────► (independent)
11.4 CI Timeouts ─────────────────────────────────────► (independent)
11.6 Split Files ─────────────────────────────────────► 11.9 pytest-xdist
11.7 Disk I/O ────────────────────────────────────────► (independent)
11.8 Loop Stall ──────────────────────────────────────► (independent)
```

Most stories are independent and can proceed in parallel. Only 11.9 (pytest-xdist) depends on 11.6 (split files) for maximum benefit.

## Key Metrics

### Current State

| Metric | Value |
|--------|-------|
| Flaky tests | 0 |
| Test files > 1,000 lines | 4 |
| Hypothesis `max_examples` overrides | 7 |
| Parallel test execution | Not enabled |
| Hard sleeps in integration tests | 2 (1.0s, 1.2s) — low priority, tests skipped in normal CI |

### Target State

| Metric | Target |
|--------|--------|
| Flaky tests | 0 |
| Test files > 1,000 lines | 0 |
| Hypothesis `max_examples` overrides | 0 |
| Parallel test execution | Enabled |

## Success Metrics

- [x] Zero tests marked `@pytest.mark.flaky`
- [ ] Hypothesis tests controlled by environment variable only
- [ ] CI timeout multiplier available via `CI_TIMEOUT_MULTIPLIER`
- [ ] All test files under 1,000 lines
- [ ] pytest-xdist enabled with `-n auto`
- [ ] CI test suite completes 30%+ faster

## Risks / Mitigation

| Risk | Impact | Mitigation |
|------|--------|------------|
| Splitting files breaks import assumptions | Low | Run full test suite after each split |
| pytest-xdist reveals hidden shared state | Medium | Fix isolation issues as discovered |
| Timeout multiplier causes tests to hang | Low | Cap maximum multiplier at 5x |
| Flaky test fixes introduce new failure modes | Medium | Test each fix 10x before removing flaky marker |

## Related Documents

- [Test Categories Documentation](../contributing/test-categories.md)
- [CI Configuration Guide](../ci-configuration.md)
- [Story 7.0 - Test Suite Value & Confidence](./7.0-series-overview.md) (complementary series)

---

## Change Log

| Date | Change | Author |
|------|--------|--------|
| 2026-01-14 | Initial series creation | Claude |
| 2026-01-14 | Remove Story 11.5 (rejected - premise incorrect, no blocking sleep in async contexts) | Claude |
