# Story 11.4: Add CI Timeout Multipliers

**Status:** Complete
**Priority:** High
**Depends on:** None

---

## Context / Background

Tests with timing-sensitive operations often fail on slow CI runners because timeouts are calibrated for fast developer machines. Rather than individually adjusting each timeout, a global multiplier allows all timeouts to scale based on the environment.

**Current problem:**
```python
# test_sink_circuit_breaker.py:80-100
config = SinkCircuitBreakerConfig(
    failure_threshold=1,
    recovery_timeout_seconds=0.05,  # 50ms - too tight for slow CI
)
breaker = SinkCircuitBreaker("test_sink", config)
breaker.record_failure()
time.sleep(0.06)  # Wait for recovery timeout
# Fails intermittently on loaded CI runners
```

**Solution:**
```python
from conftest import get_test_timeout

config = SinkCircuitBreakerConfig(
    failure_threshold=1,
    recovery_timeout_seconds=get_test_timeout(0.05),  # 50ms locally, 150ms in CI (3x)
)
breaker = SinkCircuitBreaker("test_sink", config)
breaker.record_failure()
time.sleep(get_test_timeout(0.06))
```

This pattern is already partially implemented for loop stall detection (`FAPILOG_TEST_MAX_LOOP_STALL_SECONDS`), but not generalized to all test timeouts.

---

## Scope (In / Out)

### In Scope

- Add `CI_TIMEOUT_MULTIPLIER` environment variable support
- Create a `get_test_timeout(base: float) -> float` helper function
- Apply multiplier to timing-sensitive tests
- Document the pattern in test guidelines
- Set default multiplier in CI configuration

### Out of Scope

- Changing absolute timeout values (just adding multiplier)
- Refactoring tests beyond timeout multiplication
- Production code changes (test-only)

---

## Acceptance Criteria

### AC1: Timeout Helper Function Exists

**Description:** A reusable function applies the timeout multiplier.

**Validation:**
```python
from conftest import get_test_timeout

# Without env var
assert get_test_timeout(0.05) == 0.05

# With env var
os.environ["CI_TIMEOUT_MULTIPLIER"] = "3"
assert get_test_timeout(0.05) == 0.15
```

### AC2: CI Configuration Sets Multiplier

**Description:** CI workflow sets `CI_TIMEOUT_MULTIPLIER=3` (or appropriate value).

**Validation:**
```yaml
# .github/workflows/ci.yml (uses Hatch)
jobs:
  test:
    steps:
      - name: Run tests
        env:
          CI_TIMEOUT_MULTIPLIER: "3"
        run: hatch run test:test
```

### AC3: Timing-Sensitive Tests Use Helper

**Description:** Tests with tight timeouts use `get_test_timeout()`.

**Validation:**
```python
# Before
timeout = 0.05
await asyncio.wait_for(op, timeout=timeout)

# After
timeout = get_test_timeout(0.05)
await asyncio.wait_for(op, timeout=timeout)
```

### AC4: Multiplier Has Reasonable Bounds

**Description:** Multiplier is capped to prevent tests from hanging.

**Validation:**
```python
# Cap at 5x to prevent infinite waits
os.environ["CI_TIMEOUT_MULTIPLIER"] = "100"
assert get_test_timeout(0.05) == 0.25  # 0.05 * 5 max
```

---

## Implementation Notes

### Helper Function

Add to `conftest.py` (root level, next to existing marker definitions):

```python
import os

def get_test_timeout(base: float, max_multiplier: float = 5.0) -> float:
    """Apply CI timeout multiplier to a base timeout value.

    Args:
        base: Base timeout in seconds
        max_multiplier: Maximum allowed multiplier (default 5x)

    Returns:
        Scaled timeout value

    Environment:
        CI_TIMEOUT_MULTIPLIER: Multiplier for CI environments (default: 1.0)

    Note:
        Reads env var on each call to support per-test monkeypatching.
    """
    raw = os.getenv("CI_TIMEOUT_MULTIPLIER", "1.0")
    try:
        multiplier = min(float(raw), max_multiplier)
    except ValueError:
        multiplier = 1.0
    return base * multiplier
```

### Usage Pattern

```python
from conftest import get_test_timeout

class TestCircuitBreaker:
    def test_half_open_transition(self):
        config = SinkCircuitBreakerConfig(
            failure_threshold=1,
            recovery_timeout_seconds=get_test_timeout(0.05),
        )
        breaker = SinkCircuitBreaker("test_sink", config)
        # ... test logic ...
        time.sleep(get_test_timeout(0.06))
```

### CI Configuration

```yaml
# .github/workflows/ci.yml
jobs:
  test:
    steps:
      - name: Run quick tests with coverage (PR)
        env:
          CI_TIMEOUT_MULTIPLIER: "3"
        run: hatch run test:test-cov -- -m "not slow and not integration"
```

### tox Configuration

```ini
# tox.ini
setenv =
    CI_TIMEOUT_MULTIPLIER = 3
```

### File Changes

```
conftest.py (MODIFIED - add get_test_timeout helper)
tests/unit/test_sink_circuit_breaker.py (MODIFIED - use helper)
tests/unit/test_container.py (MODIFIED - use helper)
tests/unit/test_high_performance_lru_cache.py (MODIFIED - use helper)
.github/workflows/ci.yml (MODIFIED - set env var)
tox.ini (MODIFIED - set env var)
docs/contributing/test-categories.md (MODIFIED - document pattern)
```

---

## Tasks

### Phase 1: Infrastructure

- [ ] Add `get_test_timeout()` function to `conftest.py` (root level)
- [ ] Add unit test for the helper function
- [ ] Add `CI_TIMEOUT_MULTIPLIER` to tox.ini
- [ ] Add `CI_TIMEOUT_MULTIPLIER` to `.github/workflows/ci.yml`

### Phase 2: Apply to Tests

- [ ] Update `test_sink_circuit_breaker.py` timeout tests
- [ ] Update `test_container.py` performance tests
- [ ] Update `test_high_performance_lru_cache.py` timing tests
- [ ] Search for other hardcoded timeouts and update

### Phase 3: Documentation

- [ ] Document pattern in `docs/contributing/test-categories.md`
- [ ] Add example usage in docstring
- [ ] Update CHANGELOG

---

## Tests

### Unit Test for Helper

```python
# tests/unit/test_conftest_helpers.py
from conftest import get_test_timeout

def test_get_test_timeout_default(monkeypatch):
    monkeypatch.delenv("CI_TIMEOUT_MULTIPLIER", raising=False)
    assert get_test_timeout(0.1) == 0.1

def test_get_test_timeout_with_multiplier(monkeypatch):
    monkeypatch.setenv("CI_TIMEOUT_MULTIPLIER", "3")
    assert get_test_timeout(0.1) == 0.3

def test_get_test_timeout_capped(monkeypatch):
    monkeypatch.setenv("CI_TIMEOUT_MULTIPLIER", "100")
    assert get_test_timeout(0.1) == 0.5  # 0.1 * 5 max

def test_get_test_timeout_invalid_value(monkeypatch):
    monkeypatch.setenv("CI_TIMEOUT_MULTIPLIER", "invalid")
    assert get_test_timeout(0.1) == 0.1  # Falls back to 1.0
```

### Integration Verification

```bash
# Verify multiplier works
CI_TIMEOUT_MULTIPLIER=3 pytest tests/unit/test_sink_circuit_breaker.py -v
```

---

## Definition of Done

### Code Complete

- [x] `get_test_timeout()` helper implemented
- [x] CI configuration sets multiplier
- [x] Timing-sensitive tests use helper
- [x] Multiplier capped at 5x

### Quality Assurance

- [x] Helper function has unit tests
- [x] CI tests pass with multiplier
- [x] Local tests pass without multiplier
- [x] `ruff check` passes
- [x] `mypy` passes

### Documentation

- [x] Test guidelines document pattern
- [x] CHANGELOG updated

---

## Risks / Rollback

### Risks

1. **Risk:** Multiplier too high causes tests to hang
   - **Mitigation:** Cap at 5x maximum

2. **Risk:** Developers forget to use helper
   - **Mitigation:** Add lint rule to warn on raw timeout values

3. **Risk:** Multiplier hides real performance issues
   - **Mitigation:** Log when multiplier > 1 is used

### Rollback Plan

If issues occur:
1. Set `CI_TIMEOUT_MULTIPLIER=1` to disable
2. Revert individual test changes
3. Investigate which tests need multiplier

---

## Related Stories

- **Enables:** Story 11.2 - flaky tests can use multiplier for timing
- **Related:** Story 11.8 - loop stall thresholds use similar pattern
- **Related:** Story 11.1 - event-based waits are better than multiplied sleeps

---

## Change Log

| Date | Change | Author |
|------|--------|--------|
| 2026-01-14 | Initial draft | Claude |
| 2026-01-14 | Fix paths: conftest.py at root, ci.yml not test.yml; fix helper to be testable | Claude |
