# Story 11.2: Fix Flaky Test Root Causes

**Status:** Complete
**Priority:** Critical
**Depends on:** None

---

## Context / Background

Nine tests are marked `@pytest.mark.flaky` due to timing-dependent behavior. The CI runs these with `continue-on-error: true`, which prevents build failures but doesn't fix the underlying issues.

**Current flaky tests:**

| File | Line | Test | Root Cause |
|------|------|------|------------|
| `test_logger_threading.py` | 164 | `test_drain_under_backpressure_drops_excess` | Race condition |
| `test_logger_threading.py` | 265 | `test_queue_full_drops_from_cross_thread_submission` | Race condition |
| `test_high_performance_lru_cache.py` | 281 | `test_performance_characteristics` | Absolute timing threshold (100ms) |
| `test_high_performance_lru_cache.py` | 529 | `test_event_loop_validation_performance` | Absolute timing threshold (200ms) |
| `test_loki_sink_unit.py` | 459 | `test_retry_after_invalid_value` | Monkeypatch timing |
| `test_hanging_prevention.py` | 123 | `test_concurrent_shutdown_robustness` | Race condition |
| `test_rotating_file_sink.py` | 171 | `test_timestamp_collision_suffix_with_datetime_monkeypatch` | Monkeypatch timing |
| `test_container.py` | 546 | `test_component_retrieval_performance` | Absolute timing threshold (1s) |
| `test_sink_circuit_breaker.py` | 80 | `test_circuit_transitions_to_half_open_after_timeout` | 50ms timeout too tight |

Per project policy (docs/contributing/test-categories.md:27): `@pytest.mark.flaky` requires an issue link and expiry date, and is never allowed on `critical` or `security` tests.

---

## Scope (In / Out)

### In Scope

- Fix or remove all 9 `@pytest.mark.flaky` tests
- For each test, choose one of:
  1. **Fix:** Address root cause (timing, race condition)
  2. **Rewrite:** Replace with deterministic alternative
  3. **Skip in CI:** Mark with `@pytest.mark.skipif(os.getenv("CI"))` if inherently non-deterministic
  4. **Remove:** Delete if test provides no value
- Document fix approach for each test

### Out of Scope

- Adding new tests beyond what's needed to replace removed ones
- Refactoring non-flaky tests
- Performance optimization (just make tests reliable)

---

## Acceptance Criteria

### AC1: Zero Flaky Test Markers

**Description:** No tests in the codebase use `@pytest.mark.flaky`.

**Validation:**
```bash
grep -r "@pytest.mark.flaky" tests/ | grep -v "test_verify_test_markers.py" | wc -l
# Expected: 0
```

### AC2: All Fixed Tests Pass Consistently

**Description:** Each fixed test passes 10 times in a row.

**Validation:**
```bash
pytest tests/unit/test_logger_threading.py::TestBackpressure::test_drain_under_backpressure_drops_excess --count=10
# All 10 should pass
```

### AC3: Race Conditions Eliminated

**Description:** Tests with race conditions use proper synchronization primitives.

**Validation:**
- Threading tests use `threading.Event` for coordination
- Async tests use `asyncio.Event` or `asyncio.Condition`
- No reliance on `time.sleep()` for synchronization

### AC4: Timeout Tests Use CI-Appropriate Values

**Description:** Tests with timing thresholds work on slow CI runners.

**Validation:**
- Circuit breaker timeout increased from 50ms to 200ms+ or uses `CI_TIMEOUT_MULTIPLIER`
- Performance threshold tests use relative comparisons, not absolute timings

---

## Implementation Notes

### Fix Strategy by Test

#### Category A: Threading Race Conditions (3 tests)

##### 1. `test_logger_threading.py:164` - Backpressure drops

**Problem:** Test floods 50 messages into a queue of 3 with a slow sink. The flakiness comes from timing variance in how many messages get dropped vs processed before `stop_and_drain()` completes.

**Actual test logic:**
```python
logger = SyncLoggerFacade(queue_capacity=3, ...)
for i in range(50):
    logger.info(f"flood {i}")
result = asyncio.run(logger.stop_and_drain())
assert result.dropped > 0  # This is the flaky assertion
```

**Fix options:**
1. **Relax assertion:** The test verifies backpressure works — assert `result.dropped >= 0` or remove exact count checks
2. **Increase queue pressure:** Use smaller queue or slower sink to guarantee drops
3. **Add synchronization:** Wait for queue to be full before continuing flood

```python
# Option 1: Relax assertion (simplest)
assert result.submitted == 50
assert result.submitted == result.processed + result.dropped
# Remove: assert result.dropped > 0  (timing-dependent)

# Option 2: Guarantee drops by using queue_capacity=1
logger = SyncLoggerFacade(queue_capacity=1, batch_max_size=1, ...)
```

##### 2. `test_logger_threading.py:265` - Cross-thread queue full

**Problem:** Similar to #1 — 3 threads flood 150 messages, race in how many get dropped.

**Fix:** Same options as #1. Relax assertions or guarantee drops with smaller queue.

##### 3. `test_hanging_prevention.py:123` - Concurrent shutdown

**Problem:** Test submits from 3 threads while calling `stop_and_drain()`. Race between thread.join() and shutdown timing.

**Fix:**
```python
# Current: assert result.submitted >= 20 (flaky)
# Fix: Just verify shutdown completes without hanging
assert shutdown_time < 15.0  # Keep this
# Relax or remove the submitted count assertion
```

---

#### Category B: Absolute Timing Thresholds (4 tests)

##### 4-5. `test_high_performance_lru_cache.py:281,529` - Performance tests

**Problem:** Tests assert operations complete in <100ms or <200ms. Slow CI runners fail.

**Current:**
```python
assert set_time < 0.1  # 100ms - fails on slow CI
assert get_time < 0.1
```

**Fix:** Skip in CI (performance tests aren't critical for correctness):
```python
@pytest.mark.skipif(os.getenv("CI"), reason="Performance test, skip in CI")
def test_performance_characteristics(self):
    ...
```

##### 6. `test_container.py:546` - Component retrieval performance

**Problem:** Asserts 1000 retrievals complete in <1s.

**Fix:** Same as above — skip in CI.

##### 7. `test_sink_circuit_breaker.py:80` - 50ms recovery timeout

**Problem:** Test uses 50ms recovery timeout, then sleeps 60ms. On slow CI, sleep completes but state check races.

**Current:**
```python
config = SinkCircuitBreakerConfig(recovery_timeout_seconds=0.05)  # 50ms
time.sleep(0.06)  # 60ms - only 10ms margin
```

**Fix:** Increase timeout and sleep to have more margin:
```python
config = SinkCircuitBreakerConfig(recovery_timeout_seconds=0.2)  # 200ms
time.sleep(0.25)  # 250ms - 50ms margin
```

---

#### Category C: Monkeypatch Timing (2 tests)

##### 8. `test_loki_sink_unit.py:459` - asyncio.sleep monkeypatch

**Problem:** Test monkeypatches `asyncio.sleep` to track retry delays. The monkeypatch races with async execution.

**Current:**
```python
monkeypatch.setattr(loki.asyncio, "sleep", lambda s: slept.append(s) or original_sleep(0))
```

**Fix:** Use `unittest.mock.patch` as context manager to ensure clean setup:
```python
async def test_retry_after_invalid_value():
    slept = []
    async def tracking_sleep(s):
        slept.append(s)

    with patch.object(loki.asyncio, 'sleep', tracking_sleep):
        # ... test code ...
```

##### 9. `test_rotating_file_sink.py:171` - datetime monkeypatch

**Problem:** Custom datetime class monkeypatched, but rotation timing can race.

**Fix:** Consider using `freezegun` library for more robust datetime mocking:
```python
from freezegun import freeze_time

@freeze_time("2025-01-01 12:00:00", auto_tick_seconds=2)
async def test_timestamp_collision_suffix():
    # freezegun handles datetime consistently
    ...
```

Or simplify: the test verifies collision suffix works — ensure the fake datetime is applied before sink starts.

---

## Tasks

### Phase 1: Threading Race Conditions (3 tests)

- [ ] Fix `test_drain_under_backpressure_drops_excess` — relax assertions or use queue_capacity=1
- [ ] Fix `test_queue_full_drops_from_cross_thread_submission` — same approach
- [ ] Fix `test_concurrent_shutdown_robustness` — focus on hang prevention, relax count assertions
- [ ] Remove `@pytest.mark.flaky` from each
- [ ] Verify each passes 10x

### Phase 2: Performance Tests (3 tests) — Skip in CI

- [ ] Add `@pytest.mark.skipif(os.getenv("CI"))` to `test_performance_characteristics`
- [ ] Add skip to `test_event_loop_validation_performance`
- [ ] Add skip to `test_component_retrieval_performance`
- [ ] Remove `@pytest.mark.flaky` from each

### Phase 3: Timing-Sensitive Tests (1 test)

- [ ] Fix circuit breaker test — increase timeout from 50ms to 200ms, sleep from 60ms to 250ms
- [ ] Remove `@pytest.mark.flaky`
- [ ] Verify passes 10x

### Phase 4: Monkeypatch Tests (2 tests)

- [ ] Fix `test_retry_after_invalid_value` — use `patch` context manager
- [ ] Fix `test_timestamp_collision_suffix_with_datetime_monkeypatch` — consider freezegun or ensure patch before sink start
- [ ] Remove `@pytest.mark.flaky` from each
- [ ] Verify each passes 10x

### Phase 5: Validation

- [ ] Verify zero `@pytest.mark.flaky` in codebase (excluding test_verify_test_markers.py)
- [ ] Run full test suite
- [ ] Document fixes in CHANGELOG

---

## Tests

### Stability Verification

For each fixed test, verify stability:
```bash
pytest <test_path>::<test_name> --count=10 -v
```

### Full Suite Regression

```bash
pytest tests/ -m "not slow and not integration" -v
```

---

## Definition of Done

### Code Complete

- [ ] All 9 flaky tests fixed, skipped-in-CI, or removed
- [ ] Zero `@pytest.mark.flaky` markers in test code (excluding test_verify_test_markers.py)
- [ ] Threading tests use deterministic assertions
- [ ] Performance tests skip in CI rather than flaky-marking

### Quality Assurance

- [ ] Each fixed test passes 10x in a row
- [ ] `ruff check` passes
- [ ] `mypy` passes
- [ ] No regression in existing tests

### Documentation

- [ ] Each fix documented in commit message
- [ ] CHANGELOG updated
- [ ] Consider adding patterns to test guidelines

---

## Risks / Rollback

### Risks

1. **Risk:** Fix introduces new failure mode
   - **Mitigation:** Test 10x before removing flaky marker

2. **Risk:** Removing test loses coverage
   - **Mitigation:** Only remove if test has no value; otherwise rewrite

3. **Risk:** Performance tests inherently non-deterministic
   - **Mitigation:** Use relative comparisons or skip in CI

### Rollback Plan

If issues occur:
1. Re-add `@pytest.mark.flaky` temporarily
2. Open issue to track proper fix
3. Investigate with more CI run data

---

## Related Stories

- **Related:** Story 11.4 - CI timeout multipliers could help with timing tests as alternative to skipping
- **Related:** Story 11.5 - blocking sleep issues may overlap with some fixes here
- **Enables:** Removal of `continue-on-error: true` for flaky tests in CI

---

## Change Log

| Date | Change | Author |
|------|--------|--------|
| 2026-01-14 | Initial draft | Claude |
