# Story 7.7: Add Concurrency Tests for Circuit Breaker

**Status:** Complete
**Priority:** P1
**Depends on:** None
**Effort:** M (4-8 hours)

---

## Problem Statement

The `SinkCircuitBreaker` class manages state transitions (CLOSED → OPEN → HALF_OPEN → CLOSED) that can be called from multiple async tasks or threads simultaneously. The original implementation had no locking, which could lead to:

- **Race conditions**: Multiple tasks calling `record_failure()` could exceed threshold checks
- **State corruption**: Concurrent `record_success()` and `record_failure()` during HALF_OPEN
- **Lost updates**: `_failure_count` increments could be lost under contention

**Resolution:** Added `threading.Lock()` to protect all state mutations. Also fixed `should_allow()` to atomically increment `_half_open_calls` counter.

---

## Goals

1. **Verify circuit breaker behavior under concurrent access**
2. **Identify race conditions** if they exist
3. **Document concurrency guarantees** (or lack thereof)
4. **Add thread-safety if needed** (or document why it's not needed)

---

## Acceptance Criteria

- [x] Concurrent access test with multiple threads calling `record_failure()`
- [x] Concurrent access test with mixed `record_success()` / `record_failure()`
- [x] Concurrent access test during HALF_OPEN → CLOSED/OPEN transitions
- [x] Race condition behavior documented or fixed
- [x] Stress test with high contention (100+ concurrent operations)
- [x] Clear documentation of concurrency model

---

## Technical Approach

### 1. Basic Concurrency Test

**File: `tests/unit/test_circuit_breaker_concurrency.py`**

```python
"""
Concurrency tests for SinkCircuitBreaker.

These tests verify behavior under concurrent access from multiple
threads or async tasks.
"""

from __future__ import annotations

import asyncio
import threading
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from typing import Callable

import pytest

from fapilog.core.circuit_breaker import (
    SinkCircuitBreaker,
    SinkCircuitBreakerConfig,
    CircuitState,
)


class TestCircuitBreakerConcurrency:
    """Concurrency tests for circuit breaker."""

    def test_concurrent_failures_open_circuit(self) -> None:
        """Multiple threads recording failures should eventually open circuit."""
        config = SinkCircuitBreakerConfig(
            failure_threshold=10,
            recovery_timeout_seconds=60.0,
        )
        breaker = SinkCircuitBreaker("test-sink", config)

        num_threads = 20
        failures_per_thread = 5

        def record_failures() -> int:
            count = 0
            for _ in range(failures_per_thread):
                breaker.record_failure()
                count += 1
            return count

        with ThreadPoolExecutor(max_workers=num_threads) as executor:
            futures = [executor.submit(record_failures) for _ in range(num_threads)]
            total = sum(f.result() for f in as_completed(futures))

        # All failures recorded
        assert total == num_threads * failures_per_thread

        # Circuit should be open (threshold was 10, we recorded 100)
        assert breaker.state == CircuitState.OPEN

    def test_concurrent_success_and_failure_during_half_open(self) -> None:
        """Concurrent success/failure during HALF_OPEN should not corrupt state."""
        config = SinkCircuitBreakerConfig(
            failure_threshold=1,
            recovery_timeout_seconds=0.0,  # Immediate recovery
            half_open_max_calls=10,
        )
        breaker = SinkCircuitBreaker("test-sink", config)

        # Force into HALF_OPEN state
        breaker.record_failure()
        assert breaker.state == CircuitState.OPEN
        breaker.should_allow()  # Triggers transition to HALF_OPEN
        assert breaker.state == CircuitState.HALF_OPEN

        results: list[CircuitState] = []
        lock = threading.Lock()

        def record_success() -> None:
            breaker.record_success()
            with lock:
                results.append(breaker.state)

        def record_failure() -> None:
            breaker.record_failure()
            with lock:
                results.append(breaker.state)

        with ThreadPoolExecutor(max_workers=20) as executor:
            futures = []
            for i in range(50):
                if i % 2 == 0:
                    futures.append(executor.submit(record_success))
                else:
                    futures.append(executor.submit(record_failure))

            for f in as_completed(futures):
                f.result()

        # State should be valid (not corrupted)
        final_state = breaker.state
        assert final_state in (CircuitState.CLOSED, CircuitState.OPEN, CircuitState.HALF_OPEN)

    def test_high_contention_stress(self) -> None:
        """Stress test with high contention."""
        config = SinkCircuitBreakerConfig(
            failure_threshold=50,
            recovery_timeout_seconds=0.001,
            half_open_max_calls=5,
        )
        breaker = SinkCircuitBreaker("stress-sink", config)

        num_threads = 100
        operations_per_thread = 100
        errors: list[Exception] = []
        lock = threading.Lock()

        def stress_operations() -> None:
            try:
                for i in range(operations_per_thread):
                    breaker.should_allow()
                    if i % 3 == 0:
                        breaker.record_failure()
                    else:
                        breaker.record_success()
            except Exception as e:
                with lock:
                    errors.append(e)

        with ThreadPoolExecutor(max_workers=num_threads) as executor:
            futures = [
                executor.submit(stress_operations)
                for _ in range(num_threads)
            ]
            for f in as_completed(futures):
                f.result()

        # No exceptions during stress
        assert len(errors) == 0, f"Errors during stress test: {errors}"

        # State should be valid
        assert breaker.state in (
            CircuitState.CLOSED,
            CircuitState.OPEN,
            CircuitState.HALF_OPEN,
        )


class TestCircuitBreakerAsyncConcurrency:
    """Async concurrency tests for circuit breaker."""

    @pytest.mark.asyncio
    async def test_concurrent_async_failures(self) -> None:
        """Multiple async tasks recording failures concurrently."""
        config = SinkCircuitBreakerConfig(
            failure_threshold=10,
            recovery_timeout_seconds=60.0,
        )
        breaker = SinkCircuitBreaker("async-sink", config)

        async def record_failures(count: int) -> int:
            for _ in range(count):
                breaker.record_failure()
                await asyncio.sleep(0)  # Yield to other tasks
            return count

        # Run 20 tasks, each recording 5 failures
        tasks = [
            asyncio.create_task(record_failures(5))
            for _ in range(20)
        ]
        results = await asyncio.gather(*tasks)

        assert sum(results) == 100
        assert breaker.state == CircuitState.OPEN

    @pytest.mark.asyncio
    async def test_async_state_transitions_under_load(self) -> None:
        """State transitions should be consistent under async load."""
        config = SinkCircuitBreakerConfig(
            failure_threshold=5,
            recovery_timeout_seconds=0.001,
            half_open_max_calls=2,
        )
        breaker = SinkCircuitBreaker("async-transitions", config)

        state_history: list[CircuitState] = []

        async def observer() -> None:
            """Record state transitions."""
            for _ in range(100):
                state_history.append(breaker.state)
                await asyncio.sleep(0.001)

        async def operator() -> None:
            """Perform operations that cause transitions."""
            for i in range(50):
                breaker.should_allow()
                if i % 4 == 0:
                    breaker.record_failure()
                else:
                    breaker.record_success()
                await asyncio.sleep(0.001)

        await asyncio.gather(
            observer(),
            operator(),
            operator(),
            operator(),
        )

        # All observed states should be valid
        for state in state_history:
            assert state in (
                CircuitState.CLOSED,
                CircuitState.OPEN,
                CircuitState.HALF_OPEN,
            )
```

### 2. Document Concurrency Model

Add documentation to `circuit_breaker.py`:

```python
class SinkCircuitBreaker:
    """
    Simple circuit breaker for individual sink protection.

    Concurrency Model:
    ------------------
    This implementation is NOT thread-safe. Each sink should have its own
    circuit breaker instance, and access should be serialized by the caller
    (e.g., through the async worker's event loop).

    In fapilog's architecture, circuit breakers are accessed from:
    - The async worker task (single-threaded within the event loop)
    - Potentially from sync logger facades (via thread-local instances)

    If thread-safe access is required, wrap calls in a lock or use one
    breaker per thread/task.
    """
```

### 3. Optional: Add Thread Safety

If tests reveal problematic races, add locking:

```python
import threading

class SinkCircuitBreaker:
    def __init__(self, sink_name: str, config: SinkCircuitBreakerConfig) -> None:
        self._lock = threading.Lock()
        # ... rest of init ...

    def should_allow(self) -> bool:
        with self._lock:
            # ... existing logic ...

    def record_success(self) -> None:
        with self._lock:
            # ... existing logic ...

    def record_failure(self) -> None:
        with self._lock:
            # ... existing logic ...
```

---

## Files Changed

| File | Action |
|------|--------|
| `tests/unit/test_circuit_breaker_concurrency.py` | New file |
| `src/fapilog/core/circuit_breaker.py` | Add concurrency documentation (and possibly locking) |

**Lines added:** ~200-300

---

## Risk Assessment

**Risk Level: Medium**

- **May reveal bugs**: Concurrency tests might expose race conditions
- **May require production changes**: If races are found, `circuit_breaker.py` needs fixes
- **False positives**: Concurrency tests can be flaky

**Mitigations:**
- Run tests multiple times to verify consistency
- Use `pytest-repeat` for stress testing
- Document any known races as acceptable if they don't cause incorrect behavior

---

## Rollback Plan

1. Delete `tests/unit/test_circuit_breaker_concurrency.py`
2. Revert any changes to `circuit_breaker.py`

---

## Definition of Done

- [x] Concurrent failure recording test passes
- [x] Mixed success/failure concurrency test passes
- [x] High contention stress test passes
- [x] Async concurrency tests pass
- [x] Concurrency model documented in code
- [x] Any discovered races fixed or documented as acceptable
- [x] All tests pass (including existing circuit breaker tests)
- [x] Tests run reliably (no flakiness over 10 runs)

---

## Related Stories

- **7.6**: Property-Based Tests (could add stateful property tests for circuit breaker)

---

## Future Considerations

- Add lock-free atomic implementation if performance is critical
- Consider per-sink-per-thread breakers for thread isolation
- Evaluate circuit breaker patterns from other libraries (resilience4j, pybreaker)
