# Story 4.35: Sink Fault Isolation and Circuit Breaker

**Status:** Complete  
**Priority:** High  
**Depends on:** None  
**Effort:** 2-3 days

---

## Problem Statement

When multiple sinks are configured (fanout), fapilog has two reliability problems:

### 1. No Fault Isolation Between Sinks

Currently, `_fanout_writer` writes to all sinks sequentially:

```python
async def _write(entry: dict) -> None:
    for write, _ in writers:
        await write(entry)  # If this blocks/hangs, all subsequent sinks wait
```

**Problems:**

- A slow sink blocks all subsequent sinks
- An exception in one sink affects timing of others
- No parallelization of sink writes

### 2. No Circuit Breaker for Failing Sinks

If a sink starts failing (network issues, disk full), fapilog continues trying every write:

```python
async def _sink_write(entry: dict) -> None:
    # ... start logic ...
    await sink.write(entry)  # Called every time, even if failing repeatedly
```

**Problems:**

- Wasted resources on a broken sink
- Latency spikes from retry attempts
- No automatic recovery detection

---

## Goals

1. **Isolate sink failures**: One failing sink doesn't affect others
2. **Implement circuit breaker**: Stop calling a sink after repeated failures
3. **Auto-recovery**: Periodically probe failed sinks to detect recovery
4. **Parallel writes**: Optionally write to sinks in parallel for performance
5. **Observability**: Emit metrics and diagnostics for circuit state changes

---

## Design

### Circuit Breaker States

```
CLOSED ─────(failures >= threshold)────► OPEN
   ▲                                        │
   │                                        │
   └────(success)◄── HALF_OPEN ◄──(timeout)─┘
```

- **CLOSED**: Normal operation, all writes go through
- **OPEN**: Sink is broken, skip writes, emit diagnostics
- **HALF_OPEN**: Probe mode, try one write to detect recovery

### Configuration

```python
@dataclass
class CircuitBreakerConfig:
    enabled: bool = True
    failure_threshold: int = 5          # Open after N consecutive failures
    recovery_timeout_seconds: float = 30.0  # Wait before probing
    half_open_max_calls: int = 1        # Probes before closing
```

### Per-Sink Wrapper

```python
class CircuitProtectedSink:
    def __init__(
        self,
        sink: Any,
        config: CircuitBreakerConfig,
        metrics: MetricsCollector | None = None,
    ):
        self._sink = sink
        self._config = config
        self._metrics = metrics
        self._state = CircuitState.CLOSED
        self._failure_count = 0
        self._last_failure_time: float | None = None
        self._half_open_calls = 0

    async def write(self, entry: dict) -> None:
        if not self._should_attempt():
            return  # Skip - circuit is open

        try:
            await self._sink.write(entry)
            self._on_success()
        except Exception as exc:
            self._on_failure(exc)
            raise  # Re-raise for logging, but don't crash pipeline

    def _should_attempt(self) -> bool:
        if self._state == CircuitState.CLOSED:
            return True

        if self._state == CircuitState.OPEN:
            # Check if recovery timeout elapsed
            if self._last_failure_time is not None:
                elapsed = time.monotonic() - self._last_failure_time
                if elapsed >= self._config.recovery_timeout_seconds:
                    self._state = CircuitState.HALF_OPEN
                    self._half_open_calls = 0
                    return True
            return False

        if self._state == CircuitState.HALF_OPEN:
            return self._half_open_calls < self._config.half_open_max_calls

        return True

    def _on_success(self) -> None:
        if self._state == CircuitState.HALF_OPEN:
            # Recovery confirmed
            self._state = CircuitState.CLOSED
            self._emit_state_change("closed")
        self._failure_count = 0

    def _on_failure(self, exc: Exception) -> None:
        self._failure_count += 1
        self._last_failure_time = time.monotonic()

        if self._state == CircuitState.HALF_OPEN:
            # Probe failed, back to open
            self._state = CircuitState.OPEN
            self._emit_state_change("open")
        elif self._failure_count >= self._config.failure_threshold:
            self._state = CircuitState.OPEN
            self._emit_state_change("open")
```

---

## Implementation Plan

### Phase 1: Circuit Breaker Module

**File: `src/fapilog/core/circuit_breaker.py`** (update existing)

Add sink-specific circuit breaker:

```python
from enum import Enum
from dataclasses import dataclass


class CircuitState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"


@dataclass
class SinkCircuitBreakerConfig:
    enabled: bool = True
    failure_threshold: int = 5
    recovery_timeout_seconds: float = 30.0
    half_open_max_calls: int = 1


class SinkCircuitBreaker:
    """Circuit breaker for individual sink protection."""

    def __init__(
        self,
        sink_name: str,
        config: SinkCircuitBreakerConfig,
    ):
        self.sink_name = sink_name
        self._config = config
        self._state = CircuitState.CLOSED
        self._failure_count = 0
        self._last_failure_time: float | None = None
        self._half_open_calls = 0

    @property
    def state(self) -> CircuitState:
        return self._state

    @property
    def is_open(self) -> bool:
        return self._state == CircuitState.OPEN

    def should_allow(self) -> bool:
        """Return True if a call should be attempted."""
        # ... implementation ...

    def record_success(self) -> None:
        """Record a successful call."""
        # ... implementation ...

    def record_failure(self) -> None:
        """Record a failed call."""
        # ... implementation ...
```

### Phase 2: Parallel Fanout Writer

**File: `src/fapilog/__init__.py`**

Update fanout to support parallel writes:

```python
def _fanout_writer(
    sinks: list[object],
    *,
    parallel: bool = False,
    circuit_config: SinkCircuitBreakerConfig | None = None,
) -> tuple[Any, Any]:
    """Create fanout writer with optional parallelization and circuit breakers."""

    # Create circuit breakers for each sink
    breakers = {}
    if circuit_config and circuit_config.enabled:
        for sink in sinks:
            name = getattr(sink, "name", type(sink).__name__)
            breakers[id(sink)] = SinkCircuitBreaker(name, circuit_config)

    writers = [_make_sink_writer(s) for s in sinks]

    async def _write(entry: dict) -> None:
        if parallel and len(writers) > 1:
            await _write_parallel(entry)
        else:
            await _write_sequential(entry)

    async def _write_sequential(entry: dict) -> None:
        for i, (write, _) in enumerate(writers):
            sink = sinks[i]
            breaker = breakers.get(id(sink))

            if breaker and not breaker.should_allow():
                continue  # Skip - circuit open

            try:
                await write(entry)
                if breaker:
                    breaker.record_success()
            except Exception:
                if breaker:
                    breaker.record_failure()
                # Contain error, continue to next sink

    async def _write_parallel(entry: dict) -> None:
        tasks = []
        for i, (write, _) in enumerate(writers):
            sink = sinks[i]
            breaker = breakers.get(id(sink))

            if breaker and not breaker.should_allow():
                continue

            tasks.append(_write_one(write, breaker, entry))

        if tasks:
            await asyncio.gather(*tasks, return_exceptions=True)

    async def _write_one(write, breaker, entry) -> None:
        try:
            await write(entry)
            if breaker:
                breaker.record_success()
        except Exception:
            if breaker:
                breaker.record_failure()

    # ... return writers ...
```

### Phase 3: Configuration Settings

**File: `src/fapilog/core/settings.py`**

Add settings for circuit breaker:

```python
class SinkCircuitBreakerSettings(BaseModel):
    enabled: bool = Field(default=False, description="Enable circuit breaker for sinks")
    failure_threshold: int = Field(default=5, description="Failures before opening circuit")
    recovery_timeout_seconds: float = Field(default=30.0, description="Seconds before probing")
    parallel_writes: bool = Field(default=False, description="Write to sinks in parallel")
```

### Phase 4: Metrics and Diagnostics

Emit metrics for circuit state:

```python
def _emit_state_change(self, new_state: str) -> None:
    try:
        from ..core.diagnostics import warn
        warn(
            "circuit-breaker",
            f"sink circuit {new_state}",
            sink=self.sink_name,
            failure_count=self._failure_count,
        )
    except Exception:
        pass
```

---

## Acceptance Criteria

- [x] Circuit breaker implemented for sinks
- [x] Open circuit skips writes and emits diagnostics
- [x] Half-open state probes for recovery
- [x] Successful probe closes circuit
- [x] Failed probe re-opens circuit
- [x] Parallel write option available
- [x] Settings configurable via env vars
- [x] Metrics emitted for state changes
- [x] Existing sink behavior unchanged when disabled

---

## Test Plan

```python
@pytest.mark.asyncio
async def test_circuit_opens_after_failures():
    """Circuit opens after threshold failures."""
    breaker = SinkCircuitBreaker("test", SinkCircuitBreakerConfig(failure_threshold=3))

    for _ in range(3):
        breaker.record_failure()

    assert breaker.is_open
    assert not breaker.should_allow()


@pytest.mark.asyncio
async def test_circuit_half_open_after_timeout():
    """Circuit transitions to half-open after recovery timeout."""
    config = SinkCircuitBreakerConfig(
        failure_threshold=1,
        recovery_timeout_seconds=0.1,
    )
    breaker = SinkCircuitBreaker("test", config)

    breaker.record_failure()
    assert breaker.is_open

    await asyncio.sleep(0.15)
    assert breaker.should_allow()  # Should transition to half-open
    assert breaker.state == CircuitState.HALF_OPEN


@pytest.mark.asyncio
async def test_parallel_fanout_isolates_failures():
    """Failure in one sink doesn't block others in parallel mode."""
    # Test implementation...
```

---

## Files Changed

| File                                      | Change                         |
| ----------------------------------------- | ------------------------------ |
| `src/fapilog/core/circuit_breaker.py`     | Add `SinkCircuitBreaker` class |
| `src/fapilog/core/settings.py`            | Add circuit breaker settings   |
| `src/fapilog/__init__.py`                 | Update `_fanout_writer`        |
| `tests/unit/test_sink_circuit_breaker.py` | New test file                  |
