# Circuit Breaker Best Practices

Best practices for implementing the Circuit Breaker pattern in async Python workflow engines.

## Overview

The Circuit Breaker pattern prevents cascade failures by stopping calls to failing services after a threshold is reached.

```
CLOSED → (failures >= threshold) → OPEN
OPEN → (timeout elapsed) → HALF-OPEN
HALF-OPEN → (success) → CLOSED
HALF-OPEN → (failure) → OPEN
```

## Core Implementation

### State Machine

```python
class CircuitState(Enum):
    CLOSED = "closed"      # Normal operation
    OPEN = "open"          # Blocking calls
    HALF_OPEN = "half_open"  # Testing recovery

class CircuitBreaker:
    def __init__(
        self,
        failure_threshold: int = 5,
        reset_timeout: float = 60.0,
        half_open_max_calls: int = 3,
    ):
        self.failure_threshold = failure_threshold
        self.reset_timeout = reset_timeout
        self.half_open_max_calls = half_open_max_calls
        
        self._state = CircuitState.CLOSED
        self._failure_count = 0
        self._last_failure_time: float | None = None
        self._half_open_calls = 0
        self._lock = asyncio.Lock()  # Thread-safe state
```

### Async-Safe State Transitions

```python
async def record_success(self) -> None:
    """Record successful call."""
    async with self._lock:
        self._failure_count = 0
        self._half_open_calls = 0
        self._state = CircuitState.CLOSED

async def record_failure(self) -> None:
    """Record failed call."""
    async with self._lock:
        self._failure_count += 1
        self._last_failure_time = time.monotonic()
        
        if self._state == CircuitState.HALF_OPEN:
            self._state = CircuitState.OPEN
        elif self._failure_count >= self.failure_threshold:
            self._state = CircuitState.OPEN
```

### Non-Blocking Execution

```python
async def call(self, func: Callable[[], Awaitable[T]]) -> T:
    """Execute function with circuit breaker protection."""
    if not await self.can_execute():
        raise CircuitBreakerOpenError("Circuit is open")
    
    try:
        result = await func()
        await self.record_success()
        return result
    except Exception as e:
        await self.record_failure()
        raise
```

## Best Practices

### 1. Specify Exception Types

Only count specific exceptions as failures:

```python
class CircuitBreaker:
    def __init__(
        self,
        failure_exceptions: tuple[type[Exception], ...] = (
            ConnectionError,
            TimeoutError,
            aiohttp.ClientError,
        ),
    ):
        self.failure_exceptions = failure_exceptions

    async def call(self, func):
        try:
            result = await func()
            await self.record_success()
            return result
        except self.failure_exceptions as e:
            await self.record_failure()
            raise
        except Exception:
            # Non-tracked exceptions don't trip the breaker
            raise
```

### 2. Fallback Strategies

Always provide fallbacks when circuit is open:

```python
async def get_data_with_fallback(id: str) -> dict:
    try:
        return await breaker.call(lambda: fetch_from_api(id))
    except CircuitBreakerOpenError:
        # Fallback options:
        # 1. Return cached data
        cached = await cache.get(id)
        if cached:
            return cached
        
        # 2. Return default/degraded response
        return {"id": id, "status": "unavailable", "cached": False}
        
        # 3. Queue for later processing
        await queue.push({"action": "retry_fetch", "id": id})
        return {"id": id, "status": "queued"}
```

### 3. Distributed State (Multi-Instance)

For multiple service instances, share circuit state via Redis:

```python
class RedisCircuitBreaker:
    def __init__(self, redis: Redis, name: str):
        self.redis = redis
        self.key = f"circuit:{name}"
    
    async def get_state(self) -> CircuitState:
        data = await self.redis.hgetall(self.key)
        if not data:
            return CircuitState.CLOSED
        
        state = CircuitState(data.get("state", "closed"))
        if state == CircuitState.OPEN:
            last_failure = float(data.get("last_failure", 0))
            if time.time() - last_failure > self.reset_timeout:
                return CircuitState.HALF_OPEN
        return state
    
    async def record_failure(self) -> None:
        async with self.redis.pipeline() as pipe:
            await pipe.hincrby(self.key, "failures", 1)
            await pipe.hset(self.key, "last_failure", time.time())
            failures = await pipe.hget(self.key, "failures")
            if int(failures) >= self.failure_threshold:
                await pipe.hset(self.key, "state", CircuitState.OPEN.value)
            await pipe.execute()
```

### 4. Metrics and Monitoring

Track circuit breaker state for observability:

```python
from prometheus_client import Counter, Gauge, Histogram

CIRCUIT_STATE = Gauge(
    'circuit_breaker_state',
    'Current circuit state (0=closed, 1=open, 2=half_open)',
    ['service']
)
CIRCUIT_FAILURES = Counter(
    'circuit_breaker_failures_total',
    'Total failures recorded',
    ['service']
)
CIRCUIT_TRIPS = Counter(
    'circuit_breaker_trips_total',
    'Times circuit opened',
    ['service']
)

async def record_failure(self) -> None:
    CIRCUIT_FAILURES.labels(service=self.name).inc()
    
    async with self._lock:
        self._failure_count += 1
        if self._failure_count >= self.failure_threshold:
            self._state = CircuitState.OPEN
            CIRCUIT_TRIPS.labels(service=self.name).inc()
            CIRCUIT_STATE.labels(service=self.name).set(1)
```

### 5. Combine with Other Patterns

**With Retry:**
```python
async def resilient_call():
    for attempt in range(3):
        try:
            return await breaker.call(external_api)
        except TransientError:
            if attempt < 2:
                await asyncio.sleep(2 ** attempt)  # Backoff
            else:
                raise
        except CircuitBreakerOpenError:
            break  # Don't retry if circuit is open
    return fallback_value()
```

**With Timeout:**
```python
async def call_with_timeout():
    try:
        return await asyncio.wait_for(
            breaker.call(external_api),
            timeout=10.0
        )
    except asyncio.TimeoutError:
        # Timeout counts as failure
        await breaker.record_failure()
        raise
```

**With Bulkhead:**
```python
# Limit concurrent calls to protect resources
semaphore = asyncio.Semaphore(10)

async def bulkhead_call():
    async with semaphore:
        return await breaker.call(external_api)
```

### 6. Per-Service Breakers

Use separate breakers for each external dependency:

```python
class ServiceBreakers:
    def __init__(self):
        self._breakers: dict[str, CircuitBreaker] = {}
    
    def get(self, service: str) -> CircuitBreaker:
        if service not in self._breakers:
            self._breakers[service] = CircuitBreaker(
                name=service,
                failure_threshold=5,
                reset_timeout=60.0,
            )
        return self._breakers[service]

breakers = ServiceBreakers()

async def call_payment_api():
    return await breakers.get("payment").call(payment_client.charge)

async def call_inventory_api():
    return await breakers.get("inventory").call(inventory_client.check)
```

### 7. Graceful Degradation Modes

```python
class DegradationMode(Enum):
    FULL = "full"           # All features
    REDUCED = "reduced"     # Non-critical features disabled
    MINIMAL = "minimal"     # Only essential features

async def get_product_details(id: str, mode: DegradationMode) -> dict:
    base = await get_basic_info(id)  # Always available
    
    if mode == DegradationMode.MINIMAL:
        return base
    
    try:
        reviews = await breakers.get("reviews").call(
            lambda: get_reviews(id)
        )
        base["reviews"] = reviews
    except CircuitBreakerOpenError:
        base["reviews"] = []  # Graceful degradation
    
    if mode == DegradationMode.FULL:
        try:
            recommendations = await breakers.get("recommendations").call(
                lambda: get_recommendations(id)
            )
            base["recommendations"] = recommendations
        except CircuitBreakerOpenError:
            pass  # Skip non-critical feature
    
    return base
```

## Configuration Guidelines

| Parameter | Low Latency Service | Background Job | External API |
|-----------|---------------------|----------------|--------------|
| failure_threshold | 3 | 10 | 5 |
| reset_timeout | 10s | 60s | 30s |
| half_open_max_calls | 1 | 5 | 3 |
| timeout per call | 1s | 60s | 10s |

## Libraries

- [`circuitbreaker`](https://pypi.org/project/circuitbreaker/) - Decorator-based, async support
- [`aiobreaker`](https://github.com/zuhao/aiobreaker) - Native asyncio
- [`async-circuit-breaker`](https://github.com/tsayin4/async-circuit-breaker) - No dependencies
- [`resilient-circuit`](https://resilient-circuit.readthedocs.io/) - Redis backing option

## References

- [Circuit Breaker Pattern - Martin Fowler](https://martinfowler.com/bliki/CircuitBreaker.html)
- [Python Circuit Breaker Guide](https://python.elitedev.in/python/python-circuit-breaker-pattern-complete-guide-with-redis-async-support-and-implementation-examples-24514c38/)
- [Microservices Resilience Patterns](https://johal.in/microservices-resilience-patterns-python-circuit-breaker-with-hystrix-like-libraries/)
