# Story 4.21: Add Health Check Protocol to Plugin System

**Status:** Complete  
**Priority:** High  
**Depends on:** None  
**Effort:** 2-3 days  
**PR:** [#167](https://github.com/chris-haste/fapilog/pull/167)

---

## Problem Statement

The current plugin protocol definitions are inconsistent regarding health checking:

1. **`BaseSink` protocol lacks `health_check()`** method despite it being implemented by `HttpSink` and `WebhookSink`
2. **File-based sinks (`StdoutJsonSink`, `RotatingFileSink`) have no health check** capability
3. **No aggregated health API** exists to query the health status of all loaded plugins
4. **Documentation shows health_check examples** but the protocol doesn't require it

This inconsistency makes it difficult for:
- Library consumers to introspect plugin state programmatically
- Applications to verify logging pipeline readiness before processing
- Diagnosing issues when log output stops unexpectedly

---

## Goals

1. **Define `HealthCheckable` protocol** as an optional mixin for plugins that support health checks
2. **Add optional `health_check()` to all plugin base protocols** with sensible defaults
3. **Implement health checks for file-based sinks** (check file/directory accessibility)
4. **Create aggregated health API** via the logger facade
5. **Document health check patterns** for plugin authors

---

## Design

### Option A: Optional Protocol Method (Recommended)

Add `health_check()` as an optional method with a default implementation in each protocol:

```python
# plugins/sinks/__init__.py

@runtime_checkable
class BaseSink(Protocol):
    """Authoring contract for sinks that emit finalized log entries."""

    async def start(self) -> None: ...
    async def stop(self) -> None: ...
    async def write(self, _entry: dict) -> None: ...

    # NEW: Optional health check with default implementation
    async def health_check(self) -> bool:
        """Check if the sink is healthy and able to accept writes.
        
        Returns:
            True if healthy, False otherwise.
            
        Default implementation returns True (assume healthy).
        Override for sinks that can detect degraded state.
        """
        return True
```

### Option B: Separate HealthCheckable Protocol

Define a separate protocol for health-checkable plugins:

```python
# plugins/health.py

from typing import Protocol, runtime_checkable

@runtime_checkable
class HealthCheckable(Protocol):
    """Optional protocol for plugins that support health checking."""
    
    async def health_check(self) -> bool:
        """Return True if the plugin is healthy."""
        ...


@runtime_checkable
class DetailedHealthCheckable(Protocol):
    """Extended protocol with detailed health information."""
    
    async def health_check(self) -> bool: ...
    
    async def health_details(self) -> dict[str, Any]:
        """Return detailed health status.
        
        Returns:
            Dict with keys: status, last_error, latency_ms, etc.
        """
        ...
```

**Recommendation:** Use Option A for simplicity with Option B's `DetailedHealthCheckable` as an optional extension.

---

## Implementation Plan

### Phase 1: Update Protocol Definitions

**File: `src/fapilog/plugins/sinks/__init__.py`**

```python
@runtime_checkable
class BaseSink(Protocol):
    """Authoring contract for sinks that emit finalized log entries."""

    async def start(self) -> None: ...
    async def stop(self) -> None: ...
    async def write(self, _entry: dict) -> None: ...
    
    async def health_check(self) -> bool:
        """Check if sink is healthy. Default returns True."""
        return True
```

**File: `src/fapilog/plugins/enrichers/__init__.py`**

```python
@runtime_checkable
class BaseEnricher(Protocol):
    async def start(self) -> None: ...
    async def stop(self) -> None: ...
    async def enrich(self, event: dict) -> dict: ...
    
    async def health_check(self) -> bool:
        """Check if enricher is healthy. Default returns True."""
        return True
```

**File: `src/fapilog/plugins/redactors/__init__.py`**

```python
@runtime_checkable
class BaseRedactor(Protocol):
    name: str
    async def start(self) -> None: ...
    async def stop(self) -> None: ...
    async def redact(self, event: dict) -> dict: ...
    
    async def health_check(self) -> bool:
        """Check if redactor is healthy. Default returns True."""
        return True
```

**File: `src/fapilog/plugins/processors/__init__.py`**

```python
@runtime_checkable
class BaseProcessor(Protocol):
    async def start(self) -> None: ...
    async def stop(self) -> None: ...
    async def process(self, view: memoryview) -> memoryview: ...
    
    async def health_check(self) -> bool:
        """Check if processor is healthy. Default returns True."""
        return True
```

### Phase 2: Implement Health Checks for Built-in Sinks

**File: `src/fapilog/plugins/sinks/stdout_json.py`**

```python
class StdoutJsonSink:
    # ... existing code ...
    
    async def health_check(self) -> bool:
        """Check if stdout is writable."""
        try:
            # Verify stdout buffer is accessible
            import sys
            return sys.stdout.buffer.writable()
        except Exception:
            return False
```

**File: `src/fapilog/plugins/sinks/rotating_file.py`**

```python
class RotatingFileSink:
    # ... existing code ...
    
    async def health_check(self) -> bool:
        """Check if log directory is accessible and writable."""
        try:
            if self._active_path is None:
                return False
            
            def _check() -> bool:
                # Verify directory exists and is writable
                if not self._cfg.directory.exists():
                    return False
                if not os.access(self._cfg.directory, os.W_OK):
                    return False
                # Verify active file is writable if open
                if self._active_file is not None:
                    return not self._active_file.closed
                return True
            
            return await asyncio.to_thread(_check)
        except Exception:
            return False
```

**File: `src/fapilog/plugins/sinks/mmap_persistence.py`**

```python
class MemoryMappedPersistence:
    # ... existing code ...
    
    async def health_check(self) -> bool:
        """Check if mmap file is open and accessible."""
        return self.is_open and self._mmap is not None and self._fd is not None
```

### Phase 3: Add Health Details Model

**File: `src/fapilog/plugins/health.py` (NEW)**

```python
"""
Plugin health check models and utilities.
"""

from __future__ import annotations

from dataclasses import dataclass, field
from datetime import datetime, timezone
from enum import Enum
from typing import Any, Protocol, runtime_checkable


class HealthStatus(Enum):
    """Plugin health status levels."""
    HEALTHY = "healthy"
    DEGRADED = "degraded"
    UNHEALTHY = "unhealthy"
    UNKNOWN = "unknown"


@dataclass
class PluginHealth:
    """Health status for a single plugin."""
    
    name: str
    plugin_type: str  # sink, enricher, redactor, processor
    status: HealthStatus = HealthStatus.UNKNOWN
    healthy: bool = True
    last_check: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
    last_error: str | None = None
    latency_ms: float | None = None
    details: dict[str, Any] = field(default_factory=dict)


@dataclass
class AggregatedHealth:
    """Aggregated health status across all plugins."""
    
    overall_status: HealthStatus
    timestamp: datetime
    plugins: list[PluginHealth]
    healthy_count: int
    unhealthy_count: int
    degraded_count: int
    
    @property
    def all_healthy(self) -> bool:
        return self.unhealthy_count == 0 and self.degraded_count == 0


@runtime_checkable
class DetailedHealthCheckable(Protocol):
    """Extended protocol for plugins with detailed health information."""
    
    async def health_check(self) -> bool: ...
    async def health_details(self) -> PluginHealth: ...


async def check_plugin_health(
    plugin: Any,
    plugin_type: str,
    timeout_seconds: float = 5.0,
) -> PluginHealth:
    """Check health of a single plugin with timeout."""
    import asyncio
    import time
    
    name = getattr(plugin, "name", type(plugin).__name__)
    health = PluginHealth(name=name, plugin_type=plugin_type)
    
    if not hasattr(plugin, "health_check"):
        health.status = HealthStatus.UNKNOWN
        return health
    
    start = time.perf_counter()
    try:
        result = await asyncio.wait_for(
            plugin.health_check(),
            timeout=timeout_seconds,
        )
        health.latency_ms = (time.perf_counter() - start) * 1000
        health.healthy = bool(result)
        health.status = HealthStatus.HEALTHY if result else HealthStatus.UNHEALTHY
    except asyncio.TimeoutError:
        health.healthy = False
        health.status = HealthStatus.UNHEALTHY
        health.last_error = f"Health check timed out after {timeout_seconds}s"
    except Exception as exc:
        health.healthy = False
        health.status = HealthStatus.UNHEALTHY
        health.last_error = str(exc)
    
    return health
```

### Phase 4: Add Aggregated Health API to Logger

**File: `src/fapilog/core/logger.py`**

Add to `SyncLoggerFacade` and `AsyncLoggerFacade`:

```python
async def check_health(self) -> AggregatedHealth:
    """Check health of all loaded plugins.
    
    Returns:
        AggregatedHealth with status of all sinks, enrichers, redactors.
    """
    from ..plugins.health import (
        AggregatedHealth,
        HealthStatus,
        check_plugin_health,
    )
    from datetime import datetime, timezone
    
    plugin_healths = []
    
    # Check enrichers
    for e in self._enrichers:
        health = await check_plugin_health(e, "enricher")
        plugin_healths.append(health)
    
    # Check redactors
    for r in self._redactors:
        health = await check_plugin_health(r, "redactor")
        plugin_healths.append(health)
    
    # Check sinks (would need sink references stored)
    # ... implementation details ...
    
    healthy = sum(1 for p in plugin_healths if p.healthy)
    unhealthy = sum(1 for p in plugin_healths if not p.healthy and p.status == HealthStatus.UNHEALTHY)
    degraded = sum(1 for p in plugin_healths if p.status == HealthStatus.DEGRADED)
    
    overall = HealthStatus.HEALTHY
    if unhealthy > 0:
        overall = HealthStatus.UNHEALTHY
    elif degraded > 0:
        overall = HealthStatus.DEGRADED
    
    return AggregatedHealth(
        overall_status=overall,
        timestamp=datetime.now(timezone.utc),
        plugins=plugin_healths,
        healthy_count=healthy,
        unhealthy_count=unhealthy,
        degraded_count=degraded,
    )
```

### Phase 5: Update Documentation

Add `docs/plugins/health-checks.md`:

```markdown
# Plugin Health Checks

fapilog plugins can expose an async `health_check()` method to report readiness.
Base protocols return `True` by default, so existing plugins stay compatible.

## Checking Health from Code

```python
logger = get_logger()

# Check all plugin health
health = await logger.check_health()

if health.all_healthy:
    print("All plugins healthy")
else:
    for plugin in health.plugins:
        if not plugin.healthy:
            print(f"Unhealthy: {plugin.name} - {plugin.last_error}")
```

## Implementing Health Checks in Custom Plugins

```python
class MyCustomSink(BaseSink):
    async def health_check(self) -> bool:
        # Check connection to remote service
        try:
            response = await self._client.ping()
            return response.status == 200
        except Exception:
            return False
```
```

---

## Acceptance Criteria

### Must Have

- [x] `health_check()` method added to all base protocols with default implementation
- [x] `StdoutJsonSink` implements health check (stdout writable)
- [x] `RotatingFileSink` implements health check (directory writable)
- [x] `MemoryMappedPersistence` implements health check (file open)
- [x] `PluginHealth` and `AggregatedHealth` models created
- [x] `logger.check_health()` returns aggregated health
- [x] All existing tests pass (967 passed)
- [x] New tests for health check functionality (10 tests)
- [x] 90%+ coverage maintained (91.0% overall, 97% on health.py)

### Should Have

- [x] `DetailedHealthCheckable` protocol for extended health info
- [x] Health check timeout configuration (5s default)
- [x] Documentation for health checks (`docs/plugins/health-checks.md`)

### Out of Scope

These are application-level concerns, not library concerns:

- HTTP health endpoints
- Prometheus health metrics  
- Auto-healing based on health status

---

## Implementation Summary

**Files Changed:**
- `src/fapilog/plugins/health.py` (new)
- `src/fapilog/plugins/sinks/__init__.py`
- `src/fapilog/plugins/enrichers/__init__.py`
- `src/fapilog/plugins/redactors/__init__.py`
- `src/fapilog/plugins/processors/__init__.py`
- `src/fapilog/plugins/sinks/stdout_json.py`
- `src/fapilog/plugins/sinks/rotating_file.py`
- `src/fapilog/plugins/sinks/mmap_persistence.py`
- `src/fapilog/core/logger.py`
- `src/fapilog/__init__.py`
- `docs/plugins/health-checks.md` (new)
- `tests/unit/test_plugin_health.py` (new)

**Key Design Decisions:**
- Used Option A (protocol method) with Option B's `DetailedHealthCheckable` as extension
- Added `aggregate_plugin_health()` helper to avoid code duplication in logger facades
- Default `health_check()` returns `True` for backward compatibility

---

## Test Plan

### Implemented Tests (`tests/unit/test_plugin_health.py`)

1. `test_default_health_check_true` - Happy path
2. `test_health_check_failure` - Plugin returns False
3. `test_plugin_without_health_check_method` - UNKNOWN status
4. `test_health_check_timeout` - Timeout handling
5. `test_health_check_exception` - Exception handling
6. `test_stdout_health_check` - StdoutJsonSink
7. `test_rotating_file_health_check` - RotatingFileSink
8. `test_mmap_persistence_health_check` - MemoryMappedPersistence
9. `test_logger_check_health_aggregates` - Logger facade integration
10. `test_aggregated_health_helpers` - AggregatedHealth properties

---

## Risks & Mitigations

| Risk | Impact | Mitigation |
|------|--------|------------|
| Breaking existing custom plugins | Medium | Default implementation returns True |
| Health check overhead | Low | Optional, async, with timeout |
| False positives | Medium | Clear documentation on implementation |

---

## Estimated vs Actual Effort

| Task | Estimated | Actual |
|------|-----------|--------|
| Update protocol definitions | 0.5 day | ✓ |
| Implement file sink health checks | 0.5 day | ✓ |
| Create health models | 0.5 day | ✓ |
| Add aggregated health API | 0.5 day | ✓ |
| Tests | 0.5 day | ✓ |
| Documentation | 0.25 day | ✓ |
| **Total** | **~2.5 days** | **~1 day** |

---

## Related Stories

- **4.21** - This story
- **4.23** - Plugin Metadata Completeness
- **4.27** - Plugin Testing Utilities

