# Story 4.27: Create Plugin Testing Utilities and Documentation

**Status:** Draft  
**Priority:** Medium  
**Depends on:** 4.21 (Health Check Protocol)  
**Effort:** 2-3 days

---

## Problem Statement

Plugin authors currently have no guidance or utilities for testing their plugins:

1. **No test fixtures** for common plugin testing patterns
2. **No mock sinks/enrichers** for integration testing
3. **No validation utilities** for testing protocol compliance
4. **No documentation** on testing best practices
5. **No example test files** in the plugin_examples directory

This makes it harder for third-party developers to build reliable plugins.

---

## Goals

1. **Create `fapilog.testing` module** with test utilities
2. **Provide mock implementations** for all plugin types
3. **Add protocol compliance validators** that verify plugin contracts
4. **Create pytest fixtures** for common testing scenarios
5. **Document testing best practices** with examples
6. **Add example tests** to the plugin_examples directory

---

## Design

### Testing Module Structure

```
src/fapilog/testing/
├── __init__.py         # Public exports
├── mocks.py            # Mock implementations
├── fixtures.py         # Pytest fixtures
├── validators.py       # Protocol compliance checking
├── assertions.py       # Custom assertions
└── factories.py        # Test data factories
```

---

## Implementation

### Mock Implementations

```python
# src/fapilog/testing/mocks.py

from __future__ import annotations

import asyncio
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Any


@dataclass
class MockSinkConfig:
    """Configuration for mock sink."""
    
    fail_after: int | None = None  # Fail after N writes
    fail_with: Exception | None = None  # Exception to raise
    latency_seconds: float = 0.0  # Simulated latency
    health_status: bool = True


class MockSink:
    """Mock sink for testing that captures written events.
    
    Features:
    - Captures all written events for inspection
    - Configurable failure behavior
    - Configurable latency
    - Tracks lifecycle calls
    """
    
    name = "mock"
    
    def __init__(self, config: MockSinkConfig | None = None) -> None:
        self._config = config or MockSinkConfig()
        self.events: list[dict[str, Any]] = []
        self.write_count: int = 0
        self.start_called: bool = False
        self.stop_called: bool = False
        self.health_check_count: int = 0
    
    async def start(self) -> None:
        self.start_called = True
    
    async def stop(self) -> None:
        self.stop_called = True
    
    async def write(self, entry: dict[str, Any]) -> None:
        if self._config.latency_seconds > 0:
            await asyncio.sleep(self._config.latency_seconds)
        
        self.write_count += 1
        
        if self._config.fail_after is not None:
            if self.write_count > self._config.fail_after:
                raise self._config.fail_with or RuntimeError("Mock failure")
        
        self.events.append(entry.copy())
    
    async def health_check(self) -> bool:
        self.health_check_count += 1
        return self._config.health_status
    
    def reset(self) -> None:
        """Reset state for reuse in multiple tests."""
        self.events.clear()
        self.write_count = 0
        self.start_called = False
        self.stop_called = False
        self.health_check_count = 0
    
    def assert_event_count(self, expected: int) -> None:
        """Assert number of events written."""
        assert len(self.events) == expected, (
            f"Expected {expected} events, got {len(self.events)}"
        )
    
    def assert_event_contains(self, index: int, **kwargs: Any) -> None:
        """Assert event at index contains specified fields."""
        if index >= len(self.events):
            raise AssertionError(f"No event at index {index}")
        
        event = self.events[index]
        for key, value in kwargs.items():
            assert key in event, f"Event missing field: {key}"
            assert event[key] == value, (
                f"Event[{key}] expected {value!r}, got {event[key]!r}"
            )


@dataclass
class MockEnricherConfig:
    """Configuration for mock enricher."""
    
    fields_to_add: dict[str, Any] = field(default_factory=dict)
    fail_on_call: int | None = None  # Fail on Nth call
    latency_seconds: float = 0.0


class MockEnricher:
    """Mock enricher for testing that adds configurable fields.
    
    Features:
    - Adds configured fields to events
    - Tracks all enriched events
    - Configurable failure behavior
    """
    
    name = "mock"
    
    def __init__(self, config: MockEnricherConfig | None = None) -> None:
        self._config = config or MockEnricherConfig()
        self.enriched_events: list[dict[str, Any]] = []
        self.call_count: int = 0
        self.start_called: bool = False
        self.stop_called: bool = False
    
    async def start(self) -> None:
        self.start_called = True
    
    async def stop(self) -> None:
        self.stop_called = True
    
    async def enrich(self, event: dict[str, Any]) -> dict[str, Any]:
        self.call_count += 1
        
        if self._config.fail_on_call is not None:
            if self.call_count == self._config.fail_on_call:
                raise RuntimeError("Mock enricher failure")
        
        if self._config.latency_seconds > 0:
            await asyncio.sleep(self._config.latency_seconds)
        
        self.enriched_events.append(event.copy())
        return self._config.fields_to_add.copy()
    
    async def health_check(self) -> bool:
        return True
    
    def reset(self) -> None:
        self.enriched_events.clear()
        self.call_count = 0
        self.start_called = False
        self.stop_called = False


@dataclass
class MockRedactorConfig:
    """Configuration for mock redactor."""
    
    fields_to_mask: list[str] = field(default_factory=list)
    mask_string: str = "***MOCK***"


class MockRedactor:
    """Mock redactor for testing that masks configured fields."""
    
    name = "mock"
    
    def __init__(self, config: MockRedactorConfig | None = None) -> None:
        self._config = config or MockRedactorConfig()
        self.redacted_events: list[dict[str, Any]] = []
        self.call_count: int = 0
    
    async def start(self) -> None:
        pass
    
    async def stop(self) -> None:
        pass
    
    async def redact(self, event: dict[str, Any]) -> dict[str, Any]:
        self.call_count += 1
        result = event.copy()
        
        for field_path in self._config.fields_to_mask:
            self._mask_field(result, field_path.split("."))
        
        self.redacted_events.append(result)
        return result
    
    def _mask_field(self, obj: dict[str, Any], path: list[str]) -> None:
        if not path:
            return
        
        key = path[0]
        if len(path) == 1:
            if key in obj:
                obj[key] = self._config.mask_string
        else:
            if key in obj and isinstance(obj[key], dict):
                self._mask_field(obj[key], path[1:])
    
    async def health_check(self) -> bool:
        return True


class MockProcessor:
    """Mock processor for testing."""
    
    name = "mock"
    
    def __init__(self) -> None:
        self.processed_views: list[memoryview] = []
        self.call_count: int = 0
    
    async def start(self) -> None:
        pass
    
    async def stop(self) -> None:
        pass
    
    async def process(self, view: memoryview) -> memoryview:
        self.call_count += 1
        self.processed_views.append(view)
        return view
    
    async def health_check(self) -> bool:
        return True
```

### Protocol Validators

```python
# src/fapilog/testing/validators.py

from __future__ import annotations

import asyncio
import inspect
from dataclasses import dataclass
from typing import Any, Protocol, get_type_hints

from ..plugins import BaseEnricher, BaseProcessor, BaseRedactor, BaseSink


@dataclass
class ValidationResult:
    """Result of protocol validation."""
    
    valid: bool
    plugin_type: str
    errors: list[str]
    warnings: list[str]
    
    def raise_if_invalid(self) -> None:
        if not self.valid:
            raise ProtocolViolationError(
                f"Plugin violates {self.plugin_type} protocol: "
                + "; ".join(self.errors)
            )


class ProtocolViolationError(Exception):
    """Raised when a plugin violates its protocol."""
    pass


def validate_sink(sink: Any) -> ValidationResult:
    """Validate that a sink implements BaseSink protocol correctly.
    
    Checks:
    - Required methods exist and are async
    - Methods have correct signatures
    - Lifecycle methods don't raise on call
    """
    errors: list[str] = []
    warnings: list[str] = []
    
    # Check required methods
    required_methods = ["start", "stop", "write"]
    for method_name in required_methods:
        if not hasattr(sink, method_name):
            errors.append(f"Missing required method: {method_name}")
            continue
        
        method = getattr(sink, method_name)
        if not asyncio.iscoroutinefunction(method):
            errors.append(f"{method_name} must be async")
    
    # Check optional methods
    if hasattr(sink, "health_check"):
        if not asyncio.iscoroutinefunction(sink.health_check):
            warnings.append("health_check should be async")
    
    # Check write signature
    if hasattr(sink, "write"):
        sig = inspect.signature(sink.write)
        params = list(sig.parameters.keys())
        if len(params) < 1 or (len(params) == 1 and params[0] == "self"):
            errors.append("write must accept entry parameter")
    
    return ValidationResult(
        valid=len(errors) == 0,
        plugin_type="BaseSink",
        errors=errors,
        warnings=warnings,
    )


def validate_enricher(enricher: Any) -> ValidationResult:
    """Validate that an enricher implements BaseEnricher protocol correctly."""
    errors: list[str] = []
    warnings: list[str] = []
    
    required_methods = ["start", "stop", "enrich"]
    for method_name in required_methods:
        if not hasattr(enricher, method_name):
            errors.append(f"Missing required method: {method_name}")
            continue
        
        method = getattr(enricher, method_name)
        if not asyncio.iscoroutinefunction(method):
            errors.append(f"{method_name} must be async")
    
    # Check enrich returns dict
    if hasattr(enricher, "enrich"):
        sig = inspect.signature(enricher.enrich)
        params = list(sig.parameters.keys())
        if "event" not in params and "entry" not in params:
            warnings.append("enrich should accept 'event' parameter")
    
    return ValidationResult(
        valid=len(errors) == 0,
        plugin_type="BaseEnricher",
        errors=errors,
        warnings=warnings,
    )


def validate_redactor(redactor: Any) -> ValidationResult:
    """Validate that a redactor implements BaseRedactor protocol correctly."""
    errors: list[str] = []
    warnings: list[str] = []
    
    # Check name attribute
    if not hasattr(redactor, "name"):
        errors.append("Redactor must have 'name' attribute")
    
    required_methods = ["start", "stop", "redact"]
    for method_name in required_methods:
        if not hasattr(redactor, method_name):
            errors.append(f"Missing required method: {method_name}")
            continue
        
        method = getattr(redactor, method_name)
        if not asyncio.iscoroutinefunction(method):
            errors.append(f"{method_name} must be async")
    
    return ValidationResult(
        valid=len(errors) == 0,
        plugin_type="BaseRedactor",
        errors=errors,
        warnings=warnings,
    )


def validate_processor(processor: Any) -> ValidationResult:
    """Validate that a processor implements BaseProcessor protocol correctly."""
    errors: list[str] = []
    warnings: list[str] = []
    
    required_methods = ["start", "stop", "process"]
    for method_name in required_methods:
        if not hasattr(processor, method_name):
            errors.append(f"Missing required method: {method_name}")
            continue
        
        method = getattr(processor, method_name)
        if not asyncio.iscoroutinefunction(method):
            errors.append(f"{method_name} must be async")
    
    return ValidationResult(
        valid=len(errors) == 0,
        plugin_type="BaseProcessor",
        errors=errors,
        warnings=warnings,
    )


async def validate_plugin_lifecycle(plugin: Any) -> ValidationResult:
    """Validate that a plugin's lifecycle methods work correctly.
    
    Actually calls start() and stop() to verify they don't raise.
    """
    errors: list[str] = []
    warnings: list[str] = []
    plugin_type = "unknown"
    
    # Detect type
    if hasattr(plugin, "write"):
        plugin_type = "sink"
    elif hasattr(plugin, "enrich"):
        plugin_type = "enricher"
    elif hasattr(plugin, "redact"):
        plugin_type = "redactor"
    elif hasattr(plugin, "process"):
        plugin_type = "processor"
    
    # Test start
    if hasattr(plugin, "start"):
        try:
            await plugin.start()
        except Exception as e:
            errors.append(f"start() raised: {e}")
    
    # Test stop
    if hasattr(plugin, "stop"):
        try:
            await plugin.stop()
        except Exception as e:
            errors.append(f"stop() raised: {e}")
    
    # Test stop is idempotent
    if hasattr(plugin, "stop"):
        try:
            await plugin.stop()  # Second call should not raise
        except Exception as e:
            warnings.append(f"stop() not idempotent: {e}")
    
    return ValidationResult(
        valid=len(errors) == 0,
        plugin_type=plugin_type,
        errors=errors,
        warnings=warnings,
    )
```

### Pytest Fixtures

```python
# src/fapilog/testing/fixtures.py

from __future__ import annotations

from typing import TYPE_CHECKING, Any, Generator

import pytest

from .mocks import (
    MockEnricher,
    MockEnricherConfig,
    MockRedactor,
    MockRedactorConfig,
    MockSink,
    MockSinkConfig,
)

if TYPE_CHECKING:
    from _pytest.fixtures import FixtureRequest


@pytest.fixture
def mock_sink() -> MockSink:
    """Provide a clean mock sink for each test."""
    return MockSink()


@pytest.fixture
def failing_mock_sink() -> MockSink:
    """Provide a mock sink that fails after first write."""
    return MockSink(MockSinkConfig(
        fail_after=1,
        fail_with=RuntimeError("Simulated failure"),
    ))


@pytest.fixture
def slow_mock_sink() -> MockSink:
    """Provide a mock sink with simulated latency."""
    return MockSink(MockSinkConfig(latency_seconds=0.1))


@pytest.fixture
def mock_enricher() -> MockEnricher:
    """Provide a clean mock enricher for each test."""
    return MockEnricher()


@pytest.fixture
def mock_enricher_with_fields() -> MockEnricher:
    """Provide a mock enricher that adds test fields."""
    return MockEnricher(MockEnricherConfig(
        fields_to_add={"test_field": "test_value", "timestamp": "2025-01-01T00:00:00Z"}
    ))


@pytest.fixture
def mock_redactor() -> MockRedactor:
    """Provide a clean mock redactor for each test."""
    return MockRedactor()


@pytest.fixture
def mock_redactor_with_fields() -> MockRedactor:
    """Provide a mock redactor that masks password fields."""
    return MockRedactor(MockRedactorConfig(
        fields_to_mask=["password", "secret", "user.password"]
    ))


@pytest.fixture
def sample_log_event() -> dict[str, Any]:
    """Provide a sample log event for testing."""
    return {
        "level": "INFO",
        "message": "Test message",
        "timestamp": "2025-01-02T10:00:00Z",
        "logger": "test",
        "metadata": {
            "user_id": "user123",
            "request_id": "req456",
        },
    }


@pytest.fixture
def sample_error_event() -> dict[str, Any]:
    """Provide a sample error event with exception info."""
    return {
        "level": "ERROR",
        "message": "Something went wrong",
        "timestamp": "2025-01-02T10:00:00Z",
        "logger": "test",
        "exception": {
            "type": "ValueError",
            "message": "Invalid input",
            "traceback": "...",
        },
    }


@pytest.fixture
def sample_sensitive_event() -> dict[str, Any]:
    """Provide a sample event with sensitive data for redaction testing."""
    return {
        "level": "INFO",
        "message": "User login",
        "user": {
            "id": "user123",
            "email": "user@example.com",
            "password": "secret123",
        },
        "api_key": "sk-xxx-xxx",
        "credit_card": "4111-1111-1111-1111",
    }
```

### Test Data Factories

```python
# src/fapilog/testing/factories.py

from __future__ import annotations

import random
import string
from datetime import datetime, timezone
from typing import Any


def create_log_event(
    level: str = "INFO",
    message: str | None = None,
    **metadata: Any,
) -> dict[str, Any]:
    """Create a log event with sensible defaults.
    
    Args:
        level: Log level (DEBUG, INFO, WARNING, ERROR)
        message: Log message (auto-generated if None)
        **metadata: Additional metadata fields
    
    Returns:
        Complete log event dict
    """
    if message is None:
        message = f"Test message {random.randint(1000, 9999)}"
    
    return {
        "level": level,
        "message": message,
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "logger": "test",
        "correlation_id": generate_correlation_id(),
        "metadata": metadata,
    }


def create_batch_events(
    count: int,
    level: str = "INFO",
    **metadata: Any,
) -> list[dict[str, Any]]:
    """Create a batch of log events.
    
    Args:
        count: Number of events to create
        level: Log level for all events
        **metadata: Metadata to include in all events
    
    Returns:
        List of log events
    """
    return [
        create_log_event(
            level=level,
            message=f"Batch message {i}",
            **metadata,
        )
        for i in range(count)
    ]


def generate_correlation_id() -> str:
    """Generate a random correlation ID."""
    return "".join(random.choices(string.hexdigits.lower(), k=32))


def create_sensitive_event() -> dict[str, Any]:
    """Create an event with various sensitive fields for redaction testing."""
    return {
        "level": "INFO",
        "message": "User action",
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "user": {
            "id": "user_123",
            "email": "test@example.com",
            "password": "supersecret123",
            "ssn": "123-45-6789",
        },
        "payment": {
            "card_number": "4111111111111111",
            "cvv": "123",
            "expiry": "12/25",
        },
        "auth": {
            "api_key": "sk-abcdef123456",
            "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
        },
        "url": "https://user:pass@api.example.com/data",
    }
```

### Public Exports

```python
# src/fapilog/testing/__init__.py

"""
Testing utilities for fapilog plugins.

This module provides mocks, fixtures, and validators for testing
custom plugins.

Example:
    from fapilog.testing import MockSink, validate_sink
    
    def test_my_sink():
        sink = MockSink()
        result = validate_sink(sink)
        assert result.valid
"""

from .factories import (
    create_batch_events,
    create_log_event,
    create_sensitive_event,
    generate_correlation_id,
)
from .mocks import (
    MockEnricher,
    MockEnricherConfig,
    MockProcessor,
    MockRedactor,
    MockRedactorConfig,
    MockSink,
    MockSinkConfig,
)
from .validators import (
    ProtocolViolationError,
    ValidationResult,
    validate_enricher,
    validate_plugin_lifecycle,
    validate_processor,
    validate_redactor,
    validate_sink,
)

__all__ = [
    # Mocks
    "MockSink",
    "MockSinkConfig",
    "MockEnricher",
    "MockEnricherConfig",
    "MockRedactor",
    "MockRedactorConfig",
    "MockProcessor",
    # Validators
    "validate_sink",
    "validate_enricher",
    "validate_redactor",
    "validate_processor",
    "validate_plugin_lifecycle",
    "ValidationResult",
    "ProtocolViolationError",
    # Factories
    "create_log_event",
    "create_batch_events",
    "create_sensitive_event",
    "generate_correlation_id",
]
```

---

## Documentation

### Create `docs/plugins/testing.md`

```markdown
# Testing Plugins

fapilog provides utilities to help you test your custom plugins effectively.

## Quick Start

```python
from fapilog.testing import MockSink, validate_sink, create_log_event

# Test that your sink implements the protocol correctly
def test_my_sink_protocol():
    my_sink = MySink()
    result = validate_sink(my_sink)
    result.raise_if_invalid()

# Use mock sinks for integration testing
async def test_my_enricher_adds_fields():
    sink = MockSink()
    enricher = MyEnricher()
    
    event = create_log_event(level="INFO", message="test")
    enriched = await enricher.enrich(event)
    
    assert "my_field" in enriched

# Test with the mock in a pipeline
async def test_full_pipeline():
    sink = MockSink()
    # ... configure pipeline with sink ...
    
    logger.info("test message")
    await logger.stop_and_drain()
    
    sink.assert_event_count(1)
    sink.assert_event_contains(0, level="INFO")
```

## Mock Classes

### MockSink

```python
from fapilog.testing import MockSink, MockSinkConfig

# Basic usage
sink = MockSink()
await sink.write({"level": "INFO", "message": "test"})
assert len(sink.events) == 1

# Simulate failures
failing_sink = MockSink(MockSinkConfig(
    fail_after=5,  # Fail after 5 writes
    fail_with=ConnectionError("Simulated timeout"),
))

# Simulate latency
slow_sink = MockSink(MockSinkConfig(latency_seconds=0.5))
```

### MockEnricher

```python
from fapilog.testing import MockEnricher, MockEnricherConfig

enricher = MockEnricher(MockEnricherConfig(
    fields_to_add={"service": "myapp", "env": "test"}
))

event = {"level": "INFO"}
added = await enricher.enrich(event)
assert added == {"service": "myapp", "env": "test"}
```

## Protocol Validators

Validate that your plugin correctly implements its protocol:

```python
from fapilog.testing import validate_sink, validate_enricher

result = validate_sink(my_sink)
if not result.valid:
    print("Errors:", result.errors)
    print("Warnings:", result.warnings)

# Or raise on violations
result.raise_if_invalid()
```

## Lifecycle Testing

Test that your plugin's lifecycle methods work correctly:

```python
from fapilog.testing import validate_plugin_lifecycle

result = await validate_plugin_lifecycle(my_sink)
assert result.valid, result.errors
```

## Pytest Fixtures

Register fapilog fixtures in your `conftest.py`:

```python
# conftest.py
pytest_plugins = ["fapilog.testing.fixtures"]
```

Then use in tests:

```python
async def test_with_mock_sink(mock_sink, sample_log_event):
    await mock_sink.write(sample_log_event)
    mock_sink.assert_event_count(1)
```

## Test Data Factories

Generate test data easily:

```python
from fapilog.testing import create_log_event, create_batch_events

event = create_log_event(level="ERROR", user_id="123")
events = create_batch_events(100, level="INFO")
```

## Best Practices

1. **Test protocol compliance first** using validators
2. **Test lifecycle methods** are idempotent and don't raise
3. **Test error containment** - plugins should not crash the pipeline
4. **Test with realistic data** using factories
5. **Test edge cases** - empty events, large payloads, special characters
```

---

## Acceptance Criteria

### Must Have

- [ ] `fapilog.testing` module with public exports
- [ ] `MockSink` with configurable behavior
- [ ] `MockEnricher` with configurable fields
- [ ] `MockRedactor` with configurable masking
- [ ] `validate_sink()`, `validate_enricher()`, `validate_redactor()`
- [ ] `create_log_event()` factory function
- [ ] Documentation for testing plugins
- [ ] Tests for the testing utilities themselves

### Should Have

- [ ] Pytest fixtures for common mocks
- [ ] `validate_plugin_lifecycle()` async validator
- [ ] Example tests in plugin_examples directory

### Won't Have (This Story)

- [ ] Property-based testing utilities
- [ ] Performance testing utilities
- [ ] Coverage analysis utilities

---

## Test Plan

1. **Unit Tests**
   - `test_mock_sink_captures_events()`
   - `test_mock_sink_fails_after_configured()`
   - `test_mock_enricher_adds_fields()`
   - `test_validate_sink_catches_missing_methods()`
   - `test_validate_sink_catches_sync_methods()`
   - `test_create_log_event_generates_valid_event()`
   - `test_create_batch_events_generates_many()`

2. **Integration Tests**
   - Full pipeline with mocks
   - Validator catches real protocol violations

---

## Estimated Effort

| Task | Effort |
|------|--------|
| Mock implementations | 0.5 day |
| Validators | 0.5 day |
| Factories and fixtures | 0.25 day |
| Documentation | 0.5 day |
| Tests | 0.5 day |
| Example tests | 0.25 day |
| **Total** | **~2.5 days** |

---

## Related Stories

- **4.21** - Health Check Protocol
- **4.23** - Plugin Metadata Completeness



