# Story 5.2: Add Filter Plugin Type

## Status: Complete

## Priority: High

## Estimated Effort: Medium (4-5 days)

## Dependencies: Story 5.0 (processor wiring pattern)

## Epic: Plugin System Expansion

---

## Context

Enterprise logging libraries (structlog, loguru, python-json-logger) provide filtering capabilities to drop, modify, or route logs before they enter the processing pipeline. fapilog currently lacks this capability.

### Audit Findings (Validated)

A code audit confirmed:

- **No filter plugin type exists** - only sinks, enrichers, redactors, processors
- **`core.log_level` setting is completely unused** - DEBUG logs process even when set to INFO
- **No sampling or rate limiting** for log events
- **All log methods enqueue unconditionally** - no level gating

Filters are essential for:

- Dropping debug logs in production
- Routing errors to specific sinks
- Rate-limiting verbose log sources
- PII pre-screening before enrichment
- Conditional log suppression

---

## Problem Statement

Users cannot currently:

1. Drop logs based on content before enrichment (saves processing cost)
2. Implement custom log level filtering beyond global level
3. Route specific logs to specific sinks
4. Apply rate limiting at the filter level

**Additionally**: The `core.log_level` setting exists but is **completely unused** - DEBUG logs are processed even when `log_level=INFO`. This is confusing and wasteful.

---

## Acceptance Criteria

### AC1: BaseFilter Protocol

- [ ] Define `BaseFilter` protocol in `plugins/filters/__init__.py`
- [ ] Protocol includes `async def filter(event: dict) -> dict | None`
- [ ] Return `None` to drop the event; return modified event to continue
- [ ] Include `name`, `start()`, `stop()`, `health_check()` methods

### AC2: Filter Stage in Pipeline

- [ ] Filters execute BEFORE enrichers (earliest stage)
- [ ] Filters run in configured order
- [ ] Filter returning `None` stops pipeline for that event
- [ ] Filter errors are contained; event continues

### AC3: Built-in Filters

- [ ] `LevelFilter` - Filter by log level (drop below threshold)
- [ ] `SamplingFilter` - Probabilistic sampling (drop X% of logs)
- [ ] `RateLimitFilter` - Token bucket rate limiting per key

### AC4: Configuration

- [ ] `core.filters` setting for filter plugin names
- [ ] `filter_config` settings block for per-filter config
- [ ] Filters respect allow/deny lists

### AC5: Metrics and Diagnostics

- [ ] Filter drops recorded in metrics (`events_filtered`)
- [ ] Filter execution time tracked
- [ ] Diagnostics emitted on filter errors
- [ ] Add `record_events_filtered(count)` method to `MetricsCollector`

### AC6: Wire core.log_level Setting

- [ ] When `core.log_level` is set, automatically prepend a `LevelFilter`
- [ ] `core.log_level=INFO` drops DEBUG events without explicit filter config
- [ ] Explicit `core.filters` config takes precedence (user can override)
- [ ] Maintains backward compatibility (existing configs work as expected)

### AC7: Optional Enqueue Short-Circuit (Stretch Goal)

- [ ] For simple level filtering, check level in `_enqueue()` before queuing
- [ ] Avoids queue overhead for obviously-dropped events
- [ ] Only applies when sole filter is LevelFilter with no custom filters

---

## Technical Design

### 1. BaseFilter Protocol

```python
# src/fapilog/plugins/filters/__init__.py
from __future__ import annotations

from typing import Protocol, runtime_checkable

from ..loader import register_builtin
from .level import LevelFilter
from .sampling import SamplingFilter
from .rate_limit import RateLimitFilter


@runtime_checkable
class BaseFilter(Protocol):
    """Authoring contract for filters that gate log events.

    Filters run before enrichers and can drop or modify events.
    Return None to drop an event; return the (optionally modified)
    event dict to continue processing.

    Attributes:
        name: Unique identifier for this filter type.
    """

    name: str

    async def start(self) -> None:
        """Initialize filter resources (optional)."""

    async def stop(self) -> None:
        """Release filter resources (optional)."""

    async def filter(self, event: dict) -> dict | None:
        """Filter an event.

        Args:
            event: Log event dict with level, message, metadata, etc.

        Returns:
            The event dict to continue processing, or None to drop.
        """

    async def health_check(self) -> bool:
        """Return True if the filter is healthy."""
        return True


async def filter_in_order(
    event: dict,
    filters: Iterable[BaseFilter],
    *,
    metrics: MetricsCollector | None = None,
) -> dict | None:
    """Apply filters sequentially.

    Returns None if any filter drops the event.
    Returns the (possibly modified) event if all filters pass.
    """
    current: dict = dict(event)

    for f in filters:
        plugin_name = getattr(f, "name", type(f).__name__)
        try:
            async with plugin_timer(metrics, plugin_name):
                result = await f.filter(dict(current))

            if result is None:
                # Event dropped
                if metrics is not None:
                    await metrics.record_events_filtered(1)
                return None

            current = result
        except Exception as exc:
            # Contain filter errors; continue with current event
            try:
                diagnostics.warn(
                    "filter",
                    "filter exception",
                    filter=plugin_name,
                    reason=str(exc),
                )
            except Exception:
                pass
            continue

    return current


# Register built-in filters
register_builtin("fapilog.filters", "level", LevelFilter)
register_builtin("fapilog.filters", "sampling", SamplingFilter)
register_builtin("fapilog.filters", "rate_limit", RateLimitFilter, aliases=["rate-limit"])

__all__ = [
    "BaseFilter",
    "filter_in_order",
    "LevelFilter",
    "SamplingFilter",
    "RateLimitFilter",
]
```

### 2. LevelFilter Implementation

```python
# src/fapilog/plugins/filters/level.py
from dataclasses import dataclass
from typing import Any


LEVEL_PRIORITY = {
    "DEBUG": 10,
    "INFO": 20,
    "WARNING": 30,
    "WARN": 30,
    "ERROR": 40,
    "CRITICAL": 50,
    "FATAL": 50,
}


@dataclass
class LevelFilterConfig:
    min_level: str = "INFO"
    drop_below: bool = True


class LevelFilter:
    """Filter events by log level."""

    name = "level"

    def __init__(self, *, config: LevelFilterConfig | None = None) -> None:
        cfg = config or LevelFilterConfig()
        self._min_priority = LEVEL_PRIORITY.get(cfg.min_level.upper(), 20)
        self._drop_below = cfg.drop_below

    async def start(self) -> None:
        pass

    async def stop(self) -> None:
        pass

    async def filter(self, event: dict) -> dict | None:
        level = str(event.get("level", "INFO")).upper()
        priority = LEVEL_PRIORITY.get(level, 20)

        if self._drop_below and priority < self._min_priority:
            return None

        return event

    async def health_check(self) -> bool:
        return True


PLUGIN_METADATA = {
    "name": "level",
    "version": "1.0.0",
    "plugin_type": "filter",
    "entry_point": "fapilog.plugins.filters.level:LevelFilter",
    "description": "Filter events by log level threshold.",
    "author": "Fapilog Core",
    "compatibility": {"min_fapilog_version": "0.4.0"},
    "api_version": "1.0",
}
```

### 3. SamplingFilter Implementation

```python
# src/fapilog/plugins/filters/sampling.py
import random
from dataclasses import dataclass


@dataclass
class SamplingFilterConfig:
    sample_rate: float = 1.0  # 1.0 = keep all, 0.1 = keep 10%
    seed: int | None = None


class SamplingFilter:
    """Probabilistic sampling filter."""

    name = "sampling"

    def __init__(self, *, config: SamplingFilterConfig | None = None) -> None:
        cfg = config or SamplingFilterConfig()
        self._rate = max(0.0, min(1.0, cfg.sample_rate))
        if cfg.seed is not None:
            random.seed(cfg.seed)

    async def start(self) -> None:
        pass

    async def stop(self) -> None:
        pass

    async def filter(self, event: dict) -> dict | None:
        if self._rate >= 1.0:
            return event
        if self._rate <= 0.0:
            return None

        if random.random() < self._rate:
            return event
        return None

    async def health_check(self) -> bool:
        return True
```

### 4. Pipeline Integration

Update `worker.py`:

```python
class LoggerWorker:
    def __init__(
        self,
        *,
        # ... existing params ...
        filters_getter: Callable[[], list[BaseFilter]],
    ) -> None:
        # ... existing init ...
        self._filters_getter = filters_getter

    async def _flush_batch(self, batch: list[dict[str, Any]]) -> None:
        if not batch:
            return

        for entry in batch:
            # NEW: Apply filters first
            entry = await self._apply_filters(entry)
            if entry is None:
                continue  # Event was dropped

            entry = await self._apply_enrichers(entry)
            entry = await self._apply_redactors(entry)
            # ... rest of pipeline ...

    async def _apply_filters(self, entry: dict[str, Any]) -> dict[str, Any] | None:
        filters = self._filters_getter()
        if not filters:
            return entry

        return await filter_in_order(entry, filters, metrics=self._metrics)
```

### 5. Add events_filtered Metric

Update `metrics/metrics.py`:

```python
class MetricsCollector:
    def __init__(self, enabled: bool = False) -> None:
        # ... existing ...
        self._events_filtered = 0

    async def record_events_filtered(self, count: int = 1) -> None:
        """Record count of events dropped by filters."""
        if not self._enabled:
            return
        self._events_filtered += count

    async def snapshot(self) -> MetricsSnapshot:
        # ... add events_filtered to snapshot ...
```

### 6. Wire core.log_level to Default LevelFilter

Update `__init__.py` in `_build_pipeline()`:

```python
def _build_pipeline(settings: _Settings) -> tuple[...]:
    # ... existing code ...

    # Build filters list
    filter_names = list(core_cfg.filters or [])

    # Auto-prepend LevelFilter if core.log_level is set and not DEBUG
    # (DEBUG means "show everything" - no filtering needed)
    if core_cfg.log_level != "DEBUG" and "level" not in filter_names:
        # Prepend implicit level filter
        filter_names.insert(0, "level")
        # Inject config for implicit filter
        filter_configs = _filter_configs(settings)
        if "level" not in filter_configs:
            filter_configs["level"] = {"min_level": core_cfg.log_level}

    filters = _load_plugins(
        "fapilog.filters", filter_names, settings, filter_configs
    )

    return sinks, enrichers, redactors, processors, filters, metrics
```

### 7. Optional Enqueue Short-Circuit (Stretch Goal)

For maximum efficiency, check level before queuing:

```python
# In SyncLoggerFacade._enqueue():

def _enqueue(self, level: str, message: str, ...) -> None:
    # Fast-path level check (only if simple level filtering)
    if self._level_gate is not None:
        if LEVEL_PRIORITY.get(level, 20) < self._level_gate:
            return  # Drop before queue

    # ... rest of enqueue logic ...
```

This avoids queue overhead entirely for DEBUG logs in production. Implementation is optional and should only activate when:

- No custom filters configured
- Only implicit LevelFilter from `core.log_level`

---

## Test Plan

### Unit Tests

1. **test_level_filter.py**

   - Test drops DEBUG when min_level=INFO
   - Test passes INFO when min_level=INFO
   - Test case insensitivity

2. **test_sampling_filter.py**

   - Test 1.0 rate keeps all
   - Test 0.0 rate drops all
   - Test 0.5 rate is approximately 50%

3. **test_filter_pipeline.py**
   - Test multiple filters chain correctly
   - Test first filter drop stops pipeline
   - Test filter errors contained

### Integration Tests

1. **test_filter_with_enrichers.py**

   - Verify filters run before enrichers
   - Verify dropped events skip enrichment

2. **test_core_log_level_wiring.py**

   - Test `core.log_level=INFO` drops DEBUG events
   - Test `core.log_level=DEBUG` keeps all events
   - Test explicit `core.filters` overrides implicit level filter
   - Test `core.log_level=ERROR` drops INFO and WARNING

3. **test_events_filtered_metric.py**
   - Test `events_filtered` counter increments on drop
   - Test counter visible in metrics snapshot

---

## Documentation Updates

1. Create `docs/plugins/filters.md`
2. Update `docs/plugins/index.md` to include Filters section
3. Update `docs/core-concepts/pipeline-architecture.md`
4. Add filter examples to `docs/examples/`
5. Update `docs/configuration.md` to document `core.log_level` behavior
6. Document that `core.log_level` now auto-creates a LevelFilter

---

## Rollout Plan

1. Add `record_events_filtered()` to MetricsCollector
2. Implement BaseFilter protocol and `filter_in_order()` helper
3. Implement LevelFilter (simplest)
4. Wire filters into pipeline (`_apply_filters()` in worker)
5. Wire `core.log_level` to auto-prepend LevelFilter
6. Implement SamplingFilter
7. Implement RateLimitFilter
8. (Stretch) Add enqueue short-circuit for simple level filtering
9. Documentation
10. Release

---

## Related Stories

- Story 5.0: Wire processors into pipeline (similar wiring pattern)
- Story 5.3: Sampling plugin (extends this work)
