# Story 5.3: Enhanced Sampling and Rate Limiting Plugins

## Status: Completed

## Priority: High

## Estimated Effort: Medium (3-4 days)

## Dependencies: Story 5.2 (BaseFilter plugin type)

## Epic: Plugin System Expansion

---

## Context

High-volume logging systems require sophisticated sampling and rate limiting to:

- Reduce log storage costs (often 80%+ of observability budget)
- Prevent log storms during incidents
- Maintain representative samples for debugging
- Protect downstream systems from overload

### What Story 5.2 Implemented

Story 5.2 introduced the **BaseFilter** protocol and three **basic** filters:

| Filter            | Description                 | Limitations                      |
| ----------------- | --------------------------- | -------------------------------- |
| `LevelFilter`     | Drop below level threshold  | ✅ Complete                      |
| `SamplingFilter`  | Random probability sampling | Fixed rate only                  |
| `RateLimitFilter` | Token bucket rate limiting  | Basic config, no per-key metrics |

This story adds **advanced** enterprise-grade filters that build on the foundation.

### Existing Hardcoded Sampling (Deprecated)

The codebase has a legacy hardcoded sampling mechanism:

```python
# src/fapilog/core/logger.py (DEPRECATED by this story)
rate = float(s.observability.logging.sampling_rate)
if rate < 1.0 and level in {"DEBUG", "INFO"}:
    if random.random() > rate:
        return  # Drop
```

This story deprecates `observability.logging.sampling_rate` in favor of the plugin-based approach.

---

## Problem Statement

The basic filters from Story 5.2 are insufficient for production:

1. **SamplingFilter** uses fixed random rate - may drop all instances of rare events
2. No adaptive sampling based on traffic patterns
3. No trace-aware sampling (breaks distributed trace correlation)
4. **RateLimitFilter** lacks advanced features (max_keys eviction, overflow marking, per-key metrics)
5. Legacy hardcoded sampling creates confusion with filter-based approach

---

## Acceptance Criteria

### AC1: Adaptive Sampling Filter (`adaptive_sampling`)

- [ ] Sample rate adjusts based on throughput (target EPS)
- [ ] Configurable min/max sample rates (bounds)
- [ ] Sliding window for rate calculation
- [ ] Higher sample rate during low traffic, lower during high
- [ ] Priority levels (ERROR+) always pass

### AC2: Trace-Aware Sampling Filter (`trace_sampling`)

- [ ] Deterministic sampling based on trace_id hash
- [ ] Same trace_id always gets same decision (all or none)
- [ ] Fallback to random sampling when no trace_id
- [ ] Configurable trace_id field name
- [ ] Priority levels bypass

### AC3: Enhanced Rate Limiter (`rate_limit` enhancements)

- [ ] Add `max_keys` config with LRU eviction
- [ ] Add `overflow_action`: "drop" (default) or "mark" (add `rate_limited=True`)
- [ ] Per-key bucket isolation
- [ ] Health check warns when approaching max_keys

### AC4: First-Occurrence Filter (`first_occurrence`)

- [ ] First instance of unique message always passes
- [ ] Subsequent duplicates subject to sampling
- [ ] Configurable uniqueness window
- [ ] Memory-bounded with LRU eviction

### AC5: Deprecate Legacy Sampling

- [ ] Mark `observability.logging.sampling_rate` as deprecated
- [ ] Emit deprecation warning when used
- [ ] Document migration path to `SamplingFilter`
- [ ] Legacy behavior preserved for backward compatibility

### AC6: Metrics and Observability

- [ ] Add `record_sample_rate(filter_name, rate)` to MetricsCollector
- [ ] Add `fapilog_filter_sample_rate` Prometheus gauge
- [ ] Add `fapilog_rate_limit_keys_tracked` gauge
- [ ] Emit diagnostic when rate limiter approaches max_keys

---

## Technical Design

### 1. Adaptive Sampling Filter

```python
# src/fapilog/plugins/filters/adaptive_sampling.py
import time
from dataclasses import dataclass, field
from collections import deque


@dataclass
class AdaptiveSamplingConfig:
    """Configuration for adaptive sampling."""

    # Target events per second
    target_eps: float = 100.0

    # Sample rate bounds
    min_sample_rate: float = 0.01  # Never drop more than 99%
    max_sample_rate: float = 1.0   # Never sample up

    # Window for rate calculation
    window_seconds: float = 10.0

    # Priority levels that always pass
    always_pass_levels: list[str] = field(
        default_factory=lambda: ["ERROR", "CRITICAL", "FATAL"]
    )

    # Adjustment sensitivity (0-1)
    smoothing_factor: float = 0.3


class AdaptiveSamplingFilter:
    """Dynamically adjusts sample rate based on throughput."""

    name = "adaptive_sampling"

    def __init__(self, *, config: AdaptiveSamplingConfig | None = None) -> None:
        cfg = config or AdaptiveSamplingConfig()
        self._target_eps = cfg.target_eps
        self._min_rate = cfg.min_sample_rate
        self._max_rate = cfg.max_sample_rate
        self._window = cfg.window_seconds
        self._always_pass = set(level.upper() for level in cfg.always_pass_levels)
        self._smoothing = cfg.smoothing_factor

        # State
        self._current_rate = 1.0
        self._timestamps: deque[float] = deque()
        self._last_adjustment = time.monotonic()

    async def start(self) -> None:
        self._timestamps.clear()
        self._current_rate = 1.0

    async def stop(self) -> None:
        pass

    async def filter(self, event: dict) -> dict | None:
        level = str(event.get("level", "INFO")).upper()

        # Priority levels always pass
        if level in self._always_pass:
            self._record_event()
            return event

        # Apply current sample rate
        import random
        if random.random() > self._current_rate:
            return None

        self._record_event()
        self._maybe_adjust_rate()
        return event

    def _record_event(self) -> None:
        now = time.monotonic()
        self._timestamps.append(now)

        # Prune old timestamps
        cutoff = now - self._window
        while self._timestamps and self._timestamps[0] < cutoff:
            self._timestamps.popleft()

    def _maybe_adjust_rate(self) -> None:
        now = time.monotonic()

        # Adjust every second
        if now - self._last_adjustment < 1.0:
            return

        self._last_adjustment = now

        # Calculate current EPS
        if not self._timestamps:
            current_eps = 0.0
        else:
            elapsed = max(now - self._timestamps[0], 0.001)
            current_eps = len(self._timestamps) / elapsed

        # Calculate ideal sample rate
        if current_eps <= 0:
            ideal_rate = self._max_rate
        else:
            ideal_rate = self._target_eps / current_eps

        ideal_rate = max(self._min_rate, min(self._max_rate, ideal_rate))

        # Smooth adjustment
        self._current_rate = (
            self._smoothing * ideal_rate +
            (1 - self._smoothing) * self._current_rate
        )

    @property
    def current_sample_rate(self) -> float:
        """Current sample rate for metrics."""
        return self._current_rate

    async def health_check(self) -> bool:
        return True


PLUGIN_METADATA = {
    "name": "adaptive_sampling",
    "version": "1.0.0",
    "plugin_type": "filter",
    "entry_point": "fapilog.plugins.filters.adaptive_sampling:AdaptiveSamplingFilter",
    "description": "Dynamically adjusts sample rate based on throughput.",
    "author": "Fapilog Core",
    "compatibility": {"min_fapilog_version": "0.4.0"},
    "api_version": "1.0",
}
```

### 2. Trace-Aware Sampling Filter

```python
# src/fapilog/plugins/filters/trace_sampling.py
import hashlib
from dataclasses import dataclass


@dataclass
class TraceSamplingConfig:
    """Configuration for trace-aware sampling."""

    sample_rate: float = 0.1
    trace_id_field: str = "trace_id"
    always_pass_levels: list[str] = None

    def __post_init__(self):
        if self.always_pass_levels is None:
            self.always_pass_levels = ["ERROR", "CRITICAL"]


class TraceSamplingFilter:
    """Sample consistently by trace ID.

    All events with the same trace_id are either all sampled or all dropped,
    ensuring complete trace context is preserved.
    """

    name = "trace_sampling"

    def __init__(self, *, config: TraceSamplingConfig | None = None) -> None:
        cfg = config or TraceSamplingConfig()
        self._rate = cfg.sample_rate
        self._trace_field = cfg.trace_id_field
        self._always_pass = set(
            level.upper() for level in (cfg.always_pass_levels or [])
        )

    async def start(self) -> None:
        pass

    async def stop(self) -> None:
        pass

    async def filter(self, event: dict) -> dict | None:
        level = str(event.get("level", "INFO")).upper()

        if level in self._always_pass:
            return event

        trace_id = event.get(self._trace_field)

        if trace_id is None:
            # No trace ID - use random sampling
            import random
            if random.random() > self._rate:
                return None
            return event

        # Deterministic sampling based on trace ID
        # Same trace ID always gets same decision
        hash_value = int(hashlib.md5(str(trace_id).encode()).hexdigest(), 16)
        threshold = int(self._rate * (2**128))

        if hash_value < threshold:
            return event
        return None

    async def health_check(self) -> bool:
        return True
```

### 3. Enhance Existing RateLimitFilter

Story 5.2 implemented a basic `RateLimitFilter`. Enhance it with:

```python
# src/fapilog/plugins/filters/rate_limit.py (ENHANCED)

@dataclass
class RateLimitFilterConfig:
    capacity: int = 10
    refill_rate_per_sec: float = 5.0
    key_field: str | None = None

    # NEW in Story 5.3:
    max_keys: int = 10000  # Memory bound with LRU eviction
    overflow_action: str = "drop"  # "drop" or "mark"

class RateLimitFilter:
    # ... existing implementation ...

    async def filter(self, event: dict) -> dict | None:
        key = self._resolve_key(event)

        # LRU eviction when at capacity
        if key not in self._buckets and len(self._buckets) >= self._max_keys:
            self._evict_oldest()

        # ... token bucket logic ...

        if not allowed:
            if self._overflow_action == "mark":
                event = dict(event)
                event["rate_limited"] = True
                return event
            return None  # Drop

        return event

    async def health_check(self) -> bool:
        # Warn when approaching max_keys
        if len(self._buckets) > self._max_keys * 0.9:
            return False
        return True
```

### 4. First-Occurrence Filter

```python
# src/fapilog/plugins/filters/first_occurrence.py
import time
from collections import OrderedDict
from dataclasses import dataclass


@dataclass
class FirstOccurrenceConfig:
    """Configuration for first-occurrence filter."""

    # Field(s) to determine uniqueness (default: message)
    key_fields: list[str] = None

    # Window in seconds to track occurrences
    window_seconds: float = 60.0

    # Max unique keys to track
    max_keys: int = 10000

    # Sample rate for subsequent occurrences (0 = drop all)
    subsequent_sample_rate: float = 0.0

    def __post_init__(self):
        if self.key_fields is None:
            self.key_fields = ["message"]


class FirstOccurrenceFilter:
    """First occurrence of unique message always passes."""

    name = "first_occurrence"

    def __init__(self, *, config: FirstOccurrenceConfig | None = None) -> None:
        cfg = config or FirstOccurrenceConfig()
        self._key_fields = cfg.key_fields
        self._window = cfg.window_seconds
        self._max_keys = cfg.max_keys
        self._subsequent_rate = cfg.subsequent_sample_rate
        self._seen: OrderedDict[str, float] = OrderedDict()

    async def start(self) -> None:
        self._seen.clear()

    async def stop(self) -> None:
        pass

    async def filter(self, event: dict) -> dict | None:
        key = self._make_key(event)
        now = time.monotonic()

        # Prune expired entries
        self._prune_expired(now)

        if key not in self._seen:
            # First occurrence - always pass
            self._seen[key] = now
            self._seen.move_to_end(key)

            # Evict oldest if at capacity
            while len(self._seen) > self._max_keys:
                self._seen.popitem(last=False)

            return event

        # Subsequent occurrence
        if self._subsequent_rate <= 0:
            return None

        import random
        if random.random() < self._subsequent_rate:
            return event
        return None

    def _make_key(self, event: dict) -> str:
        parts = [str(event.get(f, "")) for f in self._key_fields]
        return "|".join(parts)

    def _prune_expired(self, now: float) -> None:
        cutoff = now - self._window
        while self._seen:
            oldest_key, oldest_time = next(iter(self._seen.items()))
            if oldest_time < cutoff:
                self._seen.popitem(last=False)
            else:
                break

    async def health_check(self) -> bool:
        return True


PLUGIN_METADATA = {
    "name": "first_occurrence",
    "version": "1.0.0",
    "plugin_type": "filter",
    "entry_point": "fapilog.plugins.filters.first_occurrence:FirstOccurrenceFilter",
    "description": "First occurrence of unique message always passes.",
    "author": "Fapilog Core",
    "compatibility": {"min_fapilog_version": "0.4.0"},
    "api_version": "1.0",
}
```

### 5. Deprecate Legacy Sampling

```python
# src/fapilog/core/logger.py

def _enqueue(self, level: str, message: str, ...) -> None:
    # ... existing level gate check ...

    # DEPRECATED: Legacy probabilistic sampling
    # Will be removed in v1.0 - use SamplingFilter instead
    try:
        s = Settings()
        rate = float(s.observability.logging.sampling_rate)
        if rate < 1.0:
            import warnings
            warnings.warn(
                "observability.logging.sampling_rate is deprecated. "
                "Use core.filters=['sampling'] with filter_config.sampling instead.",
                DeprecationWarning,
                stacklevel=3,
            )
            if level in {"DEBUG", "INFO"} and random.random() > rate:
                return
    except Exception:
        pass
```

---

## Test Plan

### Unit Tests

1. **test_adaptive_sampling.py**

   - Test rate adjusts down during high load
   - Test rate adjusts up during low load
   - Test priority levels always pass
   - Test bounds respected
   - Test window pruning

2. **test_trace_sampling.py**

   - Test same trace_id always same decision
   - Test different trace_ids independent
   - Test events without trace_id random sampled
   - Test configurable trace_id field

3. **test_rate_limit_enhanced.py**

   - Test max_keys eviction
   - Test mark vs drop behavior
   - Test health_check warns at 90% capacity

4. **test_first_occurrence.py**

   - Test first occurrence passes
   - Test subsequent occurrences dropped/sampled
   - Test window expiration
   - Test max_keys eviction

5. **test_legacy_sampling_deprecation.py**
   - Test deprecation warning emitted
   - Test legacy behavior preserved
   - Test no warning when filter-based sampling used

### Performance Tests

1. **test_sampling_performance.py**
   - Benchmark adaptive sampling overhead
   - Measure memory usage with many keys
   - Test under high concurrency

### Integration Tests

1. **test_sampling_with_traces.py**
   - Verify trace_sampling preserves complete traces
   - Test with ContextVarsEnricher providing trace_id

---

## Documentation Updates

1. Create `docs/plugins/filters/sampling.md`
2. Create `docs/plugins/filters/rate-limiting.md`
3. Add sampling examples to `docs/examples/sampling-debug-logs.md`
4. Update `docs/configuration.md` with deprecation notice for `observability.logging.sampling_rate`
5. Add migration guide from legacy sampling to filter-based

---

## Migration Guide

### From Legacy Sampling to Filter-Based

**Before (deprecated):**

```yaml
observability:
  logging:
    sampling_rate: 0.25
```

**After (recommended):**

```yaml
core:
  filters: ["sampling"]
filter_config:
  sampling:
    sample_rate: 0.25
```

**For adaptive sampling:**

```yaml
core:
  filters: ["adaptive_sampling"]
filter_config:
  adaptive_sampling:
    target_eps: 100
    min_sample_rate: 0.01
    max_sample_rate: 1.0
```

---

## Rollout Plan

1. Implement `AdaptiveSamplingFilter`
2. Implement `TraceSamplingFilter`
3. Enhance `RateLimitFilter` with max_keys, overflow_action
4. Implement `FirstOccurrenceFilter`
5. Add deprecation warning to legacy sampling
6. Add new metrics to MetricsCollector
7. Documentation and migration guide
8. Release

---

## Related Stories

- Story 5.2: Add filter plugin type (prerequisite - implemented)
- Story 5.7: Plugin observability metrics
