# Story 5.0: Wire Processors into Pipeline

## Status: Complete

## Priority: Critical

## Estimated Effort: Medium (3-5 days)

## Dependencies: None

## Epic: Plugin System Completion

---

## Context

The plugin audit revealed that while `BaseProcessor` is fully defined and documented, and the `core.processors` setting exists, **processors are never actually loaded or executed** in the logging pipeline. The `worker.py` only invokes enrichers and redactors, completely bypassing the processor stage.

This is a significant gap because:

1. Documentation claims processors run after serialization
2. Users may configure processors expecting them to work
3. The zero_copy processor exists but cannot be used
4. Compression, encryption, and format conversion use cases are blocked

---

## Problem Statement

Processors are defined but not wired into the pipeline. The current flow is:

```
Event → Enrichers → Redactors → Serialize → Sinks
```

The intended flow should be:

```
Event → Enrichers → Redactors → Serialize → Processors → Sinks
```

---

## Acceptance Criteria

### AC1: Processor Loading

- [ ] `_build_pipeline()` in `__init__.py` loads processors from `core.processors` setting
- [ ] Processors are loaded via `_load_plugins("fapilog.processors", ...)`
- [ ] Allow/deny lists apply to processors
- [ ] Failed processor loads emit diagnostics and continue

### AC2: Pipeline Integration

- [ ] `LoggerWorker._flush_batch()` applies processors after serialization
- [ ] Processors receive `SerializedView` or `memoryview` from serialization
- [ ] Processor output feeds into sink `write_serialized()` fast path
- [ ] Processors run in configured order (sequential by default)

### AC3: Error Containment

- [ ] Processor exceptions are caught and logged via diagnostics
- [ ] Failed processor does not block other processors or sinks
- [ ] Original serialized view is preserved if processor fails
- [ ] Metrics record processor errors when enabled

### AC4: Lifecycle Management

- [ ] Processors are started via `_start_plugins()` during logger initialization
- [ ] Processors are stopped during `runtime()` cleanup
- [ ] Processor `health_check()` is included in `logger.check_health()`

### AC5: Configuration

- [ ] `core.processors` setting accepts list of processor names
- [ ] `processor_config` settings block exists for per-processor config
- [ ] Processors can be enabled/disabled at runtime (stretch goal)

### AC6: Fix process_many() Return Type

- [ ] `BaseProcessor.process_many()` returns `list[memoryview]` not `int`
- [ ] Default implementation delegates to `process()` for each view
- [ ] Processors can override for batch optimization
- [ ] Existing `ZeroCopyProcessor` updated to new signature

---

## Technical Design

### 0. Fix BaseProcessor.process_many() Return Type

```python
# src/fapilog/plugins/processors/__init__.py

@runtime_checkable
class BaseProcessor(Protocol):
    """Authoring contract for processors that transform serialized views."""

    name: str

    async def start(self) -> None:
        """Initialize processor resources (optional)."""

    async def stop(self) -> None:
        """Release processor resources (optional)."""

    async def process(self, view: memoryview) -> memoryview:
        """Transform a single view and return the processed view."""

    async def process_many(self, views: Iterable[memoryview]) -> list[memoryview]:
        """Process multiple views, returning processed results in order.

        Default implementation delegates to process() for each view.
        Override for batch optimization (shared compression dictionary,
        encryption context reuse, etc.).
        """
        return [await self.process(v) for v in views]

    async def health_check(self) -> bool:
        """Return True if the processor is healthy."""
        return True
```

Also update `ZeroCopyProcessor`:

```python
# src/fapilog/plugins/processors/zero_copy.py

async def process_many(self, views: Iterable[memoryview]) -> list[memoryview]:
    """Process many payloads; returns processed views."""
    async with self._lock:
        return [await self.process(v) for v in views]
```

### 1. Update `__init__.py`

```python
def _build_pipeline(
    settings: _Settings,
) -> tuple[list[object], list[object], list[object], list[object], _MetricsCollector | None]:
    # ... existing code ...

    # Add processor loading
    processor_names = list(core_cfg.processors or [])
    processors = _load_plugins(
        "fapilog.processors", processor_names, settings, _processor_configs(settings)
    )

    return sinks, enrichers, redactors, processors, metrics
```

### 2. Add Processor Config Settings

In `core/settings.py`:

```python
class ProcessorConfigSettings(BaseModel):
    """Per-processor configuration blocks."""

    zero_copy: dict[str, Any] = Field(default_factory=dict)
    gzip: GzipProcessorConfig = Field(default_factory=GzipProcessorConfig)
    extra: dict[str, dict[str, Any]] = Field(default_factory=dict)
```

### 3. Update LoggerWorker

In `core/worker.py`:

```python
class LoggerWorker:
    def __init__(
        self,
        *,
        # ... existing params ...
        processors_getter: Callable[[], list[BaseProcessor]],
    ) -> None:
        # ... existing init ...
        self._processors_getter = processors_getter

    async def _flush_batch(self, batch: list[dict[str, Any]]) -> None:
        # ... existing enricher/redactor logic ...

        for entry in batch:
            entry = await self._apply_enrichers(entry)
            entry = await self._apply_redactors(entry)

            if self._serialize_in_flush:
                view, drop_entry = await self._try_serialize(entry)
                if drop_entry:
                    continue

                # NEW: Apply processors to serialized view
                if view is not None:
                    view = await self._apply_processors(view)

                if view is not None and self._sink_write_serialized is not None:
                    try:
                        await self._sink_write_serialized(view)
                        self._counters["processed"] += 1
                        continue
                    except Exception:
                        pass

            await self._sink_write(entry)
            self._counters["processed"] += 1

    async def _apply_processors(
        self, view: SerializedView
    ) -> SerializedView | None:
        """Apply configured processors to serialized view."""
        processors = self._processors_getter()
        if not processors:
            return view

        current_view: memoryview = view.data
        for processor in processors:
            try:
                current_view = await processor.process(current_view)
            except Exception as exc:
                if self._emit_processor_diagnostics:
                    try:
                        warn(
                            "processor",
                            "processor error",
                            processor=getattr(processor, "name", type(processor).__name__),
                            error=str(exc),
                            _rate_limit_key="process",
                        )
                    except Exception:
                        pass
                # Continue with unprocessed view on error
                continue

        # Wrap result back into SerializedView
        return SerializedView(data=current_view)
```

### 4. Update Logger Facades

Both `SyncLoggerFacade` and `AsyncLoggerFacade` need:

```python
def __init__(self, *, ..., processors: list[BaseProcessor] | None = None):
    # ... existing init ...
    self._processors: list[BaseProcessor] = list(processors or [])

def _make_worker(self) -> LoggerWorker:
    return LoggerWorker(
        # ... existing params ...
        processors_getter=lambda: list(self._processors),
    )
```

---

## Test Plan

### Unit Tests

1. **test_processor_loading.py**

   - Test processors loaded from settings
   - Test allow/deny list filtering
   - Test empty processor list (no-op)

2. **test_processor_pipeline.py**

   - Test processor receives serialized bytes
   - Test processor output passed to sinks
   - Test multiple processors chain correctly
   - Test processor order preserved

3. **test_processor_error_handling.py**

   - Test processor exception contained
   - Test diagnostics emitted on error
   - Test fallback to original view on error
   - Test subsequent processors still run

4. **test_processor_lifecycle.py**
   - Test processor start() called
   - Test processor stop() called on cleanup
   - Test processor health_check() included

### Integration Tests

1. **test_gzip_processor_integration.py**

   - Create a gzip processor
   - Log events through pipeline
   - Verify output is gzipped

2. **test_multi_processor_chain.py**
   - Chain gzip + encrypt processors
   - Verify correct order of transformation

---

## Migration Notes

- No breaking changes; `core.processors = []` (default) maintains existing behavior
- Existing processor plugins will automatically work when configured
- Documentation must be updated to reflect working processor support

---

## Documentation Updates

1. Update `docs/plugins/processors.md` to remove "planned" language
2. Add processor configuration examples to `docs/plugins/configuration.md`
3. Update `docs/core-concepts/pipeline-architecture.md` with processor stage
4. Add processor troubleshooting to `docs/troubleshooting/`

---

## Rollout Plan

1. Implement core wiring (AC1, AC2)
2. Add error handling (AC3)
3. Add lifecycle management (AC4)
4. Add configuration (AC5)
5. Update all documentation
6. Release in next minor version

---

## Design Decisions

### Q1: Sequential vs Parallel Processors?

**Decision: Sequential**

Processors form a transformation chain where order matters (compress → encrypt → sign). Parallel processing within a single chain doesn't make semantic sense. If per-sink processing is needed, that's a separate feature.

### Q2: Batch Processing Efficiency?

**Decision: Fix `process_many()` return type**

The current `process_many()` returns `int` but discards results. Fix the signature to:

```python
async def process_many(self, views: Iterable[memoryview]) -> list[memoryview]:
    """Process multiple views, returning processed results in order."""
    return [await self.process(v) for v in views]
```

This enables processors to override for batch optimization (compression with shared dictionary, encryption with shared context) while keeping `process()` as the primary single-item method.

### Q3: Separate Order Configuration?

**Decision: Order = list order (no separate setting)**

Keep `core.processors = ["compress", "encrypt"]` where list order IS execution order. This matches enrichers/redactors pattern and avoids sync issues between two settings. Environment-specific ordering uses different settings files or env vars.

---

## Related Stories

- Story 5.1: Fix PLUGIN_METADATA inconsistencies
- Story 5.6: HTTP sink batching (may use processors)
- Story 5.8: OpenTelemetry integration
