# Story 4.40: Processor Use Cases Documentation

**Status:** Draft  
**Priority:** Medium  
**Depends on:** None  
**Effort:** 0.5 days

---

## Problem Statement

The processor plugin type exists but is poorly documented:

1. **What processors do** is unclear - they work on `memoryview` not `dict`
2. **When to use processors vs enrichers** is not explained
3. **Built-in processors** - only `ZeroCopyProcessor` exists, which is a no-op
4. **Real-world use cases** are missing

### Current Documentation

From `docs/plugins/processors.md`:
```markdown
# Processors

Data processing and transformation pipelines.

## Implementing a processor

```python
from fapilog.plugins import BaseProcessor

class MyProcessor(BaseProcessor):
    name = "my-processor"

    async def process(self, view: memoryview) -> memoryview:
        # Transform the serialized view
        ...
```
```

This tells you HOW to implement but not WHY or WHEN.

### The Problem

Processors operate on **serialized** data (`memoryview`), not events (`dict`). This makes them fundamentally different from enrichers:

| Aspect | Enricher | Processor |
|--------|----------|-----------|
| Input | `dict` (event) | `memoryview` (bytes) |
| Output | `dict` (additions) | `memoryview` (bytes) |
| When called | Before serialization | After serialization |
| Use case | Add fields | Transform bytes |

Without clear use case guidance, developers don't know when to use processors.

---

## Goals

1. Document processor use cases clearly
2. Explain difference from enrichers
3. Provide practical examples
4. Consider if processors are even needed (or should be deprecated)

---

## Research: When Are Processors Useful?

### Valid Use Cases

1. **Compression**: Compress serialized JSON before writing
2. **Encryption**: Encrypt serialized data at rest
3. **Format conversion**: Convert JSON to another format (BSON, MessagePack)
4. **Checksumming**: Add integrity checksums to serialized data
5. **Batching/framing**: Add message framing for streaming protocols

### Why Not Just Do This in Sinks?

Processors provide a **reusable transformation** that can be applied before any sink:

```
Event → Enrichers → Redactors → Serialize → [Processors] → Sinks
```

A compression processor works with any sink. Implementing compression per-sink leads to duplication.

### Current Reality

The only built-in processor is `ZeroCopyProcessor` - a pass-through for benchmarking. No real processors exist.

---

## Implementation Plan

### Phase 1: Expand Processor Documentation

**File: `docs/plugins/processors.md`**

Complete rewrite:

```markdown
# Processors

Processors transform **serialized** log data before it reaches sinks.

## When to Use Processors

Use processors when you need to transform the raw bytes of serialized log entries.
Common use cases:

| Use Case | Description |
|----------|-------------|
| **Compression** | Compress JSON before writing to disk/network |
| **Encryption** | Encrypt logs at rest |
| **Format Conversion** | Convert JSON to MessagePack, BSON, etc. |
| **Checksums** | Add integrity checksums for verification |
| **Framing** | Add message boundaries for streaming protocols |

### Processors vs Enrichers

| Question | Enricher | Processor |
|----------|----------|-----------|
| Do I need to add fields? | ✅ Use enricher | ❌ |
| Do I need to transform bytes? | ❌ | ✅ Use processor |
| Do I work with dict? | ✅ | ❌ (memoryview) |
| When am I called? | Before serialization | After serialization |

**Rule of thumb:** If you're working with the log message content, use an enricher.
If you're working with the raw bytes, use a processor.

## Implementing a Processor

```python
from fapilog.plugins import BaseProcessor


class GzipProcessor:
    """Compress log entries with gzip."""
    
    name = "gzip"
    
    def __init__(self, level: int = 6):
        self._level = level
    
    async def start(self) -> None:
        pass
    
    async def stop(self) -> None:
        pass
    
    async def process(self, view: memoryview) -> memoryview:
        import gzip
        compressed = gzip.compress(bytes(view), compresslevel=self._level)
        return memoryview(compressed)
    
    async def health_check(self) -> bool:
        return True
```

## Example: Encryption Processor

```python
from cryptography.fernet import Fernet


class EncryptProcessor:
    """Encrypt log entries before storage."""
    
    name = "encrypt"
    
    def __init__(self, key: bytes):
        self._fernet = Fernet(key)
    
    async def start(self) -> None:
        pass
    
    async def stop(self) -> None:
        pass
    
    async def process(self, view: memoryview) -> memoryview:
        encrypted = self._fernet.encrypt(bytes(view))
        return memoryview(encrypted)
    
    async def health_check(self) -> bool:
        # Verify key is still valid
        try:
            test = self._fernet.encrypt(b"test")
            self._fernet.decrypt(test)
            return True
        except Exception:
            return False
```

## Example: MessagePack Conversion

```python
import json
import msgpack


class MsgPackProcessor:
    """Convert JSON to MessagePack for smaller payloads."""
    
    name = "msgpack"
    
    async def start(self) -> None:
        pass
    
    async def stop(self) -> None:
        pass
    
    async def process(self, view: memoryview) -> memoryview:
        # Parse JSON, convert to MessagePack
        data = json.loads(bytes(view))
        packed = msgpack.packb(data)
        return memoryview(packed)
    
    async def health_check(self) -> bool:
        return True
```

## Built-in Processors

| Processor | Description |
|-----------|-------------|
| `zero_copy` | Pass-through processor for benchmarking (no transformation) |

## Configuration

Enable processors in settings:

```python
from fapilog import Settings

settings = Settings(
    core__processors=["gzip", "encrypt"],
)
```

Or via environment:

```bash
FAPILOG_CORE__PROCESSORS='["gzip"]'
```

## Processing Order

Processors run in the order specified:

```python
core__processors=["gzip", "encrypt"]
# Event → Serialize → gzip → encrypt → Sink
```

## Performance Considerations

- Processors add latency to every log write
- Use async I/O or thread pools for CPU-intensive operations
- Consider whether transformation is better done in the sink

## Batch Processing

Processors can implement `process_many()` for batch optimization:

```python
async def process_many(self, views: Iterable[memoryview]) -> int:
    """Process multiple views, returning count processed."""
    count = 0
    for view in views:
        await self.process(view)
        count += 1
    return count
```
```

### Phase 2: Add to Processors API Reference

**File: `docs/api-reference/plugins/processors.md`**

Expand with use cases and complete method reference.

### Phase 3: Update Plugin Index

**File: `docs/plugins/index.md`**

Update processors description:

```markdown
### [Processors](processors.md)

Transform serialized log data (compression, encryption, format conversion).
```

---

## Acceptance Criteria

- [ ] `docs/plugins/processors.md` rewritten with use cases
- [ ] Clear comparison with enrichers
- [ ] Real-world examples (compression, encryption, format)
- [ ] API reference updated
- [ ] Plugin index updated
- [ ] Documentation builds without warnings

---

## Open Question

**Should processors be kept or deprecated?**

Arguments for keeping:
- Clear use case (byte transformation)
- Reusable across sinks
- Clean separation of concerns

Arguments for deprecating:
- Only one built-in (no-op)
- Not widely used
- Same can be done in sinks

**Recommendation:** Keep but improve documentation. If usage remains low after 1-2 releases, consider deprecation.

---

## Files Changed

| File | Change |
|------|--------|
| `docs/plugins/processors.md` | Complete rewrite |
| `docs/api-reference/plugins/processors.md` | Expand with examples |
| `docs/plugins/index.md` | Update description |

