# Story 5.13: SizeGuardProcessor - Payload Size Enforcement

## Status: Ready for Implementation

## Priority: Medium

## Estimated Effort: Small (2-3 days)

## Dependencies

- **Depends on:** Story 5.0 (processors wired into pipeline) ✅ Complete

## Epic / Series

Part of Epic 3: Plugin Ecosystem Foundation / Series 5.x

---

## Context / Background

The processor stage in fapilog's pipeline operates on serialized bytes after JSON serialization but before sinks. While the architecture exists, the only built-in processor (`zero_copy`) is a no-op pass-through that provides no practical value.

This story adds `SizeGuardProcessor` — a genuinely useful processor that enforces maximum payload size. This is critical because many log destinations have hard size limits:

| Destination         | Limit                       | Behavior on Exceed       |
| ------------------- | --------------------------- | ------------------------ |
| AWS CloudWatch Logs | 256 KB per event            | Event rejected, log lost |
| Datadog Logs        | 1 MB per event              | Silent truncation        |
| Kafka               | Configurable (default 1 MB) | Producer exception       |
| Elasticsearch       | Configurable                | Request rejected         |
| Splunk HEC          | 1 MB default                | Event dropped            |
| Loki                | 256 KB per log line         | Event rejected           |

Without size enforcement, users discover these limits in production when logs silently disappear.

---

## User Story

**As a** developer sending logs to size-limited destinations  
**I want** automatic payload size enforcement  
**So that** logs are not silently dropped when they exceed destination limits

---

## Scope (In / Out)

### In Scope

- `SizeGuardProcessor` implementation with truncate/drop/warn modes
- Safe JSON truncation preserving valid JSON structure
- Configuration via Settings and environment variables
- Metrics and diagnostics for size enforcement events
- Integration with processor pipeline

### Out of Scope

- Custom truncation strategies (beyond safe JSON truncation)
- Real-time size monitoring dashboards
- Automatic destination limit detection

---

## Problem Statement

Users sending logs to size-limited destinations have no way to:

1. Detect oversized payloads before transmission fails
2. Gracefully handle oversized logs (truncate, drop, or warn)
3. Get visibility into how often size limits are being hit

Additionally, the processor stage lacks a concrete, useful built-in that demonstrates its value and serves as a reference implementation for custom processors.

---

## Why This Must Be a Processor (Not a Filter)

Size enforcement **cannot** be implemented at other pipeline stages:

| Stage    | Why Not                                                                                               |
| -------- | ----------------------------------------------------------------------------------------------------- |
| Filter   | Operates on dict; would need to serialize to check size, then discard — wasteful double-serialization |
| Enricher | Operates on dict, not final bytes                                                                     |
| Redactor | Same limitation as enricher                                                                           |
| Sink     | Too late — sink may have already attempted transmission and failed                                    |

Only the processor stage has access to the **actual serialized bytes** that will be transmitted, making it the correct place for size enforcement.

---

## Acceptance Criteria

### AC1: Core Functionality

- [ ] `SizeGuardProcessor` class in `src/fapilog/plugins/processors/size_guard.py`
- [ ] Configurable `max_bytes` threshold (default: 256,000 for CloudWatch compatibility)
- [ ] Three action modes: `truncate`, `drop`, `warn`
- [ ] Implements `BaseProcessor` protocol fully

### AC2: Action Behaviors

- [ ] `truncate`: Safely truncate JSON to fit within limit, preserving valid JSON structure
- [ ] `drop`: Replace with minimal marker payload indicating truncation occurred
- [ ] `warn`: Pass through unchanged but emit diagnostic warning

### AC3: Safe JSON Truncation

- [ ] Truncated output is always valid JSON
- [ ] Truncation adds `"_truncated": true` and `"_original_size": N` fields
- [ ] Message field is truncated first (largest string typically)
- [ ] If message truncation insufficient, metadata fields are removed
- [ ] Never truncates critical fields: `level`, `timestamp`, `logger`

### AC4: Metrics and Diagnostics

- [ ] Emit diagnostic on every truncation/drop (rate-limited)
- [ ] Record `processor_size_guard_truncated` metric when truncating
- [ ] Record `processor_size_guard_dropped` metric when dropping
- [ ] Include original size and limit in diagnostic context

### AC5: Configuration

- [ ] `processor_config.size_guard.max_bytes` setting
- [ ] `processor_config.size_guard.action` setting (`truncate` | `drop` | `warn`)
- [ ] `processor_config.size_guard.preserve_fields` setting (fields to never remove)
- [ ] Environment variable support: `FAPILOG_SIZE_GUARD__MAX_BYTES`, etc.

### AC6: Registration and Discovery

- [ ] Registered as built-in: `register_builtin("fapilog.processors", "size_guard", ...)`
- [ ] `PLUGIN_METADATA` with correct name, version, description
- [ ] Loadable via `core.processors = ["size_guard"]`

### AC7: Health Check

- [ ] `health_check()` returns True when properly configured
- [ ] Returns False if configuration is invalid (e.g., max_bytes <= 0)

---

## Technical Design

### 1. Configuration Dataclass

```python
# src/fapilog/plugins/processors/size_guard.py

from dataclasses import dataclass, field
from typing import Literal

@dataclass
class SizeGuardConfig:
    """Configuration for SizeGuardProcessor."""

    max_bytes: int = 256_000  # CloudWatch default
    action: Literal["truncate", "drop", "warn"] = "truncate"
    preserve_fields: list[str] = field(
        default_factory=lambda: ["level", "timestamp", "logger", "correlation_id"]
    )
```

### 2. Processor Implementation

```python
import json
from typing import Any

from ...core import diagnostics
from ...core.errors import ErrorCategory, ErrorSeverity, create_error_context


class SizeGuardProcessor:
    """Enforce maximum payload size for destination compatibility.

    Operates on serialized JSON bytes to ensure payloads don't exceed
    size limits of downstream log aggregators (CloudWatch, Datadog, Kafka, etc.).

    Actions:
    - truncate: Safely shorten the payload while preserving valid JSON
    - drop: Replace with minimal marker indicating the event was dropped
    - warn: Pass through unchanged but emit diagnostic warning
    """

    name = "size_guard"

    def __init__(self, *, config: SizeGuardConfig | dict | None = None) -> None:
        if isinstance(config, dict):
            cfg = SizeGuardConfig(**config.get("config", config))
        elif config is None:
            cfg = SizeGuardConfig()
        else:
            cfg = config

        self._max_bytes = max(100, int(cfg.max_bytes))  # Minimum 100 bytes
        self._action = cfg.action
        self._preserve_fields = set(cfg.preserve_fields)
        self._truncated_count = 0
        self._dropped_count = 0

    async def start(self) -> None:
        pass

    async def stop(self) -> None:
        pass

    async def process(self, view: memoryview) -> memoryview:
        size = len(view)

        if size <= self._max_bytes:
            return view

        # Payload exceeds limit
        if self._action == "warn":
            self._emit_warning(size, "passing through oversized payload")
            return view

        if self._action == "drop":
            self._dropped_count += 1
            self._emit_warning(size, "dropping oversized payload")
            return self._create_drop_marker(size)

        # action == "truncate"
        self._truncated_count += 1
        self._emit_warning(size, "truncating oversized payload")
        return self._truncate_payload(view, size)

    async def health_check(self) -> bool:
        return self._max_bytes > 0

    def _emit_warning(self, original_size: int, message: str) -> None:
        try:
            diagnostics.warn(
                "processor",
                message,
                processor="size_guard",
                original_size=original_size,
                max_bytes=self._max_bytes,
                action=self._action,
                _rate_limit_key="size_guard",
            )
        except Exception:
            pass

    def _create_drop_marker(self, original_size: int) -> memoryview:
        """Create minimal JSON marker indicating payload was dropped."""
        marker = {
            "_dropped": True,
            "_reason": "payload_size_exceeded",
            "_original_size": original_size,
            "_max_bytes": self._max_bytes,
        }
        return memoryview(json.dumps(marker).encode("utf-8"))

    def _truncate_payload(self, view: memoryview, original_size: int) -> memoryview:
        """Safely truncate JSON payload while preserving structure."""
        try:
            # Parse the JSON to work with it
            data = json.loads(bytes(view))

            # Add truncation markers
            data["_truncated"] = True
            data["_original_size"] = original_size

            # Calculate overhead for markers
            overhead = len(json.dumps({
                "_truncated": True,
                "_original_size": original_size,
            }).encode("utf-8"))

            target_size = self._max_bytes - overhead - 50  # Safety margin

            # Strategy 1: Truncate message field (usually largest)
            if "message" in data and isinstance(data["message"], str):
                data = self._truncate_field(data, "message", target_size)
                result = json.dumps(data, default=str).encode("utf-8")
                if len(result) <= self._max_bytes:
                    return memoryview(result)

            # Strategy 2: Remove non-preserved metadata fields
            if "metadata" in data and isinstance(data["metadata"], dict):
                data = self._prune_metadata(data, target_size)
                result = json.dumps(data, default=str).encode("utf-8")
                if len(result) <= self._max_bytes:
                    return memoryview(result)

            # Strategy 3: Aggressive - keep only preserved fields
            data = self._keep_only_preserved(data, original_size)
            result = json.dumps(data, default=str).encode("utf-8")
            return memoryview(result)

        except Exception:
            # If truncation fails, return drop marker
            return self._create_drop_marker(original_size)

    def _truncate_field(
        self, data: dict[str, Any], field: str, target_size: int
    ) -> dict[str, Any]:
        """Truncate a specific string field."""
        current = json.dumps(data, default=str).encode("utf-8")
        if len(current) <= target_size:
            return data

        value = data.get(field, "")
        if not isinstance(value, str):
            return data

        excess = len(current) - target_size
        new_len = max(50, len(value) - excess - 20)  # Keep at least 50 chars
        data[field] = value[:new_len] + "...[truncated]"
        return data

    def _prune_metadata(
        self, data: dict[str, Any], target_size: int
    ) -> dict[str, Any]:
        """Remove metadata fields until size target is met."""
        metadata = data.get("metadata", {})
        if not isinstance(metadata, dict):
            return data

        # Sort by value size (largest first)
        sorted_keys = sorted(
            metadata.keys(),
            key=lambda k: len(str(metadata[k])),
            reverse=True,
        )

        for key in sorted_keys:
            if key in self._preserve_fields:
                continue
            del metadata[key]
            current = json.dumps(data, default=str).encode("utf-8")
            if len(current) <= target_size:
                break

        data["metadata"] = metadata
        return data

    def _keep_only_preserved(
        self, data: dict[str, Any], original_size: int
    ) -> dict[str, Any]:
        """Last resort: keep only critical fields."""
        minimal = {
            "_truncated": True,
            "_original_size": original_size,
            "_heavily_truncated": True,
        }
        for field in self._preserve_fields:
            if field in data:
                minimal[field] = data[field]
        return minimal


# Plugin metadata
PLUGIN_METADATA = {
    "name": "size_guard",
    "version": "1.0.0",
    "plugin_type": "processor",
    "entry_point": "fapilog.plugins.processors.size_guard:SizeGuardProcessor",
    "description": "Enforces maximum payload size for destination compatibility.",
    "author": "Fapilog Core",
    "compatibility": {"min_fapilog_version": "0.4.0"},
    "api_version": "1.0",
    "config_schema": {
        "type": "object",
        "properties": {
            "max_bytes": {"type": "integer", "default": 256000},
            "action": {"type": "string", "enum": ["truncate", "drop", "warn"]},
            "preserve_fields": {"type": "array", "items": {"type": "string"}},
        },
    },
    "default_config": {
        "max_bytes": 256000,
        "action": "truncate",
        "preserve_fields": ["level", "timestamp", "logger", "correlation_id"],
    },
}
```

### 3. Registration

```python
# src/fapilog/plugins/processors/__init__.py

from .size_guard import SizeGuardProcessor

register_builtin(
    "fapilog.processors",
    "size_guard",
    SizeGuardProcessor,
    aliases=["size-guard"],
)

__all__ = [
    "BaseProcessor",
    "process_parallel",
    "ZeroCopyProcessor",
    "SizeGuardProcessor",  # Add export
]
```

### 4. Settings Integration

```python
# src/fapilog/core/settings.py (add to ProcessorConfigSettings)

class SizeGuardSettings(BaseModel):
    max_bytes: int = Field(default=256000, ge=100, description="Maximum payload size in bytes")
    action: Literal["truncate", "drop", "warn"] = Field(default="truncate")
    preserve_fields: list[str] = Field(
        default_factory=lambda: ["level", "timestamp", "logger", "correlation_id"]
    )

class ProcessorConfigSettings(BaseModel):
    zero_copy: dict[str, Any] = Field(default_factory=dict)
    size_guard: SizeGuardSettings = Field(default_factory=SizeGuardSettings)
    extra: dict[str, dict[str, Any]] = Field(default_factory=dict)
```

---

## Tasks

### Phase 1: Core Implementation
- [ ] Implement `SizeGuardProcessor` class in `src/fapilog/plugins/processors/size_guard.py`
- [ ] Implement safe JSON truncation logic
- [ ] Implement truncate/drop/warn action modes
- [ ] Add configuration support in Settings

### Phase 2: Integration
- [ ] Register as built-in processor
- [ ] Add processor configuration to Settings
- [ ] Integrate with metrics system
- [ ] Add diagnostics support

### Phase 3: Testing
- [ ] Write comprehensive unit tests
- [ ] Write integration tests
- [ ] Test all action modes
- [ ] Test edge cases

### Phase 4: Documentation
- [ ] Update processor documentation
- [ ] Add usage examples
- [ ] Document configuration options
- [ ] Update CHANGELOG

---

### Unit Tests

**test_size_guard_processor.py**

```python
import pytest
from fapilog.plugins.processors.size_guard import SizeGuardProcessor, SizeGuardConfig

@pytest.fixture
def small_payload():
    return memoryview(b'{"level":"INFO","message":"hello"}')

@pytest.fixture
def large_payload():
    # Create payload larger than default 256KB
    msg = "x" * 300_000
    return memoryview(f'{{"level":"INFO","message":"{msg}"}}'.encode())

class TestSizeGuardBasics:
    async def test_small_payload_passes_through(self, small_payload):
        proc = SizeGuardProcessor()
        result = await proc.process(small_payload)
        assert bytes(result) == bytes(small_payload)

    async def test_large_payload_truncated_by_default(self, large_payload):
        proc = SizeGuardProcessor()
        result = await proc.process(large_payload)
        assert len(result) <= 256_000
        data = json.loads(bytes(result))
        assert data["_truncated"] is True

    async def test_large_payload_dropped_when_configured(self, large_payload):
        proc = SizeGuardProcessor(config=SizeGuardConfig(action="drop"))
        result = await proc.process(large_payload)
        data = json.loads(bytes(result))
        assert data["_dropped"] is True

    async def test_large_payload_warns_when_configured(self, large_payload):
        proc = SizeGuardProcessor(config=SizeGuardConfig(action="warn"))
        result = await proc.process(large_payload)
        assert bytes(result) == bytes(large_payload)  # Unchanged

class TestTruncationStrategies:
    async def test_message_truncated_first(self):
        payload = {"level": "INFO", "message": "a" * 300_000, "metadata": {"key": "value"}}
        view = memoryview(json.dumps(payload).encode())

        proc = SizeGuardProcessor(config=SizeGuardConfig(max_bytes=1000))
        result = await proc.process(view)
        data = json.loads(bytes(result))

        assert data["_truncated"] is True
        assert "[truncated]" in data["message"]
        assert data["metadata"]["key"] == "value"  # Preserved

    async def test_preserve_fields_respected(self):
        payload = {"level": "ERROR", "timestamp": "2024-01-01", "message": "x" * 300_000}
        view = memoryview(json.dumps(payload).encode())

        proc = SizeGuardProcessor(config=SizeGuardConfig(max_bytes=500))
        result = await proc.process(view)
        data = json.loads(bytes(result))

        assert data["level"] == "ERROR"
        assert data["timestamp"] == "2024-01-01"

class TestConfiguration:
    async def test_custom_max_bytes(self):
        proc = SizeGuardProcessor(config=SizeGuardConfig(max_bytes=100))
        assert proc._max_bytes == 100

    async def test_min_max_bytes_enforced(self):
        proc = SizeGuardProcessor(config=SizeGuardConfig(max_bytes=10))
        assert proc._max_bytes == 100  # Minimum enforced

    async def test_health_check_valid_config(self):
        proc = SizeGuardProcessor()
        assert await proc.health_check() is True

class TestOutputValidity:
    async def test_truncated_output_is_valid_json(self, large_payload):
        proc = SizeGuardProcessor(config=SizeGuardConfig(max_bytes=500))
        result = await proc.process(large_payload)
        # Should not raise
        json.loads(bytes(result))

    async def test_dropped_output_is_valid_json(self, large_payload):
        proc = SizeGuardProcessor(config=SizeGuardConfig(action="drop"))
        result = await proc.process(large_payload)
        data = json.loads(bytes(result))
        assert "_dropped" in data
```

### Integration Tests

**test_size_guard_pipeline.py**

```python
async def test_size_guard_in_full_pipeline():
    """Verify size_guard works when wired into actual logging pipeline."""
    settings = Settings()
    settings.core.processors = ["size_guard"]
    settings.processor_config.size_guard.max_bytes = 1000

    captured = []
    async def capture_sink(entry):
        captured.append(entry)

    async with runtime_async(settings=settings) as logger:
        # Log something large
        await logger.info("x" * 5000)

    assert len(captured) == 1
    # Verify it was processed (truncated or marker added)
    # Actual assertion depends on whether sink receives dict or bytes
```

---

## Documentation Updates

### 1. Update `docs/plugins/processors.md`

Add comprehensive section on SizeGuardProcessor with:

- Configuration reference
- Use cases (CloudWatch, Datadog, Kafka)
- Truncation strategies explained
- Examples

### 2. Update `docs/plugin-guide.md`

Add SizeGuardProcessor to the plugin catalog.

### 3. Add Example

Create `docs/examples/size-limited-destinations.md` showing:

- CloudWatch configuration
- Kafka configuration
- Custom size limits

### 4. Update `docs/troubleshooting/`

Add "Logs being dropped by destination" troubleshooting guide.

---

## Rollout Plan

1. Implement `SizeGuardProcessor` class
2. Add configuration settings
3. Register as built-in
4. Write unit tests
5. Write integration tests
6. Update documentation
7. Release in next minor version

---

## Design Decisions

### Q1: Why 256KB default?

CloudWatch Logs has a hard 256KB limit and is one of the most common destinations. This default ensures CloudWatch compatibility out of the box.

### Q2: Why truncate message first?

The `message` field is typically the largest variable-size field and often contains verbose error details or stack traces. Truncating it preserves structured metadata which is more useful for filtering/searching.

### Q3: Why not compress instead of truncate?

Compression changes the format (no longer valid JSON lines). Destinations expecting JSON would fail to parse. Compression is better handled at the HTTP transport layer.

### Q4: What about binary/non-JSON payloads?

SizeGuardProcessor assumes JSON input. Non-JSON payloads would fail to parse during truncation and fall back to the drop marker. This is acceptable since fapilog is a JSON-structured logging library.

---

## Success Metrics

- [ ] All unit tests pass
- [ ] Integration test confirms pipeline wiring
- [ ] Documentation complete
- [ ] No performance regression (< 1ms overhead per event)
- [ ] Processor stage has a concrete, useful built-in

---

## Definition of Done

Story is complete when ALL of the following are true:

### Code Complete
- [ ] All acceptance criteria met and verified
- [ ] All tasks completed
- [ ] Code follows project style guide
- [ ] No linting errors or warnings
- [ ] Type checking passes (mypy strict)

### Quality Assurance
- [ ] Unit tests: >90% coverage of new code
- [ ] Integration tests: all scenarios passing
- [ ] Regression tests: no existing functionality broken
- [ ] Performance tests: no regression vs baseline
- [ ] Manual testing completed (if applicable)

### Documentation
- [ ] User-facing docs updated
- [ ] API reference updated (if applicable)
- [ ] Code examples added/updated
- [ ] CHANGELOG.md updated
- [ ] Inline code documentation complete

### Review & Release
- [ ] Code review approved
- [ ] Documentation reviewed for clarity
- [ ] CI/CD pipeline passing
- [ ] Ready for merge to main branch

### Backwards Compatibility
- [ ] No breaking changes OR breaking changes documented with migration guide
- [ ] Existing tests still pass
- [ ] Deprecation warnings added (if applicable)

---

## Risks / Rollback / Monitoring

### Risks

- **Risk:** Truncation breaks JSON structure
  - **Mitigation:** Comprehensive testing, safe truncation algorithm

- **Risk:** Performance impact from size checking
  - **Mitigation:** Optimize size checking, benchmark performance

- **Risk:** Users don't notice truncation happening
  - **Mitigation:** Clear diagnostics, metrics, warnings

### Rollback Plan

- Processor can be disabled via configuration
- No breaking changes to existing code
- Can remove processor if needed

### Success Metrics / Monitoring

- Track truncation/drop metrics
- Monitor processor performance
- User feedback on size enforcement

---

## Related Documents

- [Story 5.0: Wire Processors into Pipeline](./5.0.wire-processors-into-pipeline.md) ✅ Complete
- [Story 5.1: Fix PLUGIN_METADATA Inconsistencies](./5.1.fix-plugin-metadata-inconsistencies.md) ✅ Complete
- [Story 5.7: Plugin Observability Metrics](./5.7.plugin-observability-metrics.md) - SizeGuard metrics integration
- [Series 5.0 Overview](./5.0-series-overview.md)
- [Epic 3: Plugin Ecosystem Foundation](../prd/epic-3-plugin-ecosystem-foundation.md)

---

## Change Log

| Date       | Change                    | Author |
| ---------- | ------------------------- | ------ |
| 2025-01-10 | Migrated to new format    | System |
