# Story 1.27: Enricher and Pipeline Schema Alignment

**Status:** Complete
**Priority:** High
**Depends on:** Story 1.26 (Canonical Log Schema v1.1 Definition)

---

## Context / Background

Story 1.26 defines the canonical log schema v1.1 with semantic groupings:
- `context` - request/trace identifiers (correlation_id, request_id, trace_id, etc.)
- `diagnostics` - runtime/operational data (host, pid, service, k8s info, exceptions)
- `data` - user-provided structured data

**Current enricher behavior:**
Enrichers return flat dictionaries that are shallow-merged at the top level of the event:

```python
# RuntimeInfoEnricher returns:
{"service": "myapp", "host": "server1", "pid": 12345}

# ContextVarsEnricher returns:
{"request_id": "req-123", "trace_id": "abc", "span_id": "def"}

# KubernetesEnricher returns:
{"k8s_pod": "app-xyz", "k8s_namespace": "prod"}
```

After enrichment, the event has these fields scattered at the top level, not in their semantic groups.

**Required change:**
Enrichers must target the appropriate semantic group, and the merge logic must support deep-merging into nested dictionaries.

---

## Scope (In / Out)

### In Scope

- Update `RuntimeInfoEnricher` to return data targeting `diagnostics`
- Update `ContextVarsEnricher` to return data targeting `context`
- Update `KubernetesEnricher` to return data targeting `diagnostics`
- Update `enrich_parallel()` to support deep-merge into nested dicts
- Update `LogEvent` Pydantic model to match v1.1 schema
- Update enricher protocol/documentation
- Update enricher tests

### Out of Scope

- New enrichers (future stories)
- Serialization path cleanup (Story 1.28)
- Sink modifications (Story 1.28)

---

## Acceptance Criteria

### AC1: Enricher Return Format Updated

**Description:** Enrichers return data structured to target semantic groups.

**New enricher return format:**
```python
# Option A: Nested dict (enricher specifies target group)
{"diagnostics": {"host": "server1", "pid": 12345}}

# Option B: Flat dict with __target__ hint
{"host": "server1", "pid": 12345, "__target__": "diagnostics"}
```

**Validation (using Option A - nested dict):**
```python
from fapilog.plugins.enrichers import RuntimeInfoEnricher

enricher = RuntimeInfoEnricher()
await enricher.start()
result = await enricher.enrich({"level": "INFO", "message": "test"})

# Returns nested structure targeting diagnostics
assert "diagnostics" in result
assert "host" in result["diagnostics"]
assert "pid" in result["diagnostics"]
```

### AC2: RuntimeInfoEnricher Targets Diagnostics

**Description:** Runtime info (service, env, host, pid, python) goes into `diagnostics`.

**Validation:**
```python
from fapilog.plugins.enrichers import RuntimeInfoEnricher

enricher = RuntimeInfoEnricher()
await enricher.start()
result = await enricher.enrich({})

assert "diagnostics" in result
diag = result["diagnostics"]
assert "host" in diag
assert "pid" in diag
assert "python" in diag
# service and env only present if env vars set
```

### AC3: ContextVarsEnricher Targets Context

**Description:** Request/trace identifiers (request_id, user_id, trace_id, span_id) go into `context`.

**Validation:**
```python
from fapilog.plugins.enrichers import ContextVarsEnricher
from fapilog.core.errors import request_id_var, user_id_var

request_id_var.set("req-abc")
user_id_var.set("user-123")

enricher = ContextVarsEnricher()
result = await enricher.enrich({})

assert "context" in result
ctx = result["context"]
assert ctx["request_id"] == "req-abc"
assert ctx["user_id"] == "user-123"
```

### AC4: KubernetesEnricher Targets Diagnostics

**Description:** Kubernetes metadata (pod, namespace, node, cluster) goes into `diagnostics`.

**Validation:**
```python
import os
from fapilog.plugins.enrichers import KubernetesEnricher

os.environ["POD_NAME"] = "myapp-abc123-xyz"
os.environ["POD_NAMESPACE"] = "production"

enricher = KubernetesEnricher()
await enricher.start()
result = await enricher.enrich({})

assert "diagnostics" in result
diag = result["diagnostics"]
assert "k8s_pod" in diag
assert "k8s_namespace" in diag
```

### AC5: Deep-Merge in enrich_parallel()

**Description:** The `enrich_parallel()` function deep-merges enricher results into nested dicts.

**Validation:**
```python
from fapilog.plugins.enrichers import enrich_parallel, RuntimeInfoEnricher, ContextVarsEnricher

event = {
    "timestamp": "2024-01-15T10:00:00.000Z",
    "level": "INFO",
    "message": "test",
    "context": {"correlation_id": "corr-123"},
    "diagnostics": {},
    "data": {},
}

enrichers = [RuntimeInfoEnricher(), ContextVarsEnricher()]
result = await enrich_parallel(event, enrichers)

# Original context preserved and enriched
assert result["context"]["correlation_id"] == "corr-123"
# Enricher additions merged in
assert "host" in result["diagnostics"]
assert "pid" in result["diagnostics"]
```

### AC6: LogEvent Model Updated

**Description:** The `LogEvent` Pydantic model reflects the v1.1 canonical schema.

**Validation:**
```python
from fapilog.core.events import LogEvent

event = LogEvent(
    timestamp=1705312800.0,
    level="INFO",
    message="test",
    logger="myapp",
    context={"correlation_id": "abc"},
    diagnostics={"host": "server1"},
    data={"user_action": "login"},
)

mapping = event.to_mapping()
assert "context" in mapping
assert "diagnostics" in mapping
assert "data" in mapping
assert mapping["context"]["correlation_id"] == "abc"
```

### AC7: Enricher Protocol Documentation Updated

**Description:** The `BaseEnricher` protocol docstring documents the expected return format.

**Validation:**
- `BaseEnricher.enrich()` docstring explains:
  - Return dict should contain semantic group keys (`context`, `diagnostics`, or `data`)
  - Deep-merge behavior for nested dicts
  - Examples of correct return format
- Plugin authoring docs updated with enricher examples

### AC8: Documentation Alignment

**Description:** Documentation reflects the new enricher behavior and schema.

**Validation:**
- README enricher examples show new return format
- Inline docstrings in all three enrichers updated
- CHANGELOG documents the breaking change

---

## Implementation Notes

### File Changes

```
src/fapilog/plugins/enrichers/__init__.py (MODIFIED - deep-merge logic)
src/fapilog/plugins/enrichers/runtime_info.py (MODIFIED - target diagnostics)
src/fapilog/plugins/enrichers/context_vars.py (MODIFIED - target context)
src/fapilog/plugins/enrichers/kubernetes.py (MODIFIED - target diagnostics)
src/fapilog/core/events.py (MODIFIED - update LogEvent model)
tests/unit/test_runtime_info_enricher.py (NEW - runtime info doesn't have dedicated tests)
tests/unit/test_context_vars_enricher.py (MODIFIED)
tests/unit/test_kubernetes_enricher.py (MODIFIED)
tests/unit/test_enrichers_parallel.py (MODIFIED)
tests/unit/test_log_event_v1_1.py (NEW)
```

### Deep-Merge Implementation

```python
def _deep_merge(base: dict, updates: dict) -> dict:
    """Deep-merge updates into base dict.

    For nested dicts (context, diagnostics, data), merge contents.
    For other keys, updates overwrite base.
    """
    result = dict(base)
    for key, value in updates.items():
        if key in result and isinstance(result[key], dict) and isinstance(value, dict):
            result[key] = _deep_merge(result[key], value)
        else:
            result[key] = value
    return result


async def enrich_parallel(event: dict, enrichers: Iterable[BaseEnricher], ...) -> dict:
    # ... existing parallel execution ...

    merged = dict(event)
    for res in results:
        if isinstance(res, BaseException):
            continue
        merged = _deep_merge(merged, res)
    return merged
```

### Updated Enricher Example

```python
class RuntimeInfoEnricher:
    async def enrich(self, event: dict[str, Any]) -> dict[str, Any]:
        info = {
            "service": os.getenv("FAPILOG_SERVICE", "fapilog"),
            "env": os.getenv("FAPILOG_ENV", os.getenv("ENV", "dev")),
            "host": socket.gethostname(),
            "pid": os.getpid(),
            "python": platform.python_version(),
        }
        # Remove None values
        compact = {k: v for k, v in info.items() if v is not None}

        # Return targeting diagnostics group
        return {"diagnostics": compact}
```

### Updated LogEvent Model

```python
class LogEvent(BaseModel):
    timestamp: float = Field(...)
    level: str = Field(...)
    message: str = Field(...)
    logger: str | None = Field(default=None)

    # Semantic groupings (v1.1)
    context: dict[str, Any] = Field(default_factory=dict)
    diagnostics: dict[str, Any] = Field(default_factory=dict)
    data: dict[str, Any] = Field(default_factory=dict)

    # Removed in v1.1 (breaking change):
    # - metadata: dict[str, Any] → use data instead
    # - correlation_id: str → use context["correlation_id"]
    # - component: str → use context or data
```

---

## Tasks

### Phase 1: Deep-Merge Infrastructure

- [ ] Implement `_deep_merge()` helper function
- [ ] Update `enrich_parallel()` to use deep-merge
- [ ] Add unit tests for deep-merge behavior

### Phase 2: Enricher Updates

- [ ] Update `RuntimeInfoEnricher.enrich()` to return `{"diagnostics": {...}}`
- [ ] Update `ContextVarsEnricher.enrich()` to return `{"context": {...}}`
- [ ] Update `KubernetesEnricher.enrich()` to return `{"diagnostics": {...}}`
- [ ] Update enricher docstrings

### Phase 3: LogEvent Model

- [ ] Add `context`, `diagnostics`, `data` fields to LogEvent
- [ ] Remove `metadata`, `correlation_id`, `component` fields (breaking change)
- [ ] Update `to_mapping()` method
- [ ] Update model docstring

### Phase 4: Protocol and Documentation

- [ ] Update `BaseEnricher` protocol docstring
- [ ] Update enricher inline documentation
- [ ] Update README enricher examples
- [ ] Update CHANGELOG

---

## Tests

### Unit Tests

- `tests/unit/test_runtime_info_enricher.py` (NEW)
  - `test_returns_diagnostics_structure`
  - `test_diagnostics_contains_host_pid_python`

- `tests/unit/test_context_vars_enricher.py` (MODIFIED)
  - `test_returns_context_structure`
  - `test_context_contains_request_id_user_id`
  - `test_context_contains_trace_span_ids`

- `tests/unit/test_kubernetes_enricher.py` (MODIFIED)
  - `test_returns_diagnostics_structure`
  - `test_diagnostics_contains_k8s_fields`

- `tests/unit/test_enrichers_parallel.py` (MODIFIED)
  - `test_deep_merge_preserves_existing_context`
  - `test_deep_merge_combines_multiple_enrichers`
  - `test_deep_merge_handles_empty_dicts`

- `tests/unit/test_log_event_v1_1.py` (NEW)
  - `test_log_event_v1_1_schema`
  - `test_log_event_to_mapping_includes_semantic_groups`

### Integration Tests

- `tests/integration/test_enricher_pipeline.py`
  - `test_full_pipeline_produces_v1_1_schema`
  - `test_enrichers_populate_correct_semantic_groups`

---

## Definition of Done

### Code Complete

- [ ] All acceptance criteria implemented
- [ ] All three enrichers return nested structure
- [ ] Deep-merge works correctly
- [ ] LogEvent model updated

### Quality Assurance

- [ ] Unit tests written and passing
- [ ] Integration tests written and passing
- [ ] `ruff check` passes
- [ ] `mypy` passes
- [ ] No regression in existing tests

### Documentation

- [ ] BaseEnricher protocol docstring updated
- [ ] All enricher docstrings updated
- [ ] README enricher section updated
- [ ] CHANGELOG updated

---

## Risks / Rollback

### Risks

1. **Risk:** Custom enrichers in user code return flat dicts
   - **Mitigation:** Deep-merge handles both nested and flat formats for flexibility
   - **Mitigation:** Document new return format in CHANGELOG and migration guide

2. **Risk:** LogEvent model change breaks serialization
   - **Mitigation:** Ensure `to_mapping()` produces v1.1 schema output

3. **Risk:** Deep-merge performance overhead
   - **Mitigation:** Benchmark; dict operations are fast; only 3 levels deep

### Rollback Plan

If critical issues occur post-release:
1. Revert the PR in a patch release
2. Users can pin to previous version until fix available

---

## Related Stories

- **Depends on:** Story 1.26 - Canonical Log Schema v1.1 Definition
- **Enables:** Story 1.28 - Serialization Path Cleanup
- **Related:** Story 1.15 - Formal Plugin Contracts (enricher protocol)

---

## Code Review

**Date:** 2026-01-19
**Reviewer:** Claude
**Verdict:** OK to PR

### Summary

Implemented enricher and LogEvent alignment with v1.1 schema: added `_deep_merge()` helper, updated all three enrichers to return nested dicts targeting semantic groups, and updated LogEvent model with `context`, `diagnostics`, `data` fields.

### AC Verification

| Criterion | Evidence |
|-----------|----------|
| AC1: Enricher Return Format | `runtime_info.py:34` returns `{"diagnostics": compact}`; `context_vars.py:61` returns `{"context": data}`; `kubernetes.py:71` returns `{"diagnostics": dict(self._cached_fields)}` |
| AC2: RuntimeInfoEnricher Targets Diagnostics | `runtime_info.py:34` - returns `{"diagnostics": {...}}`; `test_runtime_info_enricher.py:21-25` validates structure |
| AC3: ContextVarsEnricher Targets Context | `context_vars.py:61` - returns `{"context": data}`; `test_context_vars_enricher.py:17-21` validates structure |
| AC4: KubernetesEnricher Targets Diagnostics | `kubernetes.py:71` - returns `{"diagnostics": ...}`; `test_kubernetes_enricher.py:102-105` validates structure |
| AC5: Deep-Merge in enrich_parallel() | `__init__.py:13-27` - `_deep_merge()` helper; `__init__.py:137` uses it; `test_enrichers_parallel.py:76-97` validates behavior |
| AC6: LogEvent Model Updated | `events.py:44-55` - adds `context`, `diagnostics`, `data` fields; `test_log_event_v1_1.py:21-23,29-31` validates fields |
| AC7: Protocol Documentation Updated | `__init__.py:31-59` - comprehensive docstring with return format, examples, and deep-merge behavior |
| AC8: Documentation Alignment | `CHANGELOG.md:8` - documents breaking change; all enricher docstrings updated; `plugin-guide.md` auto-updated |

### Quality Gates

- [x] ruff check passed
- [x] ruff format passed
- [x] mypy passed
- [x] diff-cover 100% (>= 90%)
- [x] No weak assertions
- [x] No dead code (vulture low-confidence only)
- [x] No Pydantic v1 syntax

---

## Change Log

| Date | Change | Author |
|------|--------|--------|
| 2025-01-19 | Initial draft | Claude |
| 2025-01-19 | Simplified for breaking release - removed deprecated field framing | Claude |
| 2026-01-19 | Implementation complete, code review passed | Claude |
