# Story 1.26: Canonical Log Schema v1.1 Definition

**Status:** Complete
**Priority:** High
**Depends on:** None (foundational)

---

## Context / Background

An external audit identified a critical schema inconsistency in the fapilog logging pipeline:

**Current State:**
- `build_envelope()` (envelope.py:84-91) produces: `{timestamp, level, message, logger, correlation_id, metadata}`
- `serialize_envelope()` (serialization.py:221-226) requires: `{timestamp, level, message, context, diagnostics}`
- These schemas are **incompatible**

**Impact:**
- Every log event triggers `TypeError: context must be a mapping` in `serialize_envelope()`
- Sinks catch this exception and fall back to `serialize_mapping_to_json_bytes()` (stdout_json.py:44-62, rotating_file.py:140-154)
- The v1.0 envelope format from Story 1.17 is **never actually produced**
- `strict_envelope_mode=True` drops ALL events because validation always fails
- Performance penalty from exceptions on every log event

**Evidence from tests:**
- `test_serialization_recovery.py:17`: *"serialize_envelope() - requires context/diagnostics fields (fails for LogEvent)"*
- `test_worker_lifecycle.py:481`: *"All events will fail serialize_envelope (missing context/diagnostics)"*

This story defines a unified canonical schema and implements it in `build_envelope()`.

---

## Scope (In / Out)

### In Scope

- Define canonical log schema v1.1 with semantic field groupings
- Update `build_envelope()` to produce the canonical schema
- Update `jsonschema/log_envelope_v1.json` to v1.1
- Add schema migration documentation
- Update inline documentation and docstrings

### Out of Scope

- Enricher changes (Story 1.27)
- LogEvent model changes (Story 1.27)
- Serialization cleanup (Story 1.28)
- Sink fallback removal (Story 1.28)

---

## Acceptance Criteria

### AC1: Canonical Schema Definition

**Description:** Define a unified schema with clear semantic groupings that serves both internal pipeline and external serialization needs.

**Schema v1.1:**
```python
{
    "timestamp": str,        # RFC3339 UTC with Z suffix (e.g., "2024-01-15T10:30:00.123Z")
    "level": str,            # DEBUG, INFO, WARNING, ERROR, CRITICAL
    "message": str,          # Human-readable log message
    "logger": str | None,    # Logger name (optional)

    # Semantic groupings:
    "context": {             # Request/trace context - identifies WHO and WHAT request
        "correlation_id": str | None,
        "request_id": str | None,
        "user_id": str | None,
        "tenant_id": str | None,
        "trace_id": str | None,
        "span_id": str | None,
        # Extensible: enrichers may add more
    },
    "diagnostics": {         # Runtime/operational context - identifies WHERE and system state
        "service": str | None,
        "env": str | None,
        "host": str | None,
        "pid": int | None,
        "python": str | None,
        # Exception info when present:
        "exception": {...} | None,
        # Extensible: enrichers may add k8s_*, etc.
    },
    "data": {                # User-provided structured data (replaces "metadata")
        # Arbitrary key-value pairs from log call extra args
    }
}
```

**Validation:**
```python
from fapilog.core.envelope import build_envelope

envelope = build_envelope(
    level="INFO",
    message="Test message",
    extra={"user_action": "login"},
    correlation_id="corr-123",
)

# All semantic groups present
assert "context" in envelope
assert "diagnostics" in envelope
assert "data" in envelope

# Context contains correlation_id
assert envelope["context"]["correlation_id"] == "corr-123"

# User data in data field
assert envelope["data"]["user_action"] == "login"

# Timestamp is RFC3339 UTC
assert envelope["timestamp"].endswith("Z")
```

### AC2: build_envelope() Produces Canonical Schema

**Description:** Update `build_envelope()` to output the v1.1 canonical schema structure.

**Validation:**
```python
from fapilog.core.envelope import build_envelope

# Basic envelope
envelope = build_envelope(
    level="ERROR",
    message="Something failed",
    logger_name="myapp.service",
)

assert envelope["level"] == "ERROR"
assert envelope["message"] == "Something failed"
assert envelope["logger"] == "myapp.service"
assert isinstance(envelope["context"], dict)
assert isinstance(envelope["diagnostics"], dict)
assert isinstance(envelope["data"], dict)

# With exception
try:
    raise ValueError("test error")
except ValueError:
    envelope = build_envelope(
        level="ERROR",
        message="Caught error",
        exc_info=True,
    )
    # Exception data now in diagnostics
    assert "exception" in envelope["diagnostics"]
```

### AC3: JSON Schema Updated to v1.1

**Description:** Update `jsonschema/log_envelope_v1.json` to validate the new schema structure.

**Validation:**
```python
import json
import jsonschema
from fapilog.core.envelope import build_envelope
from fapilog.core.serialization import serialize_envelope

# Load schema
with open("jsonschema/log_envelope_v1.json") as f:
    schema = json.load(f)

# Build and serialize envelope
envelope = build_envelope(level="INFO", message="test")
serialized = serialize_envelope(envelope)
parsed = json.loads(serialized.data)

# Should validate without errors
jsonschema.validate(parsed, schema)
```

### AC4: Migration Documentation

**Description:** Document schema changes from v1.0 to v1.1 for downstream consumers upgrading to the new release.

**Validation:**
- `docs/schema-migration-v1.0-to-v1.1.md` exists
- Documents breaking changes:
  - `metadata` field removed → use `data`
  - `correlation_id` (top-level) removed → use `context.correlation_id`
- Documents new semantic groupings (`context`, `diagnostics`, `data`)
- Includes example transformations for consumers updating their log parsers

### AC5: Code Documentation Updated

**Description:** All modified functions have updated docstrings reflecting the new schema.

**Validation:**
- `build_envelope()` docstring documents output schema structure
- `serialize_envelope()` docstring references v1.1 schema
- Module-level docstrings updated in `envelope.py` and `serialization.py`

---

## Implementation Notes

### File Changes

```
src/fapilog/core/envelope.py (MODIFIED - restructure output schema)
src/fapilog/core/serialization.py (MODIFIED - update validation for v1.1)
jsonschema/log_envelope_v1.json (MODIFIED - update to v1.1)
docs/schema-migration-v1.0-to-v1.1.md (NEW)
tests/unit/envelope/test_envelope_schema.py (MODIFIED)
tests/unit/envelope/test_envelope_v1_1.py (NEW)
tests/property/test_serialization_properties.py (MODIFIED)
```

### Key Implementation

**build_envelope() structure:**
```python
def build_envelope(
    level: str,
    message: str,
    *,
    extra: dict[str, Any] | None = None,
    bound_context: dict[str, Any] | None = None,
    exc: BaseException | None = None,
    exc_info: ...,
    exceptions_enabled: bool = True,
    exceptions_max_frames: int = 50,
    exceptions_max_stack_chars: int = 20000,
    logger_name: str = "root",
    correlation_id: str | None = None,
) -> dict[str, Any]:
    # Build context dict (request/trace identifiers)
    context: dict[str, Any] = {}
    corr_id = correlation_id if correlation_id is not None else str(uuid4())
    context["correlation_id"] = corr_id

    # Extract trace context from bound_context if present
    for key in ("request_id", "user_id", "tenant_id", "trace_id", "span_id"):
        if bound_context and key in bound_context:
            context[key] = bound_context[key]

    # Build diagnostics dict (runtime/operational)
    diagnostics: dict[str, Any] = {}
    if exceptions_enabled:
        exc_data = _serialize_exception(...)
        if exc_data:
            diagnostics["exception"] = exc_data

    # Build data dict (user-provided structured data)
    data: dict[str, Any] = {}
    if bound_context:
        # Non-context fields go to data
        for k, v in bound_context.items():
            if k not in ("request_id", "user_id", "tenant_id", "trace_id", "span_id"):
                data[k] = v
    if extra:
        data.update(extra)

    # RFC3339 timestamp
    ts = datetime.now(timezone.utc).isoformat(timespec="milliseconds").replace("+00:00", "Z")

    return {
        "timestamp": ts,
        "level": level,
        "message": message,
        "logger": logger_name,
        "context": context,
        "diagnostics": diagnostics,
        "data": data,
    }
```

---

## Tasks

### Phase 1: Schema Definition

- [ ] Document canonical schema v1.1 in code comments
- [ ] Update `jsonschema/log_envelope_v1.json` with v1.1 structure
- [ ] Create `docs/schema-migration-v1.0-to-v1.1.md`

### Phase 2: Implementation

- [ ] Update `build_envelope()` to produce v1.1 schema
- [ ] Move `correlation_id` into `context` dict
- [ ] Move exception data into `diagnostics` dict
- [ ] Rename internal `metadata` handling to `data`
- [ ] Update timestamp to RFC3339 string format

### Phase 3: Serialization Alignment

- [ ] Update `serialize_envelope()` validation for v1.1 (accept `data` field)
- [ ] Ensure `serialize_envelope(build_envelope(...))` works without exception
- [ ] Update property-based test strategies

### Phase 4: Documentation

- [ ] Update `build_envelope()` docstring
- [ ] Update `serialize_envelope()` docstring
- [ ] Update module docstrings
- [ ] Add inline comments explaining semantic groupings

---

## Tests

### Unit Tests

- `tests/unit/envelope/test_envelope_v1_1.py` (NEW)
  - `test_build_envelope_produces_v1_1_schema`
  - `test_context_contains_correlation_id`
  - `test_diagnostics_contains_exception_data`
  - `test_data_contains_user_provided_fields`
  - `test_timestamp_is_rfc3339_utc`
  - `test_bound_context_fields_routed_correctly`

- `tests/unit/envelope/test_envelope_schema.py` (MODIFIED)
  - Update existing tests for new schema structure

### Property Tests

- `tests/property/test_serialization_properties.py` (MODIFIED)
  - Update `envelope_logs` strategy for v1.1 schema
  - Add property: `serialize_envelope(build_envelope(...))` never raises

### Integration Tests

- `tests/integration/test_envelope_serialization_roundtrip.py` (NEW)
  - `test_build_serialize_roundtrip_produces_valid_json`
  - `test_json_schema_validation_passes`

---

## Definition of Done

### Code Complete

- [ ] All acceptance criteria implemented
- [ ] `serialize_envelope(build_envelope(...))` works without exception
- [ ] JSON schema validates serialized output
- [ ] Code follows project patterns

### Quality Assurance

- [ ] Unit tests written and passing
- [ ] Property tests updated and passing
- [ ] Integration tests written and passing
- [ ] `ruff check` passes
- [ ] `mypy` passes
- [ ] No regression in existing tests

### Documentation

- [ ] Schema migration guide created
- [ ] Code docstrings updated
- [ ] CHANGELOG updated with v1.1 schema changes

---

## Risks / Rollback

### Risks

1. **Risk:** Downstream consumers depend on v1.0 schema structure
   - **Mitigation:** This is a breaking release; migration guide documents all changes

2. **Risk:** Breaking change to `build_envelope()` return type affects internal code
   - **Mitigation:** Update all callers in same PR; comprehensive test coverage

3. **Risk:** Performance regression from timestamp string formatting
   - **Mitigation:** Benchmark before/after; string formatting is fast

### Rollback Plan

If critical issues occur post-release:
1. Revert the PR in a patch release
2. Users can pin to previous version until fix available

---

## Related Stories

- **Depends on:** None (foundational)
- **Enables:** Story 1.27 - Enricher and Pipeline Schema Alignment
- **Enables:** Story 1.28 - Serialization Path Cleanup
- **Supersedes:** Story 1.17 - Schema Versioned Log Envelope v1 (this fixes its incomplete implementation)

---

## Code Review

**Date:** 2026-01-19
**Reviewer:** Claude
**Verdict:** OK to PR

### Summary

Implemented canonical log schema v1.1 with semantic field groupings (`context`, `diagnostics`, `data`), fixing the schema mismatch between `build_envelope()` and `serialize_envelope()`.

### AC Verification

| Criterion | Evidence |
|-----------|----------|
| AC1: Canonical Schema Definition | `envelope.py:75-78` - `_CONTEXT_FIELDS` defines semantic routing; `envelope.py:119-127` - builds semantic dicts |
| AC2: build_envelope() Produces Canonical Schema | `envelope.py:129-135` - returns v1.1 structure; `test_envelope_v1_1.py:19-29` - validates all required fields |
| AC3: JSON Schema Updated to v1.1 | `log_envelope_v1.json:6,14` - title and schema_version updated; `test_envelope_serialization_roundtrip.py:100-117` - validates against schema |
| AC4: Migration Documentation | `docs/schema-migration-v1.0-to-v1.1.md` - complete migration guide with before/after examples |
| AC5: Code Documentation Updated | `envelope.py:41-72` - extensive docstring with v1.1 schema structure; `serialization.py:210-217` - updated docstring |

### Quality Gates

- [x] ruff check passed
- [x] ruff format passed
- [x] mypy passed
- [x] diff-cover >= 90% (96% on changed lines)
- [x] No weak assertions
- [x] No dead code
- [x] All 2108 tests pass

---

## Change Log

| Date | Change | Author |
|------|--------|--------|
| 2025-01-19 | Initial draft based on audit findings | Claude |
| 2025-01-19 | Simplified for breaking release - removed backward compat considerations | Claude |
| 2026-01-19 | Implementation complete, code review passed | Claude |
