# Story 5.27: Document Pipeline Stage Ordering

**Status:** Complete
**Priority:** Critical
**Depends on:** None

---

## Context / Background

The fapilog pipeline processes log events through multiple stages:

```
filters -> enrichers -> redactors -> processors -> sink
```

This ordering is **implicit** in the `_flush_batch()` method of `LoggerWorker` (`worker.py:235-268`). The code calls each stage in sequence without explaining:

1. **Why this order?** (e.g., why filters before enrichers?)
2. **What are the data flow guarantees?** (e.g., redactors see enriched data)
3. **What happens if a stage fails?**

This creates problems:
- **Debugging difficulty:** "Why isn't my redactor seeing the enriched field?"
- **Change risk:** Developers may reorder stages without understanding dependencies
- **Onboarding friction:** Must trace through code to understand the pipeline

The Human Understandability Audit identified this as a **P0 issue**.

Reference: `src/fapilog/core/worker.py:235-268`

---

## Scope (In / Out)

### In Scope

- Add comprehensive docstring to `_flush_batch()` explaining stage order
- Add inline comments for each stage transition
- Create architecture documentation with pipeline diagram
- Document error handling behavior per stage
- Fix the ASCII diagram in `docs/core-concepts/pipeline-architecture.md` (missing Redactors stage)

### Out of Scope

- Changing the pipeline order
- Adding new pipeline stages
- Modifying stage configuration

### Relationship to Existing Documentation

`docs/core-concepts/pipeline-architecture.md` provides a **user-facing conceptual overview** of the pipeline stages. This story adds:

1. **Code-level documentation** - Docstrings and comments in `worker.py` explaining the implementation rationale
2. **Architecture deep-dive** - `docs/architecture/pipeline-stages.md` for developers/contributors explaining internal design decisions

The existing conceptual doc should be cross-referenced but not duplicated.

---

## Acceptance Criteria

### AC1: Code Documentation

**Description:** `_flush_batch()` has docstring explaining the full pipeline.

**Validation:**
```python
async def _flush_batch(self, batch: list[dict[str, Any]]) -> None:
    """Flush a batch of log events through the processing pipeline.

    Pipeline Stage Order:
    1. FILTERS: Drop unwanted events before processing cost
    2. ENRICHERS: Add contextual data (runtime info, request context)
    3. REDACTORS: Mask sensitive data including enriched fields
    4. PROCESSORS: Transform serialized bytes (when serialize_in_flush enabled)
    5. SINK: Write to destination

    Error Handling:
    - Stages 1-4: Errors contained; original event passed through
    - Stage 5: Errors logged via diagnostics; event dropped
    """
```

### AC2: Architecture Documentation

**Description:** Pipeline stages documented with diagram.

**Validation:**
```bash
cat docs/architecture/pipeline-stages.md  # File exists with diagram
```

### AC3: No Behavior Changes

**Description:** All existing tests pass; documentation only.

**Validation:**
```bash
pytest tests/ -x  # All green, no code changes
```

---

## Implementation Notes

### File Changes

```
src/fapilog/core/worker.py (MODIFIED - add docstrings/comments)
docs/architecture/pipeline-stages.md (NEW)
docs/core-concepts/pipeline-architecture.md (MODIFIED - fix ASCII diagram)
```

### Key Documentation

```python
async def _flush_batch(self, batch: list[dict[str, Any]]) -> None:
    """Flush a batch of log events through the processing pipeline.

    Pipeline Stage Order (with rationale):

    1. FILTERS: Applied first to drop unwanted events before any
       processing cost is incurred. Filters see raw events from queue.

    2. ENRICHERS: Applied second to add contextual data. Run after
       filters so dropped events don't waste enrichment cycles.

    3. REDACTORS: Applied third to mask sensitive data. Run after
       enrichers so they can redact both original AND enriched fields.

    4. PROCESSORS: Applied fourth to transform final payload. Run on
       serialized bytes when serialize_in_flush is enabled.

    5. SINK: Final stage writes to destination.

    See: docs/architecture/pipeline-stages.md
    """
```

---

## Tasks

### Phase 1: Code Documentation

- [ ] Add comprehensive docstring to `_flush_batch()`
- [ ] Add inline comments before each `_apply_*` call
- [ ] Add docstrings to `_apply_filters()`, `_apply_enrichers()`, etc.

### Phase 2: Architecture Documentation

- [ ] Create `docs/architecture/pipeline-stages.md`
- [ ] Add pipeline diagram (ASCII or Mermaid)
- [ ] Document each stage: purpose, inputs, outputs, error handling
- [ ] Fix ASCII diagram in `docs/core-concepts/pipeline-architecture.md` to include Redactors

### Phase 3: Cross-References

- [ ] Update plugin development guide to reference pipeline docs
- [ ] Add "When does my plugin run?" to each plugin type doc

---

## Tests

No new tests needed - documentation only.

### Manual Verification

- [ ] Read through pipeline docs and verify accuracy against code
- [ ] Have someone unfamiliar with codebase review for clarity
- [ ] Verify diagram matches actual code flow

---

## Definition of Done

### Code Complete

- [ ] All docstrings and comments added
- [ ] No linting errors

### Quality Assurance

- [ ] All existing tests still pass
- [ ] Comments reviewed for technical accuracy

### Documentation

- [ ] Architecture doc created with diagram
- [ ] Cross-references added

---

## Risks / Rollback

### Risks

1. **Risk:** Documentation becomes stale if pipeline changes
   - **Mitigation:** Add comment in code noting docs must be updated

### Rollback Plan

If issues occur:
1. Revert the commit (documentation only, no risk)

---

## Related Stories

- **Related:** Story 5.0 - Wire Processors into Pipeline
- **Related:** Story 5.30 - Document Event Loop Lifecycle

---

## Change Log

| Date | Change | Author |
|------|--------|--------|
| 2026-01-14 | Initial draft | Claude |
