# Story 5.x Series: Plugin System Completion & Enhancement

## Overview

This series addresses gaps identified in the plugin system audit, bringing fapilog's plugin architecture to enterprise-grade quality. Stories are ordered by implementation dependency and priority.

---

## Implementation Order

### Phase 1: Critical Foundation (Week 1-2)

| Story   | Title                                                                               | Priority | Effort | Dependencies |
| ------- | ----------------------------------------------------------------------------------- | -------- | ------ | ------------ |
| **5.0** | [Wire Processors into Pipeline](./5.0.wire-processors-into-pipeline.md)             | Critical | Medium | None         |
| **5.1** | [Fix PLUGIN_METADATA Inconsistencies](./5.1.fix-plugin-metadata-inconsistencies.md) | Critical | Small  | None         |

These stories fix fundamental issues where documented features don't work.

### Phase 2: High-Value Features (Week 2-4)

| Story    | Title                                                                                | Priority | Effort | Dependencies    |
| -------- | ------------------------------------------------------------------------------------ | -------- | ------ | --------------- |
| **5.2**  | [Add Filter Plugin Type](./5.2.add-filter-plugin-type.md)                            | High     | Medium | 5.0 (pattern)   |
| **5.3**  | [Enhanced Sampling & Rate Limiting](./5.3.add-sampling-and-rate-limiting-plugins.md) | High     | Medium | 5.2             |
| **5.4**  | [Plugin Testing Guide & Fixtures](./5.4.plugin-testing-guide.md)                     | High     | Medium | None (parallel) |
| **5.5**  | [Cloud Sink Examples](./5.5.cloud-sink-examples.md) (Epic)                           | High     | -      | See 5.14, 5.15  |
| **5.14** | [AWS CloudWatch Sink](./5.14.aws-cloudwatch-sink.md)                                 | High     | Medium | None            |
| **5.15** | [Grafana Loki Sink](./5.15.loki-sink.md)                                             | High     | Medium | None            |

These stories add missing functionality that enterprise users expect.

**Note:** Stories 5.14 and 5.15 implement the two official cloud sinks (SDK-based and HTTP-based patterns respectively). These are fully testable in CI via LocalStack (AWS) and Docker (Loki), providing reference implementations for users building their own cloud integrations.

### Phase 3: Performance & Observability (Week 4-5)

| Story   | Title                                                                  | Priority | Effort | Dependencies |
| ------- | ---------------------------------------------------------------------- | -------- | ------ | ------------ |
| **5.6** | [HTTP Sink Batching](./5.6.http-sink-batching.md)                      | Medium   | Small  | None         |
| **5.7** | [Plugin Observability Metrics](./5.7.plugin-observability-metrics.md)  | Medium   | Small  | 5.0          |
| **5.8** | [OpenTelemetry Integration](./5.8.opentelemetry-plugin-integration.md) | Medium   | Medium | 5.7          |

These stories improve performance and monitoring capabilities.

### Phase 4: Processor Value & API Refinement (Week 5-6)

| Story    | Title                                                                     | Priority | Effort | Dependencies |
| -------- | ------------------------------------------------------------------------- | -------- | ------ | ------------ |
| **5.9**  | [Add flush() to Plugin Protocols](./5.9.add-flush-to-plugin-protocols.md) | Low      | Small  | None         |
| **5.10** | [Plugin Dependency Declaration](./5.10.plugin-dependency-declaration.md)  | Low      | Medium | None         |
| **5.13** | [SizeGuardProcessor](./5.13.size-guard-processor.md)                      | Medium   | Small  | 5.0          |

These stories refine the plugin API and add practical processor functionality.

### Phase 5: Human Understandability Improvements (Week 6-8)

These stories address findings from the Human Understandability Audit to improve code maintainability and developer experience.

#### P0: Critical Clarity Issues

| Story    | Title                                                                                       | Priority | Effort | Dependencies |
| -------- | ------------------------------------------------------------------------------------------- | -------- | ------ | ------------ |
| **5.25** | [Extract Config Builders from __init__.py](./5.25.extract-config-builders-from-init.md)    | Critical | Medium | None         |
| **5.26** | [DRY Environment Variable Alias Validators](./5.26.dry-env-alias-validators.md)            | Critical | Medium | None         |
| **5.27** | [Document Pipeline Stage Ordering](./5.27.document-pipeline-stage-ordering.md)             | Critical | Small  | None         |

#### P1: High-Leverage Improvements

| Story    | Title                                                                                       | Priority | Effort | Dependencies |
| -------- | ------------------------------------------------------------------------------------------- | -------- | ------ | ------------ |
| **5.28** | [Unify Logger Factory Functions](./5.28.unify-logger-factory-functions.md)                  | High     | Medium | 5.25         |
| **5.29** | [Refactor Sink Writer Closures to Class](./5.29.refactor-sink-writer-to-class.md)           | High     | Medium | 5.25         |
| **5.30** | [Document Event Loop Lifecycle Logic](./5.30.document-event-loop-lifecycle.md)              | High     | Small  | None         |
| **5.31** | [Explicit Plugin Name Normalization](./5.31.explicit-plugin-name-normalization.md)          | High     | Small  | None         |

These stories reduce cognitive load, improve discoverability, and make the codebase easier to maintain.

### Out of Scope

The following stories are **not currently in scope** for fapilog:

| Story    | Title                         | Reason                                                        |
| -------- | ----------------------------- | ------------------------------------------------------------- |
| ~~5.11~~ | ~~Plugin Hot-Reload~~         | Library focus: users implement their own runtime management   |
| ~~5.12~~ | ~~Plugin Marketplace Client~~ | Library focus: standard Python packaging (PyPI) is sufficient |

fapilog is designed as a **plugin-enabling library** — it provides the infrastructure for users to build their own plugins (Kafka, cloud sinks, custom processors, etc.) rather than bundling them.

---

## Dependency Graph

```
5.0 Wire Processors ─────────────────┬──► 5.2 Filter Plugins ──► 5.3 Sampling
                                     │
                                     └──► 5.13 SizeGuardProcessor
5.1 Fix Metadata ────────────────────────────────────────────► (parallel)

5.4 Testing Guide ───────────────────────────────────────────► (parallel)

5.5 Cloud Sink Examples (Epic)
├── 5.14 AWS CloudWatch Sink ────────────────────────────────► (SDK pattern)
└── 5.15 Grafana Loki Sink ──────────────────────────────────► (HTTP pattern)

5.6 HTTP Batching ───────────────────────────────────────────► (independent)

5.7 Metrics ─────────────────────────► 5.8 OpenTelemetry

5.9 flush() ─────────────────────────────────────────────────► (independent)

5.10 Dependencies ───────────────────────────────────────────► (independent)

Phase 5: Human Understandability
5.25 Extract Config Builders ────────┬──► 5.28 Unify Logger Factories
                                     └──► 5.29 Refactor Sink Writers
5.26 DRY Env Validators ─────────────────────────────────────► (parallel)
5.27 Document Pipeline ──────────────────────────────────────► (parallel)
5.30 Document Event Loop ────────────────────────────────────► (parallel)
5.31 Plugin Name Normalization ──────────────────────────────► (parallel)
```

---

## Total Effort Estimate

| Priority  | Story Count | Total Effort    |
| --------- | ----------- | --------------- |
| Critical  | 5           | ~9-13 days      |
| High      | 10          | ~24-32 days     |
| Medium    | 4           | ~8-10 days      |
| Low       | 2           | ~4-6 days       |
| **Total** | **21**      | **~45-61 days** |

_Note: Stories 5.11 (Hot-Reload) and 5.12 (Marketplace) are out of scope._
_Cloud sinks (5.14, 5.15) are fully CI-testable via LocalStack and Docker._
_Phase 5 stories (5.25-5.31) address Human Understandability Audit findings._

---

## Success Metrics

### Phase 1 Complete

- [ ] Processors execute in pipeline and appear in logs
- [ ] All PLUGIN_METADATA names match class `name` attributes
- [ ] Documentation matches implementation

### Phase 2 Complete

- [ ] Filter plugins can drop events before enrichment
- [ ] Adaptive sampling reduces log volume under load
- [ ] Plugin authors have pytest fixtures and testing guide
- [ ] AWS CloudWatch sink is production-ready with LocalStack CI tests
- [ ] Grafana Loki sink is production-ready with Docker CI tests
- [ ] Both cloud sinks serve as reference implementations for users

### Phase 3 Complete

- [ ] HTTP sink batches requests (configurable)
- [ ] Plugin load/start times visible in Prometheus
- [ ] OpenTelemetry spans created for plugin operations

### Phase 4 Complete

- [ ] `flush()` available on all buffering sinks
- [ ] Plugins can declare dependencies on other plugins
- [ ] `SizeGuardProcessor` enforces payload size limits
- [ ] Processor stage has a practical, useful built-in

### Phase 5 Complete

- [ ] `__init__.py` reduced to <500 lines (from 1,308)
- [ ] Environment variable validators share common implementation
- [ ] Pipeline stage ordering documented with rationale
- [ ] Logger factory functions share common implementation
- [ ] Sink writer logic encapsulated in class (no nested closures)
- [ ] Event loop lifecycle logic documented inline
- [ ] Plugin name normalization explicit in error messages

---

## Parallel Work Tracks

For teams with multiple developers, these can be worked in parallel:

**Track A (Core Pipeline):** 5.0 → 5.2 → 5.3 → 5.13

**Track B (DX/Docs):** 5.1, 5.4 (independent)

**Track C (Cloud Sinks):** 5.14, 5.15 (parallel, both independent)

**Track D (Observability):** 5.6 → 5.7 → 5.8

**Track E (API Refinement):** 5.9, 5.10 (independent)

**Track F (Human Understandability - P0):** 5.25, 5.26, 5.27 (parallel, all independent)

**Track G (Human Understandability - P1):** 5.25 → 5.28, 5.29 (sequential), then 5.30, 5.31 (parallel)

---

## Related Documents

- [Plugin Audit Report](./plugin-audit-report.md) - Original audit findings
- [Human Understandability Audit Report](#) - Phase 5 source findings
- [Epic 3: Plugin Ecosystem Foundation](../prd/epic-3-plugin-ecosystem-foundation.md)
- [Plugin Development Guide](../plugins/index.md)

---

## Change Log

| Date       | Change                                                                   |
| ---------- | ------------------------------------------------------------------------ |
| 2026-01-14 | Added Phase 5 (Stories 5.25-5.31) from Human Understandability Audit     |
| 2026-01-05 | Added Stories 5.14 (AWS CloudWatch) and 5.15 (Loki) as official sinks    |
| 2026-01-05 | Added Story 5.13 (SizeGuardProcessor); marked 5.11, 5.12 as out of scope |
| 2026-01-03 | Initial creation from audit recommendations                              |
