# Story 5.7: Plugin Observability and Startup Metrics

## Status: Cancelled — Deferred to Enterprise

## Priority: ~~Medium~~ N/A

## Estimated Effort: ~~Small (2 days)~~ N/A

## Dependencies: N/A

## Epic: Enterprise

---

## Cancellation Rationale

**Decision Date:** January 2026

This story has been cancelled and deferred to a future enterprise package (`fapilog-observability` or similar) for the following reasons:

### 1. Core Library Already Provides Adequate Metrics

The existing `MetricsCollector` in `src/fapilog/metrics/metrics.py` already supports:

- Event processing counts (`fapilog_events_processed_total`)
- Plugin error tracking (`fapilog_plugin_errors_total`)
- Plugin execution latency histograms (`fapilog_plugin_exec_seconds`)
- In-memory `PluginStats` for profiling
- Graceful fallback when `prometheus_client` is not installed

This covers the needs of 95% of users.

### 2. Enterprise-Specific Scope

The story explicitly targets "enterprise deployments" with needs like:

- Per-plugin startup profiling for deployment optimization
- Registry introspection APIs for compliance/management tooling
- High-cardinality labeled metrics for sophisticated monitoring stacks

These are specialized requirements that don't belong in the core library.

### 3. Cardinality Risk

Proposed labels (`plugin_name × plugin_type × operation × error_type`) can cause Prometheus cardinality explosion. Enterprise teams with dedicated SRE know how to handle this; average users shouldn't be exposed to this footgun.

### 4. Lean Core Philosophy

fapilog's core value proposition is structured logging that "just works." Advanced observability is a specialized concern that should be opt-in via a separate package.

### 5. Precedent

The project already has `packages/fapilog-tamper/` as an enterprise add-on. A future `packages/fapilog-observability/` would follow the same pattern.

---

## Future Enterprise Package Scope

If/when customer demand warrants, a `fapilog-observability` package could include:

- Plugin startup timing (load + start duration)
- Plugin health status gauges
- Plugin registry introspection API
- OpenTelemetry trace context propagation
- Grafana dashboard templates

---

## Original Context (Archived)

> Enterprise deployments need visibility into plugin behavior:
>
> - Which plugins are loaded and healthy?
> - How long did each plugin take to start?
> - What's the latency/throughput of each plugin?
> - Which plugins are causing errors?

---

## Original Problem Statement (Archived)

> Operations teams cannot easily:
>
> 1. Identify slow-starting plugins during deployment
> 2. Correlate latency spikes to specific plugins
> 3. Monitor plugin health aggregated by type
> 4. Debug plugin loading failures

---

## Original Acceptance Criteria (Archived)

> **Note:** These criteria are preserved for historical reference. This story has been cancelled.

### ~~AC1: Startup Metrics~~

- ~~Record plugin load time (load_plugin call duration)~~
- ~~Record plugin start() duration~~
- ~~Expose via Prometheus metrics when enabled~~
- ~~Include plugin name and type as labels~~

### ~~AC2: Runtime Metrics~~

- ~~Per-plugin latency histogram~~ → **Already exists in MetricsCollector**
- ~~Per-plugin error counter~~ → **Already exists in MetricsCollector**
- ~~Per-plugin call counter~~ → **Already exists in MetricsCollector**
- ~~Labels: plugin_name, plugin_type, operation~~

### ~~AC3: Health Metrics~~

- ~~Gauge for plugin health status (1=healthy, 0=unhealthy)~~
- ~~Health check latency histogram~~
- ~~Last health check timestamp~~

### ~~AC4: Registry API~~

- ~~`list_loaded_plugins()` returns all loaded plugins with metadata~~
- ~~`get_plugin_stats(name)` returns per-plugin statistics~~ → **Already exists in MetricsCollector**
- ~~Expose via logger property or standalone function~~

### ~~AC5: Prometheus Integration~~

- ~~All metrics use `fapilog_plugin_` prefix~~
- ~~Labels are consistent and cardinality-aware~~
- ~~Histogram buckets appropriate for plugin operations~~

---

## Original Technical Design (Archived)

> **Note:** This design is preserved for reference if an enterprise package is created in the future.

### 1. Plugin Metrics Definitions

```python
# src/fapilog/metrics/plugin_metrics.py
"""Plugin-specific Prometheus metrics."""

from __future__ import annotations

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from prometheus_client import Counter, Gauge, Histogram

# Lazy initialization to avoid import errors when prometheus not installed
_metrics_initialized = False
_plugin_load_duration: "Histogram | None" = None
_plugin_start_duration: "Histogram | None" = None
_plugin_operation_duration: "Histogram | None" = None
_plugin_operation_count: "Counter | None" = None
_plugin_error_count: "Counter | None" = None
_plugin_health_status: "Gauge | None" = None


def _ensure_metrics() -> bool:
    """Initialize metrics if prometheus is available."""
    global _metrics_initialized
    global _plugin_load_duration, _plugin_start_duration
    global _plugin_operation_duration, _plugin_operation_count
    global _plugin_error_count, _plugin_health_status

    if _metrics_initialized:
        return True

    try:
        from prometheus_client import Counter, Gauge, Histogram

        _plugin_load_duration = Histogram(
            "fapilog_plugin_load_duration_seconds",
            "Time to load a plugin",
            ["plugin_name", "plugin_type"],
            buckets=[0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0],
        )

        _plugin_start_duration = Histogram(
            "fapilog_plugin_start_duration_seconds",
            "Time for plugin start() to complete",
            ["plugin_name", "plugin_type"],
            buckets=[0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0],
        )

        _plugin_operation_duration = Histogram(
            "fapilog_plugin_operation_duration_seconds",
            "Time for plugin operation (write, enrich, redact, process)",
            ["plugin_name", "plugin_type", "operation"],
            buckets=[0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1],
        )

        _plugin_operation_count = Counter(
            "fapilog_plugin_operations_total",
            "Total plugin operations",
            ["plugin_name", "plugin_type", "operation"],
        )

        _plugin_error_count = Counter(
            "fapilog_plugin_errors_total",
            "Total plugin errors",
            ["plugin_name", "plugin_type", "operation", "error_type"],
        )

        _plugin_health_status = Gauge(
            "fapilog_plugin_health_status",
            "Plugin health status (1=healthy, 0=unhealthy)",
            ["plugin_name", "plugin_type"],
        )

        _metrics_initialized = True
        return True

    except ImportError:
        return False


def record_plugin_load(
    plugin_name: str,
    plugin_type: str,
    duration_seconds: float,
) -> None:
    """Record plugin load duration."""
    if _ensure_metrics() and _plugin_load_duration:
        _plugin_load_duration.labels(
            plugin_name=plugin_name,
            plugin_type=plugin_type,
        ).observe(duration_seconds)


def record_plugin_start(
    plugin_name: str,
    plugin_type: str,
    duration_seconds: float,
) -> None:
    """Record plugin start() duration."""
    if _ensure_metrics() and _plugin_start_duration:
        _plugin_start_duration.labels(
            plugin_name=plugin_name,
            plugin_type=plugin_type,
        ).observe(duration_seconds)


def record_plugin_operation(
    plugin_name: str,
    plugin_type: str,
    operation: str,
    duration_seconds: float,
) -> None:
    """Record plugin operation duration and count."""
    if not _ensure_metrics():
        return

    if _plugin_operation_duration:
        _plugin_operation_duration.labels(
            plugin_name=plugin_name,
            plugin_type=plugin_type,
            operation=operation,
        ).observe(duration_seconds)

    if _plugin_operation_count:
        _plugin_operation_count.labels(
            plugin_name=plugin_name,
            plugin_type=plugin_type,
            operation=operation,
        ).inc()


def record_plugin_error(
    plugin_name: str,
    plugin_type: str,
    operation: str,
    error_type: str,
) -> None:
    """Record plugin error."""
    if _ensure_metrics() and _plugin_error_count:
        _plugin_error_count.labels(
            plugin_name=plugin_name,
            plugin_type=plugin_type,
            operation=operation,
            error_type=error_type,
        ).inc()


def set_plugin_health(
    plugin_name: str,
    plugin_type: str,
    healthy: bool,
) -> None:
    """Set plugin health status gauge."""
    if _ensure_metrics() and _plugin_health_status:
        _plugin_health_status.labels(
            plugin_name=plugin_name,
            plugin_type=plugin_type,
        ).set(1 if healthy else 0)
```

### 2. Integration with Loader

```python
# Update plugins/loader.py

import time
from ..metrics.plugin_metrics import record_plugin_load

def load_plugin(
    group: str,
    name: str,
    config: dict[str, Any] | None = None,
    *,
    validation_mode: ValidationMode | None = None,
) -> Any:
    """Load a plugin by group and name from built-ins or entry points."""

    start_time = time.perf_counter()
    plugin_type = _group_to_type(group)

    try:
        # ... existing loading logic ...
        instance = _instantiate(cls, config, group=group, validation_mode=mode)

        # Record load time
        duration = time.perf_counter() - start_time
        plugin_name = getattr(instance, "name", name)
        record_plugin_load(plugin_name, plugin_type, duration)

        return instance

    except Exception:
        duration = time.perf_counter() - start_time
        record_plugin_load(name, plugin_type, duration)
        raise


def _group_to_type(group: str) -> str:
    """Convert entry point group to plugin type."""
    mapping = {
        "fapilog.sinks": "sink",
        "fapilog.enrichers": "enricher",
        "fapilog.redactors": "redactor",
        "fapilog.processors": "processor",
        "fapilog.filters": "filter",
    }
    return mapping.get(group, "unknown")
```

### 3. Integration with Plugin Startup

```python
# Update __init__.py

import time
from .metrics.plugin_metrics import record_plugin_start

async def _start_plugins(
    plugins: list[Any],
    plugin_type: str,
) -> list[Any]:
    """Start plugins, recording metrics."""
    started: list[Any] = []

    for plugin in plugins:
        plugin_name = getattr(plugin, "name", type(plugin).__name__)

        try:
            if hasattr(plugin, "start"):
                start_time = time.perf_counter()
                await plugin.start()
                duration = time.perf_counter() - start_time
                record_plugin_start(plugin_name, plugin_type, duration)

            started.append(plugin)

        except Exception as exc:
            # ... existing error handling ...
            pass

    return started
```

### 4. Plugin Registry API

```python
# src/fapilog/plugins/registry.py
"""Plugin registry for inspection and management."""

from __future__ import annotations

from dataclasses import dataclass, field
from datetime import datetime
from typing import Any


@dataclass
class LoadedPlugin:
    """Information about a loaded plugin."""

    name: str
    plugin_type: str
    instance: Any
    load_time_seconds: float
    start_time_seconds: float | None = None
    loaded_at: datetime = field(default_factory=datetime.now)
    healthy: bool = True
    error_count: int = 0


class PluginRegistry:
    """Registry of loaded plugins for inspection."""

    def __init__(self) -> None:
        self._plugins: dict[str, LoadedPlugin] = {}

    def register(self, plugin: LoadedPlugin) -> None:
        key = f"{plugin.plugin_type}:{plugin.name}"
        self._plugins[key] = plugin

    def get(self, name: str, plugin_type: str) -> LoadedPlugin | None:
        key = f"{plugin_type}:{name}"
        return self._plugins.get(key)

    def list_all(self) -> list[LoadedPlugin]:
        return list(self._plugins.values())

    def list_by_type(self, plugin_type: str) -> list[LoadedPlugin]:
        return [p for p in self._plugins.values() if p.plugin_type == plugin_type]

    def get_stats(self) -> dict[str, Any]:
        """Get aggregate statistics."""
        return {
            "total_plugins": len(self._plugins),
            "by_type": {
                ptype: len(self.list_by_type(ptype))
                for ptype in {"sink", "enricher", "redactor", "processor", "filter"}
            },
            "healthy": sum(1 for p in self._plugins.values() if p.healthy),
            "unhealthy": sum(1 for p in self._plugins.values() if not p.healthy),
            "total_errors": sum(p.error_count for p in self._plugins.values()),
        }


# Global registry instance
_registry = PluginRegistry()


def get_plugin_registry() -> PluginRegistry:
    """Get the global plugin registry."""
    return _registry
```

---

## Prometheus Metrics Summary

| Metric                                      | Type      | Labels                                          |
| ------------------------------------------- | --------- | ----------------------------------------------- |
| `fapilog_plugin_load_duration_seconds`      | Histogram | plugin_name, plugin_type                        |
| `fapilog_plugin_start_duration_seconds`     | Histogram | plugin_name, plugin_type                        |
| `fapilog_plugin_operation_duration_seconds` | Histogram | plugin_name, plugin_type, operation             |
| `fapilog_plugin_operations_total`           | Counter   | plugin_name, plugin_type, operation             |
| `fapilog_plugin_errors_total`               | Counter   | plugin_name, plugin_type, operation, error_type |
| `fapilog_plugin_health_status`              | Gauge     | plugin_name, plugin_type                        |

---

## Test Plan

### Unit Tests

1. **test_plugin_metrics.py**

   - Test metric recording functions
   - Test graceful handling when prometheus not installed
   - Test label values are sanitized

2. **test_plugin_registry.py**
   - Test registration and lookup
   - Test list_by_type filtering
   - Test stats aggregation

### Integration Tests

1. **test_metrics_integration.py**
   - Test metrics recorded during plugin loading
   - Test metrics recorded during plugin operations
   - Verify Prometheus exposition format

---

## Documentation Updates

1. Add `docs/core-concepts/plugin-metrics.md`
2. Update `docs/examples/prometheus-metrics.md` with plugin metrics
3. Add Grafana dashboard example for plugin monitoring

---

## Related Stories

- Story 5.0: Wire processors (complete metrics coverage)
- Story 5.8: OpenTelemetry integration — **Also review for enterprise scope**

---

## See Also

- `src/fapilog/metrics/metrics.py` — Existing MetricsCollector with plugin profiling
- `packages/fapilog-tamper/` — Example of enterprise add-on package pattern
