# Story 5.19: Standardize Plugin Configuration Parsing Pattern

## Status: Complete

## Priority: Medium

## Estimated Effort: Medium (2-3 days)

## Dependencies: None

## Epic: Plugin System Consistency

---

## Context

Plugin configuration parsing is inconsistent across plugin types. Different plugins use different patterns for accepting and parsing configuration, making it harder for users to configure plugins and for developers to author new ones.

This inconsistency creates:

1. Confusion for users about how to pass configuration
2. Duplicated parsing logic across plugins
3. Inconsistent error handling for invalid configuration
4. No type coercion (e.g., `"100"` → `100` fails)
5. Silent acceptance of typos in config keys
6. Documentation complexity

---

## Problem Statement

### Current Patterns in Use

#### Pattern A: Filters (Complex Dict Unwrapping with Dataclasses)

```python
# Used by all filters - manual parsing, no validation
def __init__(
    self, *, config: SamplingFilterConfig | dict | None = None, **kwargs: Any
) -> None:
    if isinstance(config, dict):
        raw = config.get("config", config)  # Double unwrap
        cfg = SamplingFilterConfig(**raw)  # Fails on type mismatch
    elif config is None:
        raw_kwargs = kwargs.get("config", kwargs)  # kwargs unwrap too
        cfg = SamplingFilterConfig(**raw_kwargs) if raw_kwargs else SamplingFilterConfig()
    else:
        cfg = config
```

#### Pattern B: Redactors (Simple Optional Config)

```python
# Used by redactors - no dict support
def __init__(self, *, config: FieldMaskConfig | None = None) -> None:
    cfg = config or FieldMaskConfig(fields_to_mask=[])
```

#### Pattern C: Sinks (Config Object Required)

```python
# Used by HttpSink, WebhookSink - forces user to import config class
def __init__(
    self,
    config: HttpSinkConfig,
    *,
    metrics: Any | None = None,
    pool: HttpClientPool | None = None,
) -> None:
```

#### Pattern D: Cloud Sinks (Config or Kwargs)

```python
# Used by CloudWatchSink, LokiSink - reasonable but inconsistent
def __init__(
    self, config: CloudWatchSinkConfig | None = None, **kwargs: Any
) -> None:
    if config is None:
        if kwargs:
            config = CloudWatchSinkConfig(**kwargs)
        else:
            config = CloudWatchSinkConfig()
```

### Issues

| Issue                             | Impact                                       |
| --------------------------------- | -------------------------------------------- |
| Pattern A is overly complex       | 15+ lines of boilerplate per plugin          |
| Pattern B doesn't accept dicts    | Users must import config classes             |
| Pattern C requires config objects | Poor DX for simple use cases                 |
| Dataclasses have no validation    | `{"sample_rate": "oops"}` fails cryptically  |
| No type coercion                  | `{"port": "8080"}` fails instead of coercing |
| Extra fields silently accepted    | Typos like `sampl_rate` go unnoticed         |
| Duplicated `_parse_config`        | Same code in 6 filter files                  |

---

## Acceptance Criteria

### AC1: Migrate Config Classes to Pydantic v2 BaseModel

- [ ] Convert all plugin config dataclasses to Pydantic v2 `BaseModel`
- [ ] Add `ConfigDict(frozen=True, extra="forbid")` for safety
- [ ] Ensure all fields have appropriate types and defaults

### AC2: Create Shared Parsing Utility

- [ ] Create `parse_plugin_config()` helper in `plugins/utils.py`
- [ ] Use Pydantic's `model_validate()` for dict → object conversion
- [ ] Handle nested `"config"` key unwrapping for loader compatibility
- [ ] Provide clear error messages leveraging Pydantic's validation

### AC3: Update Filter Plugins

- [ ] Refactor all 6 filter plugins to use shared utility
- [ ] Remove duplicate `_parse_config` methods
- [ ] Ensure backwards compatibility with existing config formats
- [ ] Verify type coercion works (e.g., `"0.5"` → `0.5` for sample_rate)

### AC4: Update Redactor Plugins

- [ ] Update redactors to accept dict configuration
- [ ] Convert config classes to Pydantic BaseModel
- [ ] Maintain simple pattern for config-only usage

### AC5: Update Sink Plugins

- [ ] Update cloud sinks to use shared utility
- [ ] Convert config classes to Pydantic BaseModel where applicable

### AC6: Documentation

- [ ] Document standard Pydantic-based pattern in authoring guide
- [ ] Add examples for each configuration style
- [ ] Document type coercion and validation behavior
- [ ] Update plugin examples to show dict and object configs

---

## Technical Design

### 1. Why Pydantic v2 Over Dataclasses

| Feature        | Dataclass + Custom Parser | Pydantic v2                       |
| -------------- | ------------------------- | --------------------------------- |
| Dict → Object  | Manual parsing code       | Built-in `model_validate()`       |
| Type coercion  | ❌ `"100"` → `int` fails  | ✅ Auto-coerces `"100"` → `100`   |
| Validation     | ❌ None                   | ✅ Automatic with clear errors    |
| Extra fields   | ❌ Silently accepted      | ✅ `extra="forbid"` catches typos |
| Immutability   | ❌ Manual                 | ✅ `frozen=True`                  |
| Nested configs | ❌ Complex                | ✅ Automatic recursive validation |
| JSON Schema    | ❌ None                   | ✅ `model_json_schema()`          |
| IDE support    | ⚠️ Basic                  | ✅ Full autocomplete              |
| Error messages | ❌ Cryptic `TypeError`    | ✅ Field-level validation errors  |

### 2. Standard Config Class Pattern

All plugin configs should use Pydantic v2 `BaseModel` with safety defaults:

```python
from pydantic import BaseModel, ConfigDict, Field

class SamplingFilterConfig(BaseModel):
    """Configuration for sampling filter."""

    model_config = ConfigDict(
        frozen=True,       # Immutable after creation
        extra="forbid",    # Reject unknown fields (catches typos)
        validate_default=True,  # Validate default values too
    )

    sample_rate: float = Field(default=1.0, ge=0.0, le=1.0)
    seed: int | None = None
```

### 3. Shared Parsing Utility

```python
# src/fapilog/plugins/utils.py

from typing import Any, TypeVar

from pydantic import BaseModel, ValidationError

T = TypeVar("T", bound=BaseModel)


def parse_plugin_config(
    config_cls: type[T],
    config: T | dict[str, Any] | None = None,
    **kwargs: Any,
) -> T:
    """Parse plugin configuration using Pydantic v2 model_validate.

    Accepts:
    - config_cls instance: returned as-is (validated on construction)
    - dict: validated and converted via model_validate()
    - kwargs: converted to dict and validated
    - None with no kwargs: default config_cls()

    Handles nested "config" key for loader compatibility:
    - {"config": {"field": "value"}} -> config_cls(field="value")
    - {"field": "value"} -> config_cls(field="value")

    Args:
        config_cls: The Pydantic BaseModel config class
        config: Config object, dict, or None
        **kwargs: Keyword arguments to pass to config_cls

    Returns:
        Validated instance of config_cls

    Raises:
        ValidationError: If config fails Pydantic validation
        TypeError: If config is an unsupported type

    Example:
        >>> cfg = parse_plugin_config(SamplingFilterConfig, {"sample_rate": "0.5"})
        >>> cfg.sample_rate
        0.5  # Coerced from string

        >>> parse_plugin_config(SamplingFilterConfig, {"sampl_rate": 0.5})
        ValidationError: Extra inputs are not permitted
    """
    # Case 1: Already the right type (already validated)
    if isinstance(config, config_cls):
        return config

    # Case 2: Dict - use Pydantic's model_validate
    if isinstance(config, dict):
        # Unwrap nested "config" key for loader compatibility
        if "config" in config and isinstance(config["config"], (dict, config_cls)):
            config = config["config"]
        if isinstance(config, config_cls):
            return config
        return config_cls.model_validate(config)

    # Case 3: None - check kwargs or use defaults
    if config is None:
        # Check for nested "config" in kwargs (loader compatibility)
        if "config" in kwargs:
            nested = kwargs.pop("config")
            if isinstance(nested, config_cls):
                return nested
            if isinstance(nested, dict):
                return config_cls.model_validate(nested)
        # Use remaining kwargs or defaults
        if kwargs:
            return config_cls.model_validate(kwargs)
        return config_cls()

    raise TypeError(
        f"Cannot parse config: expected {config_cls.__name__}, dict, or None; "
        f"got {type(config).__name__}"
    )
```

### 4. Refactor Filter Config Classes

Before (dataclass):

```python
from dataclasses import dataclass

@dataclass
class SamplingFilterConfig:
    sample_rate: float = 1.0
    seed: int | None = None
```

After (Pydantic v2):

```python
from pydantic import BaseModel, ConfigDict, Field

class SamplingFilterConfig(BaseModel):
    model_config = ConfigDict(frozen=True, extra="forbid")

    sample_rate: float = Field(default=1.0, ge=0.0, le=1.0)
    seed: int | None = None
```

### 5. Refactor Filter Plugin Constructors

Before:

```python
class SamplingFilter:
    def __init__(
        self, *, config: SamplingFilterConfig | dict | None = None, **kwargs: Any
    ) -> None:
        if isinstance(config, dict):
            raw = config.get("config", config)
            cfg = SamplingFilterConfig(**raw)
        elif config is None:
            raw_kwargs = kwargs.get("config", kwargs)
            cfg = (
                SamplingFilterConfig(**raw_kwargs)
                if raw_kwargs
                else SamplingFilterConfig()
            )
        else:
            cfg = config
        self._rate = max(0.0, min(1.0, float(cfg.sample_rate)))
```

After:

```python
from ..utils import parse_plugin_config

class SamplingFilter:
    def __init__(
        self, *, config: SamplingFilterConfig | dict | None = None, **kwargs: Any
    ) -> None:
        cfg = parse_plugin_config(SamplingFilterConfig, config, **kwargs)
        # No need for manual clamping - Pydantic Field(ge=0.0, le=1.0) validates
        self._rate = cfg.sample_rate
```

### 6. Update Redactor Plugins

Before:

```python
@dataclass
class FieldMaskConfig:
    fields_to_mask: list[str] = field(default_factory=list)

class FieldMaskRedactor:
    def __init__(self, *, config: FieldMaskConfig | None = None) -> None:
        cfg = config or FieldMaskConfig(fields_to_mask=[])
```

After:

```python
from pydantic import BaseModel, ConfigDict, Field

class FieldMaskConfig(BaseModel):
    model_config = ConfigDict(frozen=True, extra="forbid")

    fields_to_mask: list[str] = Field(default_factory=list)

class FieldMaskRedactor:
    def __init__(
        self, *, config: FieldMaskConfig | dict | None = None, **kwargs: Any
    ) -> None:
        cfg = parse_plugin_config(FieldMaskConfig, config, **kwargs)
        self._fields = cfg.fields_to_mask
```

### 7. Document Standard Pattern

````markdown
## Plugin Configuration Pattern

All plugins use Pydantic v2 for configuration with automatic validation and coercion.

### Defining a Config Class

```python
from pydantic import BaseModel, ConfigDict, Field

class MyPluginConfig(BaseModel):
    """Configuration for MyPlugin."""

    model_config = ConfigDict(
        frozen=True,       # Immutable
        extra="forbid",    # Reject unknown fields
    )

    threshold: int = Field(default=100, ge=1, description="Min threshold")
    enabled: bool = True
```
````

### Using parse_plugin_config

```python
from fapilog.plugins.utils import parse_plugin_config

class MyPlugin:
    name = "my_plugin"

    def __init__(
        self,
        *,
        config: MyPluginConfig | dict | None = None,
        **kwargs: Any,
    ) -> None:
        cfg = parse_plugin_config(MyPluginConfig, config, **kwargs)
        self._threshold = cfg.threshold
        self._enabled = cfg.enabled
```

### All Input Formats Supported

```python
# Option 1: Config object
plugin = MyPlugin(config=MyPluginConfig(threshold=50))

# Option 2: Dict
plugin = MyPlugin(config={"threshold": 50})

# Option 3: Kwargs
plugin = MyPlugin(threshold=50)

# Option 4: Defaults
plugin = MyPlugin()

# Option 5: Loader format (nested "config" key)
plugin = MyPlugin(config={"config": {"threshold": 50}})

# Option 6: String coercion (Pydantic handles this!)
plugin = MyPlugin(config={"threshold": "50"})  # "50" -> 50
```

### Validation Errors

Pydantic provides clear, actionable error messages:

```python
# Typo in field name
MyPlugin(config={"threshhold": 50})
# ValidationError: Extra inputs are not permitted ['threshhold']

# Invalid value
MyPlugin(config={"threshold": -1})
# ValidationError: threshold - Input should be >= 1

# Wrong type (non-coercible)
MyPlugin(config={"threshold": "not-a-number"})
# ValidationError: threshold - Input should be a valid integer
```

````

---

## Test Plan

### Unit Tests

**test_parse_plugin_config.py**

```python
import pytest
from pydantic import BaseModel, ConfigDict, Field, ValidationError
from fapilog.plugins.utils import parse_plugin_config


class SampleConfig(BaseModel):
    model_config = ConfigDict(frozen=True, extra="forbid")

    value: int = 10
    name: str = "default"


class StrictConfig(BaseModel):
    """Config with validation constraints."""
    model_config = ConfigDict(frozen=True, extra="forbid")

    rate: float = Field(default=1.0, ge=0.0, le=1.0)


class TestParsePluginConfigBasic:
    """Basic parsing tests."""

    def test_parse_config_object(self):
        """Config object should be returned as-is."""
        cfg = SampleConfig(value=20)
        result = parse_plugin_config(SampleConfig, cfg)
        assert result is cfg

    def test_parse_config_dict(self):
        """Dict should be validated and converted."""
        result = parse_plugin_config(SampleConfig, {"value": 30, "name": "test"})
        assert result.value == 30
        assert result.name == "test"

    def test_parse_config_nested_dict(self):
        """Nested 'config' key should be unwrapped."""
        result = parse_plugin_config(SampleConfig, {"config": {"value": 40}})
        assert result.value == 40

    def test_parse_config_kwargs(self):
        """Kwargs should be used when config is None."""
        result = parse_plugin_config(SampleConfig, None, value=50)
        assert result.value == 50

    def test_parse_config_defaults(self):
        """Default config should be created when nothing provided."""
        result = parse_plugin_config(SampleConfig, None)
        assert result.value == 10
        assert result.name == "default"

    def test_parse_config_nested_kwargs(self):
        """Nested 'config' in kwargs should be unwrapped."""
        result = parse_plugin_config(SampleConfig, None, config={"value": 60})
        assert result.value == 60

    def test_parse_config_invalid_type(self):
        """Invalid config type should raise TypeError."""
        with pytest.raises(TypeError, match="expected SampleConfig"):
            parse_plugin_config(SampleConfig, "invalid")


class TestPydanticFeatures:
    """Tests for Pydantic-specific features."""

    def test_type_coercion_string_to_int(self):
        """Pydantic should coerce compatible types."""
        result = parse_plugin_config(SampleConfig, {"value": "42"})
        assert result.value == 42
        assert isinstance(result.value, int)

    def test_type_coercion_string_to_float(self):
        """Pydantic should coerce string to float."""
        result = parse_plugin_config(StrictConfig, {"rate": "0.5"})
        assert result.rate == 0.5

    def test_extra_fields_rejected(self):
        """Unknown fields should raise ValidationError."""
        with pytest.raises(ValidationError, match="Extra inputs are not permitted"):
            parse_plugin_config(SampleConfig, {"value": 10, "unknown_field": "oops"})

    def test_validation_constraints(self):
        """Field constraints should be enforced."""
        with pytest.raises(ValidationError, match="greater than or equal to 0"):
            parse_plugin_config(StrictConfig, {"rate": -0.5})

        with pytest.raises(ValidationError, match="less than or equal to 1"):
            parse_plugin_config(StrictConfig, {"rate": 1.5})

    def test_invalid_type_clear_error(self):
        """Non-coercible types should give clear errors."""
        with pytest.raises(ValidationError, match="Input should be a valid integer"):
            parse_plugin_config(SampleConfig, {"value": "not-a-number"})

    def test_immutable_config(self):
        """Frozen config should reject attribute assignment."""
        cfg = parse_plugin_config(SampleConfig, {"value": 10})
        with pytest.raises(ValidationError):
            cfg.value = 20  # type: ignore


class TestLoaderCompatibility:
    """Tests for plugin loader compatibility."""

    def test_nested_config_in_dict(self):
        """Loader-style nested config should work."""
        result = parse_plugin_config(
            SampleConfig,
            {"config": {"value": 100, "name": "loader"}}
        )
        assert result.value == 100
        assert result.name == "loader"

    def test_nested_config_in_kwargs(self):
        """Nested config kwarg should work."""
        result = parse_plugin_config(
            SampleConfig,
            None,
            config={"value": 200}
        )
        assert result.value == 200
````

### Integration Tests

1. Test all filter plugins accept dict configuration with coercion
2. Test all redactor plugins accept dict configuration
3. Test extra fields are rejected across all plugins
4. Test loader passes configuration correctly with nested "config" key
5. Test validation errors are user-friendly

---

## Migration Guide

### Step 1: Convert Config Class to Pydantic

Before:

```python
from dataclasses import dataclass, field

@dataclass
class MyPluginConfig:
    threshold: int = 100
    fields: list[str] = field(default_factory=list)
```

After:

```python
from pydantic import BaseModel, ConfigDict, Field

class MyPluginConfig(BaseModel):
    model_config = ConfigDict(frozen=True, extra="forbid")

    threshold: int = Field(default=100, ge=1)
    fields: list[str] = Field(default_factory=list)
```

### Step 2: Update Constructor

Before:

```python
def __init__(self, *, config: MyConfig | dict | None = None, **kwargs: Any):
    if isinstance(config, dict):
        raw = config.get("config", config)
        cfg = MyConfig(**raw)
    elif config is None:
        raw_kwargs = kwargs.get("config", kwargs)
        cfg = MyConfig(**raw_kwargs) if raw_kwargs else MyConfig()
    else:
        cfg = config
```

After:

```python
from fapilog.plugins.utils import parse_plugin_config

def __init__(self, *, config: MyConfig | dict | None = None, **kwargs: Any):
    cfg = parse_plugin_config(MyConfig, config, **kwargs)
```

### For Users

No changes required. All existing configuration methods continue to work, plus:

- String values are now auto-coerced to correct types
- Typos in config keys now raise clear errors instead of being silently ignored
- Invalid values fail fast with descriptive messages

---

## Backwards Compatibility

| Scenario               | Before              | After                                        |
| ---------------------- | ------------------- | -------------------------------------------- |
| Config object          | ✅ Works            | ✅ Works                                     |
| Dict config            | ✅ Works            | ✅ Works + validation                        |
| Kwargs                 | ✅ Works            | ✅ Works + validation                        |
| Nested "config" key    | ✅ Works            | ✅ Works                                     |
| String numbers `"100"` | ❌ Fails            | ✅ Coerced to `100`                          |
| Extra fields           | ⚠️ Silently ignored | ❌ ValidationError (breaking but beneficial) |

**Note:** The `extra="forbid"` change is technically breaking but catches real bugs. Users with typos in their config will now get helpful errors instead of silent failures.

---

## Files to Modify

### New Files

1. `src/fapilog/plugins/utils.py` - Add `parse_plugin_config` utility

### Config Classes (dataclass → Pydantic)

2. `src/fapilog/plugins/filters/sampling.py` - `SamplingFilterConfig`
3. `src/fapilog/plugins/filters/rate_limit.py` - `RateLimitFilterConfig`
4. `src/fapilog/plugins/filters/adaptive_sampling.py` - `AdaptiveSamplingConfig`
5. `src/fapilog/plugins/filters/trace_sampling.py` - `TraceSamplingConfig`
6. `src/fapilog/plugins/filters/first_occurrence.py` - `FirstOccurrenceConfig`
7. `src/fapilog/plugins/filters/level.py` - `LevelFilterConfig`
8. `src/fapilog/plugins/redactors/field_mask.py` - `FieldMaskConfig`
9. `src/fapilog/plugins/redactors/regex_mask.py` - `RegexMaskConfig`
10. `src/fapilog/plugins/redactors/url_credentials.py` - `UrlCredentialsConfig`

### Constructor Updates (use parse_plugin_config)

- Same files as above (config class + constructor in same file)

### Documentation

11. `docs/plugins/authoring.md` - Document Pydantic config pattern
12. `docs/plugins/testing.md` - Update examples

### Tests

13. `tests/unit/test_parse_plugin_config.py` - New utility tests
14. Update existing plugin tests to verify validation behavior

---

## Estimated Effort Breakdown

| Task                                         | Effort                   |
| -------------------------------------------- | ------------------------ |
| Create `parse_plugin_config` utility + tests | 2 hours                  |
| Convert 6 filter configs to Pydantic         | 3 hours                  |
| Convert 3 redactor configs to Pydantic       | 1.5 hours                |
| Update constructors to use utility           | 1 hour                   |
| Update/add integration tests                 | 2 hours                  |
| Documentation updates                        | 1.5 hours                |
| **Total**                                    | **~11 hours (1.5 days)** |

---

## Related Stories

- Story 5.1: Fix PLUGIN_METADATA name inconsistencies
- Story 4.19: Simple plugin configuration API
- Story 5.4: Plugin testing guide
