# Story 4.47: Redaction Defaults Alignment

**Status:** Complete
**Priority:** High
**Depends on:** None

---

## Context / Background

An external audit found that users cannot reliably answer "am I protected by default?" because docs, presets, and settings don't align on redaction behavior. This creates a dangerous situation where users may deploy believing secrets are masked when they aren't.

### Evidence of Mismatch

**1. Docs claim broad regex-based secret matching** (`docs/user-guide/reliability-defaults.md:24-25`):
```markdown
- Order: `field-mask` → `regex-mask` → `url-credentials`
- Patterns: regex-mask uses a broad secret matcher (password/pass/secret/api key/token/authorization/set-cookie/ssn/email).
```

**2. Production preset enables only field_mask + url_credentials** (`src/fapilog/core/presets.py:32`):
```python
"redactors": ["field_mask", "url_credentials"],  # NO regex_mask!
```
And field_mask only masks specific `metadata.*` paths, not a broad pattern:
```python
"fields_to_mask": [
    "metadata.password",
    "metadata.api_key",
    # ... only metadata.* fields
],
```

**3. Core settings defaults redactors to empty** (`src/fapilog/core/settings.py:688-690`):
```python
redactors: list[str] = Field(
    default_factory=list,  # Empty! Redaction effectively off
    description=("Redactor plugins to use (by name); empty to disable"),
)
```

The `redactors_order` field (`settings.py:669`) includes `regex-mask`, but that only defines *order* when redactors are active—it doesn't enable them.

### Impact

- Users reading docs believe regex-based secret masking is active by default
- Users using `preset="production"` get only field_mask + url_credentials (no regex)
- Users with no preset and no explicit config get NO redaction at all
- Secrets may leak to logs in production deployments

---

## Scope (In / Out)

### In Scope

- Decide on the canonical redaction behavior (Option A or B below)
- Align docs, presets, and settings to match the decision
- Create a "Redaction Guarantee" reference page documenting exactly what's redacted
- Add CI snapshot test to prevent future drift between defaults and docs

### Out of Scope

- Adding new redactor plugins
- Changing the redactor plugin architecture
- Backward-compatible shims for existing deployments (breaking change is acceptable with migration guide)

---

## API Design Decision

### Option A: Make Production Preset Include regex_mask (Recommended)

Add `regex_mask` to production preset with documented default patterns:

```python
"production": {
    "core": {
        "redactors": ["field_mask", "regex_mask", "url_credentials"],
    },
    "redactor_config": {
        "field_mask": {
            "fields_to_mask": [
                "password", "api_key", "token", "secret",
                "authorization", "api_secret", "private_key",
                "ssn", "credit_card",
            ],
        },
        "regex_mask": {
            "patterns": [
                r"(?i)(password|passwd|pwd)\s*[:=]\s*\S+",
                r"(?i)(api[_-]?key|apikey)\s*[:=]\s*\S+",
                r"(?i)(secret|token)\s*[:=]\s*\S+",
                r"(?i)bearer\s+[a-zA-Z0-9\-._~+/]+=*",
            ],
        },
        "url_credentials": {},
    },
}
```

**Pros:**
- Matches documentation claims
- More secure by default
- Users get protection without extra config

**Cons:**
- Slightly more processing overhead
- May mask false positives in log messages

### Option B: Rewrite Docs to Match Current Behavior

Update `reliability-defaults.md` to accurately describe what's actually enabled:

```markdown
## Redaction defaults

- **With no preset**: Redaction is **disabled** by default. Set `core.redactors` to enable.
- **With `preset="production"`**: Enables `field_mask` and `url_credentials` only.
  - `field_mask` masks these fields: password, api_key, token, secret, authorization, api_secret, private_key, ssn, credit_card
  - `url_credentials` strips userinfo from URLs
  - `regex_mask` is **not enabled** by default (add to `core.redactors` if needed)
```

**Pros:**
- No behavior change
- Accurate documentation

**Cons:**
- Less secure by default
- Users must opt-in to regex protection

---

## Acceptance Criteria

### AC1: Docs Match Actual Behavior

**Description:** `docs/user-guide/reliability-defaults.md` accurately describes what redaction is enabled by default and per preset.

**Validation:**
```python
from fapilog import Settings
from fapilog.core.presets import get_preset

# What docs claim should match what code does
settings = Settings()
assert settings.core.redactors == []  # Docs should say "empty by default"

prod = get_preset("production")
assert prod["core"]["redactors"] == ["field_mask", "url_credentials"]  # Or include regex_mask if Option A
```

### AC2: Redaction Guarantee Page Exists

**Description:** A dedicated page documents exactly what's redacted under each configuration.

**Validation:**
```markdown
# docs/user-guide/redaction-guarantee.md should include:

## What's Redacted by Preset

| Preset | field_mask | regex_mask | url_credentials | Effective Protection |
|--------|------------|------------|-----------------|---------------------|
| None (default) | ❌ | ❌ | ❌ | None |
| `dev` | ❌ | ❌ | ❌ | None |
| `production` | ✅ | ✅/❌ | ✅ | Standard fields + URLs |
| `fastapi` | ❌ | ❌ | ❌ | None |
| `minimal` | ❌ | ❌ | ❌ | None |

## Fields Masked by field_mask (production)
- password, api_key, token, secret, authorization, api_secret, private_key, ssn, credit_card
```

### AC3: CI Prevents Drift

**Description:** A snapshot test validates that Settings() defaults and preset behavior match documentation claims.

**Validation:**
```python
# tests/unit/test_redaction_defaults.py
def test_default_redactors_match_docs():
    """Ensure Settings() defaults match reliability-defaults.md claims."""
    settings = Settings()
    # If docs say "empty by default", this should pass
    assert settings.core.redactors == []
    # If docs say regex-mask is included, this should fail until fixed

def test_production_preset_redactors_match_docs():
    """Ensure production preset matches documentation claims."""
    prod = get_preset("production")
    # Should match what docs/user-guide/redaction-guarantee.md says
    assert "field_mask" in prod["core"]["redactors"]
    assert "url_credentials" in prod["core"]["redactors"]
    # Add or remove regex_mask assertion based on Option A/B decision
```

### AC4: No Conflicting Claims

**Description:** No documentation page claims redaction behavior that contradicts another page or actual code.

**Validation:**
- `reliability-defaults.md` consistent with `redaction-guarantee.md`
- `configuration.md` examples consistent with actual defaults
- README preset comparison table accurate

---

## Implementation Notes

### File Changes

```
docs/user-guide/reliability-defaults.md (MODIFIED - fix claims)
docs/user-guide/redaction-guarantee.md (NEW - comprehensive reference)
src/fapilog/core/presets.py (MODIFIED if Option A - add regex_mask)
tests/unit/test_redaction_defaults.py (NEW - snapshot test)
docs/user-guide/configuration.md (MODIFIED - update preset table if needed)
```

### Redaction Guarantee Page Structure

```markdown
# Redaction Guarantee

This page documents exactly what fapilog redacts under each configuration.

## Quick Reference

| Configuration | Secrets Protected? | URL Credentials? | Regex Patterns? |
|--------------|-------------------|------------------|-----------------|
| `Settings()` (no preset) | ❌ No | ❌ No | ❌ No |
| `preset="dev"` | ❌ No | ❌ No | ❌ No |
| `preset="production"` | ✅ Yes | ✅ Yes | ✅/❌ See below |
| `preset="fastapi"` | ❌ No | ❌ No | ❌ No |

## Production Preset Details

### field_mask Redactor
Masks these field names (case-insensitive, any nesting level):
- `password`, `api_key`, `token`, `secret`
- `authorization`, `api_secret`, `private_key`
- `ssn`, `credit_card`

### url_credentials Redactor
Strips userinfo from URLs:
- `https://user:pass@example.com` → `https://***:***@example.com`

### regex_mask Redactor
[Document patterns if Option A, or state "not enabled by default" if Option B]

## Enabling Additional Protection

```python
settings = Settings(
    core={"redactors": ["field_mask", "regex_mask", "url_credentials"]},
    redactor_config={
        "regex_mask": {
            "patterns": [r"(?i)secret[:=]\s*\S+"]
        }
    }
)
```
```

---

## Tasks

### Phase 1: Decision and Alignment

- [ ] Decide Option A (add regex_mask to production) or Option B (fix docs only)
- [ ] Update `docs/user-guide/reliability-defaults.md` to match decision
- [ ] If Option A: Update `src/fapilog/core/presets.py` to include regex_mask

### Phase 2: Documentation

- [ ] Create `docs/user-guide/redaction-guarantee.md`
- [ ] Update preset comparison table in `docs/user-guide/configuration.md`
- [ ] Review and fix any other redaction claims in docs

### Phase 3: CI Protection

- [ ] Add `tests/unit/test_redaction_defaults.py` snapshot test
- [ ] Ensure test fails if docs/code drift apart
- [ ] Update CHANGELOG

---

## Tests

### Unit Tests

- `tests/unit/test_redaction_defaults.py`
  - `test_default_settings_redactors_empty`
  - `test_production_preset_redactors_match_docs`
  - `test_dev_preset_has_no_redactors`
  - `test_fastapi_preset_has_no_redactors`
  - `test_field_mask_fields_match_docs`

### Documentation Tests

- Validate that claims in `reliability-defaults.md` match code behavior
- Validate `redaction-guarantee.md` table matches presets

---

## Definition of Done

### Code Complete

- [ ] Docs accurately describe redaction defaults
- [ ] Redaction guarantee page exists
- [ ] CI test prevents future drift
- [ ] Presets match documentation (or vice versa)

### Quality Assurance

- [ ] Snapshot tests passing
- [ ] `ruff check` passes
- [ ] `mypy` passes
- [ ] Documentation builds without warnings

### Documentation

- [ ] `reliability-defaults.md` accurate
- [ ] `redaction-guarantee.md` complete
- [ ] CHANGELOG updated

---

## Risks / Rollback

### Risks

1. **Risk (Option A):** Adding regex_mask to production may cause false positives
   - **Mitigation:** Use conservative patterns; document how to disable

2. **Risk (Option B):** Users may be surprised their secrets aren't masked
   - **Mitigation:** Clear documentation and migration guide

3. **Risk:** Breaking change for users relying on current behavior
   - **Mitigation:** Document in CHANGELOG; provide opt-out

### Rollback Plan

If issues occur:
1. Option A: Users can remove `regex_mask` from their redactors list
2. Option B: No code change to rollback, only docs

---

## Related Stories

- **Related:** Story 4.46 - Fallback sink PII protection (redaction theme)
- **Related:** Story 10.15 - Documentation correctness (docs accuracy theme)

---

## Code Review

**Date:** 2026-01-18
**Reviewer:** Claude
**Verdict:** OK to PR

### Summary

Added `regex_mask` to production preset (Option A), created comprehensive redaction documentation, and added CI snapshot tests to prevent docs/code drift.

### AC Verification

| Criterion | Evidence |
|-----------|----------|
| AC1: Docs Match Actual Behavior | `reliability-defaults.md:24-29` - accurately describes no preset = empty, production = 3 redactors |
| AC2: Redaction Guarantee Page Exists | `redaction-guarantee.md:1-164` - preset table, field lists, regex patterns, examples |
| AC3: CI Prevents Drift | `test_redaction_defaults.py:1-173` - 14 tests covering all presets and configurations |
| AC4: No Conflicting Claims | Both docs consistent; cross-reference link at `reliability-defaults.md:33` |

### Quality Gates

- [x] ruff check passed
- [x] ruff format passed
- [x] mypy passed
- [x] No weak assertions
- [x] No dead code
- [x] No Pydantic v1 syntax

### Issues Addressed

- [P1] CHANGELOG not updated - Fixed in `CHANGELOG.md:7-8`

---

## Change Log

| Date | Change | Author |
|------|--------|--------|
| 2026-01-18 | Initial draft from audit findings | Claude |
| 2026-01-18 | Implementation complete, code review passed | Claude |
