# Story 4.48: Fallback Redaction List Recursion

**Status:** Complete
**Priority:** High
**Depends on:** Story 4.46 (Fallback Sink PII Protection) - Complete

---

## Context / Background

The GPT-5.2 audit identified a security gap in the fallback minimal redaction implemented in Story 4.46:

> "fallback 'minimal redaction' explicitly does not redact lists (`src/fapilog/plugins/sinks/fallback.py`) → secrets in arrays can leak to stderr during sink failures."

**Current implementation in `src/fapilog/plugins/sinks/fallback.py:14-34`:**

```python
def minimal_redact(payload: dict[str, Any]) -> dict[str, Any]:
    """Apply minimal redaction for fallback safety.

    Masks values of keys that match FALLBACK_SENSITIVE_FIELDS (case-insensitive).
    Recursively processes nested dictionaries but not lists.  # <-- THE GAP
    ...
    """
    result: dict[str, Any] = {}
    for key, value in payload.items():
        if key.lower() in FALLBACK_SENSITIVE_FIELDS:
            result[key] = "***"
        elif isinstance(value, dict):
            result[key] = minimal_redact(value)
        else:
            result[key] = value  # Lists pass through unredacted!
    return result
```

**Problem:**

A payload like:
```python
{"users": [{"password": "secret123", "name": "alice"}, {"token": "abc"}]}
```

Would have `password` and `token` **leak to stderr** because they're inside a list.

**Why this matters:**

- Secrets in arrays are common (user lists, batch API responses, audit logs)
- The fallback path is specifically for failure scenarios
- Leaking secrets during failures is worse than losing the log entry

---

## Scope (In / Out)

### In Scope

- Extend `minimal_redact` to recurse into lists
- Handle arbitrarily nested list/dict combinations
- Add protection against infinite recursion (depth limit)
- Add tests for list-based redaction scenarios

### Out of Scope

- Changing the sensitive field list (covered by 4.46)
- Performance optimization (fallback is rare path)
- Handling non-JSON-serializable types in lists

---

## Acceptance Criteria

### AC1: List Contents Are Recursively Redacted

**Description:** When a list contains dicts with sensitive fields, those fields are masked.

**Validation:**
```python
payload = {"users": [{"password": "secret", "name": "alice"}]}
result = minimal_redact(payload)
assert result == {"users": [{"password": "***", "name": "alice"}]}
```

### AC2: Nested Lists Are Handled

**Description:** Lists within lists containing dicts are also redacted.

**Validation:**
```python
payload = {"matrix": [[{"token": "abc"}], [{"api_key": "xyz"}]]}
result = minimal_redact(payload)
assert result == {"matrix": [[{"token": "***"}], [{"api_key": "***"}]]}
```

### AC3: Mixed Nesting Works Correctly

**Description:** Arbitrary combinations of dicts and lists are handled.

**Validation:**
```python
payload = {
    "data": {
        "items": [
            {"credentials": {"password": "secret", "username": "admin"}},
            {"api_key": "sk-xxx", "public": True}
        ]
    }
}
result = minimal_redact(payload)
assert result["data"]["items"][0]["credentials"]["password"] == "***"
assert result["data"]["items"][1]["api_key"] == "***"
assert result["data"]["items"][1]["public"] is True
```

### AC4: Non-Dict List Items Preserved

**Description:** Lists containing primitives (strings, numbers) are preserved unchanged.

**Validation:**
```python
payload = {"tags": ["prod", "critical"], "counts": [1, 2, 3]}
result = minimal_redact(payload)
assert result == {"tags": ["prod", "critical"], "counts": [1, 2, 3]}
```

### AC5: Depth Limit Prevents Stack Overflow

**Description:** Extremely deep nesting doesn't cause stack overflow.

**Validation:**
```python
# Create deeply nested structure (100+ levels)
deep = {"password": "secret"}
for _ in range(150):
    deep = {"nested": [deep]}

# Should not raise RecursionError, should redact up to limit
result = minimal_redact(deep)
# Redaction stops at depth limit, preserving structure
```

### AC6: Contract Test - Fallback Uses Updated Function

**Description:** The fallback handler uses the updated `minimal_redact` function.

**Validation:**
```python
# Integration test: sink failure with list-containing payload
async def test_fallback_redacts_lists_in_payload(capsys):
    payload = {"users": [{"password": "secret"}]}
    await handle_sink_write_failure(
        payload,
        sink=mock_sink,
        error=Exception("test"),
        serialized=False,
        redact_mode="minimal",
    )
    captured = capsys.readouterr()
    assert '"password":"***"' in captured.err
    assert "secret" not in captured.err
```

---

## Implementation Notes

### File Changes

```
src/fapilog/plugins/sinks/fallback.py (MODIFIED - extend minimal_redact)
tests/unit/test_sink_fallback.py (MODIFIED - add list redaction tests)
tests/integration/test_sink_fallback_signaling.py (MODIFIED - add list scenario)
```

### Key Code Change

```python
# src/fapilog/plugins/sinks/fallback.py

_MAX_REDACT_DEPTH = 32  # Prevent stack overflow on pathological input


def minimal_redact(
    payload: dict[str, Any],
    *,
    _depth: int = 0,
) -> dict[str, Any]:
    """Apply minimal redaction for fallback safety.

    Masks values of keys that match FALLBACK_SENSITIVE_FIELDS (case-insensitive).
    Recursively processes nested dictionaries AND lists.

    Args:
        payload: The dictionary to redact.
        _depth: Internal recursion depth counter (do not set manually).

    Returns:
        A new dictionary with sensitive fields masked as "***".
    """
    if _depth >= _MAX_REDACT_DEPTH:
        return payload  # Stop recursion at depth limit

    result: dict[str, Any] = {}
    for key, value in payload.items():
        if key.lower() in FALLBACK_SENSITIVE_FIELDS:
            result[key] = "***"
        elif isinstance(value, dict):
            result[key] = minimal_redact(value, _depth=_depth + 1)
        elif isinstance(value, list):
            result[key] = _redact_list(value, _depth=_depth + 1)
        else:
            result[key] = value
    return result


def _redact_list(items: list[Any], *, _depth: int) -> list[Any]:
    """Recursively redact sensitive fields within list items."""
    if _depth >= _MAX_REDACT_DEPTH:
        return items  # Stop recursion at depth limit

    result: list[Any] = []
    for item in items:
        if isinstance(item, dict):
            result.append(minimal_redact(item, _depth=_depth + 1))
        elif isinstance(item, list):
            result.append(_redact_list(item, _depth=_depth + 1))
        else:
            result.append(item)
    return result
```

---

## Tasks

### Phase 1: Core Implementation

- [ ] Add `_MAX_REDACT_DEPTH` constant
- [ ] Add `_depth` parameter to `minimal_redact`
- [ ] Implement `_redact_list` helper function
- [ ] Update `minimal_redact` to handle list values

### Phase 2: Testing

- [ ] Add unit tests for list redaction scenarios
- [ ] Add test for nested list/dict combinations
- [ ] Add test for depth limit protection
- [ ] Add integration test for fallback with list payloads

### Phase 3: Documentation

- [ ] Update docstring to reflect list handling
- [ ] Update CHANGELOG

---

## Tests

### Unit Tests

- `tests/unit/test_sink_fallback.py`
  - `test_minimal_redact_handles_list_of_dicts`
  - `test_minimal_redact_handles_nested_lists`
  - `test_minimal_redact_handles_mixed_nesting`
  - `test_minimal_redact_preserves_primitive_lists`
  - `test_minimal_redact_depth_limit_prevents_overflow`

### Integration Tests

- `tests/integration/test_sink_fallback_signaling.py`
  - `test_fallback_redacts_sensitive_fields_in_lists`

---

## Definition of Done

### Code Complete

- [ ] `minimal_redact` recurses into lists
- [ ] Depth limit protects against stack overflow
- [ ] All acceptance criteria implemented

### Quality Assurance

- [ ] Unit tests written and passing
- [ ] Integration tests passing
- [ ] `ruff check` passes
- [ ] `mypy` passes
- [ ] No regression in existing tests

### Documentation

- [ ] Function docstring updated
- [ ] CHANGELOG updated

---

## Risks / Rollback

### Risks

1. **Risk:** Performance impact from list recursion
   - **Mitigation:** Fallback is rare path; depth limit bounds worst case

2. **Risk:** Breaking existing behavior for edge cases
   - **Mitigation:** Only adds functionality; existing dict-only redaction unchanged

3. **Risk:** Memory usage on large list payloads
   - **Mitigation:** Depth limit prevents unbounded recursion; fallback payloads typically small

### Rollback Plan

If issues occur:
1. Revert the list handling changes
2. Original dict-only redaction still functional

---

## Related Stories

- **Depends on:** Story 4.46 - Fallback Sink PII Protection (provides foundation)
- **Related:** Story 4.45 - Webhook secure defaults (security theme)
- **Related:** Story 3.5 - Plugin allowlist secure default (security defaults theme)

---

## Code Review

**Date:** 2026-01-19
**Reviewer:** Claude
**Verdict:** OK to PR

### Summary

Reviewed implementation of list recursion in fallback redaction. All 6 acceptance criteria verified with evidence from code and tests.

### AC Verification

| Criterion | Evidence |
|-----------|----------|
| AC1: List contents recursively redacted | `fallback.py:52-53` - list branch calls `_redact_list`; `test_sink_fallback.py:409-412` |
| AC2: Nested lists handled | `fallback.py:22-23` - `_redact_list` recurses; `test_sink_fallback.py:414-418` |
| AC3: Mixed nesting works | `test_sink_fallback.py:420-434` verifies deep nested dict/list combinations |
| AC4: Primitive lists preserved | `fallback.py:24-25` - else branch; `test_sink_fallback.py:436-440` |
| AC5: Depth limit prevents overflow | `fallback.py:14,17,45` - `_MAX_REDACT_DEPTH=32`; `test_sink_fallback.py:442-451` |
| AC6: Contract test | `test_sink_fallback_signaling.py:221-239` - end-to-end via `handle_sink_write_failure` |

### Quality Gates

- [x] ruff check passed
- [x] ruff format passed
- [x] mypy passed
- [x] diff-cover >= 90% (93.42%)
- [x] No weak assertions
- [x] No dead code

---

## Change Log

| Date | Change | Author |
|------|--------|--------|
| 2026-01-19 | Initial draft from GPT-5.2 audit follow-up | Claude |
| 2026-01-19 | Implementation complete, code review passed | Claude |
