# Story 7.3: Add Redaction Integration Tests for All Sink Types

**Status:** Complete
**Priority:** P0
**Depends on:** None
**Effort:** 2 days

---

## Problem Statement

The Test Value Audit identified a critical gap: **redaction is tested with mocks but not verified at real sink boundaries**.

Current redaction tests verify:
- Field masking works in isolation (unit tests)
- Nested paths are handled
- Array wildcards work
- Redactor ordering and error handling (`tests/integration/test_redactors_stage.py`)

But they do NOT verify:
- Masked data appears in PostgreSQL rows
- Masked data appears in HTTP request bodies
- Masked data appears in CloudWatch payloads
- Masked data appears in file sink output
- Redaction happens *before* serialization (not after)

This is a **security-critical gap**. If redaction occurs at the wrong point in the pipeline, sensitive data could reach sinks unmasked.

**Note:** Existing integration tests in `test_redactors_stage.py` verify redactor pipeline ordering using stub redactors and captured sinks. This story adds tests that verify **real redactors** produce **masked output at real sink boundaries**.

---

## Goals

1. **Verify redaction at every sink boundary** — Not just in-memory
2. **Verify ordering** — Redaction must happen before serialization and sink write
3. **Verify no bypass paths** — All log methods (info, error, exception) apply redaction
4. **Verify with real sinks** — Not mocks

---

## Acceptance Criteria

- [x] Integration test for redaction → stdout JSON sink
- [x] Integration test for redaction → rotating file sink
- [x] Integration test for redaction → HTTP sink (with captured request body)
- [x] Integration test for redaction → PostgreSQL sink (with database query)
- [x] Test verifies redaction via `logger.info()`, `logger.error()`, `logger.exception()`
- [x] Test verifies nested field redaction reaches sinks correctly
- [x] All tests use real sinks (not mocks)

---

## Technical Approach

**Shared Imports** (at top of test file):

```python
from __future__ import annotations

import json
from io import StringIO
from typing import Any, cast
from pathlib import Path

import pytest

from fapilog import get_logger
from fapilog.plugins.redactors import BaseRedactor
from fapilog.plugins.redactors.field_mask import FieldMaskRedactor
```

### Test 1: Stdout JSON Sink

Capture stdout and verify masked data appears.

```python
@pytest.mark.asyncio
@pytest.mark.security
async def test_redaction_reaches_stdout_sink(capsys):
    """Verify redacted data appears in stdout JSON output."""

    # Create real field mask redactor
    redactor = FieldMaskRedactor(
        fields_to_mask=["password", "credit_card", "ssn"],
        mask_nested=True,
    )

    logger = get_logger(name="redaction-stdout-test")

    # Inject redactor (following pattern from test_redactors_stage.py)
    logger._redactors = cast(list[BaseRedactor], [redactor])

    logger.info(
        "user_login",
        username="alice",
        password="secret123",
        credit_card="4111-1111-1111-1111",
        user={"name": "Alice", "ssn": "123-45-6789"},
    )

    await logger.stop_and_drain()

    captured = capsys.readouterr()
    output = captured.out

    # Parse the JSON output
    log_entry = json.loads(output.strip())

    # Verify sensitive fields are masked
    assert log_entry["password"] == "***"
    assert log_entry["credit_card"] == "***"
    assert log_entry["user"]["ssn"] == "***"

    # Verify non-sensitive fields are NOT masked
    assert log_entry["username"] == "alice"
    assert log_entry["user"]["name"] == "Alice"

    # Verify the actual secret values do NOT appear anywhere in raw output
    assert "secret123" not in output
    assert "4111-1111-1111-1111" not in output
    assert "123-45-6789" not in output
```

### Test 2: Rotating File Sink

Write to a file and verify masked data.

```python
from fapilog.plugins.sinks.rotating_file import RotatingFileSink, RotatingFileSinkConfig


@pytest.mark.asyncio
@pytest.mark.security
async def test_redaction_reaches_file_sink(tmp_path: Path):
    """Verify redacted data appears in file sink output."""

    log_file = tmp_path / "test.log"

    # Create real file sink
    sink = RotatingFileSink(
        RotatingFileSinkConfig(
            directory=str(tmp_path),
            filename="test.log",
            max_bytes=10_000_000,
            backup_count=1,
        )
    )
    await sink.start()

    # Create redactor
    redactor = FieldMaskRedactor(
        fields_to_mask=["api_key", "token"],
        mask_nested=True,
    )

    logger = get_logger(name="redaction-file-test")
    logger._sink_write = sink.write  # type: ignore[attr-defined]
    logger._redactors = cast(list[BaseRedactor], [redactor])

    logger.info("api_call", api_key="sk-12345", token="bearer-xyz", endpoint="/users")

    await logger.stop_and_drain()
    await sink.stop()

    # Read the log file
    content = log_file.read_text()

    # Verify sensitive values are NOT in file
    assert "sk-12345" not in content
    assert "bearer-xyz" not in content

    # Verify masked values ARE in file
    assert '"api_key":"***"' in content or '"api_key": "***"' in content
    assert '"token":"***"' in content or '"token": "***"' in content

    # Verify non-sensitive data IS in file
    assert "/users" in content
```

### Test 3: HTTP Sink with Request Capture

Use a capturing sender to verify the actual HTTP body sent.

```python
import httpx
from fapilog.plugins.sinks.http_client import HttpSink, HttpSinkConfig, AsyncHttpSender
from fapilog.core.resources import HttpClientPool


class CapturingHttpSender(AsyncHttpSender):
    """Test double that captures all requests instead of sending them."""

    def __init__(self) -> None:
        self.captured: list[dict[str, Any]] = []

    async def post(
        self,
        url: str,
        *,
        json: Any | None = None,
        content: bytes | None = None,
        headers: Any = None,
    ) -> httpx.Response:
        if json is not None:
            self.captured.extend(json if isinstance(json, list) else [json])
        return httpx.Response(200)


@pytest.mark.asyncio
@pytest.mark.security
async def test_redaction_reaches_http_sink():
    """Verify redacted data appears in HTTP request body."""

    capturing_sender = CapturingHttpSender()

    # Create HTTP sink with capturing sender
    sink = HttpSink(
        HttpSinkConfig(endpoint="https://logs.example.com/ingest"),
        sender=capturing_sender,
    )
    await sink.start()

    # Create redactor
    redactor = FieldMaskRedactor(
        fields_to_mask=["password", "secret"],
        mask_nested=True,
    )

    logger = get_logger(name="redaction-http-test")
    logger._sink_write = sink.write  # type: ignore[attr-defined]
    logger._redactors = cast(list[BaseRedactor], [redactor])

    logger.info("login", username="bob", password="hunter2", secret="abc123")

    await logger.stop_and_drain()
    await sink.stop()

    # Verify the captured HTTP request body
    assert len(capturing_sender.captured) == 1

    sent_event = capturing_sender.captured[0]
    assert sent_event["password"] == "***"
    assert sent_event["secret"] == "***"
    assert sent_event["username"] == "bob"

    # Verify raw secrets never sent
    all_content = json.dumps(capturing_sender.captured)
    assert "hunter2" not in all_content
    assert "abc123" not in all_content
```

### Test 4: PostgreSQL Sink with Database Query

Verify masked data in actual database rows.

```python
@pytest.mark.asyncio
@pytest.mark.integration
@pytest.mark.security
async def test_redaction_reaches_postgres_sink(postgres_pool):
    """Verify redacted data appears in PostgreSQL rows."""

    pytest.importorskip("asyncpg")

    from fapilog.plugins.sinks.contrib.postgres import PostgresSink, PostgresSinkConfig

    # Create table for test
    async with postgres_pool.acquire() as conn:
        await conn.execute("""
            CREATE TABLE IF NOT EXISTS redaction_test_logs (
                id SERIAL PRIMARY KEY,
                timestamp TIMESTAMPTZ,
                level TEXT,
                message TEXT,
                data JSONB
            )
        """)
        await conn.execute("TRUNCATE redaction_test_logs")

    sink = PostgresSink(
        PostgresSinkConfig(
            dsn="postgresql://fapilog:fapilog@localhost:5432/fapilog_test",
            table_name="redaction_test_logs",
        )
    )
    await sink.start()

    # Create redactor
    redactor = FieldMaskRedactor(
        fields_to_mask=["password", "credit_card"],
        mask_nested=True,
    )

    logger = get_logger(name="redaction-postgres-test")
    logger._sink_write = sink.write  # type: ignore[attr-defined]
    logger._redactors = cast(list[BaseRedactor], [redactor])

    logger.info(
        "payment",
        user_id="u-123",
        password="secret",
        credit_card="4111111111111111",
        amount=99.99,
    )

    await logger.stop_and_drain()
    await sink.stop()

    # Query the database directly
    async with postgres_pool.acquire() as conn:
        row = await conn.fetchrow(
            "SELECT * FROM redaction_test_logs WHERE message = 'payment'"
        )

    assert row is not None

    # Parse the JSON data column
    event_data = json.loads(row["data"]) if isinstance(row["data"], str) else row["data"]

    # Verify sensitive fields are masked IN THE DATABASE
    assert event_data.get("password") == "***"
    assert event_data.get("credit_card") == "***"

    # Verify non-sensitive fields are NOT masked
    assert event_data.get("user_id") == "u-123"
    assert event_data.get("amount") == 99.99

    # Verify raw secret never stored
    full_row_str = str(row)
    assert "4111111111111111" not in full_row_str
```

### Test 5: All Log Levels Apply Redaction

Verify redaction works for info, warning, error, and exception.

```python
@pytest.mark.asyncio
@pytest.mark.security
async def test_redaction_applies_to_all_log_levels():
    """Verify redaction is applied regardless of log level."""

    collected: list[dict[str, Any]] = []

    async def collecting_sink(event: dict[str, Any]) -> None:
        collected.append(event)

    redactor = FieldMaskRedactor(
        fields_to_mask=["secret"],
        mask_nested=True,
    )

    logger = get_logger(name="redaction-levels-test")
    logger._sink_write = collecting_sink  # type: ignore[attr-defined]
    logger._redactors = cast(list[BaseRedactor], [redactor])

    logger.debug("debug-msg", secret="debug-secret")
    logger.info("info-msg", secret="info-secret")
    logger.warning("warning-msg", secret="warning-secret")
    logger.error("error-msg", secret="error-secret")

    try:
        raise ValueError("test error")
    except ValueError:
        logger.exception("exception-msg", secret="exception-secret")

    await logger.stop_and_drain()

    # Verify we got all 5 events (depends on log level config)
    # Filter to only our test messages
    test_events = [e for e in collected if e.get("message", "").endswith("-msg")]

    # All captured events should have masked secret
    for event in test_events:
        assert event.get("secret") == "***", f"Event {event['message']} has unmasked secret"

    # Verify none of the raw secrets appear
    all_content = json.dumps(collected)
    assert "debug-secret" not in all_content
    assert "info-secret" not in all_content
    assert "warning-secret" not in all_content
    assert "error-secret" not in all_content
    assert "exception-secret" not in all_content
```

### Test 6: Redaction Before Serialization

Verify redaction happens before orjson serializes the event.

```python
@pytest.mark.asyncio
@pytest.mark.security
async def test_redaction_happens_before_serialization():
    """Verify redaction occurs before JSON serialization."""

    captured_payloads: list[dict[str, Any]] = []

    async def capturing_sink(event: dict[str, Any]) -> None:
        # Capture a deep copy to preserve the state at sink time
        captured_payloads.append(dict(event))

    redactor = FieldMaskRedactor(
        fields_to_mask=["password"],
        mask_nested=True,
    )

    logger = get_logger(name="redaction-order-test")
    logger._sink_write = capturing_sink  # type: ignore[attr-defined]
    logger._redactors = cast(list[BaseRedactor], [redactor])

    logger.info("login", password="supersecret")

    await logger.stop_and_drain()

    # The payload at sink should already have masked value
    assert len(captured_payloads) == 1

    payload = captured_payloads[0]

    # Verify redaction occurred BEFORE reaching sink
    assert payload.get("password") == "***"
    assert "supersecret" not in str(payload)

    # Verify the unmasked value never reaches the sink
    all_content = json.dumps(captured_payloads)
    assert "supersecret" not in all_content
```

---

## Files Changed

| File | Action |
|------|--------|
| `tests/integration/test_redaction_sinks.py` | New file |

**Lines added:** ~400

---

## Test Infrastructure

Tests use direct redactor/sink injection following the pattern from `test_redactors_stage.py`:

1. **Stdout**: Use `capsys` pytest fixture
2. **File**: Use `tmp_path` pytest fixture + `RotatingFileSink` direct instantiation
3. **HTTP**: Use `CapturingHttpSender` test double injected via `sender=` parameter
4. **PostgreSQL**: Use existing `postgres_pool` fixture or skip if unavailable

No external infrastructure required except PostgreSQL for that specific test (can be skipped in CI without Postgres).

---

## Risk Assessment

**Risk Level: Low**

- Tests only, no production code changes
- May reveal security bugs (which is critical)
- No coverage impact (additive)

---

## Rollback Plan

Delete `tests/integration/test_redaction_sinks.py`; no other files affected.

---

## Security Note

These tests are **security-critical**. If any test fails, it indicates a potential data leak where sensitive information could reach external systems unmasked. Failures should be treated as P0 bugs.

---

## Definition of Done

- [x] Integration test for stdout sink redaction
- [x] Integration test for file sink redaction
- [x] Integration test for HTTP sink redaction
- [x] Integration test for PostgreSQL sink redaction
- [x] Integration test for all log levels
- [x] Integration test verifying redaction-before-serialization ordering
- [x] All tests pass
- [x] Tests added to CI
- [x] Tests marked as `@pytest.mark.security` for visibility


