# Story 7.9: Tag Tests by Risk Category

**Status:** Complete
**Priority:** P2
**Depends on:** None
**Effort:** M (4-8 hours)

---

## Problem Statement

The test suite treats all tests equally, but not all tests are equal:

- **Security tests** (redaction, auth) should never be skipped
- **Integration tests** may require external dependencies
- **Slow tests** might be skipped during development
- **Flaky tests** need monitoring and eventual fixing

Without categorization:

- Developers don't know which tests are critical
- CI can't prioritize security tests on every PR
- Slow tests slow down the feedback loop
- Test failures lack context about severity

---

## Goals

1. **Differentiate risk and type** with minimal per-test overhead
2. **Add pytest markers** for filtering by risk and runtime constraints
3. **Enable selective test runs** (critical/security on PR, skip slow/integration in quick)
4. **Document marker policy** including defaults and flaky handling

---

## Sweet Spot Principles

- Default risk is **standard**; only critical/security need explicit marking.
- Prefer module-level `pytestmark` or class decorators to avoid per-test tagging.
- `flaky` is temporary quarantine with an issue and expiry; never for critical/security.
- CI stays fast: critical/security on PR, full suite on main/nightly.

---

## Acceptance Criteria

- [ ] Risk markers defined (`critical`, `security`, `standard` where `standard` is the default)
- [ ] Type markers defined (`integration`, `slow`, `flaky`, `postgres`, `property`)
- [ ] All security-critical tests tagged with `@pytest.mark.security` (module/class markers allowed)
- [ ] All integration tests tagged with `@pytest.mark.integration`
- [ ] CI runs critical/security on every PR; quick CI skips slow/integration and excludes or quarantines flaky tests
- [ ] Flaky policy documented: issue + expiry, never for critical/security, separate CI behavior
- [ ] Verification script validates marker names, supports module/class markers, and reports unmarked tests (treated as `standard`); strict mode available for later enforcement
- [ ] Documentation for test categories and markers

---

## Technical Approach

### 1. Define Markers

**conftest.py (root):**

```python
import pytest


def pytest_configure(config: pytest.Config) -> None:
    """Register custom markers for test categorization."""

    # Risk-based markers
    config.addinivalue_line(
        "markers",
        "critical: Tests that must never fail - core functionality",
    )
    config.addinivalue_line(
        "markers",
        "security: Security-critical tests (redaction, auth, secrets)",
    )
    config.addinivalue_line(
        "markers",
        "standard: Default risk category for typical unit tests",
    )
    config.addinivalue_line(
        "markers",
        "integration: Tests requiring external dependencies",
    )
    config.addinivalue_line(
        "markers",
        "slow: Tests that take >1 second",
    )
    config.addinivalue_line(
        "markers",
        "flaky: Tests with known intermittent failures (tracked for fixing)",
    )

    # Feature-based markers (for filtering)
    config.addinivalue_line(
        "markers",
        "asyncio: Async tests",
    )
    config.addinivalue_line(
        "markers",
        "postgres: Tests requiring PostgreSQL",
    )
    config.addinivalue_line(
        "markers",
        "property: Property-based tests (may be slow)",
    )
```

**pyproject.toml:**

```toml
[tool.pytest.ini_options]
markers = [
    "critical: Tests that must never fail - core functionality",
    "security: Security-critical tests (redaction, auth, secrets)",
    "standard: Default risk category for typical unit tests",
    "integration: Tests requiring external dependencies",
    "slow: Tests that take >1 second",
    "flaky: Tests with known intermittent failures",
    "asyncio: Async tests",
    "postgres: Tests requiring PostgreSQL",
    "property: Property-based tests",
]
```

### 2. Marker Policy

Risk markers (choose at most one; `standard` is the default if omitted):

| Marker | When to Use | CI Behavior |
|--------|-------------|-------------|
| `@pytest.mark.critical` | Core logger, worker, queue functionality | Always run, fail-fast |
| `@pytest.mark.security` | Redaction, auth, secret handling | Always run on PR; never flaky |
| `@pytest.mark.standard` | Default for typical unit tests (optional) | Runs in full CI |

Type markers (optional; can be combined with risk):

| Marker | When to Use | CI Behavior |
|--------|-------------|-------------|
| `@pytest.mark.integration` | External deps (Postgres, HTTP) | Run in full CI, skip in quick |
| `@pytest.mark.slow` | Tests >1 second | Skip in quick CI |
| `@pytest.mark.flaky` | Known intermittent failures (time-boxed) | Quarantined; allow-fail or xfail |
| `@pytest.mark.postgres` | Needs PostgreSQL | Skip if Postgres unavailable |
| `@pytest.mark.property` | Hypothesis tests | Run with reduced examples in quick CI |

### 3. Apply Markers to Existing Tests

**Module-level default (preferred):**

```python
# tests/integration/test_postgres.py
pytestmark = [pytest.mark.integration, pytest.mark.security]
```

**Security tests (7.3):**

```python
@pytest.mark.security
@pytest.mark.integration
async def test_redaction_reaches_postgres_sink(postgres_pool):
    """Verify redacted data appears in PostgreSQL rows."""
    ...
```

**Critical tests:**

```python
@pytest.mark.critical
def test_logger_enqueue_does_not_block():
    """Core invariant: logging should never block the caller."""
    ...
```

**Integration tests:**

```python
@pytest.mark.integration
@pytest.mark.postgres
async def test_postgres_sink_writes_events():
    ...
```

### 4. CI Configuration

**Quick CI (every PR):**

```yaml
# .github/workflows/test.yml
- name: Run critical and security tests
  run: pytest -m "(critical or security) and not flaky" tests/

- name: Run unit tests (skip slow)
  run: pytest -m "not slow and not integration and not flaky" tests/unit/

- name: Run flaky tests (allowed to fail)
  run: pytest -m flaky tests/
  continue-on-error: true
```

**Full CI (main branch, nightly):**

```yaml
- name: Run all tests
  run: pytest tests/
```

### 5. Verification Script

**scripts/verify_test_markers.py:**

```python
#!/usr/bin/env python3
"""
Verify test markers and report unmarked tests (treated as standard).

Usage:
    python scripts/verify_test_markers.py [--strict] tests/
"""

import ast
import sys
from pathlib import Path
from dataclasses import dataclass


RISK_MARKERS = {"critical", "security", "standard"}
TYPE_MARKERS = {"integration", "slow", "flaky", "postgres", "property"}
ALLOWED_MARKERS = RISK_MARKERS | TYPE_MARKERS


@dataclass
class MarkerIssue:
    file: Path
    line: int
    name: str
    issue: str


class TestMarkerVisitor(ast.NodeVisitor):
    """Find test functions and check their markers."""

    def __init__(self, filepath: Path, module_markers: set[str]):
        self.filepath = filepath
        self.module_markers = module_markers
        self.class_markers_stack: list[set[str]] = []
        self.unmarked: list[MarkerIssue] = []
        self.unknown: list[MarkerIssue] = []
        self.conflicts: list[MarkerIssue] = []

    def visit_FunctionDef(self, node: ast.FunctionDef) -> None:
        self._check_function(node)
        self.generic_visit(node)

    def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None:
        self._check_function(node)
        self.generic_visit(node)

    def visit_ClassDef(self, node: ast.ClassDef) -> None:
        class_markers = self._get_pytest_markers(node)
        self.class_markers_stack.append(class_markers)
        self.generic_visit(node)
        self.class_markers_stack.pop()

    def _check_function(self, node: ast.FunctionDef | ast.AsyncFunctionDef) -> None:
        if not node.name.startswith("test_"):
            return

        inherited = set(self.module_markers)
        for class_markers in self.class_markers_stack:
            inherited |= class_markers
        markers = inherited | self._get_pytest_markers(node)

        unknown = markers - ALLOWED_MARKERS
        if unknown:
            self.unknown.append(
                MarkerIssue(
                    self.filepath,
                    node.lineno,
                    node.name,
                    f"unknown markers: {sorted(unknown)}",
                )
            )

        if "flaky" in markers and ("critical" in markers or "security" in markers):
            self.conflicts.append(
                MarkerIssue(
                    self.filepath,
                    node.lineno,
                    node.name,
                    "flaky cannot be combined with critical/security",
                )
            )

        # Check if has any risk marker (unmarked defaults to standard)
        has_risk = bool(markers & RISK_MARKERS)
        if not has_risk:
            self.unmarked.append(
                MarkerIssue(
                    self.filepath,
                    node.lineno,
                    node.name,
                    "missing risk marker",
                )
            )

    def _get_pytest_markers(
        self, node: ast.FunctionDef | ast.AsyncFunctionDef | ast.ClassDef
    ) -> set[str]:
        """Extract pytest.mark.X decorators from a node with decorator_list."""
        markers = set()

        for decorator in node.decorator_list:
            marker = _extract_marker_name(decorator)
            if marker:
                markers.add(marker)

        return markers


def _extract_marker_name(node: ast.AST) -> str | None:
    """Extract pytest.mark.X name from decorator or marker node."""
    target = None
    if isinstance(node, ast.Attribute):
        target = node
    elif isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute):
        target = node.func

    if not target:
        return None

    if (
        isinstance(target.value, ast.Attribute)
        and isinstance(target.value.value, ast.Name)
        and target.value.value.id == "pytest"
        and target.value.attr == "mark"
    ):
        return target.attr

    return None


def _extract_markers_from_node(node: ast.AST) -> set[str]:
    """Extract markers from pytestmark assignments."""
    markers = set()
    if isinstance(node, ast.List):
        for elt in node.elts:
            markers |= _extract_markers_from_node(elt)
    else:
        marker = _extract_marker_name(node)
        if marker:
            markers.add(marker)
    return markers


def _module_markers(tree: ast.Module) -> set[str]:
    """Extract module-level pytestmark markers."""
    markers = set()
    for node in tree.body:
        if not isinstance(node, ast.Assign):
            continue
        if not any(isinstance(t, ast.Name) and t.id == "pytestmark" for t in node.targets):
            continue
        markers |= _extract_markers_from_node(node.value)
    return markers


def check_file(filepath: Path) -> tuple[list[MarkerIssue], list[MarkerIssue], list[MarkerIssue]]:
    """Check a single file for marker issues."""
    try:
        source = filepath.read_text()
        tree = ast.parse(source)
    except SyntaxError:
        return [], [], []

    module_markers = _module_markers(tree)
    visitor = TestMarkerVisitor(filepath, module_markers)
    visitor.visit(tree)
    return visitor.unmarked, visitor.unknown, visitor.conflicts


def main() -> int:
    strict = "--strict" in sys.argv
    args = [arg for arg in sys.argv[1:] if arg != "--strict"]
    if len(args) < 1:
        print("Usage: python scripts/verify_test_markers.py [--strict] <path>")
        return 1

    path = Path(args[0])
    unmarked: list[MarkerIssue] = []
    unknown: list[MarkerIssue] = []
    conflicts: list[MarkerIssue] = []

    if path.is_file():
        unmarked, unknown, conflicts = check_file(path)
    elif path.is_dir():
        for filepath in path.rglob("test_*.py"):
            file_unmarked, file_unknown, file_conflicts = check_file(filepath)
            unmarked.extend(file_unmarked)
            unknown.extend(file_unknown)
            conflicts.extend(file_conflicts)

    if unknown:
        print(f"Found {len(unknown)} tests with unknown markers:\n")
        for issue in unknown[:20]:
            print(f"  {issue.file}:{issue.line}: {issue.name} ({issue.issue})")

    if conflicts:
        print(f"\nFound {len(conflicts)} tests with marker conflicts:\n")
        for issue in conflicts[:20]:
            print(f"  {issue.file}:{issue.line}: {issue.name} ({issue.issue})")

    if unmarked:
        print(f"\nFound {len(unmarked)} tests without risk markers (defaulting to standard):\n")
        for issue in unmarked[:20]:
            print(f"  {issue.file}:{issue.line}: {issue.name} ({issue.issue})")

    if any(len(items) > 20 for items in (unknown, conflicts, unmarked)):
        print("  ... and more")

    if unknown or conflicts or (strict and unmarked):
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())
```

### 6. Documentation

**docs/contributing/test-categories.md:**

```markdown
# Test Categories

Tests are categorized by risk level and type for selective execution.

## Risk Markers

| Marker | Description | Required for |
|--------|-------------|--------------|
| `@pytest.mark.critical` | Core functionality that must never break | Logger, worker, queue |
| `@pytest.mark.security` | Security-critical paths | Redaction, auth |
| `@pytest.mark.standard` | Default for typical unit tests (optional) | Everything else |

## Type Markers

| Marker | Description |
|--------|-------------|
| `@pytest.mark.integration` | Requires external dependencies |
| `@pytest.mark.slow` | Takes >1 second |
| `@pytest.mark.flaky` | Known intermittent failures (time-boxed) |
| `@pytest.mark.postgres` | Requires PostgreSQL |
| `@pytest.mark.property` | Property-based (Hypothesis) |

Unmarked tests are treated as `standard`.

## Flaky Policy

- `@pytest.mark.flaky` requires an issue link and expiry date in the test docstring.
- `flaky` is never allowed on `critical` or `security`.
- Flaky tests run in a quarantined allow-fail job or as `xfail`.

## Running Subsets

```bash
# Only critical tests (fast feedback)
pytest -m critical

# Security tests
pytest -m security

# Skip slow tests
pytest -m "not slow"

# Skip integration tests
pytest -m "not integration"

# Unit tests only
pytest tests/unit/ -m "not slow"

# Skip flaky tests
pytest -m "not flaky"
```

## Adding Markers

Prefer module-level `pytestmark` or class decorators to reduce per-test overhead. Unmarked tests
default to `standard`, so only critical/security/integration/slow/flaky need
explicit markers.

Module-level example:

```python
# tests/integration/test_postgres.py
pytestmark = [pytest.mark.integration, pytest.mark.security]
```

Function-level example:

```python
@pytest.mark.security
@pytest.mark.integration
async def test_redaction_reaches_sink():
    ...
```
```

---

## Files Changed

| File | Action |
|------|--------|
| `conftest.py` | Add marker definitions |
| `pyproject.toml` | Add markers to pytest config |
| `scripts/verify_test_markers.py` | New verification script |
| `docs/contributing/test-categories.md` | New documentation |
| `tests/**/*.py` | Add markers to existing tests (incremental) |
| `.github/workflows/test.yml` | Update to use marker-based runs |

**Lines changed:** ~300 (mostly marker additions to tests)

---

## Rollout Plan

### Phase 1: Define Markers

1. Add marker definitions to conftest.py and pyproject.toml
2. Create verification script
3. Document categories

### Phase 2: Tag Critical Tests

1. Tag all security tests (`@pytest.mark.security`)
2. Tag all integration tests (`@pytest.mark.integration`)
3. Tag known slow tests (`@pytest.mark.slow`)

### Phase 3: Update CI

1. Add quick CI job that runs critical + security
2. Keep full CI for all tests
3. Monitor for missing markers

---

## Risk Assessment

**Risk Level: Low**

- Adding markers doesn't change test behavior
- Incremental rollout minimizes risk
- Can be reverted by removing markers

---

## Rollback Plan

1. Remove marker definitions from conftest.py
2. Revert pyproject.toml changes
3. Remove verification script
4. Markers in test files can remain (harmless)

---

## Definition of Done

- [ ] Marker definitions added to conftest.py
- [ ] pyproject.toml updated with markers
- [ ] All security tests tagged with `@pytest.mark.security`
- [ ] All integration tests tagged with `@pytest.mark.integration`
- [ ] Verification script created
- [ ] CI updated to use marker-based runs
- [ ] Documentation created
- [ ] Flaky policy documented and enforced in CI
- [ ] All tests still pass

---

## Related Stories

- **7.3**: Redaction Integration Tests (should have `@pytest.mark.security`)
- **7.6**: Property-Based Tests (should have `@pytest.mark.property`)
- **7.7**: Concurrency Tests (could be `@pytest.mark.slow`)

---

## Future Considerations

- Enforce `--strict` in CI once unmarked tests are addressed
- Consider auto-detection of slow tests (pytest-timeout)
- Add test coverage reporting per category
- Consider risk-weighted test selection for CI optimization
