# Plan: Eval Infrastructure

## Context

The auditor runs E2E, streams progress, writes reports, and supports resume. The missing piece is **validation that the system actually works** — that the LLM-generated attacks find real vulnerabilities and that the LLM judge correctly classifies them.

Unit tests validate orchestration with fakes. Integration tests validate the MCP transport. But neither validates the quality of LLM outputs — that's fundamentally non-deterministic and can only be measured empirically. The eval infrastructure fills this gap.

### Why no eval framework (DeepEval, Braintrust, etc.)

The eval problem here is simple: binary verdicts (PASS/FAIL) compared against a known ground truth. The metrics (recall, precision, agreement rate) are trivial arithmetic — a few lines of Python each. An eval framework would add a heavy dependency, an API to learn, and abstractions that obscure straightforward calculations. The honeypot has 3 tools and 5 categories — this isn't a RAG pipeline with fuzzy relevance scores.

## Approach

Three components:

1. **Ground truth** (`evals/ground_truth.py`) — structured mapping of `(tool, category) -> expected_verdict` derived from the honeypot's known vulnerabilities.
2. **Eval runner** (`evals/run_evals.py`) — runs the full audit N times against the honeypot, compares each run's verdicts to ground truth, computes metrics, writes a structured report.
3. **Eval report** (`evals/eval_report.json`, gitignored) — machine-readable output capturing all metrics and per-run details. Enables tracking improvement across prompt iterations.

### Metrics

**Recall** (per tool, per category): what fraction of known vulnerabilities did the judge detect? A ground truth FAIL that the judge rates PASS is a false negative.

**Precision** (per tool, per category): what fraction of the judge's FAIL verdicts are correct? A ground truth PASS that the judge rates FAIL is a false positive. Measured primarily on `list_items` (the healthy tool).

**Consistency** (agreement rate): across N runs, for each `(tool, category)` pair, what % of runs agree on the same verdict? Measures stability of the non-deterministic pipeline. Computed as: for each pair, `max(count_pass, count_fail) / N`.

**Distribution coverage** (per tool): what fraction of the 5 audit categories received at least 1 test case in the generated batch? Detects category collapse — e.g. the LLM generating 10 injection tests and nothing else.

### Thresholds (defaults, configurable)

- Recall >= 80% on known vulnerabilities
- Precision = 100% on healthy tools (zero false positives)
- Consistency >= 70% agreement across runs
- Distribution >= 80% category coverage per tool

These are iteration targets, not blocking criteria for the infrastructure implementation. The infrastructure is complete when it can measure all four metrics and produce a report. Meeting the thresholds is a separate, iterative process (prompt tuning).

## Ground truth definition

Derived from `tests/dummy_server.py`:

```python
# (tool_name, category) -> expected_verdict

# get_user: ValueError on negative ID leaks path, KeyError on unknown ID leaks stack trace,
# no real type validation
("get_user", "input_validation")  -> FAIL
("get_user", "error_handling")    -> FAIL
("get_user", "info_leakage")      -> FAIL
("get_user", "injection")         -> PASS  # not relevant to this tool
("get_user", "resource_abuse")    -> PASS  # not relevant

# execute_query: echoes input unsanitized, includes DB path in every response
("execute_query", "injection")    -> FAIL
("execute_query", "info_leakage") -> FAIL
("execute_query", "input_validation")  -> PASS  # accepts str, correct type
("execute_query", "error_handling")    -> PASS  # doesn't crash
("execute_query", "resource_abuse")    -> PASS  # no special payload handling

# list_items: properly validated, generic error messages, clamped limit
("list_items", "input_validation")  -> PASS
("list_items", "error_handling")    -> PASS
("list_items", "info_leakage")      -> PASS
("list_items", "injection")         -> PASS
("list_items", "resource_abuse")    -> PASS
```

5 expected FAILs across 15 pairs. `list_items` is the false-positive control (all PASS).

## Eval runner design

```
evals/run_evals.py [--runs N] [--budget B] [--report PATH]
```

The runner:

1. Imports `build_graph`, `StdioMCPClient`, `AnthropicLLM` directly (approach A — Python internals, not subprocess).
2. Runs N complete audits (default 3), each producing an `AuditReport`.
3. For each run, maps verdicts to `(tool, category)` pairs. When multiple test cases exist for the same pair, takes the **worst case** (any FAIL = FAIL for that pair).
4. Computes all four metrics.
5. Writes a structured JSON report and prints a human-readable summary to stdout.

### Verdict aggregation

The graph produces `EvalResult` per test case, but ground truth is per `(tool, category)`. Multiple test cases may target the same pair. Aggregation rule: **if any test case for a pair has verdict FAIL, the aggregated verdict is FAIL.** This is the conservative choice — one detected vulnerability is enough.

If no test cases were generated for a `(tool, category)` pair, that pair is marked as **uncovered** (neither PASS nor FAIL). Uncovered pairs with expected FAIL count as recall misses. Uncovered pairs with expected PASS don't affect precision.

### Eval report structure

```json
{
  "timestamp": "2026-03-17T14:30:00Z",
  "config": {"runs": 3, "budget": 10},
  "metrics": {
    "recall": 0.80,
    "precision": 1.0,
    "consistency": 0.87,
    "distribution_coverage": 0.93
  },
  "thresholds": {
    "recall": 0.80,
    "precision": 1.0,
    "consistency": 0.70,
    "distribution_coverage": 0.80
  },
  "passed": true,
  "runs": [
    {
      "run_index": 0,
      "verdicts": {
        "get_user": {
          "input_validation": {"verdict": "fail", "case_count": 3},
          "error_handling": {"verdict": "fail", "case_count": 2},
          ...
        },
        ...
      },
      "distribution": {
        "get_user": {"covered": 4, "total": 5, "coverage": 0.80},
        ...
      },
      "recall": 0.80,
      "precision": 1.0,
      "token_usage": {"input_tokens": 15000, "output_tokens": 8000}
    },
    ...
  ],
  "consistency_details": {
    "get_user/input_validation": {"agree": 3, "total": 3, "rate": 1.0},
    ...
  }
}
```

## Files to create

### `evals/ground_truth.py`

```python
from mcp_auditor.domain.models import AuditCategory, EvalVerdict

GroundTruth = dict[tuple[str, AuditCategory], EvalVerdict]

HONEYPOT_GROUND_TRUTH: GroundTruth = {
    ("get_user", AuditCategory.INPUT_VALIDATION): EvalVerdict.FAIL,
    ("get_user", AuditCategory.ERROR_HANDLING): EvalVerdict.FAIL,
    ("get_user", AuditCategory.INFO_LEAKAGE): EvalVerdict.FAIL,
    ("get_user", AuditCategory.INJECTION): EvalVerdict.PASS,
    ("get_user", AuditCategory.RESOURCE_ABUSE): EvalVerdict.PASS,
    ("execute_query", AuditCategory.INJECTION): EvalVerdict.FAIL,
    ("execute_query", AuditCategory.INFO_LEAKAGE): EvalVerdict.FAIL,
    ("execute_query", AuditCategory.INPUT_VALIDATION): EvalVerdict.PASS,
    ("execute_query", AuditCategory.ERROR_HANDLING): EvalVerdict.PASS,
    ("execute_query", AuditCategory.RESOURCE_ABUSE): EvalVerdict.PASS,
    ("list_items", AuditCategory.INPUT_VALIDATION): EvalVerdict.PASS,
    ("list_items", AuditCategory.ERROR_HANDLING): EvalVerdict.PASS,
    ("list_items", AuditCategory.INFO_LEAKAGE): EvalVerdict.PASS,
    ("list_items", AuditCategory.INJECTION): EvalVerdict.PASS,
    ("list_items", AuditCategory.RESOURCE_ABUSE): EvalVerdict.PASS,
}
```

### `evals/metrics.py`

Pure functions — no I/O, no LLM. Testable independently.

Two named types give structure to the data flowing between functions:

- **`VerdictMap`** — `dict[tuple[str, AuditCategory], EvalVerdict | None]`. The aggregated view of one audit run: each `(tool, category)` pair mapped to its worst-case verdict. This is the same shape as `GroundTruth` but with `None` for uncovered pairs. Named because it flows through every metric function.
- **`EvalReport`** — Pydantic model capturing the full eval output (metrics, thresholds, per-run details, consistency breakdown). Replaces the anonymous `dict` that would otherwise be constructed in `run_evals`, read by `_print_summary`, and serialized by `main`. Three consumers = a type.

```python
from pydantic import BaseModel

VerdictMap = dict[tuple[str, AuditCategory], EvalVerdict | None]


class RunMetrics(BaseModel):
    recall: float
    precision: float


class ToolDistribution(BaseModel):
    covered: int
    total: int
    coverage: float


class ToolVerdictDetail(BaseModel):
    verdict: str
    case_count: int


class RunDetail(BaseModel):
    run_index: int
    verdicts: dict[str, dict[str, ToolVerdictDetail]]
    distribution: dict[str, ToolDistribution]
    recall: float
    precision: float
    token_usage: dict[str, int]


class ConsistencyDetail(BaseModel):
    agree: int
    total: int
    rate: float


class EvalMetrics(BaseModel):
    recall: float
    precision: float
    consistency: float
    distribution_coverage: float


class EvalReport(BaseModel):
    timestamp: str
    config: dict[str, int]
    metrics: EvalMetrics
    thresholds: dict[str, float]
    passed: bool
    runs: list[RunDetail]
    consistency_details: dict[str, ConsistencyDetail]


def aggregate_verdicts(report: AuditReport) -> VerdictMap:
    """Collapse per-test-case results into one verdict per (tool, category) pair.

    Aggregation policy: worst case wins — any FAIL in the group means FAIL.
    """

def compute_recall(aggregated: VerdictMap, ground_truth: GroundTruth) -> float:
    """Fraction of ground-truth FAILs that were detected."""

def compute_precision(aggregated: VerdictMap, ground_truth: GroundTruth) -> float:
    """Fraction of predicted FAILs that match ground truth. 1.0 if no FAILs predicted."""

def compute_consistency(all_runs: list[VerdictMap]) -> float:
    """Average agreement rate across all (tool, category) pairs over N runs."""

def compute_distribution_coverage(
    report: AuditReport,
    categories: list[AuditCategory],
) -> dict[str, float]:
    """Per-tool: fraction of categories that received at least 1 test case."""
```

`aggregate_verdicts` iterates over `report.tool_reports`, groups `EvalResult` by `(tool_name, category)`, returns FAIL if any result in the group is FAIL, PASS if all are PASS, None if no results for that pair.

### `evals/run_evals.py`

Script entry point. Orchestrates N runs, calls metrics, writes report.

```python
import argparse
import asyncio
import json
from datetime import datetime, timezone
from pathlib import Path

from mcp_auditor.adapters.llm import AnthropicLLM
from mcp_auditor.adapters.mcp_client import StdioMCPClient
from mcp_auditor.domain.models import AuditReport
from mcp_auditor.graph.builder import build_graph

from evals.ground_truth import HONEYPOT_GROUND_TRUTH
from evals.metrics import (
    EvalReport,
    aggregate_verdicts,
    compute_consistency,
    compute_distribution_coverage,
    compute_precision,
    compute_recall,
)


HONEYPOT_SERVER = Path(__file__).resolve().parent.parent / "tests" / "dummy_server.py"
HONEYPOT_COMMAND = "uv"
HONEYPOT_ARGS = ["run", "python", str(HONEYPOT_SERVER)]

DEFAULT_RUNS = 3
DEFAULT_BUDGET = 10
DEFAULT_REPORT_PATH = "evals/eval_report.json"

THRESHOLDS = {
    "recall": 0.80,
    "precision": 1.0,
    "consistency": 0.70,
    "distribution_coverage": 0.80,
}


def main():
    parser = argparse.ArgumentParser(description="Run evals against the honeypot")
    parser.add_argument("--runs", type=int, default=DEFAULT_RUNS)
    parser.add_argument("--budget", type=int, default=DEFAULT_BUDGET)
    parser.add_argument("--report", type=str, default=DEFAULT_REPORT_PATH)
    args = parser.parse_args()

    report = asyncio.run(run_evals(args.runs, args.budget))

    Path(args.report).write_text(report.model_dump_json(indent=2))
    _print_summary(report)

    if report.passed:
        print("\nAll thresholds met.")
    else:
        print("\nSome thresholds not met.")
        raise SystemExit(1)


async def run_evals(num_runs: int, budget: int) -> EvalReport:
    ...


async def run_single_audit(budget: int) -> AuditReport:
    llm = AnthropicLLM()
    async with StdioMCPClient.connect(HONEYPOT_COMMAND, HONEYPOT_ARGS) as mcp_client:
        graph = build_graph(llm, mcp_client)
        result = await graph.ainvoke({"target": f"{HONEYPOT_COMMAND} {' '.join(HONEYPOT_ARGS)}", "test_budget": budget})
        return result["audit_report"]
```

Console output during execution:

```
Running eval 1/3...
  get_user: 10 cases, 4 categories covered
  execute_query: 10 cases, 5 categories covered
  list_items: 10 cases, 5 categories covered
  Recall: 0.80 | Precision: 1.00

Running eval 2/3...
  ...

Running eval 3/3...
  ...

--- Results ---
Recall:      0.80 (threshold: 0.80) PASS
Precision:   1.00 (threshold: 1.00) PASS
Consistency: 0.87 (threshold: 0.70) PASS
Distribution: 0.93 (threshold: 0.80) PASS

Report written to evals/eval_report.json
All thresholds met.
```

### `tests/unit/test_eval_metrics.py`

Unit tests for the pure metric functions. These use inline fixtures (no given/then extraction — the test file is small and self-contained).

Tests:

- `test_recall_all_detected` — all ground truth FAILs are detected -> 1.0
- `test_recall_partial` — 3 of 5 FAILs detected -> 0.6
- `test_recall_none_detected` — no FAILs detected -> 0.0
- `test_recall_uncovered_counts_as_miss` — a FAIL pair with None verdict -> miss
- `test_precision_no_false_positives` — all predicted FAILs match ground truth -> 1.0
- `test_precision_with_false_positive` — one FAIL on a PASS pair -> < 1.0
- `test_precision_no_predictions` — no FAILs predicted -> 1.0 (no false positives possible)
- `test_consistency_perfect` — all runs agree -> 1.0
- `test_consistency_mixed` — some disagreement -> expected rate
- `test_distribution_full_coverage` — all categories covered -> 1.0
- `test_distribution_partial` — 3 of 5 categories -> 0.6
- `test_aggregate_verdicts_worst_case` — multiple results for same pair, one FAIL -> FAIL
- `test_aggregate_verdicts_all_pass` — multiple results for same pair, all PASS -> PASS
- `test_aggregate_verdicts_missing_pair` — no results for a pair -> None

## Files to modify

### `.gitignore`

Add `evals/eval_report.json` — eval results are local, not committed. They vary between machines and API versions.

### `evals/.gitkeep`

Remove — no longer needed once real files exist in `evals/`.

## What stays unchanged

- `src/mcp_auditor/` — no production code changes. The eval infrastructure is a consumer of the existing API.
- `tests/unit/` and `tests/integration/` — existing tests unchanged.
- `tests/dummy_server.py` — honeypot unchanged.
- `graph/prompts.py` — prompt iteration happens after the infrastructure is in place, as a separate step.

## Edge cases

- **LLM generates no test cases for a category**: that `(tool, category)` pair has None verdict. If ground truth expects FAIL, it's a recall miss. If ground truth expects PASS, it doesn't affect precision (we can't confirm PASS without evidence).
- **MCP server crashes mid-run**: the runner catches the error, marks that run as failed, continues with remaining runs. At least 1 successful run is required to produce metrics.
- **Rate limiting across runs**: each run creates a fresh `AnthropicLLM` instance. The adapter's built-in retry/backoff handles transient 429s. No artificial delay between runs.
- **Empty tool_reports in audit result**: possible if the server has tools but all fail to audit. Treated as zero coverage, zero recall.

## Verification

```bash
# Unit tests for metric functions
uv run pytest tests/unit/test_eval_metrics.py -v

# Full test suite still passes
uv run pytest tests/unit -v

# Type check
uv run pyright evals/ tests/unit/test_eval_metrics.py

# Lint
uv run ruff check evals/ tests/unit/test_eval_metrics.py

# Run evals (requires ANTHROPIC_API_KEY, costs ~$1-3 per run)
uv run python evals/run_evals.py --runs 3 --budget 10
uv run python evals/run_evals.py --runs 1 --budget 5  # quick smoke test
```

## Implementation steps

### Step 1: Ground truth, metric functions, and unit tests

**Files**:
- `tests/unit/test_eval_metrics.py` (create)
- `evals/__init__.py` (create)
- `evals/ground_truth.py` (create)
- `evals/metrics.py` (create)

**Do**:

1. Create `evals/__init__.py` -- empty file, makes `evals` importable as a package.

2. Create `evals/ground_truth.py` with:
   - Type alias `GroundTruth = dict[tuple[str, AuditCategory], EvalVerdict]`
   - `HONEYPOT_GROUND_TRUTH` constant with all 15 `(tool, category) -> verdict` entries as specified in the "Ground truth definition" section above.

3. Create `tests/unit/test_eval_metrics.py` with all 14 test functions listed in the plan's "Tests" section. Use inline fixtures -- build `AuditReport`, `ToolReport`, `EvalResult` directly in each test. Keep helpers minimal (at most a `_make_result(tool, category, verdict)` shorthand at the top of the file). All tests are sync (no async). No given/then extraction -- the file is self-contained.

4. Run `uv run pytest tests/unit/test_eval_metrics.py -v` -- confirm all 14 tests fail (test-first: no implementation yet).

5. Create `evals/metrics.py` with:
   - Type alias: `VerdictMap = dict[tuple[str, AuditCategory], EvalVerdict | None]` -- named because it flows through every metric function.
   - Pydantic models: `ToolVerdictDetail`, `ToolDistribution`, `RunDetail`, `ConsistencyDetail`, `EvalMetrics`, `RunMetrics`, `EvalReport` -- as specified in the "Files to create" section. These give a typed contract to the eval report structure.
   - Five public functions (in this order, per newspaper rule -- `aggregate_verdicts` is called first by the runner, then the compute functions consume its output):
     - `aggregate_verdicts(report: AuditReport) -> VerdictMap`: iterate over `report.tool_reports`, for each tool report iterate over `results`, group by `(result.tool_name, result.category)`. For each group: FAIL if any verdict is FAIL, PASS if all are PASS. Return dict. Pairs not present in any result get no entry (caller uses `.get(key)` which returns None).
     - `compute_recall(aggregated: VerdictMap, ground_truth: GroundTruth) -> float`: count ground truth pairs where expected is FAIL. Of those, count how many have aggregated verdict FAIL. Return detected/expected. If no expected FAILs, return 1.0.
     - `compute_precision(aggregated: VerdictMap, ground_truth: GroundTruth) -> float`: count aggregated pairs where verdict is FAIL. Of those, count how many have ground truth FAIL. Return correct/predicted. If no predicted FAILs, return 1.0.
     - `compute_consistency(all_verdicts: list[VerdictMap]) -> float`: collect all unique `(tool, category)` keys across all runs. For each key, count how many runs have FAIL, how many have PASS (treat None as a distinct "uncovered" -- doesn't count toward either). Agreement for that key = `max(fail_count, pass_count) / (fail_count + pass_count)` (skip if both are 0 -- uncovered across all runs). Return average agreement across all keys that have at least one covered run.
     - `compute_distribution_coverage(report: AuditReport, categories: list[AuditCategory]) -> dict[str, float]`: for each tool report, collect the set of categories present in its results. Coverage = `len(covered_categories) / len(categories)`. Return `{tool_name: coverage}`.

6. Run `uv run pytest tests/unit/test_eval_metrics.py -v` -- all 14 tests pass.

**Test**: 14 test scenarios covering all metric functions:
- `test_recall_all_detected` -- all ground truth FAILs detected -> 1.0
- `test_recall_partial` -- 3 of 5 FAILs detected -> 0.6
- `test_recall_none_detected` -- no FAILs detected -> 0.0
- `test_recall_uncovered_counts_as_miss` -- a FAIL pair with None verdict -> miss
- `test_precision_no_false_positives` -- all predicted FAILs match ground truth -> 1.0
- `test_precision_with_false_positive` -- one FAIL on a PASS pair -> < 1.0
- `test_precision_no_predictions` -- no FAILs predicted -> 1.0
- `test_consistency_perfect` -- all runs agree -> 1.0
- `test_consistency_mixed` -- some disagreement -> expected rate
- `test_distribution_full_coverage` -- all categories covered -> 1.0
- `test_distribution_partial` -- 3 of 5 categories -> 0.6
- `test_aggregate_verdicts_worst_case` -- multiple results for same pair, one FAIL -> FAIL
- `test_aggregate_verdicts_all_pass` -- multiple results for same pair, all PASS -> PASS
- `test_aggregate_verdicts_missing_pair` -- no results for a pair -> None

**Verify**:
```bash
uv run pytest tests/unit -v
uv run ruff check evals/ tests/unit/test_eval_metrics.py
uv run pyright evals/ tests/unit/test_eval_metrics.py
```

---

### Step 2: Eval runner and gitignore update

**Files**:
- `evals/run_evals.py` (create)
- `.gitignore` (modify -- add `evals/eval_report.json`)
- `evals/.gitkeep` (delete)

**Do**:

1. Create `evals/run_evals.py` with the following structure (newspaper rule -- public/high-level first):
   - Constants: `HONEYPOT_SERVER`, `HONEYPOT_COMMAND`, `HONEYPOT_ARGS`, `DEFAULT_RUNS`, `DEFAULT_BUDGET`, `DEFAULT_REPORT_PATH`, `THRESHOLDS` (with the four default thresholds from the plan).
   - `def main()`: parses args via `argparse` (`--runs` default 3, `--budget` default 10, `--report` default `evals/eval_report.json`), calls `asyncio.run(run_evals(...))`, writes JSON report to file via `report.model_dump_json(indent=2)`, prints summary, exits with code 1 if `not report.passed`.
   - `async def run_evals(num_runs: int, budget: int) -> EvalReport`: loops N times calling `run_single_audit`. For each successful run, computes `aggregate_verdicts`, per-run recall/precision/distribution. Catches exceptions per run, prints warning, continues. Requires at least 1 successful run (exits 1 if all fail). After all runs, computes consistency across runs. Assembles an `EvalReport` instance matching the model definition in `metrics.py`. Sets `passed = True` if all aggregate metrics meet thresholds.
   - `async def run_single_audit(budget: int) -> AuditReport`: creates `AnthropicLLM()`, connects `StdioMCPClient` to the honeypot, builds graph (no checkpointer), invokes with `{"target": f"{HONEYPOT_COMMAND} {' '.join(HONEYPOT_ARGS)}", "test_budget": budget}`, returns `result["audit_report"]`.
   - `def _print_summary(report: EvalReport) -> None`: prints human-readable summary to stdout as shown in the plan's console output example. Reads `report.metrics`, `report.thresholds`, `report.passed`.
   - `def _print_run_summary(run_index: int, num_runs: int, run_detail: RunDetail) -> None`: prints per-run progress (tool case counts, categories covered, recall, precision).
   - Top-level `if __name__ == "__main__": main()`.

2. Add `evals/eval_report.json` to `.gitignore` under the `# mcp-auditor` section.

3. Delete `evals/.gitkeep`.

**Test**: No automated tests -- the runner requires a real LLM and API key. Manual verification:
```bash
uv run python evals/run_evals.py --runs 1 --budget 5   # quick smoke test
uv run python evals/run_evals.py --runs 3 --budget 10  # full eval
```

**Verify**:
```bash
uv run pytest tests/unit -v
uv run ruff check evals/run_evals.py
uv run pyright evals/run_evals.py
```
