# llmdebug evals

This folder provides a local A/B harness to measure whether llmdebug snapshots
improve automated debugging outcomes across a deterministic synthetic suite.

- **A (`traceback_only`)**: pytest output only
- **B (`with_snapshot`)**: pytest output + `.llmdebug/latest.*`
- **C (`with_snapshot_structured`)**: same snapshot signal as B, but evidence-first packaging and
  delta-first retry protocol

Patcher backends:
- `command`: local command that returns a unified diff
- `openai`: OpenAI-compatible `/v1/chat/completions`
- `heuristic`: demo snapshot-aware patcher
- `null`: no-op patcher (smoke testing)

Artifacts are written to `evals/artifacts/` and results to `evals/results/*.jsonl`.

The suite is signal-focused:
- `snapshot_dependency=required`: bugs where runtime state should materially help
- `snapshot_dependency=helpful`: snapshots may help, but traceback may still suffice
- `snapshot_dependency=none`: traceback-control cases

## Case contract

Each `evals/cases/<case_id>/` must contain:
- `buggy.py` (or small module set),
- `test_buggy.py`,
- `case.json`,
- `oracle.patch` (required; used by validator).

`case.json` schema:

```json
{
  "description": "short summary",
  "category": "hidden_state|mutability_aliasing|async_temporal|cross_file_contract|data_shape_runtime|parse_value|path_import_env|traceback_controls|syntax_error|reference_error|logic_error|type_coercion",
  "difficulty": "easy|medium|hard",
  "snapshot_dependency": "required|helpful|none",
  "expected_exception": "IndexError",
  "tags": ["multifile", "stateful"],
  "python_args": ["-m", "pytest", "-q"],
  "timeout_sec": 30,
  "bug_source": "hand_crafted|debugbench|humanevalpack|condefects|mutmut",
  "contamination_risk": "none|low|medium|high",
  "expected_fix_lines": 3,
  "source_id": "debugbench_42",
  "difficulty_rationale": "optional explanation"
}
```

The last five fields (`bug_source` through `difficulty_rationale`) are optional
and used by imported cases for traceability.

### Categories

| Category | Description |
|----------|-------------|
| `hidden_state` | Bugs involving mutable state, caching, stale values |
| `mutability_aliasing` | Reference/aliasing issues, shared mutable objects |
| `async_temporal` | Async/await ordering, race conditions, cancellation |
| `cross_file_contract` | Multi-file contract violations, API misuse |
| `data_shape_runtime` | Array shape/dimension mismatches, wrong dtypes |
| `parse_value` | Parsing or value extraction bugs |
| `path_import_env` | Path/import/environment configuration issues |
| `traceback_controls` | Control cases where traceback alone suffices |
| `syntax_error` | Missing colons, wrong indentation, operator misuse |
| `reference_error` | Undefined variables, wrong scope, missing attributes |
| `logic_error` | Wrong comparisons, off-by-one, incorrect conditions |
| `type_coercion` | Type mismatch, wrong cast, implicit conversion bugs |

### Contamination risk

Imported cases carry a `contamination_risk` field indicating how likely the
bug-fix pair appears in LLM training data:

| Level | Description | Example sources |
|-------|-------------|-----------------|
| `none` | Freshly generated, impossible to memorize | hand-crafted, mutmut mutations |
| `low` | Post-training-cutoff or GPT-4-generated bugs | DebugBench, ConDefects |
| `medium` | Pre-cutoff but not widely published | Older benchmarks |
| `high` | Extremely well-known benchmark | HumanEvalPack |

## Validate cases (required before benchmarks)

```bash
uv run python -m evals.validate_cases --schema-only
```

`evals.validate_cases` also supports the same execution flags as `run_eval`
(`--executor`, `--container-image`, `--container-*`, `--container-env`, `--jobs`).
Example local fallback:

```bash
uv run python -m evals.validate_cases --executor local
```

Useful filters:

```bash
uv run python -m evals.validate_cases --category hidden_state --difficulty easy --snapshot-dependency required --max-cases 5
```

Full executable integrity validation (baseline fail + oracle pass) is available for
later runs:

```bash
uv run python -m evals.validate_cases
```

## Current case inventory

<!-- BEGIN AUTO-GENERATED: CASE_INVENTORY -->
_This section is auto-generated from `evals/cases` by `python -m evals.doc_sync --write`._

1664 validated cases across 12 categories and 3 sources:

| Source | Cases |
|--------|-------|
| ConDefects | 1267 |
| DebugBench | 328 |
| Hand-crafted | 69 |
| **Total** | **1664** |

Category distribution:

| Category | Count | Difficulty breakdown |
|----------|-------|---------------------|
| logic_error | 1204 | 589 easy, 302 medium, 313 hard |
| type_coercion | 294 | 198 easy, 47 medium, 49 hard |
| reference_error | 79 | 30 easy, 36 medium, 13 hard |
| async_temporal | 30 | 7 easy, 16 medium, 7 hard |
| syntax_error | 26 | 16 easy, 6 medium, 4 hard |
| data_shape_runtime | 6 | 3 easy, 2 medium, 1 hard |
| parse_value | 5 | 3 easy, 2 medium |
| cross_file_contract | 4 | 1 easy, 2 medium, 1 hard |
| hidden_state | 4 | 2 easy, 2 medium |
| mutability_aliasing | 4 | 2 easy, 1 medium, 1 hard |
| path_import_env | 4 | 3 easy, 1 medium |
| traceback_controls | 4 | 4 easy |
<!-- END AUTO-GENERATED: CASE_INVENTORY -->

## Import external cases

Install the `evals` extra for HuggingFace dataset access:

```bash
pip install llmdebug[evals]
# or: pip install datasets>=2.0
```

### DebugBench (contamination: low)

Imports Python cases from `Rtian/DebugBench` on HuggingFace. Filters by code length,
fix size, determinism, and stdlib-only imports. Generates tests from LeetCode examples.
~34% of imported cases survive full execution validation.

```bash
# Full import (unlimited cases, auto-validate)
uv run python -m evals.importers.cli debugbench --validate

# Preview first 5 cases without writing
uv run python -m evals.importers.cli debugbench --max-cases 5 --dry-run
```

When using `--validate`, importers honor the same execution flags as `run_eval`
and `validate_cases` (`--executor`, container resource/network flags, and `--container-env`).

### HumanEvalPack (~20 cases, contamination: high)

```bash
uv run python -m evals.importers.cli humanevalpack --max-cases 20 --validate
```

### ConDefects (contamination: low)

Requires cloning the ConDefects repo and downloading test data separately.

```bash
git clone https://github.com/appmlk/ConDefects.git
# Download Test.zip from OneDrive and extract into ConDefects/
uv run python -m evals.importers.cli condefects --repo-path ./ConDefects --validate
```

### mutmut mutations (contamination: none)

Generates zero-contamination cases by mutating llmdebug source code itself.
Only keeps killed mutants (where tests actually detect the bug).

```bash
pip install mutmut
uv run python -m evals.importers.cli mutmut --target src/llmdebug/capture.py --max-cases 10
uv run python -m evals.importers.cli mutmut --target src/llmdebug/serialize.py --max-cases 10
```

## Benchmark runs

### Containerized test execution (default)

`evals.run_eval`, `evals.validate_cases`, and `evals.importers.cli --validate`
use `--executor testcontainers` by default.

Build the default runner image once:

```bash
docker build -f evals/docker/runner.Dockerfile -t llmdebug-eval-runner:latest .
```

Important execution flags:
- `--executor testcontainers|local` (default: `testcontainers`)
- `--jobs <int>` (default: `1`)
- `--container-image <image>` (default: `llmdebug-eval-runner:latest`)
- `--container-network none|bridge` (default: `none`)
- `--container-cpus <float>`, `--container-memory <str>`, `--container-pids-limit <int>`
- `--container-user <uid:gid>` (default: host `uid:gid` when available; fallback `0:0`)
- `--container-tmpfs-size-mb <int>` (default: `128`)
- `--container-env KEY=VALUE` (repeatable, explicit allowlist)
- stale eval containers are pruned automatically at run start (best-effort, 5-minute grace)

Use `--executor local` if you want the legacy host-side pytest execution.

Quick sanity (small subset):

```bash
uv run python -m evals.run_eval \
  --patcher openai \
  --openai-base-url http://127.0.0.1:1234 \
  --openai-model deepseek/deepseek-r1-0528-qwen3-8b \
  --openai-response-mode edits \
  --jobs 2 \
  --no-files-context \
  --k 1 \
  --max-cases 5
```

Full suite (all discovered cases in `evals/cases`):

```bash
uv run python -m evals.run_eval \
  --patcher openai \
  --openai-base-url http://127.0.0.1:1234 \
  --openai-model deepseek/deepseek-r1-0528-qwen3-8b \
  --openai-response-mode edits \
  --no-files-context \
  --k 1
```

Filtered run (category/difficulty/snapshot dependency/bug source):

```bash
uv run python -m evals.run_eval \
  --patcher openai \
  --openai-base-url http://127.0.0.1:1234 \
  --openai-model deepseek/deepseek-r1-0528-qwen3-8b \
  --category async_temporal \
  --difficulty hard \
  --snapshot-dependency required \
  --k 3

# Filter by bug source
uv run python -m evals.run_eval \
  --patcher openai \
  --openai-base-url http://127.0.0.1:1234 \
  --openai-model deepseek/deepseek-r1-0528-qwen3-8b \
  --bug-source hand_crafted \
  --k 1
```

`--openai-response-mode edits` is recommended for local reasoning models.
`--no-files-context` is useful when prompts are too long.
Runner-level forced patch attempts are enabled by default to reduce `no_patch_proposed`
and `patch_apply_failed` records; disable with `--no-force-patch-attempt`.
For OpenAI patcher responses specifically, `--openai-force-patch-attempt` can further
force a minimal patch diff when model output is unparsable.

### Progress output and safe resume

`evals.run_eval` now prints per-unit progress lines while running:
- `START <case>/<condition>` before each unit,
- `DONE <case>/<condition>` with `status`, unit time, elapsed time, and ETA,
- `SKIP (resumed) <case>/<condition>` for units already completed in a resumed run.

Use `--run-id` to make runs resumable:

```bash
# Start a named run
uv run python -m evals.run_eval \
  --run-id z-ai-glm45air-main \
  --patcher openai \
  --openai-base-url https://api.z.ai/api/anthropic \
  --openai-model glm-4.5-air \
  --k 1

# Resume later with the exact same settings + filters
uv run python -m evals.run_eval \
  --run-id z-ai-glm45air-main \
  --patcher openai \
  --openai-base-url https://api.z.ai/api/anthropic \
  --openai-model glm-4.5-air \
  --k 1
```

Resume safety is strict: if an existing `run_id` was created with different
run-affecting settings (model, conditions, `k`, context protocol, case selection, etc.),
the runner aborts instead of mixing incomparable records.

To compare retry strategies cleanly, run separate `run_id`s with identical settings except
`--retry-context-mode`:

```bash
# Stateless retries (new context each retry)
uv run python -m evals.run_eval \
  --run-id z-ai-glm45air-fresh \
  --patcher openai \
  --openai-base-url https://api.z.ai/api/anthropic \
  --openai-model glm-4.5-air \
  --retry-context-mode fresh

# Conversational retries (carry summarized history)
uv run python -m evals.run_eval \
  --run-id z-ai-glm45air-history \
  --patcher openai \
  --openai-base-url https://api.z.ai/api/anthropic \
  --openai-model glm-4.5-air \
  --retry-context-mode history
```

### Reproducibility controls

Use deterministic mode when you need reproducible patch-generation runs:

```bash
uv run python -m evals.run_eval \
  --patcher openai \
  --openai-base-url http://127.0.0.1:1234 \
  --openai-model deepseek/deepseek-r1-0528-qwen3-8b \
  --deterministic \
  --seed 12345 \
  --record-prompt hash \
  --record-evidence hash \
  --record-deps minimal
```

Key flags:
- `--deterministic --seed <int>`: enforce deterministic mode; fails fast if OpenAI-compatible backend does not acknowledge seed support.
- `--record-prompt full|hash|none`: controls prompt persistence in `attempt_metadata`.
- `--record-evidence full|hash|none`: controls evidence persistence in `attempt_metadata`.
- `--record-env KEY` (repeatable): allowlist environment keys into `run_metadata`.
- `--record-deps none|minimal|runtime`: dependency version capture scope.
- `--jobs <int>`: number of units to execute concurrently (default `1`).
- `--executor testcontainers|local`: test executor backend (default: `testcontainers`).
- `--container-image <image>`: image used by `testcontainers` executor.
- `--container-network none|bridge`: container network mode (`none` is default and recommended).
- `--container-mount-repo-src` / `--container-no-mount-repo-src`: mount local `src/` for plugin imports.
- `--container-env KEY=VALUE`: explicit env allowlist passed into container (repeatable).
- stale eval containers are pruned automatically at run start (best-effort, 5-minute grace).
- `--openai-api-key <str>`: API key for OpenAI-compatible patcher; defaults to `Z_API`, then `OPENAI_API_KEY`.
- `--context-protocol monolithic|staged`: prompt protocol. `staged` adds a Stage-A evidence request pass before patch generation.
- `--context-stage-a-max-requests <int>`: maximum Stage-A retrieval requests (default 6).
- `--context-stage-a-max-chars <int>`: maximum Stage-A returned context size in characters (default 120000).
- `--retry-prompt-mode full|delta|auto`: retry packaging mode (`auto` defaults to `delta` for `with_snapshot_structured`).
- `--retry-context-mode fresh|history`: retry context policy (`history` includes prior attempt transcript; default `history`).
- `--retry-history-max-chars <int>`: max characters for retry-history transcript (default 48000).
- `--prompt-budget-total-chars <int>`: Stage-B prompt total character budget (0 disables cap).
- `--prompt-budget-retry-delta-chars <int>`: retry packet per-section cap.
- `--prompt-budget-stdout-chars <int>`: pytest output per-section cap.
- `--prompt-budget-snapshot-chars <int>`: snapshot section cap.
- `--prompt-budget-evidence-chars <int>`: Stage-A evidence section cap.
- `--prompt-budget-retry-history-chars <int>`: retry-history section cap (default 12000).
- `--conditions traceback_only,with_snapshot,with_snapshot_structured`: run explicit A/B/C comparison
  (default remains `traceback_only,with_snapshot` for continuity).

## Analyze and report uplift

```bash
uv run python -m evals.analyze_results evals/results/*.jsonl --out evals/results/summary.json
```

`evals.analyze_results` now enforces strict schema validation (`schema_version=3`) for
records. Older JSONL outputs without required metadata fields will be rejected and should
be regenerated with the current harness.

The analyzer reports:
- overall metrics,
- per-category metrics,
- per-difficulty metrics,
- per-snapshot-dependency metrics,
- per-bug-source metrics,
- per-contamination-risk metrics,
- explicit `uplift_success_rate = with_snapshot - traceback_only`,
- `uplift_structured_vs_snapshot`,
- `uplift_structured_vs_traceback`,
- pairwise condition uplift matrix (`pairwise_uplifts`) across all discovered conditions,
- `retry_gate_block_rate`,
- prompt-size metrics including `mean_prompt_chars_stage_b` and per-condition means,
- retry/evidence packaging telemetry (`retry_prompt_mode`, evidence-ID counts, prompt-budget fields),
- attempt-efficiency telemetry (`mean_attempts_budget`, `mean_attempt_ratio_when_successful`,
  `solve_on_attempt_distribution`),
- token telemetry (`mean_tokens_prompt`, `mean_tokens_completion`, `mean_tokens_total`,
  `mean_tokens_per_attempt`, `mean_llm_calls_per_case`),
- retry-history telemetry (`mean_retry_history_chars`, `mean_retry_history_turns`,
  `retry_history_budget_applied_rate`, `retry_history_truncation_rate`).

### Statistical analysis

Add `--stats` for rigorous significance testing:

```bash
uv run python -m evals.analyze_results evals/results/*.jsonl --stats --out evals/results/summary.json

# Evaluate structured packaging directly against traceback-only:
uv run python -m evals.analyze_results \
  evals/results/*.jsonl \
  --stats \
  --stats-pair-a traceback_only \
  --stats-pair-b with_snapshot_structured
```

The `--stats` flag adds:
- **McNemar's test**: Paired binary significance test (exact binomial for <25 discordant pairs, chi-squared otherwise).
- **Cohen's h**: Effect size for proportions (small >= 0.2, medium >= 0.5, large >= 0.8).
- **Bootstrap CI**: 95% confidence interval for uplift via paired bootstrap (10k resamples).
- **Holm-Bonferroni**: Multiple comparison correction for per-category p-values.
- **Power report**: Post-hoc power assessment to gauge result trustworthiness.
- **Pairwise condition matrix**: always reports pairwise A/B summaries for all conditions in the input.

For multi-run (k>1) analysis, use `--aggregation`:

```bash
# Majority vote: case succeeds if >50% of runs succeed
uv run python -m evals.analyze_results evals/results/*.jsonl --stats --aggregation majority_vote

# Any success: case succeeds if any run succeeds
uv run python -m evals.analyze_results evals/results/*.jsonl --stats --aggregation any_success
```

### Sample size guidance

| Cases | Discordant pairs (est.) | Power (uplift=0.2) | Recommendation |
|-------|------------------------|---------------------|----------------|
| 30 | ~10 | ~0.30 | Directional signal only |
| 60 | ~20 | ~0.55 | Moderate evidence |
| 100 | ~35 | ~0.75 | Near-publication quality |
| 150+ | ~50+ | ~0.90+ | Publication-grade |

Interpretation guidance:
- positive uplift in `required` buckets suggests llmdebug snapshots provide useful signal,
- near-zero uplift in `none` control buckets is expected and acts as a sanity check,
- positive uplift in a category suggests snapshots help that bug class,
- near-zero uplift suggests no clear advantage yet,
- negative uplift suggests prompt or patch quality regressions for that bucket.
