# llmdebug evals

This folder provides a local A/B harness to measure whether llmdebug snapshots
improve automated debugging outcomes across a deterministic synthetic suite.

- **A (`traceback_only`)**: pytest output only
- **B (`with_snapshot`)**: pytest output + `.llmdebug/latest.*`
- **C (`with_snapshot_structured`)**: same snapshot signal as B, but evidence-first packaging and
  delta-first retry protocol
- **D (`adaptive_gated`)**: starts traceback-first and invokes snapshot helper when gate policy
  detects stagnation
- **E (`adaptive_llm_discretion`)**: starts traceback-first and lets the model decide whether to
  invoke helper context

Patcher backends:
- `command`: local command that returns a unified diff
- `openai`: OpenAI-compatible `/v1/chat/completions`
- `heuristic`: demo snapshot-aware patcher
- `null`: no-op patcher (smoke testing)

Command patcher safety defaults:
- command execution uses argv mode by default (`shell=False`)
- explicit shell opt-in is available via `--patch-cmd-shell` or `LLMDEBUG_EVAL_PATCH_CMD_SHELL=1`
- timeout defaults to 30s and can be changed with
  `--patch-cmd-timeout-sec` or `LLMDEBUG_EVAL_PATCH_CMD_TIMEOUT_SEC`
- each command attempt writes the current prompt to
  `$LLMDEBUG_EVAL_PROMPT_FILE` (default: `<context_dir>/prompt.txt`)

Artifacts are written to `evals/artifacts/` and results to `evals/results/*.jsonl`.

## SWE-Bench Lite (Hybrid: HAICore generation + local Docker scoring)

SWE-Bench Lite integration lives under `evals/swebench/` and follows a
two-plane workflow:

1. Generation plane (HAICore + Enroot):
- produces paired predictions for `baseline` and `llmdebug`.
- writes run artifacts under `evals/swebench_runs/<run_id>/`.

2. Scoring plane (Docker machine):
- runs official SWE-Bench evaluator on copied predictions.
- writes raw official outputs under `official_eval/`.

Primary CLI:

```bash
# Generation on HAICore
python -m evals.swebench.cli generate-lite \
  --run-id <id> \
  --conditions baseline,llmdebug \
  --max-instances <n|0> \
  --instance-ids-file <optional> \
  --openai-base-url http://localhost:<port> \
  --openai-model <model_alias> \
  --openai-http2 \
  --openai-async \
  --jobs <workers> \
  --max-request-errors 25 \
  --max-request-error-rate 0.5

# Handoff preparation (validates both conditions + writes handoff metadata)
python -m evals.swebench.cli prepare-handoff --run-id <id>

# Official scoring on Docker host
python -m evals.swebench.cli score-official \
  --run-dir /path/to/evals/swebench_runs/<id> \
  --conditions baseline,llmdebug \
  --workers <n> \
  --cache-level <none|base|env|instance>

# Build a tiny paired scoring subset for smoke checks
python -m evals.swebench.cli prepare-scoring-smoke \
  --run-dir /path/to/evals/swebench_runs/<id> \
  --max-instances 2
```

Local Docker convenience wrapper:

```bash
./scripts/score_swebench_lite_official.sh --run-dir /path/to/evals/swebench_runs/<id>

# Smoke-check official scoring on tiny paired subset
./scripts/smoke_swebench_lite_official.sh --run-dir /path/to/evals/swebench_runs/<id> --max-instances 2
```

Run directory contract:

`evals/swebench_runs/<run_id>/`

- `baseline/predictions.jsonl`
- `llmdebug/predictions.jsonl`
- `instance_ids.txt`
- `run_metadata.json`
- `run_index.json` (written by `prepare-handoff`)
- `COPY_INSTRUCTIONS.txt` (written by `prepare-handoff`)
- `official_eval/` (written during scoring)

Policy notes:

- `baseline` + `llmdebug` are mandatory paired conditions for generation, handoff,
  and scoring (`--conditions` must include both).
- generation fails fast when patch-request errors indicate likely systemic failure
  (all requests failing, absolute error threshold, or high error-rate threshold).
- generation fails fast when the selected SWE-Bench split contains duplicate
  `instance_id` values.

Each prediction row in `predictions.jsonl`:

- `instance_id` (string)
- `model_patch` (string, unified diff)
- `model_name_or_path` (string)

Detailed operational guide:
- `docs/haicore_enroot_swebench_lite_runbook.md`

The suite is signal-focused:
- `snapshot_dependency=required`: bugs where runtime state should materially help
- `snapshot_dependency=helpful`: snapshots may help, but traceback may still suffice
- `snapshot_dependency=none`: traceback-control cases

## Case contract

Each `evals/cases/<case_id>/` must contain:
- `buggy.py` (or small module set),
- `test_buggy.py`,
- `case.json`,
- `oracle.patch` (required; used by validator).

`case.json` schema:

```json
{
  "description": "short summary",
  "category": "hidden_state|mutability_aliasing|async_temporal|cross_file_contract|data_shape_runtime|parse_value|path_import_env|traceback_controls|syntax_error|reference_error|logic_error|type_coercion",
  "difficulty": "easy|medium|hard",
  "snapshot_dependency": "required|helpful|none",
  "expected_exception": "IndexError",
  "tags": ["multifile", "stateful"],
  "python_args": ["-m", "pytest", "-q"],
  "timeout_sec": 30,
  "bug_source": "hand_crafted|debugbench|pyresbugs|humanevalpack|condefects|mutmut|bugsinpy|pybughive|trickybugs",
  "contamination_risk": "none|low|medium|high",
  "expected_fix_lines": 3,
  "source_id": "debugbench_42",
  "difficulty_rationale": "optional explanation"
}
```

The last five fields (`bug_source` through `difficulty_rationale`) are optional
and used by imported cases for traceability.

### Categories

| Category | Description |
|----------|-------------|
| `hidden_state` | Bugs involving mutable state, caching, stale values |
| `mutability_aliasing` | Reference/aliasing issues, shared mutable objects |
| `async_temporal` | Async/await ordering, race conditions, cancellation |
| `cross_file_contract` | Multi-file contract violations, API misuse |
| `data_shape_runtime` | Array shape/dimension mismatches, wrong dtypes |
| `parse_value` | Parsing or value extraction bugs |
| `path_import_env` | Path/import/environment configuration issues |
| `traceback_controls` | Control cases where traceback alone suffices |
| `syntax_error` | Missing colons, wrong indentation, operator misuse |
| `reference_error` | Undefined variables, wrong scope, missing attributes |
| `logic_error` | Wrong comparisons, off-by-one, incorrect conditions |
| `type_coercion` | Type mismatch, wrong cast, implicit conversion bugs |

### Contamination risk

Imported cases carry a `contamination_risk` field indicating how likely the
bug-fix pair appears in LLM training data:

| Level | Description | Example sources |
|-------|-------------|-----------------|
| `none` | Freshly generated, impossible to memorize | hand-crafted, mutmut mutations |
| `low` | Post-training-cutoff or GPT-4-generated bugs | DebugBench, ConDefects |
| `medium` | Pre-cutoff but not widely published | PyResBugs, BugsInPy, PyBugHive |
| `high` | Extremely well-known benchmark | HumanEvalPack, TrickyBugs |

## Validate cases (required before benchmarks)

```bash
uv run python -m evals.validate_cases --schema-only
```

`evals.validate_cases` also supports the same execution flags as `run_eval`
(`--executor`, `--container-image`, `--container-*`, `--container-env`, `--jobs`).
Example local fallback:

```bash
uv run python -m evals.validate_cases --executor local
```

Useful filters:

```bash
uv run python -m evals.validate_cases --category hidden_state --difficulty easy --snapshot-dependency required --max-cases 5
```

Full executable integrity validation (baseline fail + oracle pass) is available for
later runs:

```bash
uv run python -m evals.validate_cases
```

## Current case inventory

<!-- BEGIN AUTO-GENERATED: CASE_INVENTORY -->
_This section is auto-generated from `evals/cases` by `python -m evals.doc_sync --write`._

1664 validated cases across 12 categories and 3 sources:

| Source | Cases |
|--------|-------|
| ConDefects | 1267 |
| DebugBench | 328 |
| Hand-crafted | 69 |
| **Total** | **1664** |

Category distribution:

| Category | Count | Difficulty breakdown |
|----------|-------|---------------------|
| logic_error | 1204 | 589 easy, 302 medium, 313 hard |
| type_coercion | 294 | 198 easy, 47 medium, 49 hard |
| reference_error | 79 | 30 easy, 36 medium, 13 hard |
| async_temporal | 30 | 7 easy, 16 medium, 7 hard |
| syntax_error | 26 | 16 easy, 6 medium, 4 hard |
| data_shape_runtime | 6 | 3 easy, 2 medium, 1 hard |
| parse_value | 5 | 3 easy, 2 medium |
| cross_file_contract | 4 | 1 easy, 2 medium, 1 hard |
| hidden_state | 4 | 2 easy, 2 medium |
| mutability_aliasing | 4 | 2 easy, 1 medium, 1 hard |
| path_import_env | 4 | 3 easy, 1 medium |
| traceback_controls | 4 | 4 easy |
<!-- END AUTO-GENERATED: CASE_INVENTORY -->

## Import external cases

Install the `evals` extra for HuggingFace dataset access:

```bash
pip install llmdebug[evals]
# or: pip install datasets>=2.0
```

### DebugBench (contamination: low)

Imports Python cases from `Rtian/DebugBench` on HuggingFace. Filters by code length,
fix size, determinism, and stdlib-only imports. Generates tests from LeetCode examples.
~34% of imported cases survive full execution validation.

```bash
# Full import (unlimited cases, auto-validate)
uv run python -m evals.importers.cli debugbench --validate

# Preview first 5 cases without writing
uv run python -m evals.importers.cli debugbench --max-cases 5 --dry-run
```

When using `--validate`, importers honor the same execution flags as `run_eval`
and `validate_cases` (`--executor`, container resource/network flags, and `--container-env`).

### HumanEvalPack (~20 cases, contamination: high)

```bash
uv run python -m evals.importers.cli humanevalpack --max-cases 20 --validate
```

### ConDefects (contamination: low)

Requires cloning the ConDefects repo and downloading test data separately.

```bash
git clone https://github.com/appmlk/ConDefects.git
# Download Test.zip from OneDrive and extract into ConDefects/
uv run python -m evals.importers.cli condefects --repo-path ./ConDefects --validate
```

### mutmut mutations (contamination: none)

Generates zero-contamination cases by mutating llmdebug source code itself.
Only keeps killed mutants (where tests actually detect the bug).

```bash
pip install mutmut
uv run python -m evals.importers.cli mutmut --target src/llmdebug/capture.py --max-cases 10
uv run python -m evals.importers.cli mutmut --target src/llmdebug/serialize.py --max-cases 10
```

### Deferred-download top 3 (BugsInPy, PyBugHive, TrickyBugs)

Use a two-step workflow so local preparation does not pull large artifacts.

1. Prepare a lock manifest (safe for low-storage laptops):

```bash
uv run python -m evals.importers.cli prepare-top3 \
  --manifest evals/manifests/top3.lock.json
```

2. Materialize on a server with enough disk just before evaluation:

```bash
uv run python -m evals.importers.cli materialize-top3 \
  --manifest evals/manifests/top3.lock.json \
  --data-root /mnt/evals/top3 \
  --download-policy missing \
  --jobs 4 \
  --validate
```

`materialize-top3` writes a structured import report to
`evals/artifacts/import_reports/top3-<timestamp>.json` with attempted/imported/skipped
counts, deterministic skip reasons, and hard-failure flags per source.

Download policies:
- `never`: do not download; fail/skip if required artifacts are missing locally.
- `missing`: download only missing artifacts; existing files are integrity-verified and
  re-downloaded once if verification fails (recommended default).
- `refresh`: re-download all pinned artifacts.

Strict reproducibility semantics (top3 only):
- native dataset test commands only (`repro_mode=native_dataset_test`).
- no synthetic marker-test conversion fallback.
- imported metadata includes `source_test_command`, `source_test_selector`,
  `source_buggy_commit`, and `source_fixed_commit`.
- non-convertible entries are skipped with explicit, stable reasons.

PyBugHive policy:
- offline dumps are mandatory and authoritative.
- full offline archive list is pinned in `prepare-top3` manifest metadata.
- required artifacts must carry integrity pins (`size_bytes` + `checksum`); legacy
  manifests without required integrity pins hard-fail deterministically.
- if mandatory PyBugHive artifacts fail availability/integrity checks, materialization
  still writes a report but exits non-zero.

Manual pre-step before `run_eval` (phase 1):

```bash
uv run python -m evals.importers.cli materialize-top3 \
  --manifest evals/manifests/top3.lock.json \
  --data-root /mnt/evals/top3 \
  --download-policy missing \
  --validate
uv run python -m evals.run_eval --bug-source bugsinpy --bug-source pybughive --bug-source trickybugs
```

Storage guidance (compressed inputs, approximate):
- BugsInPy metadata archive: MB scale.
- PyBugHive offline dump: tens of GiB.
- TrickyBugs archive: ~1.5 GiB.

## Benchmark runs

### Containerized test execution (default)

`evals.run_eval`, `evals.validate_cases`, and `evals.importers.cli --validate`
use `--executor testcontainers` by default.

Build the default runner image once:

```bash
docker build -f evals/docker/runner.Dockerfile -t llmdebug-eval-runner:latest .
```

Important execution flags:
- `--executor testcontainers|local` (default: `testcontainers`)
- `--jobs <int>` (default: `1`)
- `--container-image <image>` (default: `llmdebug-eval-runner:latest`)
- `--container-network none|bridge` (default: `none`)
- `--container-cpus <float>`, `--container-memory <str>`, `--container-pids-limit <int>`
- `--container-user <uid:gid>` (default: host `uid:gid` when available; fallback `0:0`)
- `--container-tmpfs-size-mb <int>` (default: `128`)
- `--container-env KEY=VALUE` (repeatable, explicit allowlist)
- stale eval containers are pruned automatically at run start (best-effort, 5-minute grace)

Use `--executor local` if you want the legacy host-side pytest execution.

### Single-container eval runner

For running the whole eval pipeline in one container (without Docker-in-Docker),
use:

```bash
evals/docker/run_eval_docker.sh --patcher null --max-cases 5 --k 1
```

Behavior notes:
- auto-builds `llmdebug-eval:latest` when missing (or if `LLMDEBUG_EVAL_BUILD=1`)
- strips `--executor` and `--container-*` flags, then forces `--executor local`
- for `--patcher openai`, resolves API key precedence as `Z_API`, then
  `OPENAI_API_KEY`, then `LLMDEBUG_EVAL_OPENAI_API_KEY`, and passes it via
  `--openai-api-key` (instead of forwarding raw key env vars)
- rewrites localhost OpenAI URLs on macOS to `host.docker.internal`; on Linux,
  it uses `--network host` when a localhost URL is detected
- rewrites `--config <path>` to a staged path mounted inside the container, so
  host-local config files work out of the box

### TOML config files (`--config`)

Use `--config <file.toml>` to move long argument lists into a versioned config.

Resolution order is:
- defaults
- TOML config values
- explicit CLI flags (highest precedence)

Config format:
- keys can live at the top level or inside `[run_eval]`
- nested tables are supported (`[run_eval.openai]`, `[run_eval.adaptive]`, etc.)
- table keys flatten to CLI destinations (`openai.base_url` -> `openai_base_url`)

Secret policy:
- `openai_api_key` literals are rejected in config files
- use `openai_api_key_env = "OPENAI_API_KEY"` instead

Each run writes a resolved lock file:
- `evals/artifacts/configs/<run_id>.resolved.json`
- contains the final resolved args and redacts secret values

Example:

```bash
RUN_ID="local-qwen-$(date +%Y%m%dT%H%M%S)"
bash evals/docker/run_eval_docker.sh \
  --patcher openai \
  --run-id "$RUN_ID" \
  --config evals/configs/openai_adaptive.template.toml
```

See template:
- `evals/configs/openai_adaptive.template.toml`

Blackwell + vLLM Docker quickstart (known-good):

Detailed runbook and troubleshooting log:
`docs/blackwell_vllm_docker_eval_runbook.md`

```bash
# Terminal 1: launch vLLM (includes CUDA Error 803 workaround)
evals/docker/run_vllm_blackwell.sh
```

```bash
# Terminal 2: smoke run against local vLLM
bash evals/docker/run_eval_docker.sh \
  --patcher openai \
  --openai-base-url http://127.0.0.1:7777/v1 \
  --openai-model qwen3coder30ba3b \
  --openai-response-mode edits \
  --conditions traceback_only,adaptive_llm_discretion \
  --adaptive-max-helper-calls 1 \
  --retry-prompt-mode full \
  --jobs 2 \
  --k 1 \
  --max-cases 5
```

```bash
# Full run aligned with slurm/eval_gguf_eval.sh defaults on HAICore
RUN_ID="local-qwen3coder30ba3b-$(date +%Y%m%dT%H%M%S)"
bash evals/docker/run_eval_docker.sh \
  --patcher openai \
  --openai-base-url http://127.0.0.1:7777/v1 \
  --openai-model qwen3coder30ba3b \
  --openai-api-key local \
  --openai-timeout-sec 180 \
  --openai-max-propose-sec 300 \
  --openai-max-tokens 2048 \
  --openai-temperature 0.0 \
  --max-context-chars 200000 \
  --prompt-budget-total-chars 180000 \
  --conditions traceback_only,adaptive_llm_discretion \
  --adaptive-max-helper-calls 1 \
  --retry-prompt-mode full \
  --jobs 4 \
  --k 3 \
  --run-id "$RUN_ID"
```

If you want continuity with `evals.run_eval` CLI defaults instead, use:
`--conditions traceback_only,with_snapshot`.

Quick sanity (small subset):

```bash
uv run python -m evals.run_eval \
  --patcher openai \
  --openai-base-url http://127.0.0.1:1234 \
  --openai-model deepseek/deepseek-r1-0528-qwen3-8b \
  --openai-response-mode edits \
  --jobs 2 \
  --no-files-context \
  --k 1 \
  --max-cases 5
```

Full suite (all discovered cases in `evals/cases`):

```bash
uv run python -m evals.run_eval \
  --patcher openai \
  --openai-base-url http://127.0.0.1:1234 \
  --openai-model deepseek/deepseek-r1-0528-qwen3-8b \
  --openai-response-mode edits \
  --no-files-context \
  --k 1
```

Filtered run (category/difficulty/snapshot dependency/bug source):

```bash
uv run python -m evals.run_eval \
  --patcher openai \
  --openai-base-url http://127.0.0.1:1234 \
  --openai-model deepseek/deepseek-r1-0528-qwen3-8b \
  --category async_temporal \
  --difficulty hard \
  --snapshot-dependency required \
  --k 3

# Filter by bug source
uv run python -m evals.run_eval \
  --patcher openai \
  --openai-base-url http://127.0.0.1:1234 \
  --openai-model deepseek/deepseek-r1-0528-qwen3-8b \
  --bug-source hand_crafted \
  --k 1
```

`--openai-response-mode edits` is recommended for local reasoning models.
`--no-files-context` is useful when prompts are too long.
Runner-level forced patch attempts are enabled by default to reduce `no_patch_proposed`
and `patch_apply_failed` records; disable with `--no-force-patch-attempt`.
For OpenAI patcher responses specifically, `--openai-force-patch-attempt` can further
force a minimal patch diff when model output is unparsable.
Forced marker patches are automatically skipped on transport/context-limit request failures,
and those attempts are recorded with note `patcher_request_failed`.
When OpenAI-compatible patcher responses return a non-transport no-diff error,
the runner performs one compact retry with source-focused guidance before falling back
to forced-patch handling.

### Progress output and safe resume

`evals.run_eval` prints per-unit progress lines while running:
- `START <case>/<condition>` before each unit,
- `DONE <case>/<condition>` with `status`, unit time, elapsed time, and ETA,
- `SKIP (resumed) <case>/<condition>` for units already completed in a resumed run.

For long-running units, enable richer in-flight visibility:

```bash
uv run python -m evals.run_eval \
  --run-id z-ai-glm45air-main \
  --patcher openai \
  --openai-base-url https://api.z.ai/api/anthropic \
  --openai-model glm-4.5-air \
  --progress-verbosity detailed \
  --progress-heartbeat-sec 20
```

Progress controls:
- `--progress-verbosity basic|detailed|trace` (default `basic`)
- `--progress-heartbeat-sec <float>` (default `30`; `0` disables heartbeat)
- `detailed` adds `[HEARTBEAT]` and per-attempt timing lines while units are in flight
- `trace` additionally prints phase transitions (attempt start, patch propose cycle, retry phase)

Use `--run-id` to make runs resumable:

```bash
# Start a named run
uv run python -m evals.run_eval \
  --run-id z-ai-glm45air-main \
  --patcher openai \
  --openai-base-url https://api.z.ai/api/anthropic \
  --openai-model glm-4.5-air \
  --k 1

# Resume later with the exact same settings + filters
uv run python -m evals.run_eval \
  --run-id z-ai-glm45air-main \
  --patcher openai \
  --openai-base-url https://api.z.ai/api/anthropic \
  --openai-model glm-4.5-air \
  --k 1
```

Resume safety guardrails:
- if an existing `run_id` is missing `run_settings_fingerprint`, resume still aborts
  (cannot validate compatibility safely)
- if an existing `run_id` has a different `run_settings_fingerprint`, the runner now prints
  a warning and continues, reusing completed-unit state from existing records

Crash/recovery safeguards:
- result records are appended incrementally during each unit, so progress survives process crashes
- trailing malformed JSON in the last line of a results file is auto-repaired during resume (file is truncated to the last valid record and a `.corrupt-tail-*.log` sidecar backup is written)
- a per-run lock file (`<run_id>.jsonl.lock`) prevents two writers from appending to the same run simultaneously when OS file-lock support is available; unsupported platforms continue with a warning

To compare retry strategies cleanly, run separate `run_id`s with identical settings except
`--retry-context-mode`:

```bash
# Stateless retries (new context each retry)
uv run python -m evals.run_eval \
  --run-id z-ai-glm45air-fresh \
  --patcher openai \
  --openai-base-url https://api.z.ai/api/anthropic \
  --openai-model glm-4.5-air \
  --retry-context-mode fresh

# Conversational retries (carry summarized history)
uv run python -m evals.run_eval \
  --run-id z-ai-glm45air-history \
  --patcher openai \
  --openai-base-url https://api.z.ai/api/anthropic \
  --openai-model glm-4.5-air \
  --retry-context-mode history
```

### Two-pass targeted rescue workflow

Use this when full adaptive runs are expensive and you want to spend helper/context
budget only on hard cases.

```bash
# Pass 1: traceback-only baseline over a broad set
RUN_ID_BASE="tb-$(date +%Y%m%dT%H%M%S)"
uv run python -m evals.run_eval \
  --run-id "$RUN_ID_BASE" \
  --patcher openai \
  --openai-base-url http://127.0.0.1:1234 \
  --openai-model <model> \
  --conditions traceback_only \
  --k 3
```

```bash
# Build rerun list from terminal traceback_only rows:
# include failed cases only (recommended default protocol).
CASE_LIST="evals/artifacts/case_lists/${RUN_ID_BASE}.failed.txt"
uv run python -m evals.select_cases \
  --results-jsonl "evals/results/${RUN_ID_BASE}.jsonl" \
  --condition traceback_only \
  --mode failed \
  --out "$CASE_LIST"
```

```bash
# Pass 2: adaptive rescue over failed subset only
RUN_ID_RESCUE="rescue-$(date +%Y%m%dT%H%M%S)"
uv run python -m evals.run_eval \
  --run-id "$RUN_ID_RESCUE" \
  --patcher openai \
  --openai-base-url http://127.0.0.1:1234 \
  --openai-model <model> \
  --conditions adaptive_llm_discretion \
  --case-list-file "$CASE_LIST" \
  --adaptive-max-helper-calls 1 \
  --adaptive-budget-topup-max-calls 1 \
  --adaptive-online-policy-mode on \
  --adaptive-online-policy-algo linucb \
  --adaptive-online-state-path "evals/artifacts/policies/${RUN_ID_RESCUE}.json" \
  --adaptive-online-events-path "evals/results/${RUN_ID_RESCUE}.adaptive_policy.jsonl" \
  --k 3
```

Equivalent `just` shortcuts:

```bash
RUN_ID_BASE="tb-20260301T120000"
RUN_ID_RESCUE="rescue-20260301T130000"

# 1) Traceback-only baseline (uses evals/configs/openai_traceback_only.latest.toml)
just eval-traceback-only "$RUN_ID_BASE"

# 2) Extract failed traceback_only cases
just eval-select-failed "$RUN_ID_BASE"
# writes: evals/artifacts/case_lists/${RUN_ID_BASE}.failed.txt

# 3) Adaptive online-policy rescue on failed-only subset from step (2)
just eval-adaptive-online-failed "$RUN_ID_BASE" "$RUN_ID_RESCUE"

# 4) Workflow-level uplift summary (baseline + rescue)
just eval-two-pass-summary "$RUN_ID_BASE" "$RUN_ID_RESCUE"
```

Argument mapping:
- `eval-traceback-only <run_id>`: creates `evals/results/<run_id>.jsonl`.
- `eval-select-failed <run_id>`: reads `evals/results/<run_id>.jsonl`, writes `evals/artifacts/case_lists/<run_id>.failed.txt`.
- `eval-adaptive-online-failed <source_run_id> <rescue_run_id>`: reads `evals/artifacts/case_lists/<source_run_id>.failed.txt`, writes `evals/results/<rescue_run_id>.jsonl` plus online-policy state/events artifacts.
- `eval-two-pass-summary <baseline_run_id> <rescue_run_id>`: reports rescue coverage on baseline failures, rescued success rate on failed cases, and net success uplift vs baseline.

### Reproducibility controls

Use deterministic mode when you need reproducible patch-generation runs:

```bash
uv run python -m evals.run_eval \
  --patcher openai \
  --openai-base-url http://127.0.0.1:1234 \
  --openai-model deepseek/deepseek-r1-0528-qwen3-8b \
  --deterministic \
  --seed 12345 \
  --record-prompt hash \
  --record-evidence hash \
  --record-deps minimal
```

Key flags:
- `--deterministic --seed <int>`: enforce deterministic mode; fails fast if OpenAI-compatible backend does not acknowledge seed support.
- `--record-prompt full|hash|none`: controls prompt persistence in `attempt_metadata`.
- `--record-evidence full|hash|none`: controls evidence persistence in `attempt_metadata`.
- `--record-env KEY` (repeatable): allowlist environment keys into `run_metadata`.
- `--record-deps none|minimal|runtime`: dependency version capture scope.
- `--jobs <int>`: number of units to execute concurrently (default `1`).
- `--executor testcontainers|local`: test executor backend (default: `testcontainers`).
- `--case-list-file <path>`: newline-delimited case IDs to include (repeatable).
  Blank lines and `# comments` are ignored. Useful for targeted reruns.
- `--pytest-output-tail-lines <int>`: persist only the latest N lines in `pytest_before_*` / `pytest_after_*` artifacts (default `4000`, `0` disables line truncation; a hard per-file cap of `250000` chars still applies to protect disk usage).
- `--snapshot-artifact-max-string-chars <int>`: per-string truncation cap for persisted `snapshot.json` artifacts (default `16000`, `0` disables); patcher proposal calls still receive full snapshot context for compatibility.
- `--container-image <image>`: image used by `testcontainers` executor.
- `--container-network none|bridge`: container network mode (`none` is default and recommended).
- `--container-mount-repo-src` / `--container-no-mount-repo-src`: mount local `src/` for plugin imports.
- `--container-env KEY=VALUE`: explicit env allowlist passed into container (repeatable).
- stale eval containers are pruned automatically at run start (best-effort, 5-minute grace).
- `--openai-api-key <str>`: API key for OpenAI-compatible patcher; defaults to `Z_API`, then `OPENAI_API_KEY`.
- `--transport-fail-fast-streak <int>`: abort run after N consecutive terminal transport/backend failures (transport request failures, retryable HTTP transport statuses, or crash-like runtime errors such as segfault signatures; default `5`, `0` disables).
- `--openai-max-propose-sec <float>`: per-attempt shared wall-clock budget for OpenAI Stage-B propose calls (initial + context/no-diff recovery), where each call still uses its own internal retries/fallbacks (default `180`; set `0` to disable cap).
- `--openai-transport-max-retries <int>`: max immediate retries inside one OpenAI request for
  `request_failed` and retryable HTTP statuses (`408/409/425/429/5xx`) (default `6`).
- `--openai-transport-retry-backoff-base-sec <float>` and
  `--openai-transport-retry-backoff-max-sec <float>`: exponential backoff profile for those
  transport retries (defaults `1.0` and `8.0` seconds).
- per-attempt metadata records retry telemetry (`transport_retry_count`,
  `transport_retry_attempts`, and effective retry/backoff settings).
- `--openai-context-window-tokens <int>`: optional OpenAI-compatible context window guard for Stage-B prompt preflight (default `0` disables guard). Useful for LMStudio-like endpoints that return context-length HTTP 400s.
- `--openai-auto-context-window`: opt-in model metadata probe for context window tokens before execution starts (falls back to disabled guard on probe failure).
- probe adapter order is LM Studio REST (`/api/v1/models`, then `/api/v0/models`) followed by generic OpenAI-compatible `/models`.
- manual `--openai-context-window-tokens` always takes precedence over `--openai-auto-context-window`.
- `--openai-response-mode-autoswitch off|retry_to_diff`: opt-in retry policy for OpenAI patcher internal syntax/sanitize cycles (`retry_to_diff` switches `edits` → `diff` after retry-triggering failures).
- `--context-protocol monolithic|staged`: prompt protocol. `staged` adds a Stage-A evidence request pass before patch generation.
- `--context-stage-a-max-requests <int>`: maximum Stage-A retrieval requests (default 6).
- `--context-stage-a-max-chars <int>`: maximum Stage-A returned context size in characters (default 120000).
- Stage-A request schema (STRICT JSON, produced by the patcher and executed by the runner):
```json
{
  "need_more_context": true,
  "need_snapshot": false,
  "need_compact_context": false,
  "requests": [
    {"type": "file", "path": "relative/path.py"},
    {"type": "search", "path": "relative/path.py", "pattern": "literal_text"},
    {"type": "frame", "index": 1}
  ],
  "snapshot_justification": "one short sentence (only when need_snapshot=true)",
  "compact_context_reason": "one short sentence (only when need_compact_context=true)",
  "sufficiency_rationale": "one short sentence"
}
```
- `--retry-prompt-mode full|delta|auto`: retry packaging mode (`auto` defaults to `delta` for `with_snapshot_structured`).
- adaptive conditions keep full retry prompts by default; use `--retry-prompt-mode delta` only when explicitly evaluating delta-first adaptive retries.
- `--retry-context-mode fresh|history`: retry context policy (`history` includes prior attempt transcript; default `history`).
- `--retry-history-max-chars <int>`: max characters for retry-history transcript (default 48000).
- `--prompt-budget-total-chars <int>`: Stage-B prompt total character budget (0 disables cap).
- `--prompt-budget-retry-delta-chars <int>`: retry packet per-section cap.
- `--prompt-budget-stdout-chars <int>`: pytest output per-section cap.
- `--prompt-budget-snapshot-chars <int>`: snapshot section cap.
- `--prompt-budget-evidence-chars <int>`: Stage-A evidence section cap.
- `--prompt-budget-retry-history-chars <int>`: retry-history section cap (default 12000).
- `--context-compaction-mode off|auto_overflow|always`: delegated context compaction policy for Stage-B (default `auto_overflow`).
- `--context-compaction-scope snapshot,pytest,files`: raw sections eligible for delegated compaction.
- `--context-compaction-max-input-chars <int>`: max raw characters sent to compaction calls.
- `--context-compaction-target-chars <int>`: target compacted context size in characters.
- `--context-compaction-max-calls <int>`: max compaction calls per unit (default `1`, `0` disables).
- `--conditions traceback_only,with_snapshot,with_snapshot_structured,adaptive_gated,adaptive_llm_discretion`:
  run explicit A/B/C + adaptive comparisons (default remains `traceback_only,with_snapshot` for continuity).
- `--adaptive-gate-policy aggressive|balanced|conservative`: helper trigger policy for `adaptive_gated`.
- `--adaptive-max-helper-calls <int>`: helper-call cap per unit (default `1`, `0` means unlimited).
- `--adaptive-budget-topup-max-calls <int>`: one-time helper budget top-up cap (default `1`, `0` disables top-ups). Top-ups only trigger under repeated no-progress failures after prior `need_more_context=true`.
- `--adaptive-stagnation-window <int>`: repeated-signature window for gate activation.
- `--adaptive-saturation-threshold <float>`: disables helper when low-yield helper ratio is too high.
- `--adaptive-no-progress-token-cap <int>`: cumulative no-progress token budget for adaptive hard-stop (`150000` default, `0` disables).
- `--adaptive-helper-budget-profile default|compact`: prompt budget profile when helper is invoked.
- `--adaptive-openai-helper-response-mode inherit|edits|diff`: per-helper-attempt OpenAI response-mode override (default `inherit`).
- `--adaptive-informativeness-pregate`: enables rule-based traceback informativeness pre-gate for adaptive conditions.
- `--adaptive-informativeness-pregate-threshold <float>`: pre-gate threshold in `[0.0, 1.0]` (default `0.7`).
- `--adaptive-online-policy-mode off|shadow|on`: online learning mode for `adaptive_llm_discretion`.
- `adaptive_llm_discretion` now includes a generalized stagnation override: when helper usage is repeatedly skipped under no-progress signatures, helper can be forced on the next retry (with one optional budget top-up if configured).
- `--adaptive-online-policy-algo linucb`: online policy algorithm (currently `linucb`).
- `--adaptive-online-alpha <float>`: LinUCB exploration coefficient.
- `--adaptive-online-epsilon <float>`: epsilon-greedy exploration probability.
- `--adaptive-online-l2 <float>`: LinUCB regularization strength.
- `--adaptive-online-warmup-decisions <int>`: uniform warmup decisions before policy exploitation.
- `--adaptive-online-save-every <int>`: persist policy state every N updates.
- `--adaptive-online-state-path <path>`: policy state JSON path (default `evals/artifacts/policies/<run_id>.json`).
- `--adaptive-online-events-path <path>`: policy event log JSONL path (default `evals/results/<run_id>.adaptive_policy.jsonl`).
- `--adaptive-reward-lambda-tokens <float>` / `--adaptive-reward-mu-latency <float>`: reward penalties for token and latency cost.
- `--adaptive-reward-token-scale <float>` / `--adaptive-reward-latency-scale <float>`: normalization scales for reward penalties.
- Adaptive online policy feature vectors include trajectory, helper-budget/top-up, lagged helper/compaction outcomes, case priors, and pregate exception flags in addition to the base pre-gate signals.
- When loading an existing online policy state, newly observed feature keys are auto-expanded into LinUCB matrices (existing learned weights are preserved).
- `--syntax-preflight-enabled` / `--no-syntax-preflight`: enable/disable Python syntax preflight (`py_compile`) after patch apply and before test rerun (default enabled).
- `--syntax-preflight-max-cycles <int>`: max internal syntax-repair cycles per logical attempt (default `2`); these cycles do not consume extra retry attempts.
- `--patch-sanitize-enabled` / `--no-patch-sanitize`: enable/disable malformed patch-text sanitation guard before patch apply/syntax preflight (default enabled).
- `--patch-verifier-infer-test-source-scope` / `--no-patch-verifier-infer-test-source-scope`: enable/disable patch-verifier source-scope inference from failing/crash test imports (default enabled).

### Adaptive workflow (tool-like evaluation)

Compare gate-driven vs model-discretion helper usage with the same run settings:

```bash
uv run python -m evals.run_eval \
  --patcher openai \
  --openai-base-url http://127.0.0.1:1234 \
  --openai-model <model> \
  --conditions adaptive_gated,adaptive_llm_discretion \
  --k 3 \
  --adaptive-gate-policy balanced \
  --adaptive-max-helper-calls 1 \
  --retry-prompt-mode full \
  --adaptive-helper-budget-profile compact \
  --adaptive-openai-helper-response-mode inherit
```

Enable online policy learning (`shadow` logs only; `on` updates state):

```bash
uv run python -m evals.run_eval \
  --patcher openai \
  --openai-base-url http://127.0.0.1:1234 \
  --openai-model <model> \
  --conditions traceback_only,adaptive_llm_discretion \
  --adaptive-informativeness-pregate \
  --adaptive-online-policy-mode on \
  --adaptive-online-policy-algo linucb \
  --adaptive-online-alpha 0.8 \
  --adaptive-online-epsilon 0.05 \
  --adaptive-online-state-path evals/artifacts/policies/run_online.json \
  --adaptive-online-events-path evals/results/run_online.adaptive_policy.jsonl
```

## Analyze and report uplift

Quick single-run core summary:

```bash
just eval-summary <run_id>

# Equivalent direct CLI:
uv run python -m evals.eval_summary <run_id>
```

This summary command reads `evals/results/<run_id>.jsonl` and enforces strict
schema validation (`schema_version=3`).

```bash
uv run python -m evals.analyze_results evals/results/*.jsonl --out evals/results/summary.json
```

`evals.analyze_results` now enforces strict schema validation (`schema_version=3`) for
records. Older JSONL outputs without required metadata fields will be rejected and should
be regenerated with the current harness.
Runtime accounting now records canonical per-attempt wall-clock time as
`attempt_wall_clock_sec` (non-negative number) in each JSONL row. The analyzer
prefers this field when present across all attempts for a case-condition unit and
falls back to legacy cumulative/before/after duration fields for backward compatibility.
When resumed runs append duplicate attempt rows for the same
`(run_id, case_id, condition, attempt)`, the analyzer resolves them with
**last-write-wins** and reports `duplicate_attempt_rows_seen` /
`duplicate_attempt_rows_resolved` under input diagnostics.

The analyzer reports:
- overall metrics,
- per-category metrics,
- per-difficulty metrics,
- per-snapshot-dependency metrics,
- per-bug-source metrics,
- per-contamination-risk metrics,
- explicit `uplift_success_rate = with_snapshot - traceback_only`,
- `uplift_structured_vs_snapshot`,
- `uplift_structured_vs_traceback`,
- pairwise condition uplift matrix (`pairwise_uplifts`) across all discovered conditions,
- `retry_gate_block_rate`,
- prompt-size metrics including `mean_prompt_chars_stage_b` and per-condition means,
- retry/evidence packaging telemetry (`retry_prompt_mode`, evidence-ID counts, prompt-budget fields),
- attempt-efficiency telemetry (`mean_attempts_budget`, `mean_attempt_ratio_when_successful`,
  `solve_on_attempt_distribution`),
- token telemetry (`mean_tokens_prompt`, `mean_tokens_completion`, `mean_tokens_total`,
  `mean_tokens_per_attempt`, `mean_llm_calls_per_case`),
- adaptive helper telemetry (`helper_invocation_case_rate`, `helper_success_when_invoked`,
  `helper_rescue_rate`, `helper_need_more_true_rate`, budget/saturation guard rates),
- adaptive online-policy telemetry (`adaptive_online_decisions`,
  `adaptive_online_selected_invoke_rate`, `adaptive_online_applied_invoke_rate`,
  `adaptive_online_update_rate`, `adaptive_online_mean_reward`,
  `adaptive_online_mean_propensity`),
- retry-history telemetry (`mean_retry_history_chars`, `mean_retry_history_turns`,
  `retry_history_budget_applied_rate`, `retry_history_truncation_rate`).

Export online-policy logs to a flat training dataset:

```bash
uv run python -m evals.export_policy_dataset \
  --events "evals/results/*.adaptive_policy.jsonl" \
  --output-csv evals/results/adaptive_policy_dataset.csv
```

Adaptive conditions also run a pre-apply patch verifier. Low-relevance adaptive patches are
rejected before apply with note `patch_verifier_rejected` and corresponding
`patch_verifier_*` telemetry in `attempt_metadata` / `patch_meta`.

### Statistical analysis

Add `--stats` for rigorous significance testing:

```bash
uv run python -m evals.analyze_results evals/results/*.jsonl --stats --out evals/results/summary.json

# Evaluate structured packaging directly against traceback-only:
uv run python -m evals.analyze_results \
  evals/results/*.jsonl \
  --stats \
  --stats-pair-a traceback_only \
  --stats-pair-b with_snapshot_structured
```

The `--stats` flag adds:
- **McNemar's test**: Paired binary significance test (exact binomial for <25 discordant pairs, chi-squared otherwise).
- **Cohen's h**: Effect size for proportions (small >= 0.2, medium >= 0.5, large >= 0.8).
- **Bootstrap CI**: 95% confidence interval for uplift via paired bootstrap (10k resamples).
- **Holm-Bonferroni**: Multiple comparison correction for per-category p-values.
- **Power report**: Post-hoc power assessment to gauge result trustworthiness.
- **Pairwise condition matrix**: always reports pairwise A/B summaries for all conditions in the input.

For multi-run (k>1) analysis, use `--aggregation`:

```bash
# Majority vote: case succeeds if >50% of runs succeed
uv run python -m evals.analyze_results evals/results/*.jsonl --stats --aggregation majority_vote

# Any success: case succeeds if any run succeeds
uv run python -m evals.analyze_results evals/results/*.jsonl --stats --aggregation any_success
```

### Paper-ready plain-text report

Use `evals.report_run_metrics` for a narrative, paper-facing summary with
primary endpoint, secondary slices, efficiency, and failure diagnostics:

```bash
uv run python -m evals.report_run_metrics \
  --jsonl "evals/results/*.jsonl" \
  --eval-log results/<run_id>/eval-<run_id>.out
```

Failure diagnostics metric glossary:
- `http_error_attempts`: attempt-incident counter where
  `patch_meta.error == "http_error"`; retries are included, so this is not a
  terminal failure count by itself.
- `context_overflow_attempts`: subset of attempts where the HTTP response body
  indicates context-size overflow.
- `context_overflow_cases`: case-level count (grouped by `run_id`, `case_id`,
  `condition`) where at least one attempt overflowed.
- `context_overflow_case_successes` / `context_overflow_case_failures`: among
  overflowed cases, whether any attempt eventually succeeded.

### Sample size guidance

| Cases | Discordant pairs (est.) | Power (uplift=0.2) | Recommendation |
|-------|------------------------|---------------------|----------------|
| 30 | ~10 | ~0.30 | Directional signal only |
| 60 | ~20 | ~0.55 | Moderate evidence |
| 100 | ~35 | ~0.75 | Near-publication quality |
| 150+ | ~50+ | ~0.90+ | Publication-grade |

Interpretation guidance:
- positive uplift in `required` buckets suggests llmdebug snapshots provide useful signal,
- near-zero uplift in `none` control buckets is expected and acts as a sanity check,
- positive uplift in a category suggests snapshots help that bug class,
- near-zero uplift suggests no clear advantage yet,
- negative uplift suggests prompt or patch quality regressions for that bucket.
