# SWE-Bench Lite Hybrid Runbook

This runbook documents the two-machine SWE-Bench Lite workflow:

1. HAICore + Enroot generation (LLM inference only).
2. Local Docker official scoring (no LLM calls).

The workflow keeps generation and scoring decoupled so low-storage development
machines can still orchestrate runs and prepare handoff artifacts.

## Scope and decisions

- Benchmark: `SWE-Bench Lite` only.
- Conditions: `baseline` and `llmdebug` always generated together.
- Generation runtime: HAICore via Enroot (`llama-server-cuda` + `python-eval`).
- Scoring runtime: separate Docker-capable machine using official SWE-Bench evaluator.
- Milestone-1 publication artifacts: raw official evaluator outputs only.

## Paths and artifact contract

Generation writes one run directory under:

`/Users/nicolas/Documents/PhD/llmdebug/evals/swebench_runs/<run_id>/`

Required artifacts in a complete generated run:

- `baseline/predictions.jsonl`
- `llmdebug/predictions.jsonl`
- `instance_ids.txt`
- `run_metadata.json`
- `run_index.json` (created by `prepare-handoff`)
- `COPY_INSTRUCTIONS.txt` (created by `prepare-handoff`)

Scoring writes:

- `official_eval/` (per-condition raw official outputs)

Prediction JSONL rows must include:

- `instance_id` (string)
- `model_patch` (string unified diff; empty string allowed for no patch)
- `model_name_or_path` (string; condition-labeled model tag)

## A. HAICore generation plane (Enroot)

### 1) Preflight

```bash
cd /Users/nicolas/Documents/PhD/llmdebug
slurm/check_env_swebench_lite.sh
```

### 2) Mandatory 10-instance smoke gate

```bash
SBL_RUN_ID=swebench-lite-smoke-$(date +%Y%m%dT%H%M%S) \
SBL_EVAL_JOBS=1 \
WAIT_FOR_COMPLETION=1 \
slurm/run_swebench_lite_smoke.sh
```

By default this also runs post-smoke handoff validation (`prepare-handoff`).
Set `PREPARE_HANDOFF_AFTER_SMOKE=0` only if you need to skip that check.

### 3) Optional 50-instance pilot

```bash
SBL_CONDITIONS='baseline,llmdebug' \
sbatch --export=ALL,\
SBL_RUN_ID=swebench-lite-pilot-$(date +%Y%m%dT%H%M%S),\
SBL_MAX_INSTANCES=50,\
SBL_EVAL_JOBS=2,\
SBL_RESUME=1 \
slurm/eval_swebench_lite_generate.sh
```

### 4) Full Lite generation

```bash
SBL_CONDITIONS='baseline,llmdebug' \
sbatch --export=ALL,\
SBL_RUN_ID=swebench-lite-full-$(date +%Y%m%dT%H%M%S),\
SBL_MAX_INSTANCES=0,\
SBL_EVAL_JOBS=2,\
SBL_RESUME=1 \
slurm/eval_swebench_lite_generate.sh
```

If walltime chaining is needed:

```bash
slurm/submit_swebench_lite_resume_chain.sh <RUN_ID> 4
```

### 5) Prepare handoff artifacts

Run after generation completes:

```bash
python -m evals.swebench.cli prepare-handoff --run-id <RUN_ID>
```

This validates paired instance coverage and writes:

- `run_index.json`
- `COPY_INSTRUCTIONS.txt`

### 6) Optional debug bundle for failed runs

```bash
slurm/collect_swebench_lite_debug_bundle.sh <JOB_ID>
```

## B. Artifact handoff plane

Copy the full run directory to the Docker scoring machine:

```bash
rsync -av \
  /Users/nicolas/Documents/PhD/llmdebug/evals/swebench_runs/<RUN_ID>/ \
  <docker-host>:/path/to/evals/swebench_runs/<RUN_ID>/
```

No reconstruction step is required if `prepare-handoff` passed.

## C. Local Docker scoring plane (official evaluator)

On the Docker machine, from repo root:

```bash
./scripts/score_swebench_lite_official.sh \
  --run-dir /path/to/evals/swebench_runs/<RUN_ID> \
  --conditions baseline,llmdebug \
  --workers 6 \
  --cache-level env
```

The scorer enforces `swebench==4.1.0` for reproducible publication scoring.

Fast official-scoring smoke (tiny paired subset):

```bash
./scripts/smoke_swebench_lite_official.sh \
  --run-dir /path/to/evals/swebench_runs/<RUN_ID> \
  --max-instances 2
```

Equivalent Python CLI:

```bash
python -m evals.swebench.cli score-official \
  --run-dir /path/to/evals/swebench_runs/<RUN_ID> \
  --conditions baseline,llmdebug \
  --workers 6 \
  --cache-level env
```

The scorer produces condition-separated outputs under:

`/path/to/evals/swebench_runs/<RUN_ID>/official_eval/{baseline,llmdebug}/`

## Environment variables used by Slurm generation

- `SBL_RUN_ID`: required for deterministic run identity.
- `SBL_MAX_INSTANCES`: `0` means full split cardinality.
- `SBL_EVAL_JOBS`: generation concurrency for patch proposals.
- `SBL_CONDITIONS`: default `baseline,llmdebug`.
- `SBL_INSTANCE_IDS_FILE`: optional explicit instance set.
- `SBL_RESUME`: `1` resumes existing condition files.

Optional:

- `SBL_SNAPSHOT_DIR`: per-instance runtime evidence for llmdebug condition.
- `SBL_DATASET_NAME` and `SBL_SPLIT`: dataset override.
- `MODEL_PRESET`, `MODEL_DIR`, `MODEL_FILE`, `MODEL_ALIAS`: model routing.

## Failure modes and safeguards

- Missing one condition file: `prepare-handoff` fails fast.
- Condition instance skew: `prepare-handoff` and scoring both fail fast.
- Empty predictions files: scoring fails fast.
- Docker unavailable on scoring host: scorer preflight fails before job launch.
- Generation health checks fail fast when patch-request errors indicate systemic
  issues (all failing, too many errors, or high error rate).

## Publication usage

For milestone 1, use raw official outputs from `official_eval/` as the source of
truth for tables and deltas. Keep HAICore generation logs for reproducibility and
diagnostics.
