# Scripts

Helper utilities for benchmarking, HAICore SSHFS, SWE-Bench scoring, and CI
quality gates. These scripts are **not** SLURM jobs — see `slurm/README.md` for
cluster submission workflows.

## Scripts

| Script | Purpose | Where to run |
|--------|---------|--------------|
| `bench_capture_compare.py` | Compare pytest-benchmark capture-overhead medians between two JSON reports | Local / CI |
| `haicore_sshfs.sh` | OTP-safe SSHFS mount for HAICore via a persistent SSH master connection | Local (macOS / Linux) |
| `score_swebench_lite_official.sh` | Run official SWE-Bench Lite scoring on generated predictions | Docker machine |
| `smoke_swebench_lite_official.sh` | Quick smoke-test for official scoring on a small paired subset | Docker machine |
| `quality/check_coverage_no_regression.py` | Fail CI when branch coverage regresses below the configured baseline | Local / CI |

## Prerequisites

| Script | Requirements |
|--------|--------------|
| `bench_capture_compare.py` | Python 3, `llmdebug` package installed |
| `haicore_sshfs.sh` | `ssh`, `sshfs` (macFUSE + sshfs on macOS, `sshfs` package on Linux) |
| `score_swebench_lite_official.sh` | Python 3, `swebench` package, Docker |
| `smoke_swebench_lite_official.sh` | Same as `score_swebench_lite_official.sh` |
| `quality/check_coverage_no_regression.py` | Python 3, `llmdebug` package, `coverage.json` from `pytest --cov-branch --cov-report=json` |

## Usage

### bench_capture_compare.py

Compares median runtimes from two pytest-benchmark JSON reports and prints a
delta table.

```bash
python scripts/bench_capture_compare.py baseline.json current.json
```

### haicore_sshfs.sh

Manages a persistent SSH master connection and SSHFS mount to HAICore. Designed
for OTP-protected logins — authenticate once, then mount/browse without repeated
prompts. See `slurm/README.md § OTP-Safe SSHFS Workflow` for the full walkthrough.

```bash
# 1) Authenticate once (OTP/password)
scripts/haicore_sshfs.sh open --host <user>@haicore.scc.kit.edu

# 2) Mount remote home
scripts/haicore_sshfs.sh mount --remote-path /home/kastel/vy3326 \
  --mount-dir ~/mnt/haicore --volume-name haicore-home

# 3) Check status
scripts/haicore_sshfs.sh status

# 4) Tear down
scripts/haicore_sshfs.sh umount --mount-dir ~/mnt/haicore
scripts/haicore_sshfs.sh close
```

Run `scripts/haicore_sshfs.sh --help` for all options.

### score_swebench_lite_official.sh

Runs the official SWE-Bench evaluation harness against predictions produced by
the HAICore generation pipeline. Intended for a Docker-capable machine (not
HAICore).

```bash
./scripts/score_swebench_lite_official.sh \
  --run-dir /path/to/evals/swebench_runs/<RUN_ID> \
  --conditions baseline,llmdebug \
  --workers 6 \
  --cache-level env
```

Key options: `--clean` (re-run from scratch), `--cache-level none|base|env|instance`.

### smoke_swebench_lite_official.sh

Prepares a small paired subset from a full run and scores it — a fast sanity
check before committing to the full evaluation.

```bash
./scripts/smoke_swebench_lite_official.sh \
  --run-dir /path/to/evals/swebench_runs/<RUN_ID> \
  --max-instances 2
```

### quality/check_coverage_no_regression.py

Compares current branch coverage against `quality/coverage_baseline.json` and
exits non-zero if it regressed beyond a configurable tolerance.

```bash
python scripts/quality/check_coverage_no_regression.py \
  --coverage-json coverage.json \
  --baseline-json quality/coverage_baseline.json \
  --tolerance 0.05
```
