# SLURM Quickstart (HAICore + Enroot + OpenAI-Compatible Local Servers)

For the full procedure and troubleshooting playbook, see:
- `docs/haicore_enroot_eval_runbook.md`
- `docs/haicore_enroot_swebench_lite_runbook.md`

## 60-Second Workflow
1. Run preflight checks:
```bash
cd ~/llmdebug
slurm/check_env_gguf_eval.sh
```
2. Run the mandatory smoke test gate:
```bash
slurm/run_gguf_smoke.sh
```
3. Submit full run:
```bash
sbatch slurm/eval_gguf_eval.sh
```
4. Monitor:
```bash
squeue --me
tail -f ~/llmdebug/eval-<JOBID>.out
```

## Local Env File (Recommended)
Keep site/account-specific paths out of git by using a local env file:

```bash
cp slurm/.env.local.example slurm/.env.local
$EDITOR slurm/.env.local
```

Supported SLURM entry scripts auto-load `slurm/.env.local` (or
`LLMDEBUG_SLURM_ENV_FILE=/path/to/file`) with `set -a`, so variables are
exported to child processes, including `sbatch` jobs and enroot-launched
processes.

## Run From tmux (Recommended Over SSH)
Long runs should be launched from a `tmux` session so they survive SSH disconnects.

```bash
# Start a named session once per login
tmux new -s gguf-eval

# Inside tmux
cd ~/llmdebug
slurm/check_env_gguf_eval.sh

# Example: 50-case smoke gate (blocking until done)
SMOKE_SERVER_BACKEND=llama_cpp \
MODEL_PRESET=qwen3coder30ba3b-q4_k_m \
SMOKE_TIME=03:00:00 \
SMOKE_TEST_MAX_CASES=50 \
SMOKE_TEST_EVAL_K=1 \
SMOKE_TEST_CONTEXT_SIZE=196608 \
WAIT_FOR_COMPLETION=1 \
slurm/run_gguf_smoke.sh
```

`tmux` controls:
```bash
# Detach from session (job keeps running)
Ctrl-b d

# List sessions
tmux ls

# Reattach later
tmux attach -t gguf-eval
```

If you only want to submit and return immediately, use:
```bash
WAIT_FOR_COMPLETION=0 slurm/run_gguf_smoke.sh
```

## OTP-Safe SSHFS Workflow (Optional)
If your HAICore login uses OTP and you want GUI/TUI file browsing without repeated auth prompts:

```bash
# from repo root
chmod +x scripts/haicore_sshfs.sh

# open one persistent SSH master (OTP/password once)
scripts/haicore_sshfs.sh open --host <user>@haicore.scc.kit.edu

# mount your HAICore home via sshfs
scripts/haicore_sshfs.sh mount --remote-path /home/kastel/vy3326 --mount-dir ~/mnt/haicore --volume-name haicore-home

# check state
scripts/haicore_sshfs.sh status

# tear down when done
scripts/haicore_sshfs.sh umount --mount-dir ~/mnt/haicore
scripts/haicore_sshfs.sh close
```

The `mount` subcommand is fail-closed (`BatchMode=yes`) and reuses only the control socket.
If the master session dies, it fails instead of triggering repeated re-auth attempts.

Troubleshooting:
- `remote host has disconnected` on mount:
  - usually indicates a stale multiplex master; run:
    `scripts/haicore_sshfs.sh close && scripts/haicore_sshfs.sh open --host <user>@haicore.scc.kit.edu`
  - then retry mount.
- `Cannot execute command-line and remote command.`:
  - this happens when your `~/.ssh/config` host alias sets both `RemoteCommand` (for example tmux auto-attach)
    and SFTP command mode at once.
  - `scripts/haicore_sshfs.sh` now forces `RemoteCommand=none` and `RequestTTY=no` for SSHFS operations.
  - if needed, use explicit host form: `--host <user>@haicore.scc.kit.edu`.
- macOS Finder volume label:
  - macFUSE may show the mounted filesystem under `/Volumes/<volume-name>` (for example `/Volumes/haicore-home`)
    even when your CLI mountpoint is `~/mnt/haicore`.
  - use `--volume-name` to set a stable Finder label.

## Common Commands
```bash
# Dry-run resolved config (no launch)
sbatch --export=ALL,DRY_RUN=1 slurm/eval_gguf_eval.sh

# Pilot mode quick run
sbatch --export=ALL,SMOKE_TEST_MODE=1 slurm/eval_gguf_eval.sh

# Short-budget sweep (successive-halving style)
slurm/submit_gguf_sweep.sh

# Resume chain for one run-id across walltime chunks
slurm/submit_gguf_resume_chain.sh <RUN_ID> 4

# Collect support bundle for a failed job
slurm/collect_gguf_debug_bundle.sh <JOBID>
```

## Important sbatch --export Gotcha (Comma Values)
`sbatch --export=ALL,...` uses commas as separators. If a variable value itself
contains a comma (for example `EVAL_CONDITIONS=traceback_only,adaptive_llm_discretion`),
Slurm will split it and silently truncate the value.

Bad:
```bash
sbatch --export=ALL,EVAL_CONDITIONS=traceback_only,adaptive_llm_discretion slurm/eval_gguf_eval.sh
```

Good (recommended):
```bash
EVAL_CONDITIONS='traceback_only,adaptive_llm_discretion' \
sbatch --export=ALL,SERVER_BACKEND=llama_cpp,MODEL_PRESET=qwen3coder30ba3b-q4_k_m,EVAL_K=3,EVAL_MAX_CASES=0 \
slurm/eval_gguf_eval.sh
```

Apply the same pattern for any comma-containing variable (for example
`SBL_CONDITIONS='baseline,llmdebug'`).

## SWE-Bench Lite Quickstart (Hybrid)

SWE-Bench Lite uses a two-machine workflow:
- HAICore/Enroot for patch generation (`baseline` + `llmdebug`).
- Docker machine for official SWE-Bench scoring.

Generation on HAICore:

```bash
cd ~/llmdebug
slurm/check_env_swebench_lite.sh

# mandatory 10-instance gate
SBL_RUN_ID=swebench-lite-smoke-$(date +%Y%m%dT%H%M%S) \
WAIT_FOR_COMPLETION=1 \
slurm/run_swebench_lite_smoke.sh

# full Lite generation (0 = full split cardinality)
SBL_CONDITIONS='baseline,llmdebug' \
sbatch --export=ALL,\
SBL_RUN_ID=swebench-lite-full-$(date +%Y%m%dT%H%M%S),\
SBL_MAX_INSTANCES=0,\
SBL_EVAL_JOBS=2,\
SBL_RESUME=1 \
slurm/eval_swebench_lite_generate.sh

# if walltime chaining is needed
slurm/submit_swebench_lite_resume_chain.sh <RUN_ID> 4

# prepare handoff artifacts after generation
python -m evals.swebench.cli prepare-handoff --run-id <RUN_ID>
```

Note: `slurm/run_swebench_lite_smoke.sh` now runs post-smoke handoff
validation automatically (`prepare-handoff`) unless
`PREPARE_HANDOFF_AFTER_SMOKE=0`.

Official scoring on Docker machine:

```bash
./scripts/score_swebench_lite_official.sh \
  --run-dir /path/to/evals/swebench_runs/<RUN_ID> \
  --conditions baseline,llmdebug \
  --workers 6 \
  --cache-level env

# fast official-scoring smoke on a tiny paired subset
./scripts/smoke_swebench_lite_official.sh \
  --run-dir /path/to/evals/swebench_runs/<RUN_ID> \
  --max-instances 2
```

## Backend Modes

`slurm/eval_gguf_eval.sh` now supports two server backends:

- `SERVER_BACKEND=vllm` (default)
  - Uses direct Hugging Face model IDs (no GGUF required).
  - Default model: `HF_MODEL_ID=Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8`
  - Default alias: `HF_MODEL_ALIAS=qwen3coder30ba3b`
  - Default server container: `vllm-server-cuda`
- `SERVER_BACKEND=llama_cpp`
  - Uses GGUF presets/files (legacy path).
  - Default server container: `llama-server-cuda`

Quick examples:

```bash
# Default vLLM run (direct HF model)
sbatch --export=ALL,SERVER_BACKEND=vllm slurm/eval_gguf_eval.sh

# Explicit vLLM model override
sbatch --export=ALL,SERVER_BACKEND=vllm,HF_MODEL_ID=Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8,HF_MODEL_ALIAS=qwen3coder30ba3b slurm/eval_gguf_eval.sh

# Legacy GGUF path via llama.cpp
sbatch --export=ALL,SERVER_BACKEND=llama_cpp,MODEL_PRESET=qwen3coder30ba3b-q4_k_m slurm/eval_gguf_eval.sh
```

## Model Presets (llama.cpp / GGUF only)
```bash
# Default preset
sbatch --export=ALL,SERVER_BACKEND=llama_cpp,MODEL_PRESET=qwen3coder30ba3b-q4_k_m slurm/eval_gguf_eval.sh

# Alternate preset
sbatch --export=ALL,SERVER_BACKEND=llama_cpp,MODEL_PRESET=glm47flash-ud-q4_k_xl slurm/eval_gguf_eval.sh
```

Supported `MODEL_PRESET` values:
- `glm47flash-ud-q4_k_xl`
- `qwen3coder30ba3b-q4_k_m` (default)
- `qwen3coder30ba3b-q6_k`
- `custom` (set `MODEL_DIR`, `MODEL_FILE`, `MODEL_ALIAS`)

For `glm47flash-ud-q4_k_xl`, server defaults follow Unsloth guidance:
- `SERVER_TEMPERATURE=1.0`
- `SERVER_TOP_P=0.95`
- `SERVER_REPEAT_PENALTY=1.0`
- `SERVER_MIN_P=0.01`
- `OPENAI_TEMPERATURE=1.0`

For `qwen3coder30ba3b-q4_k_m` and `qwen3coder30ba3b-q6_k`, server defaults follow Unsloth guidance:
- `SERVER_TEMPERATURE=0.7`
- `SERVER_TOP_P=0.8`
- `SERVER_TOP_K=20`
- `SERVER_REPEAT_PENALTY=1.05`
- `SERVER_MIN_P=0.00`
- `OPENAI_TEMPERATURE=0.7`

Context defaults:
- `glm47flash-ud-q4_k_xl`: `CONTEXT_SIZE=202752`, `SERVER_PARALLEL=3`
- `qwen3coder30ba3b-q4_k_m`: `CONTEXT_SIZE=196608`, `SERVER_PARALLEL=2`
- `qwen3coder30ba3b-q6_k`: `CONTEXT_SIZE=262144`, `SERVER_PARALLEL=2` (higher VRAM pressure)
- `custom`: `CONTEXT_SIZE=8192` (unless overridden)

Important: llama.cpp effective per-request slot context is approximately
`floor(CONTEXT_SIZE / SERVER_PARALLEL)`. If this value is too low, requests can
fail with `exceed_context_size_error`. Keep this ratio in mind when increasing
`EVAL_JOBS` or `SERVER_PARALLEL`.

Default budget guardrails in `slurm/eval_gguf_eval.sh`:
- `GPU_FAILFAST=1` with warmup/tail thresholds tuned to avoid false positives.
- `SERVER_FAILFAST=1` with liveness checks to stop eval quickly if the local LLM server dies.
- `OPENAI_MAX_PROPOSE_SEC=300` to cut low-yield long-tail patch attempts.

Qwen practical boundary notes for overflow mitigation (llama.cpp / GGUF mode):
- `SERVER_PARALLEL=2` at `CONTEXT_SIZE=196608` is the safer A100-40 default for `qwen3coder30ba3b-q4_k_m`.
- `qwen3coder30ba3b-q6_k` often needs additional KV-cache tuning (or lower context) on A100-40.
- `SERVER_PARALLEL=4` is possible but increases residual overflow risk on long prompts.

## Override Examples
```bash
# Fast sanity run
sbatch --export=ALL,EVAL_MAX_CASES=20,EVAL_K=1 slurm/eval_gguf_eval.sh

# vLLM baseline (direct HF)
sbatch --export=ALL,SERVER_BACKEND=vllm,HF_MODEL_ID=Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8,HF_MODEL_ALIAS=qwen3coder30ba3b,VLLM_MAX_MODEL_LEN=32768 slurm/eval_gguf_eval.sh

# vLLM long-context pressure reduction (single A100; targets >79k-token prompts)
sbatch --export=ALL,SERVER_BACKEND=vllm,HF_MODEL_ID=Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8,HF_MODEL_ALIAS=qwen3coder30ba3b,VLLM_MAX_MODEL_LEN=98304,VLLM_MAX_NUM_SEQS=1,VLLM_KV_CACHE_DTYPE=fp8,VLLM_CALCULATE_KV_SCALES=1,VLLM_GPU_MEMORY_UTILIZATION=0.95 slurm/eval_gguf_eval.sh

# GLM long-context baseline (single A100)
sbatch --export=ALL,SERVER_BACKEND=llama_cpp,MODEL_PRESET=glm47flash-ud-q4_k_xl,CONTEXT_SIZE=202752,SERVER_PARALLEL=3 slurm/eval_gguf_eval.sh

# Qwen long-context baseline (single A100)
sbatch --export=ALL,SERVER_BACKEND=llama_cpp,MODEL_PRESET=qwen3coder30ba3b-q4_k_m,CONTEXT_SIZE=196608,SERVER_PARALLEL=2 slurm/eval_gguf_eval.sh

# Force deterministic mode with explicit seed
sbatch --export=ALL,EVAL_DETERMINISTIC=1,EVAL_SEED=42 slurm/eval_gguf_eval.sh

# Run only one evidence condition (budget-focused)
sbatch --export=ALL,EVAL_CONDITIONS=traceback_only slurm/eval_gguf_eval.sh

# Fix server port if needed
sbatch --export=ALL,SERVER_PORT=8000 slurm/eval_gguf_eval.sh

# Override sampling defaults if needed
sbatch --export=ALL,SERVER_BACKEND=llama_cpp,MODEL_PRESET=qwen3coder30ba3b-q4_k_m,SERVER_TEMPERATURE=0.0,OPENAI_TEMPERATURE=0.0 slurm/eval_gguf_eval.sh

# llama.cpp A100 tuning knobs (prefill/throughput experiments; flash-attn is off by default)
sbatch --export=ALL,SERVER_BACKEND=llama_cpp,SERVER_BATCH_SIZE=4096,SERVER_UBATCH_SIZE=1024,SERVER_PARALLEL=2,SERVER_THREADS=8,SERVER_THREADS_BATCH=8 slurm/eval_gguf_eval.sh

# llama.cpp long-context memory tuning (optional KV cache quantization)
sbatch --export=ALL,SERVER_BACKEND=llama_cpp,SERVER_CACHE_TYPE_K=q8_0,SERVER_CACHE_TYPE_V=q8_0 slurm/eval_gguf_eval.sh

# llama.cpp CUDA backend experiments (advanced; benchmark before keeping)
sbatch --export=ALL,SERVER_BACKEND=llama_cpp,GGML_CUDA_FORCE_CUBLAS=1 slurm/eval_gguf_eval.sh

# Explicit Slurm step binding for server launch (enabled by default)
sbatch --export=ALL,SERVER_USE_SRUN=1,SERVER_SRUN_CPU_BIND=cores,SERVER_SRUN_GPU_BIND=single:1 slurm/eval_gguf_eval.sh

# GPU-hour guardrail: fail-fast when utilization stays low after warmup
sbatch --export=ALL,GPU_FAILFAST=1,GPU_FAILFAST_WARMUP_SEC=1800,GPU_FAILFAST_UTIL_PCT=25,GPU_FAILFAST_LOW_FRAC_PCT=95,GPU_FAILFAST_CHECK_SEC=60 slurm/eval_gguf_eval.sh

# GPU-hour guardrail: fail-fast when the server becomes unhealthy mid-run
sbatch --export=ALL,SERVER_FAILFAST=1,SERVER_FAILFAST_CHECK_SEC=20,SERVER_FAILFAST_MAX_CONSEC_HEALTH_FAILS=12 slurm/eval_gguf_eval.sh

# Long-tail guardrail: cap wall-clock patch proposal time per case/attempt
sbatch --export=ALL,OPENAI_MAX_PROPOSE_SEC=300 slurm/eval_gguf_eval.sh

# Optional output-cap tuning (default 2048)
sbatch --export=ALL,OPENAI_MAX_TOKENS=1536 slurm/eval_gguf_eval.sh
```

## Logs to Check
- `~/llmdebug/eval-<JOBID>.out`
- `~/llmdebug/llm-server-<JOBID>.log`
- `~/llmdebug/eval-launch-<JOBID>.log`
- `~/llmdebug/enroot-diag-<JOBID>.log`
- `~/llmdebug/gpu-util-<JOBID>.csv`
- `~/llmdebug/gpu-failfast-<JOBID>.txt` (only when fail-fast triggers)
- `~/llmdebug/server-failfast-<JOBID>.txt` (only when server fail-fast triggers)
- `~/llmdebug/run-config-<JOBID>.json`
- `~/llmdebug/failure-dump-<JOBID>.json` (on failures)
