# SLURM Quickstart (HAICore + Enroot + GGUF Models)

For the full procedure and troubleshooting playbook, see:
- `docs/haicore_enroot_eval_runbook.md`

## 60-Second Workflow
1. Run preflight checks:
```bash
cd ~/llmdebug
slurm/check_env_gguf_eval.sh
```
2. Run the mandatory smoke test gate:
```bash
slurm/run_gguf_smoke.sh
```
3. Submit full run:
```bash
sbatch slurm/eval_gguf_eval.sh
```
4. Monitor:
```bash
squeue --me
tail -f ~/llmdebug/eval-<JOBID>.out
```

## Run From tmux (Recommended Over SSH)
Long runs should be launched from a `tmux` session so they survive SSH disconnects.

```bash
# Start a named session once per login
tmux new -s gguf-eval

# Inside tmux
cd ~/llmdebug
slurm/check_env_gguf_eval.sh

# Example: 50-case smoke gate (blocking until done)
MODEL_PRESET=glm47flash-ud-q4_k_xl \
SMOKE_TIME=03:00:00 \
SMOKE_TEST_MAX_CASES=50 \
SMOKE_TEST_EVAL_K=1 \
SMOKE_TEST_CONTEXT_SIZE=202752 \
WAIT_FOR_COMPLETION=1 \
slurm/run_gguf_smoke.sh
```

`tmux` controls:
```bash
# Detach from session (job keeps running)
Ctrl-b d

# List sessions
tmux ls

# Reattach later
tmux attach -t gguf-eval
```

If you only want to submit and return immediately, use:
```bash
WAIT_FOR_COMPLETION=0 slurm/run_gguf_smoke.sh
```

## Common Commands
```bash
# Dry-run resolved config (no launch)
sbatch --export=ALL,DRY_RUN=1 slurm/eval_gguf_eval.sh

# Pilot mode quick run
sbatch --export=ALL,SMOKE_TEST_MODE=1 slurm/eval_gguf_eval.sh

# Short-budget sweep (successive-halving style)
slurm/submit_gguf_sweep.sh

# Resume chain for one run-id across walltime chunks
slurm/submit_gguf_resume_chain.sh <RUN_ID> 4

# Collect support bundle for a failed job
slurm/collect_gguf_debug_bundle.sh <JOBID>
```

## Model Presets
```bash
# Default preset (existing)
sbatch --export=ALL,MODEL_PRESET=glm47flash-ud-q4_k_xl slurm/eval_gguf_eval.sh

# New coding preset (single A100 40GB friendly)
sbatch --export=ALL,MODEL_PRESET=qwen3coder30ba3b-q6_k slurm/eval_gguf_eval.sh
```

Supported `MODEL_PRESET` values:
- `glm47flash-ud-q4_k_xl`
- `qwen3coder30ba3b-q6_k`
- `custom` (set `MODEL_DIR`, `MODEL_FILE`, `MODEL_ALIAS`)

For `glm47flash-ud-q4_k_xl`, server defaults follow Unsloth guidance:
- `SERVER_TEMPERATURE=1.0`
- `SERVER_TOP_P=0.95`
- `SERVER_REPEAT_PENALTY=1.0`
- `SERVER_MIN_P=0.01`
- `OPENAI_TEMPERATURE=1.0`

For `qwen3coder30ba3b-q6_k`, server defaults follow Unsloth guidance:
- `SERVER_TEMPERATURE=0.7`
- `SERVER_TOP_P=0.8`
- `SERVER_TOP_K=20`
- `SERVER_REPEAT_PENALTY=1.05`
- `SERVER_MIN_P=0.00`
- `OPENAI_TEMPERATURE=0.7`

Context defaults:
- `glm47flash-ud-q4_k_xl`: `CONTEXT_SIZE=202752`, `SERVER_PARALLEL=3`
- `qwen3coder30ba3b-q6_k`: `CONTEXT_SIZE=262144`, `SERVER_PARALLEL=3`
- `custom`: `CONTEXT_SIZE=8192` (unless overridden)

Important: llama.cpp effective per-request slot context is approximately
`floor(CONTEXT_SIZE / SERVER_PARALLEL)`. If this value is too low, requests can
fail with `exceed_context_size_error`. Keep this ratio in mind when increasing
`EVAL_JOBS` or `SERVER_PARALLEL`.

Qwen practical boundary notes for overflow mitigation:
- `SERVER_PARALLEL=3` at `CONTEXT_SIZE=262144` keeps high throughput and near-zero residual overflow risk on current eval prompts.
- `SERVER_PARALLEL=4` is possible but increases residual overflow risk on long prompts.

## Override Examples
```bash
# Fast sanity run
sbatch --export=ALL,EVAL_MAX_CASES=20,EVAL_K=1 slurm/eval_gguf_eval.sh

# GLM long-context baseline (single A100)
sbatch --export=ALL,MODEL_PRESET=glm47flash-ud-q4_k_xl,CONTEXT_SIZE=202752,SERVER_PARALLEL=3 slurm/eval_gguf_eval.sh

# Qwen long-context baseline (single A100)
sbatch --export=ALL,MODEL_PRESET=qwen3coder30ba3b-q6_k,CONTEXT_SIZE=262144,SERVER_PARALLEL=3 slurm/eval_gguf_eval.sh

# Force deterministic mode with explicit seed
sbatch --export=ALL,EVAL_DETERMINISTIC=1,EVAL_SEED=42 slurm/eval_gguf_eval.sh

# Fix server port if needed
sbatch --export=ALL,SERVER_PORT=8000 slurm/eval_gguf_eval.sh

# Override sampling defaults if needed
sbatch --export=ALL,MODEL_PRESET=qwen3coder30ba3b-q6_k,SERVER_TEMPERATURE=0.0,OPENAI_TEMPERATURE=0.0 slurm/eval_gguf_eval.sh

# A100 tuning knobs (prefill/throughput experiments)
sbatch --export=ALL,SERVER_FLASH_ATTN=on,SERVER_BATCH_SIZE=4096,SERVER_UBATCH_SIZE=1024,SERVER_PARALLEL=2,SERVER_THREADS=8,SERVER_THREADS_BATCH=8 slurm/eval_gguf_eval.sh

# Long-context memory tuning (optional KV cache quantization)
sbatch --export=ALL,SERVER_CACHE_TYPE_K=q8_0,SERVER_CACHE_TYPE_V=q8_0 slurm/eval_gguf_eval.sh

# CUDA backend experiments (advanced; benchmark before keeping)
sbatch --export=ALL,GGML_CUDA_FORCE_CUBLAS=1 slurm/eval_gguf_eval.sh
```

## Logs to Check
- `~/llmdebug/eval-<JOBID>.out`
- `~/llmdebug/llama-server-<JOBID>.log`
- `~/llmdebug/eval-launch-<JOBID>.log`
- `~/llmdebug/enroot-diag-<JOBID>.log`
- `~/llmdebug/gpu-util-<JOBID>.csv`
- `~/llmdebug/run-config-<JOBID>.json`
- `~/llmdebug/failure-dump-<JOBID>.json` (on failures)
