# HAICore Enroot Eval Runbook (vLLM HF + llama.cpp GGUF + llmdebug)

## Purpose and Scope
This runbook is the canonical procedure for running `llmdebug` evals on HAICore with:
- default backend: `vLLM` server container (`vllm-server-cuda`) using direct HF model IDs
- legacy backend: `llama.cpp` server container (`llama-server-cuda`) using GGUF presets
- Python eval container (`python-eval`)
- HF default model: `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8`
- GGUF model presets (`glm47flash-ud-q4_k_xl`, `qwen3coder30ba3b-q4_k_m`, `qwen3coder30ba3b-q6_k`) for `SERVER_BACKEND=llama_cpp`
- Job script `slurm/eval_gguf_eval.sh`

Reference:
- HAICore containers docs: https://www.nhr.kit.edu/userdocs/haicore/containers/
- Unsloth Qwen3-Coder local settings: https://unsloth.ai/docs/models/qwen3-coder-how-to-run-locally

Companion scripts:
- `slurm/eval_gguf_eval.sh`: main deployment script.
- `slurm/check_env_gguf_eval.sh`: preflight validator.
- `slurm/run_gguf_smoke.sh`: smoke gate before long runs.
- `slurm/submit_gguf_sweep.sh`: short-budget sweep submitter.
- `slurm/submit_gguf_resume_chain.sh`: walltime-safe resume chains.
- `slurm/collect_gguf_debug_bundle.sh`: debug/support bundle creator.

## Prerequisites
- HAICore account with Slurm access.
- Access to a GPU partition (example: `normal` with A100).
- `enroot` available.
- Repository checked out in home directory.
- Enough storage for model, enroot rootfs, and eval artifacts.

## One-Time Setup (Login Node)
### 1) Clone
```bash
cd ~
git clone <your-repo-url> llmdebug
cd ~/llmdebug
```

### 2) Import images
```bash
cd ~
enroot import docker://vllm/vllm-openai:latest
enroot import docker://ghcr.io#ggml-org/llama.cpp:server-cuda
enroot import docker://python:3.11
ls -lah *.sqsh
```

### 3) Create containers
```bash
enroot create --name vllm-server-cuda <vllm-server-file>.sqsh
enroot create --name llama-server-cuda <llama-server-file>.sqsh
enroot create --name python-eval <python-file>.sqsh
enroot list
```

### 4) Install eval deps in `python-eval`
```bash
enroot start --root --rw -e HOME=/tmp python-eval -- \
  pip install click rich pytest

enroot start --root --rw -e HOME=/tmp \
  -m ~/llmdebug:/workspace \
  python-eval -- \
  pip install -e /workspace
```

### 5) Download model(s)
```bash
enroot start --root --rw -e HOME=/tmp python-eval -- \
  pip install "huggingface-hub[cli]"

mkdir -p ~/models

# GLM preset model
enroot start --rw \
  -m ~/models:/models \
  python-eval -- \
  hf download unsloth/GLM-4.7-Flash-GGUF \
    --include "GLM-4.7-Flash-UD-Q4_K_XL.gguf" \
    --local-dir /models/GLM-4.7-Flash-GGUF

# Qwen coding preset model (A100-40 default in Q4_K_M)
enroot start --rw \
  -m ~/models:/models \
  python-eval -- \
  hf download unsloth/Qwen3-Coder-30B-A3B-Instruct-1M-GGUF \
    --include "Qwen3-Coder-30B-A3B-Instruct-1M-Q4_K_M.gguf" \
    --local-dir /models/Qwen3-Coder-30B-A3B-Instruct-1M-GGUF
```

### 6) Verify expected model path(s)
```bash
ls -lh "$HOME/models/GLM-4.7-Flash-GGUF/GLM-4.7-Flash-UD-Q4_K_XL.gguf"
ls -lh "$HOME/models/Qwen3-Coder-30B-A3B-Instruct-1M-GGUF/Qwen3-Coder-30B-A3B-Instruct-1M-Q4_K_M.gguf"
```

## Backend Selection
Choose backend explicitly when needed:

```bash
# Default backend (vLLM + HF model)
sbatch --export=ALL,SERVER_BACKEND=vllm slurm/eval_gguf_eval.sh

# Explicit vLLM model override
sbatch --export=ALL,SERVER_BACKEND=vllm,HF_MODEL_ID=Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8,HF_MODEL_ALIAS=qwen3coder30ba3b,VLLM_MAX_MODEL_LEN=32768 slurm/eval_gguf_eval.sh

# vLLM long-context pressure reduction (single A100)
sbatch --export=ALL,SERVER_BACKEND=vllm,HF_MODEL_ID=Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8,HF_MODEL_ALIAS=qwen3coder30ba3b,VLLM_MAX_MODEL_LEN=98304,VLLM_MAX_NUM_SEQS=1,VLLM_KV_CACHE_DTYPE=fp8,VLLM_CALCULATE_KV_SCALES=1,VLLM_GPU_MEMORY_UTILIZATION=0.95 slurm/eval_gguf_eval.sh

# Legacy llama.cpp GGUF backend
sbatch --export=ALL,SERVER_BACKEND=llama_cpp,MODEL_PRESET=qwen3coder30ba3b-q4_k_m slurm/eval_gguf_eval.sh
```

## Model Presets (llama.cpp / GGUF only)
Choose a model preset at submission time:

```bash
# Default preset (A100-40 friendly)
sbatch --export=ALL,SERVER_BACKEND=llama_cpp,MODEL_PRESET=qwen3coder30ba3b-q4_k_m slurm/eval_gguf_eval.sh

# Alternate coding preset (higher VRAM pressure)
sbatch --export=ALL,SERVER_BACKEND=llama_cpp,MODEL_PRESET=qwen3coder30ba3b-q6_k slurm/eval_gguf_eval.sh

# Fully custom model (manual path/file/alias)
sbatch --export=ALL,SERVER_BACKEND=llama_cpp,MODEL_PRESET=custom,MODEL_DIR="$HOME/models/MyModel",MODEL_FILE="my-model-Q6_K.gguf",MODEL_ALIAS="MyModel" slurm/eval_gguf_eval.sh
```

For `glm47flash-ud-q4_k_xl`, server sampling defaults follow Unsloth guidance:
- `SERVER_TEMPERATURE=1.0`
- `SERVER_TOP_P=0.95`
- `SERVER_REPEAT_PENALTY=1.0`
- `SERVER_MIN_P=0.01`
- `OPENAI_TEMPERATURE=1.0`

For `qwen3coder30ba3b-q4_k_m` and `qwen3coder30ba3b-q6_k`, server sampling defaults follow Unsloth guidance:
- `SERVER_TEMPERATURE=0.7`
- `SERVER_TOP_P=0.8`
- `SERVER_TOP_K=20`
- `SERVER_REPEAT_PENALTY=1.05`
- `SERVER_MIN_P=0.00`
- `OPENAI_TEMPERATURE=0.7`

Context defaults:
- `glm47flash-ud-q4_k_xl`: `CONTEXT_SIZE=202752`, `SERVER_PARALLEL=3`
- `qwen3coder30ba3b-q4_k_m`: `CONTEXT_SIZE=196608`, `SERVER_PARALLEL=2`
- `qwen3coder30ba3b-q6_k`: `CONTEXT_SIZE=262144`, `SERVER_PARALLEL=2` (higher VRAM pressure)
- `custom`: `CONTEXT_SIZE=8192` (unless overridden)

Important: effective llama.cpp per-request slot context is approximately
`floor(CONTEXT_SIZE / SERVER_PARALLEL)`. If this value is too low, requests may
fail with `exceed_context_size_error`.

Qwen practical boundary notes for overflow mitigation:
- `SERVER_PARALLEL=2` at `CONTEXT_SIZE=196608` is the safer A100-40 default for `qwen3coder30ba3b-q4_k_m`.
- `qwen3coder30ba3b-q6_k` typically needs KV cache tuning (or lower context) on A100-40.
- `SERVER_PARALLEL=4` may improve throughput further but increases long-prompt overflow risk.

## Preflight + Smoke (Before Every Full Run)
### SSH-safe execution with tmux (recommended)
Run smoke/full jobs from `tmux` so SSH disconnects do not interrupt your workflow.

```bash
# Start or reattach a persistent session
tmux new -s gguf-eval
# or, if already created:
tmux attach -t gguf-eval

# Inside tmux
cd ~/llmdebug
```

### 1) Preflight
```bash
cd ~/llmdebug
slurm/check_env_gguf_eval.sh
```

### 2) Mandatory smoke test gate
```bash
slurm/run_gguf_smoke.sh
```

Example 50-case smoke gate (legacy llama.cpp GGUF mode):
```bash
SMOKE_SERVER_BACKEND=llama_cpp \
MODEL_PRESET=glm47flash-ud-q4_k_xl \
SMOKE_TIME=03:00:00 \
SMOKE_TEST_MAX_CASES=50 \
SMOKE_TEST_EVAL_K=1 \
SMOKE_TEST_CONTEXT_SIZE=202752 \
WAIT_FOR_COMPLETION=1 \
slurm/run_gguf_smoke.sh
```

`tmux` controls:
```bash
# Detach and leave jobs running
Ctrl-b d

# List active sessions
tmux ls

# Reattach later
tmux attach -t gguf-eval
```

If you only want to submit and return immediately:
```bash
WAIT_FOR_COMPLETION=0 slurm/run_gguf_smoke.sh
```

### Optional: OTP-safe SSHFS mount for file workflows
If OTP prompts make repeated transfers painful, use the helper script to authenticate once,
mount via sshfs, and fail closed when the session expires.

```bash
cd ~/llmdebug
chmod +x scripts/haicore_sshfs.sh

# One interactive auth (OTP/password)
scripts/haicore_sshfs.sh open --host <user>@haicore.scc.kit.edu

# Mount your HAICore home
scripts/haicore_sshfs.sh mount --remote-path /home/kastel/vy3326 --mount-dir ~/mnt/haicore --volume-name haicore-home

# Later cleanup
scripts/haicore_sshfs.sh umount --mount-dir ~/mnt/haicore
scripts/haicore_sshfs.sh close
```

The script's mount step uses `BatchMode=yes` through the SSH control socket. If the master
connection is gone, operations fail immediately instead of repeatedly prompting auth.

Troubleshooting:
- Symptom: `remote host has disconnected` during mount
  - likely stale multiplex master; run:
    `scripts/haicore_sshfs.sh close && scripts/haicore_sshfs.sh open --host <user>@haicore.scc.kit.edu`
  - retry `scripts/haicore_sshfs.sh mount ...`.
- Symptom: `Cannot execute command-line and remote command.`
  - caused by `~/.ssh/config` host aliases that combine `RemoteCommand` (often tmux auto-attach)
    with SFTP command mode.
  - helper script forces `RemoteCommand=none` and `RequestTTY=no` for SSHFS operations.
  - if alias behavior is still surprising, use explicit host syntax:
    `--host <user>@haicore.scc.kit.edu`.
- Symptom: Finder shows `macFUSE Volume ...` or `/Volumes/...`
  - expected macOS/macFUSE behavior; mount is still accessible at `--mount-dir`.
  - pass `--volume-name haicore-home` (or another label) for a stable Finder name.

### 3) Optional config-only dry run
```bash
sbatch --export=ALL,DRY_RUN=1 slurm/eval_gguf_eval.sh
```

## Submit and Monitor
### Important `sbatch --export` note
`sbatch --export=ALL,...` treats commas as separators. Do not put
comma-containing values directly inside `--export` assignments.

Example: this can silently truncate `EVAL_CONDITIONS` to only `traceback_only`:
```bash
sbatch --export=ALL,EVAL_CONDITIONS=traceback_only,with_snapshot slurm/eval_gguf_eval.sh
```

Use a shell env assignment instead:
```bash
EVAL_CONDITIONS='traceback_only,with_snapshot' \
sbatch --export=ALL,SERVER_BACKEND=llama_cpp,MODEL_PRESET=glm47flash-ud-q4_k_xl,EVAL_K=3,EVAL_MAX_CASES=0 \
slurm/eval_gguf_eval.sh
```

Same remedy for any comma-containing variable.

### Full run
```bash
cd ~/llmdebug
sbatch slurm/eval_gguf_eval.sh
```

### Monitor
```bash
squeue --me
tail -f ~/llmdebug/eval-<JOBID>.out
```

### Inspect logs
```bash
cat ~/llmdebug/eval-<JOBID>.out
tail -n 200 ~/llmdebug/llm-server-<JOBID>.log
tail -n 200 ~/llmdebug/eval-launch-<JOBID>.log
cat ~/llmdebug/enroot-diag-<JOBID>.log
```

## Throughput-Oriented Workflows
### Successive-halving style sweeps
Run many short jobs first; promote only top configs.

```bash
# default sweep grid
slurm/submit_gguf_sweep.sh

# custom compact grid
RUN_TAG=pilotA SWEEP_TIME=01:30:00 \
MAX_CASES_LIST=10,20 K_LIST=1,2 CONTEXT_LIST=2048,4096 JOBS_LIST=2,4 \
slurm/submit_gguf_sweep.sh
```

### Resume chains under walltime caps
```bash
slurm/submit_gguf_resume_chain.sh qwen3coder-prod-20260218 4
```

All chain jobs use one `EVAL_RUN_ID`, and `evals/run_eval.py` resumes unfinished units safely.

## What “Working” Looks Like
Expected milestones in `eval-<JOBID>.out`:
- `Preparing enroot bind-mount targets...`
- startup banner with effective config
- `Starting LLM server (backend=...)...`
- `Waiting for LLM server to become ready...`
- `LLM server ready after ...`
- `Running eval: python -m evals.run_eval ...`
- `Eval finished with exit code: 0`

## Output Artifacts
- `~/llmdebug/eval-<JOBID>.out`: job stdout/stderr.
- `~/llmdebug/llm-server-<JOBID>.log`: server launch/runtime.
- `~/llmdebug/eval-launch-<JOBID>.log`: eval launch/runtime.
- `~/llmdebug/enroot-diag-<JOBID>.log`: mount/filesystem diagnostics.
- `~/llmdebug/gpu-util-<JOBID>.csv`: periodic GPU utilization samples.
- `~/llmdebug/gpu-failfast-<JOBID>.txt`: utilization fail-fast reason (only when triggered).
- `~/llmdebug/server-failfast-<JOBID>.txt`: server liveness fail-fast reason (only when triggered).
- `~/llmdebug/run-config-<JOBID>.json`: resolved run contract snapshot.
- `~/llmdebug/failure-dump-<JOBID>.json`: structured failure summary (failure only).
- `~/llmdebug/evals/results/<RUN_ID>.jsonl`: eval results.
- `~/llmdebug/evals/artifacts/`: case artifacts.
- `~/llmdebug/reports/slurm-debug/gguf-eval-debug-<JOBID>-<timestamp>.tar.gz`: bundle output.

## Automatic Failure Bundle
```bash
slurm/collect_gguf_debug_bundle.sh <JOBID>
```

This collects logs plus `scontrol`/`sacct` summaries into one archive.

## Failure Signatures and Recovery
| Signature | Meaning | Recovery |
|---|---|---|
| `enroot-mount: failed to mount ... No such device` | Bind mount incompatibility | Script retries with `${TMPDIR}` staging; inspect `enroot-diag-<JOBID>.log` if still failing. |
| `ERROR: Enroot container '<name>' not found` | Missing enroot named container | Re-run `enroot import` + `enroot create`. |
| `ERROR: Model not found:` | Wrong GGUF model path/file | Verify `MODEL_DIR` + `MODEL_FILE` (`SERVER_BACKEND=llama_cpp`). |
| `LLM server process died` | Server failed before health | Inspect server + diag logs first. |
| `LLM server did not start within timeout` | Slow load/OOM/stuck startup | Lower `CONTEXT_SIZE`/`VLLM_MAX_MODEL_LEN`, or reduce vLLM KV pressure (`VLLM_MAX_NUM_SEQS=1`, `VLLM_KV_CACHE_DTYPE=fp8`, `VLLM_CALCULATE_KV_SCALES=1`), then check model/GPU memory. |
| Eval step fails with mount error | Repo bind failure | Script retries once with staged repo. |
| Low average GPU utilization | GPU often idle | Increase loader parallelism, preprocess/caching, reduce I/O overhead. |

## Path Canonicalization on HAICore
Home can appear as both:
- `/home/kastel/vy3326`
- `/hkfs/home/haicore/kastel/vy3326`

These are canonical/alias views on HAICore. Seeing `/hkfs/home/haicore/...` in logs is expected.

## Enroot vs Pyxis Option Clarification
- `--container-mount-home` is a Slurm Pyxis option (typically with `srun` container flow).
- It is **not** an `enroot start` flag.
- This workflow uses direct `enroot start` with explicit `-m` binds.

## Runtime Override Matrix
Set via `sbatch --export=ALL,KEY=value,... slurm/eval_gguf_eval.sh`.

| Variable | Default | Purpose |
|---|---|---|
| `HOME_DIR` | `$HOME` | Base user directory. |
| `REPO_DIR` | `${HOME_DIR}/llmdebug` | Repo root. |
| `SERVER_BACKEND` | `vllm` | Backend selector: `vllm` (default) or `llama_cpp` (legacy GGUF). |
| `MODEL_PRESET` | `qwen3coder30ba3b-q4_k_m` | GGUF preset key (used when `SERVER_BACKEND=llama_cpp`). |
| `MODEL_DIR` | preset-specific | Host model dir (GGUF mode). |
| `MODEL_FILE` | preset-specific | GGUF filename (GGUF mode). |
| `HF_MODEL_ID` | `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8` | HF model ID for vLLM backend. |
| `HF_MODEL_ALIAS` | `qwen3coder30ba3b` | Served model alias for vLLM/OpenAI API requests. |
| `VLLM_MAX_MODEL_LEN` | `32768` | vLLM max context length. |
| `VLLM_GPU_MEMORY_UTILIZATION` | `0.92` | vLLM GPU memory reservation fraction. |
| `VLLM_TENSOR_PARALLEL_SIZE` | `1` | vLLM tensor parallel degree (single GPU default). |
| `VLLM_DTYPE` | `auto` | vLLM dtype setting. |
| `VLLM_KV_CACHE_DTYPE` | `auto` | vLLM KV cache dtype (`auto`, `fp8`, etc.). |
| `VLLM_CALCULATE_KV_SCALES` | `0` | Add vLLM `--calculate-kv-scales` (recommended with `VLLM_KV_CACHE_DTYPE=fp8`). |
| `VLLM_MAX_NUM_SEQS` | `1` | Max concurrent active sequences in vLLM scheduler (lower is safer for long context on 40GB GPUs). |
| `VLLM_MAX_NUM_BATCHED_TOKENS` | empty | Optional vLLM scheduler cap for total batched tokens per step. |
| `VLLM_TRUST_REMOTE_CODE` | `0` | Pass `--trust-remote-code` to vLLM (enable only if model requires it). |
| `VLLM_DOWNLOAD_DIR` | empty | Host download dir mounted into server container and passed as vLLM `--download-dir`. |
| `HF_CACHE_DIR` | empty | Host HF cache dir mounted into server container at `/cache`. |
| `SERVER_CONTAINER` | backend-specific (`vllm-server-cuda` default, `llama-server-cuda` legacy) | Server container name. |
| `EVAL_CONTAINER` | `python-eval` | Eval container name. |
| `MODEL_ALIAS` | preset-specific | API model alias. |
| `SERVER_PORT` | `18000 + JOB_ID % 1000` | Server local port. |
| `CONTEXT_SIZE` | backend-specific | LLM context size (`vllm`: mirrors `VLLM_MAX_MODEL_LEN`; `llama_cpp`: preset-specific). |
| `GPU_LAYERS` | `99` | GPU offload layers. |
| `SERVER_TEMPERATURE` | preset-specific | llama-server `--temp`. |
| `SERVER_TOP_P` | preset-specific | llama-server `--top-p`. |
| `SERVER_TOP_K` | preset-specific | llama-server `--top-k`. |
| `SERVER_REPEAT_PENALTY` | preset-specific | llama-server `--repeat-penalty`. |
| `SERVER_MIN_P` | preset-specific | llama-server `--min-p`. |
| `SERVER_CACHE_TYPE_K` | empty (`f16` server default) | llama-server `--cache-type-k`. |
| `SERVER_CACHE_TYPE_V` | empty (`f16` server default) | llama-server `--cache-type-v`. |
| `SERVER_FLASH_ATTN` | `off` | llama-server `--flash-attn` (`on/off/auto`). |
| `SERVER_BATCH_SIZE` | `512` | llama-server `--batch-size`. |
| `SERVER_UBATCH_SIZE` | `128` | llama-server `--ubatch-size`. |
| `SERVER_PARALLEL` | backend-specific (`1` for `vllm`; preset-specific for `llama_cpp`) | Server concurrency/control knob (`llama_cpp`: `--parallel`). |
| `SERVER_THREADS` | `${SLURM_CPUS_PER_TASK}` | llama-server `--threads` (empty = server default). |
| `SERVER_THREADS_BATCH` | `${SERVER_THREADS}` | llama-server `--threads-batch`. |
| `SERVER_THREADS_HTTP` | empty | llama-server `--threads-http`. |
| `SERVER_METRICS` | `0` | enable llama-server `/metrics` endpoint. |
| `CUDA_SCALE_LAUNCH_QUEUES` | empty | CUDA queue depth override (advanced). |
| `GGML_CUDA_FORCE_CUBLAS` | empty | force cuBLAS kernels (advanced benchmarking). |
| `GGML_CUDA_FORCE_MMQ` | empty | force MMQ kernels (advanced benchmarking). |
| `GGML_CUDA_ENABLE_UNIFIED_MEMORY` | empty | allow VRAM spillover (slower, last-resort). |
| `SERVER_FAILFAST` | `1` | Abort eval quickly if local LLM server dies or health fails repeatedly. |
| `SERVER_FAILFAST_CHECK_SEC` | `20` | Liveness check interval for server fail-fast. |
| `SERVER_FAILFAST_MAX_CONSEC_HEALTH_FAILS` | `12` | Consecutive failed health checks before server fail-fast triggers. |
| `EVAL_RUN_ID` | `<preset-prefix>-<JOB_ID>` | Resume/output run ID. |
| `EVAL_K` | `3` | Attempts per condition. |
| `EVAL_JOBS` | `4` | Parallel workers. |
| `EVAL_MAX_CASES` | `0` | Max cases (`0`=all). |
| `EVAL_CATEGORY` | empty | Optional category filter. |
| `OPENAI_TEMPERATURE` | backend-specific (`0.0` default for `vllm`; preset-specific for `llama_cpp`) | Eval request temperature (`evals.run_eval`). |
| `MAX_WAIT` | `600` | Health timeout seconds. |
| `POLL_INTERVAL` | `5` | Health poll interval. |
| `DRY_RUN` | `0` | Print config and exit. |
| `SMOKE_TEST_MODE` | `0` | Enable tiny smoke profile. |
| `SMOKE_TEST_MAX_CASES` | `3` | Smoke case cap. |
| `SMOKE_TEST_EVAL_K` | `1` | Smoke attempts. |
| `SMOKE_TEST_CONTEXT_SIZE` | `2048` | Smoke context size. |
| `SMOKE_TEST_MAX_WAIT_SEC` | `240` | Smoke startup timeout. |
| `EVAL_DETERMINISTIC` | `0` | Pass `--deterministic`. |
| `EVAL_SEED` | empty | Seed (required if deterministic). |
| `ENABLE_ENROOT_DEBUG` | `1` | Enable enroot debug logs. |
| `ENABLE_TMPDIR_FALLBACK` | `1` | Enable tmp staging fallback. |
| `GPU_UTIL_MONITOR` | `1` | Record GPU utilization CSV. |
| `GPU_UTIL_POLL_SEC` | `10` | Util monitor interval. |
| `RUN_CONFIG_SNAPSHOT` | `1` | Write run config JSON. |
| `FAILURE_DUMP_MODE` | `1` | Write failure dump JSON. |

## Research Compute Best-Practices (Applied Here)
1. Smoke-test every run before long jobs.
2. Track effective GPU utilization, not just walltime.
3. Record a tiny run contract (`run-config-<JOBID>.json`) for reproducibility.
4. Use successive-halving sweeps (many short jobs, promote winners).
5. Prefer modular short jobs and resume chains for throughput.
6. Make failures informative automatically (`failure-dump-<JOBID>.json`).

## A100 Tuning Suggestions (llama.cpp)
Use these as controlled A/B tests rather than permanent defaults:
1. Flash attention is disabled by default (`SERVER_FLASH_ATTN=off`); enable only for explicit A/B checks.
2. Increase prefill batch gradually: `SERVER_BATCH_SIZE=4096`, `SERVER_UBATCH_SIZE=1024`
3. Decouple server slots from eval workers: tune `SERVER_PARALLEL` independently of `EVAL_JOBS`
4. Set explicit CPU thread counts: `SERVER_THREADS`, `SERVER_THREADS_BATCH`
5. Enable metrics for measurements: `SERVER_METRICS=1`
6. For long-context memory pressure, test KV cache quantization: `SERVER_CACHE_TYPE_K=q8_0`, `SERVER_CACHE_TYPE_V=q8_0`
7. Backend kernel experiments (benchmark carefully): `GGML_CUDA_FORCE_CUBLAS=1` vs default/MMQ

## Repeat Checklist
1. `cd ~/llmdebug`
2. `git pull` if needed.
3. `slurm/check_env_gguf_eval.sh`
4. `slurm/run_gguf_smoke.sh`
5. `sbatch slurm/eval_gguf_eval.sh`
6. monitor logs
7. on failure: `slurm/collect_gguf_debug_bundle.sh <JOBID>`

## Versioned Notes
- Updated workflow date: **2026-02-23**
- Source of truth script: `slurm/eval_gguf_eval.sh`
- Persistent memory docs: `docs/haicore_enroot_eval_runbook.md`, `slurm/README.md`
