# Blackwell + vLLM + Docker Eval Runbook

Last verified: March 3, 2026

This runbook captures the working setup for running:
- vLLM OpenAI server in Docker on RTX PRO 6000 Blackwell
- `llmdebug` eval pipeline via `evals/docker/run_eval_docker.sh`

## Known-good baseline

- GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition
- Driver: `590.48.01`
- Docker GPU passthrough: working (`docker run --gpus all nvidia/cuda:12.9.1-runtime-ubuntu22.04 nvidia-smi`)
- vLLM image: `vllm/vllm-openai:nightly-2cbf9656ce6013f7b531bc5a0909d03b88c14862`

## Main failure signatures and what they meant

1. `fatal error: Python.h: No such file or directory`
- Cause: local venv/native build path missing Python dev headers.
- Resolution: use Dockerized vLLM path instead of local build.

2. `nvcc fatal: Unsupported gpu architecture 'compute_120a'`
- Cause: old host NVCC toolchain for Blackwell JIT path.
- Resolution: avoid host compilation path; use suitable vLLM Docker image.

3. `Error 803: system has unsupported display driver / cuda driver combination`
- Cause: container CUDA compat library precedence issue.
- Resolution: strip `cuda-*/compat` entries from `/etc/ld.so.conf.d`, run `ldconfig`, then start vLLM.

## Start vLLM (recommended)

Use the helper script (already includes the compat-lib workaround):

```bash
evals/docker/run_vllm_blackwell.sh
```

Defaults in that script match the validated setup:
- model: `Qwen/Qwen3-Coder-30B-A3B-Instruct`
- served model name: `qwen3coder30ba3b`
- port: `7777`
- dtype: `bfloat16`
- max model len: `131072`
- gpu memory utilization: `0.96`
- kv cache: `fp8` + `--calculate-kv-scales`
- max seqs: `2`
- max batched tokens: `8192`

## Optional: llama.cpp server in Docker (OpenAI-compatible)

If you run a custom `llama-server` image (for example `llama-server-blackwell:latest`)
and append args after the image name, this image's entrypoint script keeps
Blackwell-tuned defaults and appends your args afterwards.

Build image (from repo root):

```bash
docker build --no-cache -f Dockerfile.llama -t llama-server-blackwell:latest .
```

Defaults are controlled with environment variables:
- `LLAMA_PRESET=latency` (default): `--parallel 2`, `--ctx-size 131072`, `-b 2048`, `-ub 1024`
- `LLAMA_PRESET=throughput`: `--parallel 4`, `--ctx-size 262144`, `-b 4096`, `-ub 2048`
- shared defaults include: `--flash-attn auto`, `-ngl all`, `-ctk q8_0`, `-ctv q8_0`, `--kv-unified`, `--metrics`
- optional full manual mode: `-e LLAMA_DISABLE_DEFAULT_ARGS=1`

Working pattern:

```bash
MODEL="/models/models--unsloth--Qwen3.5-35B-A3B-GGUF/snapshots/<snapshot-id>/Qwen3.5-35B-A3B-UD-Q4_K_M.gguf"

docker run --gpus all \
  -v "$(pwd)":/models \
  --ulimit memlock=-1:-1 \
  -p 12000:8080 \
  --restart on-failure:10 \
  -e LLAMA_PRESET=throughput \
  llama-server-blackwell:latest \
  -m "$MODEL"
```

Notes:
- Port mapping `-p 12000:8080` is correct (`host:container`).
- If you pass flags directly, they are appended after defaults (so later values
  override earlier ones if duplicated).
- Replace `<snapshot-id>` with the real snapshot hash from your local model directory.
- Keep the model path quoted via `"$MODEL"` to avoid shell line-wrap/splitting issues.
- Quick health check:
  - `curl http://127.0.0.1:12000/v1/models`

Troubleshooting:
- `llama-server: error while loading shared libraries: libmtmd.so.0`
  - Rebuild with latest `Dockerfile.llama` (uses `BUILD_SHARED_LIBS=OFF` in builder).
- `llama-server: error while loading shared libraries: libgomp.so.1`
  - Rebuild with latest `Dockerfile.llama` (runtime now installs `libgomp1`).
- `...UD-Q4_K_M.gguf: command not found`
  - Shell command likely got split across lines; use the `MODEL="..."` variable form above.

## Verify CUDA inside the vLLM container

```bash
docker run --rm --gpus all --ipc=host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  --entrypoint bash vllm/vllm-openai:nightly-2cbf9656ce6013f7b531bc5a0909d03b88c14862 -lc '
set -eux
for f in /etc/ld.so.conf.d/*.conf; do sed -i "/cuda-.*\/compat/d" "$f"; done
rm -f /etc/ld.so.conf.d/cuda-compat.conf || true
ldconfig
python3 -c "import torch; print(torch.__version__, torch.version.cuda); print(torch.cuda.is_available(), torch.cuda.device_count()); print(torch.cuda.get_device_name(0) if torch.cuda.is_available() else \"no-gpu\")"
'
```

Expected: `torch.cuda.is_available()` is `True` and device name shows Blackwell GPU.

## Run eval (Docker wrapper)

Smoke run:

```bash
bash evals/docker/run_eval_docker.sh \
  --patcher openai \
  --openai-base-url http://127.0.0.1:7777/v1 \
  --openai-model qwen3coder30ba3b \
  --openai-response-mode edits \
  --conditions traceback_only,adaptive_llm_discretion \
  --adaptive-max-helper-calls 1 \
  --retry-prompt-mode full \
  --jobs 2 \
  --k 1 \
  --max-cases 5
```

Full run aligned with `slurm/eval_gguf_eval.sh` defaults:

```bash
RUN_ID="local-qwen3coder30ba3b-$(date +%Y%m%dT%H%M%S)"
bash evals/docker/run_eval_docker.sh \
  --patcher openai \
  --openai-base-url http://127.0.0.1:7777/v1 \
  --openai-model qwen3coder30ba3b \
  --openai-api-key local \
  --openai-timeout-sec 180 \
  --openai-max-propose-sec 300 \
  --openai-max-tokens 2048 \
  --openai-temperature 0.0 \
  --max-context-chars 200000 \
  --prompt-budget-total-chars 180000 \
  --conditions traceback_only,adaptive_llm_discretion \
  --adaptive-max-helper-calls 1 \
  --retry-prompt-mode full \
  --jobs 4 \
  --k 3 \
  --run-id "$RUN_ID"
```

## Notes

- Manual cuDNN install on host is not required for this flow.
- For this setup, keep vLLM server in terminal/session 1 and eval in terminal/session 2.
- If you need pure `evals.run_eval` defaults instead of HAICore-like defaults, set:
  - `--conditions traceback_only,with_snapshot`
