# Blackwell + vLLM + Docker Eval Runbook

Last verified: February 25, 2026

This runbook captures the working setup for running:
- vLLM OpenAI server in Docker on RTX PRO 6000 Blackwell
- `llmdebug` eval pipeline via `evals/docker/run_eval_docker.sh`

## Known-good baseline

- GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition
- Driver: `590.48.01`
- Docker GPU passthrough: working (`docker run --gpus all nvidia/cuda:12.9.1-runtime-ubuntu22.04 nvidia-smi`)
- vLLM image: `vllm/vllm-openai:nightly-2cbf9656ce6013f7b531bc5a0909d03b88c14862`

## Main failure signatures and what they meant

1. `fatal error: Python.h: No such file or directory`
- Cause: local venv/native build path missing Python dev headers.
- Resolution: use Dockerized vLLM path instead of local build.

2. `nvcc fatal: Unsupported gpu architecture 'compute_120a'`
- Cause: old host NVCC toolchain for Blackwell JIT path.
- Resolution: avoid host compilation path; use suitable vLLM Docker image.

3. `Error 803: system has unsupported display driver / cuda driver combination`
- Cause: container CUDA compat library precedence issue.
- Resolution: strip `cuda-*/compat` entries from `/etc/ld.so.conf.d`, run `ldconfig`, then start vLLM.

## Start vLLM (recommended)

Use the helper script (already includes the compat-lib workaround):

```bash
evals/docker/run_vllm_blackwell.sh
```

Defaults in that script match the validated setup:
- model: `Qwen/Qwen3-Coder-30B-A3B-Instruct`
- served model name: `qwen3coder30ba3b`
- port: `7777`
- dtype: `bfloat16`
- max model len: `131072`
- gpu memory utilization: `0.96`
- kv cache: `fp8` + `--calculate-kv-scales`
- max seqs: `2`
- max batched tokens: `8192`

## Optional: llama.cpp server in Docker (OpenAI-compatible)

If you run a custom `llama-server` image (for example `llama-server-blackwell:latest`)
and append args after the image name, Docker replaces the image `CMD`.
That means default flags from the Dockerfile `CMD` are not applied unless you pass
them explicitly.

Working pattern:

```bash
docker run --gpus all \
  -v "$(pwd)":/models \
  --ulimit memlock=-1:-1 \
  -p 12000:8080 \
  --restart on-failure:10 \
  llama-server-blackwell:latest \
  --host 0.0.0.0 \
  --port 8080 \
  -m /models/models--unsloth--Qwen3.5-35B-A3B-GGUF/snapshots/<snapshot-id>/Qwen3.5-35B-A3B-UD-Q4_K_M.gguf
```

Notes:
- Port mapping `-p 12000:8080` is correct (`host:container`).
- Keep `--host 0.0.0.0` when overriding `CMD`; otherwise service binding may not
  be reachable from the host.
- Quick health check:
  - `curl http://127.0.0.1:12000/v1/models`

## Verify CUDA inside the vLLM container

```bash
docker run --rm --gpus all --ipc=host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  --entrypoint bash vllm/vllm-openai:nightly-2cbf9656ce6013f7b531bc5a0909d03b88c14862 -lc '
set -eux
for f in /etc/ld.so.conf.d/*.conf; do sed -i "/cuda-.*\/compat/d" "$f"; done
rm -f /etc/ld.so.conf.d/cuda-compat.conf || true
ldconfig
python3 -c "import torch; print(torch.__version__, torch.version.cuda); print(torch.cuda.is_available(), torch.cuda.device_count()); print(torch.cuda.get_device_name(0) if torch.cuda.is_available() else \"no-gpu\")"
'
```

Expected: `torch.cuda.is_available()` is `True` and device name shows Blackwell GPU.

## Run eval (Docker wrapper)

Smoke run:

```bash
bash evals/docker/run_eval_docker.sh \
  --patcher openai \
  --openai-base-url http://127.0.0.1:7777/v1 \
  --openai-model qwen3coder30ba3b \
  --openai-response-mode edits \
  --conditions traceback_only,adaptive_llm_discretion \
  --adaptive-max-helper-calls 1 \
  --retry-prompt-mode full \
  --jobs 2 \
  --k 1 \
  --max-cases 5
```

Full run aligned with `slurm/eval_gguf_eval.sh` defaults:

```bash
RUN_ID="local-qwen3coder30ba3b-$(date +%Y%m%dT%H%M%S)"
bash evals/docker/run_eval_docker.sh \
  --patcher openai \
  --openai-base-url http://127.0.0.1:7777/v1 \
  --openai-model qwen3coder30ba3b \
  --openai-api-key local \
  --openai-timeout-sec 180 \
  --openai-max-propose-sec 300 \
  --openai-max-tokens 2048 \
  --openai-temperature 0.0 \
  --max-context-chars 200000 \
  --prompt-budget-total-chars 180000 \
  --conditions traceback_only,adaptive_llm_discretion \
  --adaptive-max-helper-calls 1 \
  --retry-prompt-mode full \
  --jobs 4 \
  --k 3 \
  --run-id "$RUN_ID"
```

## Notes

- Manual cuDNN install on host is not required for this flow.
- For this setup, keep vLLM server in terminal/session 1 and eval in terminal/session 2.
- If you need pure `evals.run_eval` defaults instead of HAICore-like defaults, set:
  - `--conditions traceback_only,with_snapshot`
