## QA Results

### 2026-02-09

| Step | Mode/Service     | Result | Notes                                                                                                                      |
| ---- | ---------------- | ------ | -------------------------------------------------------------------------------------------------------------------------- |
| 1    | blank            | PASS   | vllm-embedding-code started on port 8210, returned 1024-dim embeddings                                                     |
| 2    | rag              | FAIL   | vllm-embedding OOM -- `gpu_memory_utilization=0.22` too low for Qwen3-VL-Embedding-2B. Preset config issue, not a code bug |
| 3    | code (glm-code)  | PASS   | GLM-4.7-Flash-UD-Q4_K_XL loaded via router, chat completion working                                                        |
| 4    | nemotron         | PASS   | Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL loaded, reasoning + content output                                                      |
| 5    | blank (cleanup)  | PASS   | All services stopped (manual cleanup of orphan from stale MCP)                                                             |
| 6    | devstral-small-2 | FAIL   | vllm crash: `MistralCommonTokenizer` missing `all_special_ids`. Upstream vllm bug                                          |

**Fixes applied during QA:**

- Added `--models-max` rendering to `llamacpp.service.j2` -- router model load was returning 404 without it
- llama.cpp `/models/load` API uses `{"model": "..."}` not `{"id": "..."}`

**Known issues (not gpumod bugs):**

- vllm-embedding needs `gpu_memory_utilization` bumped from 0.22 to ~0.30 in preset
- devstral-small-2 blocked on upstream vllm/mistral tokenizer compatibility
- Stale MCP server process doesn't pick up uncommitted code changes until Claude Code restarts

---

### 2026-02-09 (v0.1.1)

#### New Features QA

| Step | Feature               | Result | Notes                                                              |
| ---- | --------------------- | ------ | ------------------------------------------------------------------ |
| 1    | `gpumod watch --help` | PASS   | Command registered, shows --timeout, --debounce, --no-sync options |
| 2    | watch file filtering  | PASS   | Rejects .swp, .tmp, ~ backup files; accepts .yaml/.yml             |
| 3    | watch debounce        | PASS   | Rapid events coalesced into single sync (500ms default)            |
| 4    | watch error handling  | PASS   | Malformed YAML logged as warning, watcher continues                |
| 5    | auto-sync on startup  | PASS   | cli_context() and MCP lifespan call sync_presets/sync_modes        |
| 6    | --no-sync flag        | PASS   | Skips auto-sync when specified                                     |

#### Mode Switching QA

| Step | Mode/Service     | Result  | Notes                                                                     |
| ---- | ---------------- | ------- | ------------------------------------------------------------------------- |
| 1    | blank            | PASS    | All services stopped cleanly                                              |
| 2    | code             | PASS    | glm-code started, GLM-4.7-Flash-UD-Q4_K_XL responded via llama.cpp router |
| 3    | rag              | TIMEOUT | vllm-embedding health check timed out after 120s (see root cause below)   |
| 4    | code (2nd)       | PASS    | Clean switch after rag timeout, health check OK                           |
| 5    | devstral-small-2 | FAIL    | vllm crash during tokenizer init (see root cause below)                   |

#### Root Cause Analysis

**RAG mode timeout (vllm-embedding)**

- **Symptom**: Health check times out after 120 seconds
- **Journal output**: Model loaded in ~3.5s, but vllm was still initializing pooling task
- **Root cause**: vllm 0.11.0 pooling mode initialization takes longer than 120s on first cold start due to:
  - FlashInfer fallback to PyTorch-native top-p/top-k sampling (performance warning)
  - KV cache preallocation for max_model_len=1024
  - Chunked prefill warmup
- **Generic solution**: Sleep/wake mode switching (gpumod-4dw) eliminates cold starts entirely. Services stay warm in memory, wake in <5s instead of 120s+ cold boot.

**devstral-small-2 failure**

- **Symptom**: vllm crashes immediately on startup
- **Error**: `AttributeError: 'MistralCommonTokenizer' object has no attribute 'all_special_ids'`
- **Stack trace**: `vllm/transformers_utils/tokenizer.py:96 in get_cached_tokenizer`
- **Root cause**: Upstream vllm bug — Mistral's custom tokenizer class doesn't implement the `all_special_ids` property required by vllm's tokenizer caching layer
- **Affected versions**: vllm 0.11.0 with mistralai/Devstral-Small-2505
- **Generic solution**: Model compatibility pre-flight check (gpumod-9i4) validates tokenizer, architecture, and dependencies before attempting startup. Catches incompatibilities early instead of wasting time on doomed starts.

#### Summary

**New features in v0.1.1:**

- `gpumod watch` command for filesystem hot-reload (gpumod-7nz)
- Auto-sync presets and modes on CLI/MCP startup (gpumod-9h8)
- Mode sync from YAML with delete detection (gpumod-652)
- Preset sync from YAML with delete detection (gpumod-7ug)

**gpumod functionality verified:**

- Mode switching starts/stops services correctly
- Health check timeout detection works
- Journal tail provides useful diagnostic output on failure
- Clean mode transitions don't leave orphan processes

---

### 2026-02-09 (session 2)

#### Mode Switching QA

| Step | Mode/Service         | Result | Notes                                                                        |
| ---- | -------------------- | ------ | ---------------------------------------------------------------------------- |
| 1    | blank                | PASS   | VRAM: 18 MiB                                                                 |
| 2    | code (GLM-4.7)       | PASS   | Load OK, chat 348ms, 151 tok/s, VRAM: 16.9 GB                                |
| 3    | rag (vllm-embedding) | PASS\* | Health check timeout (120s), but service works. 2048-dim embeddings in 118ms |
| 4    | code (2nd)           | PASS   | Clean switch, 401ms latency, 114 tok/s, VRAM: 23 GB                          |
| 5    | devstral-small-2     | FAIL   | Known upstream vLLM bug: `MistralCommonTokenizer` missing `all_special_ids`  |
| 6    | blank (cleanup)      | PASS   | All services stopped, VRAM: 15 MiB (after killing orphan process)            |

#### Issues Found

1. **vllm-embedding health timeout** - Known issue, vLLM pooling init takes >120s on cold start
2. **devstral-small-2 crash** - Upstream vLLM tokenizer bug (preflight TokenizerCheck designed to catch this)
3. **Orphan process** - vllm EngineCore from previous session wasn't cleaned up by mode switch

#### Summary

- Mode switching works correctly between blank/code/rag
- GLM-4.7-Flash performs well: ~150 tok/s generation, 350-400ms latency
- vLLM embedding works but health check timeout needs adjustment (>120s cold start)
- devstral-small-2 blocked by upstream vLLM/Mistral tokenizer bug

---

### 2026-02-09 (session 5)

**See [qa/20260209_5_qa.md](qa/20260209_5_qa.md) for detailed results.**

**Bug Fixed: gpumod-77o - Orphan services not stopped during mode switch**

Test sequence: blank → code → rag → code → nemotron → blank

| Transition       | Time | Result   |
| ---------------- | ---- | -------- |
| blank → code     | 1.7s | PASS     |
| code → rag       | 58s  | PASS     |
| rag → code       | 3.4s | PASS     |
| code → nemotron  | 2.1s | PASS     |
| nemotron → blank | 1.0s | **PASS** |

**Improvements:**

- Orphan services now properly stopped (was FAIL, now PASS)
- VRAM wait timeout increased to 120s
- code → rag improved from 181s to 58s
- Final VRAM: 15 MiB (clean)

---

### 2026-02-09 (session 4)

**See [qa/20260209_4_qa.md](qa/20260209_4_qa.md) for detailed results.**

Test sequence: blank → code → rag → code → nemotron → blank

| Transition       | Time | Result |
| ---------------- | ---- | ------ |
| blank → code     | <1s  | PASS   |
| code → rag       | 181s | PASS\* |
| rag → code       | 0s   | PASS   |
| code → nemotron  | 61s  | PASS   |
| nemotron → blank | 1s   | FAIL   |

\*vLLM engine init failure on first attempt (systemd restart recovered)

**Issues found:**

1. vLLM cold start engine init failure (restart recovers)
2. VRAM wait timeout (60s) too short for large models
3. Orphan services not stopped during mode switch (needs investigation)

---

### 2026-02-09 (session 3)

#### Bugs Fixed

1. **Health check timeout fix (gpumod-pbu)** - `_wait_for_healthy` now uses `service.startup_timeout` instead of hardcoded 120s
2. **Orphan GPU process cleanup (gpumod-buv)** - Added `KillMode=control-group` and `TimeoutStopSec=30` to vLLM template
3. **D-Bus environment fix** - MCP server now sets `XDG_RUNTIME_DIR` and `DBUS_SESSION_BUS_ADDRESS` for `systemctl --user` commands

#### Config Updates

- **modes/code.yaml** - Updated description to "Qwen3-Coder", uses `glm-code` service
- **presets/llm/glm-code.yaml** - Renamed to "Qwen3-Coder (llama.cpp Router)", added `default_model: Qwen3-Coder-30B-A3B-Instruct-Q4_K_M`
- **presets/embedding/vllm-embedding.yaml** - Increased `startup_timeout: 180`
- **presets/embedding/vllm-embedding-code.yaml** - Increased `startup_timeout: 120`

#### Mode Switching QA

| Transition       | Started         | Stopped         | Time  | Result |
| ---------------- | --------------- | --------------- | ----- | ------ |
| blank → code     | glm-code        | -               | 0.55s | PASS   |
| code → rag       | vllm-embedding  | glm-code        | 3m 5s | PASS   |
| rag → code       | glm-code        | vllm-embedding  | 0.50s | PASS   |
| code → nemotron  | nemotron-3-nano | glm-code        | 1.08s | PASS   |
| nemotron → blank | -               | nemotron-3-nano | 0.56s | PASS   |

#### Qwen3-Coder Performance (via llama.cpp router)

| Metric              | Value     |
| ------------------- | --------- |
| VRAM usage          | 23 GB     |
| Generation speed    | 202 tok/s |
| Prompt processing   | 268 tok/s |
| Latency (20 tokens) | 99ms      |

#### Issues Found

**VRAM Race Condition (gpumod-scz)**

The `code → rag` transition took 3m 5s due to VRAM not being released immediately:

```
ValueError: Free memory on device (0.82/23.52 GiB) on startup is less than
desired GPU memory utilization (0.22, 5.17 GiB)
```

- glm-code (Qwen3-Coder, 23 GB) stopped by systemd
- vllm-embedding started immediately, but VRAM not yet freed
- vLLM crashed → systemd restart → crash loop (62 restarts)
- Eventually VRAM released and service started

**Fix needed**: LifecycleManager should poll nvidia-smi after stop() and wait for VRAM to be released before starting GPU-heavy services.

#### Summary

- All mode switches eventually successful, but VRAM race causes delays
- `vllm-embedding-code` stays running as shared base service across all modes
- Clean service transitions with no orphan processes (verified by `KillMode=control-group`)
- Qwen3-Coder performs well: 200+ tok/s generation on RTX 4090
