# QA Session: 2026-02-09 (Session 4)

## Purpose

Validate mode switching with VRAM wait fix (gpumod-scz) and verify all modes return valid responses.

**Test sequence:** blank → code → rag → code → nemotron → blank

## Environment

- GPU: NVIDIA RTX 4090 (24 GB)
- gpumod version: 0.1.1
- Driver: 580.65.06
- Changes since last QA:
  - gpumod-scz: VRAM wait via pynvml before starting services
  - nvidia-ml-py dependency added

---

## Results

| Transition | Time | Result | Notes |
|------------|------|--------|-------|
| blank → code | <1s | PASS | Chat: "Hello! How can I help you?", 203 tok/s prompt, 171 tok/s gen |
| code → rag | 181s | PASS* | 2048-dim embeddings. vLLM restart cycle due to initialization failure |
| rag → code | 0s | PASS | Chat: "2 + 2 = 4", instant switch |
| code → nemotron | 61s | PASS | Reasoning output. VRAM wait timed out but continued |
| nemotron → blank | 1s | FAIL | Orphan services not stopped (glm-code, vllm-embedding still running) |

*code → rag took 181s total due to vLLM engine initialization failure on first attempt (restart cycle)

---

## Detailed Test Log

### 1. blank → code

```
Starting VRAM: 513 MiB
Mode switch: instant via MCP
Model load: Qwen3-Coder-30B-A3B-Instruct-Q4_K_M (loading state ~4s)

Response:
{"choices":[{"message":{"content":"Hello! Hi there! How can I help you today?"}}],
 "timings":{"prompt_per_second":203.17,"predicted_per_second":171.24}}

Final VRAM: 23027 MiB
```

### 2. code → rag

```
Duration: 181 seconds
Issue: vLLM engine core initialization failed on first attempt
       (RuntimeError: Engine core initialization failed)
       Systemd restart succeeded on 2nd attempt

After restart stable:
  curl http://127.0.0.1:8200/v1/embeddings → 2048 dims

Final VRAM: 6013 MiB
```

### 3. rag → code

```
Duration: 0 seconds (instant)
Model reload required after switch

Response:
{"choices":[{"message":{"content":"2 + 2 = 4"}}],
 "timings":{"predicted_per_second":133.28}}

Final VRAM: 23033 MiB
```

### 4. code → nemotron

```
Duration: 61 seconds
VRAM wait status: "18068 MB free < 20512 MB required after 60.1s"
                  (timed out but continued)

Response:
{"choices":[{"message":{
  "reasoning_content":"The user asks \"What is 2+2?\" That's a simple math question. Answer is 4.",
  "content":""}}]}

Final VRAM: 22567 MiB
```

### 5. nemotron → blank

```
Duration: 1 second
Issue: Orphan services detected after switch
  - glm-code.service: still running
  - nemotron-3-nano.service: sleeping (not stopped)
  - vllm-embedding.service: still running

Manual cleanup required:
  systemctl --user stop glm-code nemotron-3-nano vllm-embedding

Final VRAM after cleanup: 15 MiB
```

---

## Issues Found

### 1. vLLM Engine Initialization Failure (code → rag)

- **Symptom**: Engine core initialization fails on first start attempt
- **Root cause**: Unknown (may be race condition with CUDA context)
- **Workaround**: Systemd Restart=on-failure handles it automatically
- **Impact**: Adds ~60s to transition time

### 2. VRAM Wait Timeout (code → nemotron)

- **Symptom**: VRAM wait times out after 60s with 18 GB free vs 20.5 GB required
- **Root cause**: llama.cpp router keeps model loaded in VRAM even when sleeping
- **Impact**: Mode switch continues anyway (non-blocking), but slower startup

### 3. Orphan Services on Mode Switch (nemotron → blank)

- **Symptom**: Services from previous modes remain running after switch
- **Root cause**: Mode switch logic not stopping services that were started by earlier modes in the same session
- **Impact**: VRAM not released, must manually stop services
- **Ticket**: Needs investigation - may be related to mode state tracking

---

## Summary

- **VRAM wait fix (gpumod-scz)**: Partially working - provides logging but 60s timeout too short for large models
- **Mode switching**: Works but has orphan service issue when chaining multiple mode switches
- **Performance**: Qwen3-Coder at 200+ tok/s, Nemotron reasoning works correctly
- **Recommendation**:
  1. Investigate orphan service issue
  2. Consider longer VRAM wait timeout (90s+) or model-specific timeouts
  3. vLLM cold start reliability needs monitoring
