# F2-Prep: Real Model Validation & Customer-Ready Engine

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Validate LLMHosts Core can load and generate coherent text from a real Qwen2.5-0.5B GGUF model, fix all gaps preventing customer-ready dogfooding on the Alienware laptop.

**Architecture:** Download real model + tokenizer, build Rust extensions, fix GGUF parsing issues, add Qwen chat template, add EOS/stop token support from GGUF metadata, add tokenizer auto-discovery. All validation on VPS (CPU) — correctness only, not speed.

**Tech Stack:** Rust (pyo3/maturin), Python, HuggingFace Hub, tokenizers lib

---

### Task 1: Build Rust Extensions

**Files:**
- Build: `llmhosts_core/`, `llmhosts_gpu/`, `relay_core/`

**Step 1: Build all three crates with maturin**

```bash
cd llmhosts_core && maturin develop --release && cd ..
cd llmhosts_gpu && maturin develop --release && cd ..
cd relay_core && maturin develop --release && cd ..
```

**Step 2: Verify extensions import in Python**

```python
import _llmhosts_core; print(_llmhosts_core.LLMHOSTS_CORE_VERSION)
import _llmhosts_gpu; print("GPU ext OK")
import _relay_core; print("Relay ext OK")
```

**Step 3: Run tests with extensions available**

```bash
pytest tests/test_llmhosts_core.py tests/test_llmhosts_gpu.py -v
```

All tests should pass (not skip).

---

### Task 2: Download Real Model & Tokenizer

**Files:**
- Download to: `~/models/qwen2.5-0.5b/`

**Step 1: Download GGUF model**

```bash
mkdir -p ~/models/qwen2.5-0.5b
huggingface-cli download Qwen/Qwen2.5-0.5B-Instruct-GGUF qwen2.5-0.5b-instruct-q4_k_m.gguf --local-dir ~/models/qwen2.5-0.5b/
```

**Step 2: Download tokenizer.json**

```bash
huggingface-cli download Qwen/Qwen2.5-0.5B-Instruct tokenizer.json --local-dir ~/models/qwen2.5-0.5b/
```

**Step 3: Verify files exist**

```bash
ls -lh ~/models/qwen2.5-0.5b/
# Should see: qwen2.5-0.5b-instruct-q4_k_m.gguf (~400MB) and tokenizer.json
```

---

### Task 3: Test Real Model Loading

**Step 1: Try loading the model via Python**

```python
from llmhosts.inference.core import LLMHostsInferenceEngine
engine = LLMHostsInferenceEngine()
info = engine.load_model("/root/models/qwen2.5-0.5b/qwen2.5-0.5b-instruct-q4_k_m.gguf")
print(info)
```

**Step 2: Fix any GGUF parsing errors**

The real model may use tensor naming conventions or metadata keys that differ from our synthetic test model. Debug and fix any:
- Missing metadata keys
- Tensor name mismatches (e.g., `blk.0.attn_q.weight` vs actual names)
- Quantization type not handled
- Dimension mismatches

**Step 3: Verify config extraction is correct**

Compare loaded config against known Qwen2.5-0.5B specs:
- hidden_size: 896
- num_layers: 24
- num_attention_heads: 14
- num_kv_heads: 2
- intermediate_size: 4864
- vocab_size: 151936

---

### Task 4: Test Real Model Generation

**Step 1: Generate tokens (raw)**

```python
tokens = engine.generate_tokens([1, 2, 3], max_tokens=10, temperature=0.0)
print(tokens)  # Should be valid token IDs within vocab range
```

**Step 2: Generate text with tokenizer**

```python
result = engine.generate("Hello, my name is", max_tokens=50, temperature=0.0)
print(result["text"])  # Should be coherent English
```

**Step 3: Fix any generation issues**

- OOB token IDs
- NaN/Inf in logits
- Incorrect output dimensions
- Memory issues (0.5B model is ~400MB — should fit in VPS RAM)

---

### Task 5: Add EOS Token Support from GGUF Metadata

**Files:**
- Modify: `llmhosts_core/src/gguf.rs` (extract EOS token ID from metadata)
- Modify: `llmhosts_core/src/lib.rs` (pass EOS to generate)
- Modify: `llmhosts_core/src/model/config.rs` (store EOS token ID)

**Step 1: Extract EOS token from GGUF metadata**

GGUF files store `tokenizer.ggml.eos_token_id` in metadata. Read it and pass to the generation loop as a stop token.

**Step 2: Wire EOS into generation**

When `generate_tokens()` is called, automatically include the EOS token in stop_tokens.

**Step 3: Test that generation stops at EOS**

---

### Task 6: Add Qwen2.5 Chat Template

**Files:**
- Modify: `llmhosts/proxy/dispatcher.py` (chat template in `_dispatch_core`)

**Step 1: Implement Qwen chat template**

Qwen2.5 expects:
```
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
```

Replace the generic `<|role|>\n{content}` with model-aware templates.

**Step 2: Add template detection**

Based on model architecture or name, select the right template:
- Qwen/Qwen2.5: `<|im_start|>role\ncontent<|im_end|>`
- Llama/Mistral: `[INST] content [/INST]`
- Default: generic format

**Step 3: Test with real model**

---

### Task 7: Add Tokenizer Auto-Discovery

**Files:**
- Modify: `llmhosts/inference/core.py` (_load_tokenizer method)

**Step 1: Search multiple locations for tokenizer.json**

1. Same directory as GGUF file
2. HuggingFace cache (`~/.cache/huggingface/hub/`)
3. Parent directory of GGUF file

**Step 2: Auto-download if not found**

If the model name is recognizable (e.g., contains "qwen2.5"), attempt to download tokenizer.json from HuggingFace.

**Step 3: Test auto-discovery**

---

### Task 8: End-to-End API Test

**Files:**
- Create: `tests/test_core_e2e_real.py` (integration test, skippable)

**Step 1: Write E2E test**

Test the full flow: HTTP request → dispatcher → Core engine → tokenizer → response.

**Step 2: Verify OpenAI-compatible response format**

POST to `/v1/chat/completions` with a Qwen model, verify response has proper structure.

**Step 3: Commit, push, CI green**

---

### Task 9: Full Verification & PR

**Step 1: Run all checks**

```bash
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
ruff check . && ruff format --check .
mypy llmhosts/ --ignore-missing-imports
pytest tests/ -v
```

**Step 2: Create PR, all CI checks green, merge**
