# LLMHosts — User Journeys & Success Criteria

**Last updated:** 2026-03-09
**Status:** LOCKED — authoritative ground truth for all implementation and acceptance testing
**Applies to:** LLMHosts Core F1-F5, Python control plane, Next.js dashboard, Relay tunnel, Marketplace

> This document defines EVERY testable success criterion for the LLMHosts product. If a feature is not listed here, it is not required. If it IS listed here, it MUST pass before the corresponding phase ships. No exceptions.

---

## Table of Contents

1. [Product Identity & Naming](#product-identity--naming)
2. [Journey 1: First Install → Working AI](#journey-1-first-install--working-ai)
3. [Journey 2: Remote Access](#journey-2-remote-access)
4. [Journey 3: OpenAI Drop-In Replacement](#journey-3-openai-drop-in-replacement)
5. [Journey 4: Hardware Maximized](#journey-4-hardware-maximized)
6. [Journey 5: Zero Vendor Lock-In](#journey-5-zero-vendor-lock-in)
7. [Journey 6: Marketplace Operator](#journey-6-marketplace-operator)
8. [Journey 7: Fleet Orchestration (Token)](#journey-7-fleet-orchestration-token)
9. [Journey 8: Privacy-First Enterprise](#journey-8-privacy-first-enterprise)
10. [Journey 9: Developer Experience](#journey-9-developer-experience)
11. [Journey 10: Monitoring & Observability](#journey-10-monitoring--observability)
12. [Success Criteria Registry — F1 Foundation](#f1-success-criteria)
13. [Success Criteria Registry — F2 Performance](#f2-success-criteria)
14. [Success Criteria Registry — F3 Intelligence](#f3-success-criteria)
15. [Success Criteria Registry — F4 Scale](#f4-success-criteria)
16. [Success Criteria Registry — F5 Polish](#f5-success-criteria)
17. [Cross-Cutting Success Criteria](#cross-cutting-success-criteria)
18. [Dogfooding Protocol — Alienware Laptop](#dogfooding-protocol)

---

## Product Identity & Naming

**LLMHosts** turns a home GPU into production AI infrastructure accessible from anywhere.

### Naming Taxonomy (MANDATORY)

| Component | Crate / Package | Struct / Class | Product Name |
|-----------|----------------|----------------|--------------|
| Inference engine | `llmhosts_core` | `LLMHostsEngine`, `LLMHostsModel`, `LLMHostsTransformer` | LLMHosts Core |
| GPU detection | `llmhosts_gpu` | `LLMHostsGpuTopology` | LLMHosts GPU |
| Tunnel relay | `relay_core` | `RelayClient`, `RelayServer` | LLMHosts Relay |
| Python wrapper | `llmhosts` (PyPI) | `LLMHostsInferenceEngine` | LLMHosts |
| CLI binary | `llmhosts` | — | `llmhosts` |
| Frontend | `llmhosts-frontend` | — | LLMHosts Dashboard |
| AI assistant | — | — | Token |

**Rule:** Every struct, class, constant, and product name MUST be grounded in `LLMHosts`. No "Forge", no "Hosts" standalone prefix. Always `LLMHosts` + descriptor.

---

## Journey 1: First Install → Working AI

### Persona
Developer who wants local AI inference. Has an NVIDIA GPU. Never used Ollama.

### J1-SC: Success Criteria (48 criteria)

#### Installation (J1-SC-001 through J1-SC-012)

| ID | Criterion | Phase | Verification |
|----|-----------|-------|-------------|
| J1-SC-001 | `pip install llmhosts` completes in <60s on 100Mbps connection | F1 | Timed install |
| J1-SC-002 | Package size is <60MB (core tier, no ONNX/PyTorch) | F1 | `pip download llmhosts --no-deps && ls -lh` |
| J1-SC-003 | Rust extensions (`_llmhosts_core`, `_llmhosts_gpu`) are pre-compiled in the wheel — no Rust toolchain needed | F1 | `pip install` on machine without rustc |
| J1-SC-004 | `pip install llmhosts[smart]` adds ONNX Runtime + FAISS (~150MB extra) | Existing | Size check |
| J1-SC-005 | `pip install llmhosts[full]` adds PyTorch (~2GB extra) | Existing | Size check |
| J1-SC-006 | Install works on Python 3.10, 3.11, 3.12 | F1 | CI matrix |
| J1-SC-007 | Install works on Linux x86_64 | F1 | CI |
| J1-SC-008 | Install works on macOS arm64 (Apple Silicon) | F5 | Manual test |
| J1-SC-009 | Install works on Windows x86_64 | F5 | Manual test |
| J1-SC-010 | No post-install binary downloads (everything in wheel) | F1 | Network monitor during install |
| J1-SC-011 | `llmhosts --version` prints version after install | Existing | CLI test |
| J1-SC-012 | `llmhosts --help` shows all available commands | Existing | CLI test |

#### Hardware Detection (J1-SC-013 through J1-SC-025)

| ID | Criterion | Phase | Verification |
|----|-----------|-------|-------------|
| J1-SC-013 | `llmhosts_gpu` detects NVIDIA GPUs via NVML (runtime loaded, no compile-time CUDA dep) | F1 | Python: `detect_gpus()` returns non-empty `gpus` list on NVIDIA machine |
| J1-SC-014 | GPU detection returns: name, VRAM total MB, VRAM free MB, driver version, compute capability | F1 | Field assertions on detect_gpus() output |
| J1-SC-015 | Multi-GPU systems report all GPUs with interconnect info (NVLink/PCIe) | F1 | Test on multi-GPU system |
| J1-SC-016 | GPU detection completes in <2s | F1 | Timed call |
| J1-SC-017 | `recommended_strategy` is one of: CpuOnly, SingleGpu, TensorParallel, PhaseSplit | F1 | Enum assertion |
| J1-SC-018 | On machines without NVIDIA GPU, returns `CpuOnly` (no crash, no error) | F1 | Test on CPU-only machine |
| J1-SC-019 | AMD GPU detection via sysfs (Linux) | F5 | Test on AMD machine |
| J1-SC-020 | Apple Silicon GPU detection via IOKit/Metal | F5 | Test on Mac |
| J1-SC-021 | Intel GPU detection via sysfs/Level Zero | F5 | Test on Intel machine |
| J1-SC-022 | VRAM reporting is accurate (within 5% of nvidia-smi) | F1 | Compare with nvidia-smi |
| J1-SC-023 | detect_gpus() returns valid JSON parseable by Python | F1 | json.loads() succeeds |
| J1-SC-024 | CPU info detected (core count, RAM total/free) | F1 | Field assertions |
| J1-SC-025 | Graceful degradation: if _llmhosts_gpu not compiled, detect_gpus() returns empty CpuOnly result | F1 | Monkeypatched import test |

#### Model Loading (J1-SC-026 through J1-SC-042)

| ID | Criterion | Phase | Verification |
|----|-----------|-------|-------------|
| J1-SC-026 | `LLMHostsEngine.load_model("path/to/model.gguf")` loads a GGUF v3 file | F1 | E2E test with real GGUF |
| J1-SC-027 | GGUF v2 files also load correctly (different field widths) | F1 | Test with v2 GGUF |
| J1-SC-028 | GGUF header parsed: magic (0x46475547), version, tensor_count, metadata_count | F1 | Rust unit test |
| J1-SC-029 | All 13 GGUF metadata value types supported (uint8-float64, string, array, bool) | F1 | Rust unit test per type |
| J1-SC-030 | Tensor descriptors parsed: name, dimensions, dtype, offset | F1 | Rust unit test |
| J1-SC-031 | Model weights accessed via mmap (memmap2) — no full file copy into RAM | F1 | Memory usage check |
| J1-SC-032 | Q8_0 dequantization correct: 32 elements/block, f16 scale + 32 int8 quants → f32 | F1 | Rust unit test with known input/output |
| J1-SC-033 | Q4_K_M dequantization correct: 256 elements/block, 6-bit packed scales (bit shift `<< 2` not `<< 4`) | F1 | Rust reference test against known values |
| J1-SC-034 | F16 → F32 conversion correct | F1 | Rust unit test |
| J1-SC-035 | F32 passthrough correct (no conversion needed) | F1 | Rust unit test |
| J1-SC-036 | Model config extracted from GGUF metadata: architecture, hidden_size, num_layers, num_heads, num_kv_heads, vocab_size, max_seq_len, intermediate_size, head_dim, rms_norm_eps, rope_freq_base | F1 | Rust unit test |
| J1-SC-037 | load_model() returns JSON with model config info | F1 | Python test |
| J1-SC-038 | load_model() on non-existent file raises FileNotFoundError | F1 | Python test |
| J1-SC-039 | load_model() on corrupted file raises RuntimeError with descriptive message | F1 | Rust error handling test |
| J1-SC-040 | Model loads in <30s for 0.5B GGUF, <60s for 7B GGUF | F1 | Timed test |
| J1-SC-041 | Attention bias tensors loaded if present in GGUF (Qwen2.5 uses attention biases) | F1 | Test with Qwen2.5 GGUF |
| J1-SC-042 | Tied embeddings: if `output.weight` not in GGUF, falls back to `token_embd.weight` | F1 | Test with tied-weight model |

#### Inference — Forward Pass (J1-SC-043 through J1-SC-070)

| ID | Criterion | Phase | Verification |
|----|-----------|-------|-------------|
| J1-SC-043 | Embedding lookup: token IDs → embedding vectors via gather_rows | F1 | Rust unit test |
| J1-SC-044 | RMS normalization: correct formula `x / sqrt(mean(x^2) + eps) * weight` | F1 | Rust unit test |
| J1-SC-045 | RoPE: precomputed frequency tables for head_dim and max_seq_len | F1 | Rust unit test |
| J1-SC-046 | RoPE: handles variable rope_theta (10000 for Llama, 1000000 for Qwen) | F1 | Rust unit test |
| J1-SC-047 | RoPE: applied to Q and K tensors only (not V) | F1 | Code inspection |
| J1-SC-048 | RoPE: preserves vector norms (rotation property) | F1 | Rust unit test |
| J1-SC-049 | RoPE: start_pos correctly offsets positions during decode step | F1 | Rust unit test |
| J1-SC-050 | GQA attention: multiple Q heads share one KV head (e.g., 14 Q / 2 KV for Qwen2.5-0.5B) | F1 | Rust unit test |
| J1-SC-051 | Causal masking: token at position P can only attend to positions 0..P | F1 | Rust unit test |
| J1-SC-052 | Causal mask correct during decode: single token at start_pos attends to ALL prior positions, not just position 0 | F1 | Rust unit test specifically for decode step |
| J1-SC-053 | Attention scaling: 1/sqrt(head_dim) applied to dot products | F1 | Code inspection |
| J1-SC-054 | Softmax: numerically stable (subtract max before exp) | F1 | Rust unit test with large values |
| J1-SC-055 | SwiGLU FFN: correct formula `silu(gate_proj(x)) * up_proj(x)` then `down_proj()` | F1 | Rust unit test |
| J1-SC-056 | SiLU activation: x * sigmoid(x) | F1 | Rust unit test |
| J1-SC-057 | KV cache: pre-allocated buffer for max_seq_len | F1 | Rust unit test |
| J1-SC-058 | KV cache: append new K/V entries during each decode step | F1 | Rust unit test |
| J1-SC-059 | KV cache: length grows correctly (3 after prefill of 3 tokens, 4 after one decode step) | F1 | Rust unit test |
| J1-SC-060 | KV cache: reset clears all entries | F1 | Rust unit test |
| J1-SC-061 | Forward pass: embed → (attn_norm → QKV → RoPE → KV cache → attention → output_proj + residual → ffn_norm → SwiGLU + residual) × N → final_norm → lm_head | F1 | Integration test |
| J1-SC-062 | Forward pass output shape: [seq_len, vocab_size] | F1 | Rust unit test |
| J1-SC-063 | All logits are finite (no NaN, no Inf) | F1 | Rust unit test assertion |
| J1-SC-064 | Residual connections: hidden state added back after attention and FFN | F1 | Code inspection |
| J1-SC-065 | Attention bias addition: if model has biases, added after Q/K/V projections | F1 | Test with bias model |
| J1-SC-066 | matmul uses matrixmultiply::sgemm (10-20x faster than naive triple loop) | F1 | Code inspection + benchmark |
| J1-SC-067 | LM head: final hidden states projected to vocab_size logits | F1 | Rust unit test |
| J1-SC-068 | Greedy sampling: argmax of logits when temperature=0 | F1 | Rust unit test |
| J1-SC-069 | Top-p (nucleus) sampling: correct cumulative probability cutoff | F1 | Rust unit test |
| J1-SC-070 | Temperature scaling: logits divided by temperature before softmax | F1 | Rust unit test |

#### Text Generation (J1-SC-071 through J1-SC-085)

| ID | Criterion | Phase | Verification |
|----|-----------|-------|-------------|
| J1-SC-071 | `engine.generate("Hello!", max_tokens=32)` accepts string input | F1 | Python E2E test |
| J1-SC-072 | Returns dict with "text" (str), "tokens" (list[int]), "token_count" (int) | F1 | Python E2E test |
| J1-SC-073 | Generated text is coherent (not random garbage) on Qwen2.5-0.5B | F1 | Manual inspection on Alienware |
| J1-SC-074 | Prefill processes all prompt tokens in one forward pass | F1 | KV cache length after prefill = prompt length |
| J1-SC-075 | Decode generates one token per forward pass | F1 | KV cache grows by 1 per step |
| J1-SC-076 | Stop tokens terminate generation early | F1 | Rust unit test |
| J1-SC-077 | max_tokens limit respected | F1 | Rust unit test |
| J1-SC-078 | EOS token terminates generation | F1 | Test with real model |
| J1-SC-079 | generate_tokens() raw API still works for power users | F1 | Python test |
| J1-SC-080 | Tokenizer loaded from co-located tokenizer.json in GGUF directory | F1 | Python test |
| J1-SC-081 | Tokenizer fallback: if no tokenizer.json, raise clear error | F1 | Python test |
| J1-SC-082 | CPU inference: >3 tok/s on Qwen2.5-0.5B (with matrixmultiply) | F1 | Benchmark on Alienware |
| J1-SC-083 | Memory usage: <3GB for Qwen2.5-0.5B (dequantized f32) | F1 | Memory measurement |
| J1-SC-084 | Synthetic test model generates valid (if nonsensical) output | F1 | E2E test with test_utils |
| J1-SC-085 | Test model generator exposed to Python via pyfunction | F1 | Python test |

#### API Integration (J1-SC-086 through J1-SC-100)

| ID | Criterion | Phase | Verification |
|----|-----------|-------|-------------|
| J1-SC-086 | `curl localhost:4000/v1/chat/completions` returns OpenAI-format response from LLMHosts Core | F1 | curl test |
| J1-SC-087 | Response has `id` (chat completion ID) | F1 | JSON field check |
| J1-SC-088 | Response has `object: "chat.completion"` | F1 | JSON field check |
| J1-SC-089 | Response has `created` (Unix timestamp) | F1 | JSON field check |
| J1-SC-090 | Response has `model` (loaded model identifier) | F1 | JSON field check |
| J1-SC-091 | Response has `choices[0].message.content` (generated text) | F1 | JSON field check |
| J1-SC-092 | Response has `choices[0].message.role: "assistant"` | F1 | JSON field check |
| J1-SC-093 | Response has `choices[0].finish_reason` ("stop" or "length") | F1 | JSON field check |
| J1-SC-094 | Response has `usage.prompt_tokens` (accurate count) | F1 | JSON field check |
| J1-SC-095 | Response has `usage.completion_tokens` (accurate count) | F1 | JSON field check |
| J1-SC-096 | Response has `usage.total_tokens` = prompt + completion | F1 | JSON field check |
| J1-SC-097 | Header `X-LLMHosts-Backend: llmhosts-core` present | F1 | Header check |
| J1-SC-098 | Header `X-LLMHosts-Model` contains model identifier | F1 | Header check |
| J1-SC-099 | Header `X-Request-ID` present for tracing | Existing | Header check |
| J1-SC-100 | `/v1/models` includes LLMHosts Core model in list | F1 | API test |

---

## Journey 2: Remote Access

### J2-SC: Success Criteria (25 criteria)

| ID | Criterion | Phase | Verification |
|----|-----------|-------|-------------|
| J2-SC-001 | `llmhosts tunnel start` connects to relay server via WSS | Relay M1 | Connection test |
| J2-SC-002 | Connection established in <3s | Relay M1 | Timed test |
| J2-SC-003 | Prints stable relay URL (e.g., `https://abc123.relay.llmhosts.com`) | Relay M1 | Output check |
| J2-SC-004 | URL persists across reconnects (same device ID = same URL) | Relay M2 | Reconnect test |
| J2-SC-005 | No port forwarding required | Relay M1 | NAT traversal test |
| J2-SC-006 | No DNS configuration required | Relay M1 | Fresh machine test |
| J2-SC-007 | No Tailscale account required | Existing | Tailscale removed |
| J2-SC-008 | No Cloudflare account required | Existing | Cloudflare removed |
| J2-SC-009 | Remote `curl` to relay URL returns inference response | Relay M1 + F1 | Remote curl test |
| J2-SC-010 | Tunnel overhead: <100ms added latency | Relay M1 | Latency measurement |
| J2-SC-011 | Streaming works through tunnel (SSE chunks forwarded) | Relay M1 | Stream test |
| J2-SC-012 | TLS end-to-end (relay.llmhosts.com has valid cert) | Relay M1 | SSL check |
| J2-SC-013 | Auth token required for relay access (401 without it) | Relay M2 | Auth test |
| J2-SC-014 | Noise NK handshake authenticates relay connection | Relay M2 | Protocol test |
| J2-SC-015 | yamux multiplexing for concurrent requests through tunnel | Relay M3 | Concurrent request test |
| J2-SC-016 | Auto-reconnect within 5s of network restoration | Relay M2 | Kill/restore test |
| J2-SC-017 | 503 during downtime with helpful message | Relay M1 | Offline device test |
| J2-SC-018 | Heartbeat keeps connection alive | Existing | Long-running connection test |
| J2-SC-019 | Tunnel status visible in TUI dashboard | Existing | Visual check |
| J2-SC-020 | Tunnel status visible in web dashboard | Existing | E2E test |
| J2-SC-021 | Multiple concurrent tunnel connections supported | Relay M3 | Multi-device test |
| J2-SC-022 | Bandwidth: >50Mbps through tunnel (not bottleneck for inference) | Relay M3 | iperf-like test |
| J2-SC-023 | Connection survives brief network interruptions (<10s) | Relay M2 | Flap test |
| J2-SC-024 | Device registration via device code auth (RFC 8628) | Existing | Auth flow test |
| J2-SC-025 | Privacy: PII never leaves local machine even through tunnel (IslandRun) | Existing | Privacy routing test |

---

## Journey 3: OpenAI Drop-In Replacement

### J3-SC: Success Criteria (35 criteria)

| ID | Criterion | Phase | Verification |
|----|-----------|-------|-------------|
| J3-SC-001 | `OpenAI(base_url="http://localhost:4000/v1")` works with no other changes | F1 | SDK test |
| J3-SC-002 | `client.chat.completions.create()` returns valid response | F1 | SDK test |
| J3-SC-003 | `model="auto"` routes to best available backend | F1 | Test |
| J3-SC-004 | `model="llmhosts"` explicitly targets LLMHosts Core | F1 | Test |
| J3-SC-005 | `stream=True` returns SSE chunks matching OpenAI format | F1 | Stream test |
| J3-SC-006 | Each SSE chunk has `data: {"choices":[{"delta":{"content":"..."}}]}` | F1 | Format test |
| J3-SC-007 | Stream ends with `data: [DONE]` | F1 | Stream test |
| J3-SC-008 | `client.models.list()` returns model list | Existing | API test |
| J3-SC-009 | Local models show `owned_by: "llmhosts-core"` | F1 | Field check |
| J3-SC-010 | Ollama models show `owned_by: "ollama"` | Existing | Field check |
| J3-SC-011 | Cloud models show appropriate owner | Existing | Field check |
| J3-SC-012 | `temperature` parameter respected | F1 | Comparison test |
| J3-SC-013 | `max_tokens` parameter respected | F1 | Length test |
| J3-SC-014 | `top_p` parameter respected | F1 | Comparison test |
| J3-SC-015 | `stop` sequences parameter respected | F1 | Stop test |
| J3-SC-016 | `n` parameter (multiple completions) handled | F2 | Test |
| J3-SC-017 | `presence_penalty` / `frequency_penalty` parameters handled | F2 | Test |
| J3-SC-018 | System message in messages array processed correctly | F1 | Test |
| J3-SC-019 | Multi-turn conversation (system + user + assistant + user) works | F1 | Test |
| J3-SC-020 | Anthropic SDK: `Anthropic(base_url="http://localhost:4000")` works | Existing | SDK test |
| J3-SC-021 | `/v1/messages` returns Anthropic-format response | Existing | SDK test |
| J3-SC-022 | Anthropic streaming (`text_delta` events) works | Existing | Stream test |
| J3-SC-023 | LangChain OpenAI adapter works | F1 | Integration test |
| J3-SC-024 | LlamaIndex OpenAI adapter works | F1 | Integration test |
| J3-SC-025 | Vercel AI SDK works | F1 | Integration test |
| J3-SC-026 | Error response matches OpenAI error format `{"error":{"message":"...","type":"...","code":...}}` | Existing | Error test |
| J3-SC-027 | 429 rate limit response matches OpenAI format | Existing | Rate limit test |
| J3-SC-028 | Three-tier router classifies request before routing | Existing | Router test |
| J3-SC-029 | Semantic cache (vCache) returns cached response when available | Existing | Cache test |
| J3-SC-030 | LLMHosts Core preferred over Ollama when loaded | F1 | Routing test |
| J3-SC-031 | Falls back to Ollama if LLMHosts Core unavailable | F1 | Fallback test |
| J3-SC-032 | Falls back to cloud BYOK if no local backend available | Existing | Fallback test |
| J3-SC-033 | `X-LLMHosts-Cost` header shows inference cost | Existing | Header test |
| J3-SC-034 | Cost tracking accurate for LLMHosts Core (local = $0) | F1 | Metrics test |
| J3-SC-035 | Request/response logged in audit system | Existing | Audit test |

---

## Journey 4: Hardware Maximized

### J4-SC: Success Criteria (30 criteria)

| ID | Criterion | Phase | Verification |
|----|-----------|-------|-------------|
| J4-SC-001 | CubeCL GPU kernels compile to CUDA | F2 | Build test |
| J4-SC-002 | CubeCL GPU kernels compile to ROCm/HIP | F5 | Build test |
| J4-SC-003 | CubeCL GPU kernels compile to WGPU (Metal/Vulkan) | F5 | Build test |
| J4-SC-004 | cudarc NVIDIA optimizations for Flash Attention | F2 | Benchmark |
| J4-SC-005 | PagedAttention KV cache: memory blocks allocated on demand | F2 | Memory test |
| J4-SC-006 | PagedAttention: >24x throughput vs simple KV cache | F2 | Benchmark |
| J4-SC-007 | Continuous batching: new requests added to running batch | F2 | Concurrent test |
| J4-SC-008 | Continuous batching: >23x throughput vs sequential | F2 | Benchmark |
| J4-SC-009 | Flash Attention on NVIDIA: fused kernel, O(N) memory | F2 | Benchmark |
| J4-SC-010 | FP8 KV cache on Hopper+ (H100, RTX 4090) | F2 | Hardware test |
| J4-SC-011 | Qwen2.5-7B on single GPU: >80 tok/s | F2 | Benchmark |
| J4-SC-012 | Qwen2.5-7B TTFT: <200ms | F2 | Benchmark |
| J4-SC-013 | Benchmark vs Ollama: within 20% on single-stream | F2 | Comparison |
| J4-SC-014 | Benchmark vs Ollama: 2x+ on 10 concurrent requests | F2 | Comparison |
| J4-SC-015 | Multi-GPU tensor parallelism across 2+ GPUs | F4 | Test on multi-GPU |
| J4-SC-016 | NVLink interconnect detected and used for tensor parallel | F4 | Hardware test |
| J4-SC-017 | Phase-split serving: prefill on powerful GPU, decode on any | F4 | Architecture test |
| J4-SC-018 | Phase-split: 40% cost reduction vs uniform | F4 | Benchmark |
| J4-SC-019 | 10 concurrent requests: >200 aggregate tok/s | F4 | Benchmark |
| J4-SC-020 | P99 latency for short requests <500ms during long-context | F4 | Latency test |
| J4-SC-021 | GPU memory utilization >80% under load | F2 | Monitoring |
| J4-SC-022 | GPU partitioning: different models on different GPU fractions | F4 | Test |
| J4-SC-023 | Automatic batch size tuning based on available memory | F2 | Adaptive test |
| J4-SC-024 | RadixAttention / prefix caching for repeated prompts | F2 | Cache test |
| J4-SC-025 | CPU fallback path works when no GPU available | F1 | Test on CPU-only |
| J4-SC-026 | Mixed GPU handling (different GPU models in same system) | F4 | Test |
| J4-SC-027 | Memory footprint: <50MB binary (vs ~260MB Ollama, ~2GB vLLM) | F1 | Size check |
| J4-SC-028 | Dashboard shows real-time tok/s per GPU | F3 | Dashboard test |
| J4-SC-029 | Dashboard shows VRAM usage per GPU | F3 | Dashboard test |
| J4-SC-030 | Dashboard shows cost savings vs cloud | Existing | Dashboard test |

---

## Journey 5: Zero Vendor Lock-In

### J5-SC: Success Criteria (20 criteria)

| ID | Criterion | Phase | Verification |
|----|-----------|-------|-------------|
| J5-SC-001 | No Ollama dependency in inference path | F1 | Dependency audit |
| J5-SC-002 | No llama.cpp dependency | F1 | Dependency audit |
| J5-SC-003 | No vLLM dependency | F1 | Dependency audit |
| J5-SC-004 | No external binary downloads during install | F1 | Network monitor |
| J5-SC-005 | Everything ships in pip wheel via maturin | F1 | Wheel contents |
| J5-SC-006 | Works completely offline after install + model download | F1 | Air-gap test |
| J5-SC-007 | No telemetry sent | F1 | Network monitor |
| J5-SC-008 | No license check | F1 | Air-gap test |
| J5-SC-009 | All Rust dependencies are MIT or Apache 2.0 licensed | F1 | `cargo license` |
| J5-SC-010 | Source fully auditable — no obfuscated binaries | F1 | Code review |
| J5-SC-011 | `cargo build` with standard Rust toolchain | F1 | Build test |
| J5-SC-012 | `maturin develop` rebuilds in <60s | F1 | Timed build |
| J5-SC-013 | User can fork and modify inference behavior | F1 | Documentation |
| J5-SC-014 | Ollama remains as optional fallback (graceful degradation) | F1 | Config test |
| J5-SC-015 | Cloud API remains as optional fallback (BYOK keys) | Existing | Config test |
| J5-SC-016 | No Docker required for basic operation | F1 | Bare metal test |
| J5-SC-017 | Config stored locally at `~/.config/llmhosts/config.toml` | Existing | Config test |
| J5-SC-018 | All data stored locally (no cloud storage) | F1 | Data audit |
| J5-SC-019 | Uninstall is clean: `pip uninstall llmhosts` removes everything | F1 | Uninstall test |
| J5-SC-020 | Uninstall never removes user's models or config | Existing | Uninstall test |

---

## Journey 6: Marketplace Operator

### J6-SC: Success Criteria (22 criteria)

| ID | Criterion | Phase | Verification |
|----|-----------|-------|-------------|
| J6-SC-001 | `llmhosts serve --marketplace` registers device | Existing | Registration test |
| J6-SC-002 | Device appears on marketplace within 30s | Existing | Timing test |
| J6-SC-003 | LLMHosts Relay tunnel auto-configured for marketplace | Existing | Tunnel test |
| J6-SC-004 | Pricing set via API/CLI | Existing | Pricing test |
| J6-SC-005 | Buyer requests routed to operator GPU | Existing | Routing test |
| J6-SC-006 | Billing event created per request (token-level accuracy) | Existing | Billing test |
| J6-SC-007 | 80% revenue to operator, 20% platform fee | Existing | Fee split test |
| J6-SC-008 | Stripe Connect payout within 7 days | Existing | Payout test |
| J6-SC-009 | Reputation scoring: weighted uptime/latency/error rate | Existing | Score test |
| J6-SC-010 | Tier system: Unverified → Standard → Verified → Elite | Existing | Tier test |
| J6-SC-011 | Auto-removal: nodes with <30% 30d uptime removed from marketplace | Existing | Auto-removal test |
| J6-SC-012 | Operator dashboard shows earnings, requests, uptime | Existing | Dashboard test |
| J6-SC-013 | GPU remit system: operator sets which % of GPU to share | Existing | Remit test |
| J6-SC-014 | Retry logic: failed requests retried on different operator | Existing | Retry test |
| J6-SC-015 | MarketplaceRefresher: 5min background sync | Existing | Sync test |
| J6-SC-016 | Buyer credits system | Existing | Credits test |
| J6-SC-017 | At-capacity cloud fallback when all operators busy | Existing | Overflow test |
| J6-SC-018 | Buyer can filter by model, latency, price | Existing | Filter test |
| J6-SC-019 | SLA enforcement: operators must meet tier requirements | Existing | SLA test |
| J6-SC-020 | Device heartbeat with 5 fields (GPU util, VRAM, temp, load, uptime) | Existing | Heartbeat test |
| J6-SC-021 | Control routes: /control/pull, /control/restart, /control/status | Existing | Control test |
| J6-SC-022 | SSE device status updates to dashboard | Existing | SSE test |

---

## Journey 7: Fleet Orchestration (Token)

### J7-SC: Success Criteria (20 criteria)

| ID | Criterion | Phase | Verification |
|----|-----------|-------|-------------|
| J7-SC-001 | Token chat available at /dashboard/token | Existing | Nav test |
| J7-SC-002 | System prompt enriched with fleet context (device count, status) | Existing | Prompt test |
| J7-SC-003 | Intent classifier routes to correct model (chat→Flash, research→Sonnet) | Existing | Classifier test |
| J7-SC-004 | Fleet actions: propose → approve → dispatch | Existing | Action flow test |
| J7-SC-005 | `POST /control/pull` dispatched to device tunnel URL | Existing | Dispatch test |
| J7-SC-006 | Cost analytics: monthly spend, per-model breakdown | Existing | Analytics test |
| J7-SC-007 | Cost savings vs OpenAI calculated and displayed | Existing | Savings test |
| J7-SC-008 | Proactive alerts: device offline, budget exceeded | Existing | Alert test |
| J7-SC-009 | Email notifications with cooldown (1h offline, 4h budget) | Existing | Email test |
| J7-SC-010 | Research caching: 24h TTL, no repeated Sonnet calls | Existing | Cache test |
| J7-SC-011 | SSE streaming protocol: conversationId+intent → token deltas → metadata | Existing | Stream test |
| J7-SC-012 | Multi-turn conversations persisted in database | Existing | DB test |
| J7-SC-013 | Markdown rendering in chat UI | Existing | UI test |
| J7-SC-014 | Action cards with approve/reject buttons | Existing | UI test |
| J7-SC-015 | Intent badges (chat, research, fleet, analysis) | Existing | UI test |
| J7-SC-016 | 169 Token tests pass | Existing | `npm run test` |
| J7-SC-017 | Conversation export (Phase 4) | Token P4 | Test |
| J7-SC-018 | Model performance benchmarking (Phase 4) | Token P4 | Test |
| J7-SC-019 | Fleet scheduling (Phase 4) | Token P4 | Test |
| J7-SC-020 | Monthly email summary | Existing | Email test |

---

## Journey 8: Privacy-First Enterprise

### J8-SC: Success Criteria (18 criteria)

| ID | Criterion | Phase | Verification |
|----|-----------|-------|-------------|
| J8-SC-001 | `--privacy strict` flag blocks all external routing | Existing + F1 | Config test |
| J8-SC-002 | IslandRun trust levels: personal=1.0, private=0.7, external=0.4 | Existing | Router test |
| J8-SC-003 | PII detection before external API routing | Existing | PromptGuard test |
| J8-SC-004 | Strict mode: zero network connections except localhost | F1 | netstat verification |
| J8-SC-005 | Even with cloud keys configured, strict mode blocks them | F1 | Config test |
| J8-SC-006 | `X-LLMHosts-Privacy: strict` header on all responses in strict mode | F1 | Header test |
| J8-SC-007 | Audit log: every request logged with routing decision | Existing | Audit test |
| J8-SC-008 | `llmhosts audit export --format json` works | Existing | Export test |
| J8-SC-009 | No PII in audit logs (redacted) | Existing | Log scan |
| J8-SC-010 | TrustShield PromptGuard: injection detection | Existing | Security test |
| J8-SC-011 | TrustLedger: HMAC chain for audit integrity | Existing | Chain test |
| J8-SC-012 | AdaptiveShield: sliding window rate limiting | Existing | Rate limit test |
| J8-SC-013 | SecureMesh: mTLS for inter-service communication | Existing | TLS test |
| J8-SC-014 | TransparencyDashboard: real-time security metrics | Existing | Dashboard test |
| J8-SC-015 | SQLite audit log (no external database required) | Existing | DB test |
| J8-SC-016 | Encrypted key storage (Fernet) for BYOK keys | Existing | Encryption test |
| J8-SC-017 | OWASP top 10 mitigated (XSS, injection, CSRF, etc.) | Existing | Security tests |
| J8-SC-018 | Rate limiting per API key | Existing | Rate limit test |

---

## Journey 9: Developer Experience

### J9-SC: Success Criteria (25 criteria)

| ID | Criterion | Phase | Verification |
|----|-----------|-------|-------------|
| J9-SC-001 | `llmhosts doctor` runs full diagnostics | Existing | CLI test |
| J9-SC-002 | Doctor checks: Python version, GPU drivers, network, config, model status | Existing | Output check |
| J9-SC-003 | `llmhosts setup` interactive wizard for first-time users | Existing | Wizard test |
| J9-SC-004 | Config priority: defaults → TOML → env vars → CLI flags | Existing | Config test |
| J9-SC-005 | `LLMHOSTS_*` environment variables override config | Existing | Env test |
| J9-SC-006 | `--config` flag specifies custom config file | Existing | Flag test |
| J9-SC-007 | TUI dashboard during `llmhosts serve` (Textual) | Existing | Visual test |
| J9-SC-008 | `--no-tui` flag disables TUI for headless operation | Existing | Flag test |
| J9-SC-009 | Web dashboard at localhost:3000 | Existing | HTTP test |
| J9-SC-010 | Dashboard: device list with status | Existing | E2E test |
| J9-SC-011 | Dashboard: live pipeline stream (SSE) | Existing | SSE test |
| J9-SC-012 | Dashboard: 9 documentation pages (5 public, 4 auth-gated) | Existing | Page test |
| J9-SC-013 | Auth: NextAuth v5 with device code auth | Existing | Auth test |
| J9-SC-014 | Prisma + SQLite for frontend data | Existing | DB test |
| J9-SC-015 | Docker support: `docker compose build dev` | Existing | Docker test |
| J9-SC-016 | Production Docker: Traefik routing | Existing | Docker test |
| J9-SC-017 | `/health` endpoint returns 200 with backend status | Existing | Health test |
| J9-SC-018 | `/ready` endpoint returns 200 only when fully initialized | Existing | Ready test |
| J9-SC-019 | `/openapi.json` returns valid OpenAPI spec | Existing | Spec test |
| J9-SC-020 | Prometheus metrics endpoint | Existing | Metrics test |
| J9-SC-021 | Grafana dashboard (via monitoring stack) | Existing | Dashboard test |
| J9-SC-022 | Loki log aggregation | Existing | Logging test |
| J9-SC-023 | `pip install -e ".[dev]"` editable install for development | Existing | Install test |
| J9-SC-024 | `ruff check . && ruff format .` lint and format | Existing | CI check |
| J9-SC-025 | `mypy llmhosts/ --ignore-missing-imports` type check | Existing | CI check |

---

## Journey 10: Monitoring & Observability

### J10-SC: Success Criteria (15 criteria)

| ID | Criterion | Phase | Verification |
|----|-----------|-------|-------------|
| J10-SC-001 | Cost tracking per request (cloud cost, local cost = $0) | Existing | Metrics test |
| J10-SC-002 | Savings calculation: actual cost vs what it would cost on OpenAI | Existing | Savings test |
| J10-SC-003 | `X-LLMHosts-Cost` header on every response | Existing | Header test |
| J10-SC-004 | Prometheus metrics: request count, latency histogram, error count | Existing | Metrics test |
| J10-SC-005 | Per-backend metrics (LLMHosts Core vs Ollama vs cloud) | F1 | Metrics test |
| J10-SC-006 | GPU utilization tracking (NVML) | F1 | Monitoring test |
| J10-SC-007 | VRAM usage tracking | F1 | Monitoring test |
| J10-SC-008 | Model load/unload events logged | F1 | Audit test |
| J10-SC-009 | Token throughput (tok/s) tracking | F1 | Metrics test |
| J10-SC-010 | Time-to-first-token (TTFT) tracking | F1 | Metrics test |
| J10-SC-011 | Request queue depth tracking | F2 | Metrics test |
| J10-SC-012 | Batch size tracking | F2 | Metrics test |
| J10-SC-013 | KV cache utilization tracking | F2 | Metrics test |
| J10-SC-014 | Alerting: GPU temperature thresholds | F3 | Alert test |
| J10-SC-015 | Alerting: memory pressure warnings | F3 | Alert test |

---

<a id="f1-success-criteria"></a>
## F1 Success Criteria Registry

**Total F1 criteria: 112**
(From journeys: J1-SC-001 through J1-SC-100, J4-SC-025, J4-SC-027, J5-SC-001 through J5-SC-020, J10-SC-005 through J10-SC-010, plus the below)

### F1 Test Suite Requirements

| ID | Criterion | Verification |
|----|-----------|-------------|
| F1-TS-001 | `cargo test --workspace` passes (>60 Rust tests) | CI |
| F1-TS-002 | `pytest tests/ -v` passes (all existing + new tests, zero regressions) | CI |
| F1-TS-003 | `cargo clippy --workspace` — zero warnings | CI |
| F1-TS-004 | `ruff check .` — zero violations | CI |
| F1-TS-005 | `ruff format --check .` — zero violations | CI |
| F1-TS-006 | Python tests run against real compiled Rust (no mocks except 1 degradation test) | Test inspection |
| F1-TS-007 | Synthetic GGUF test model exercises full pipeline (not real multi-GB model) | Test inspection |
| F1-TS-008 | E2E test: load test model → generate tokens → valid output | pytest |
| F1-TS-009 | Benchmark script produces tok/s measurement | Manual run |
| F1-TS-010 | All tests pass on Alienware laptop | Dogfooding |

### F1 Performance Floor

| Metric | Target | Model |
|--------|--------|-------|
| tok/s (CPU, decode) | >3 | Qwen2.5-0.5B Q4_K_M |
| TTFT (CPU, 50-token prompt) | <10s | Qwen2.5-0.5B Q4_K_M |
| Model load time | <30s | Qwen2.5-0.5B Q4_K_M |
| Memory usage | <3GB | Qwen2.5-0.5B Q4_K_M (dequantized f32) |
| Binary size (Rust extensions) | <50MB | Both crates combined |

---

<a id="f2-success-criteria"></a>
## F2 Success Criteria Registry

**Total F2 criteria: 28**
(J4-SC-001 through J4-SC-014, J4-SC-021 through J4-SC-024, J10-SC-011 through J10-SC-013, plus below)

| ID | Criterion | Verification |
|----|-----------|-------------|
| F2-PERF-001 | Qwen2.5-7B single GPU: >80 tok/s | Benchmark |
| F2-PERF-002 | TTFT <200ms for 7B model | Benchmark |
| F2-PERF-003 | Within 20% of Ollama on single-stream throughput | Comparison |
| F2-PERF-004 | 2x+ Ollama on 10 concurrent requests | Comparison |
| F2-PERF-005 | PagedAttention: >24x throughput improvement | Benchmark |
| F2-PERF-006 | Continuous batching: >23x throughput improvement | Benchmark |

---

<a id="f3-success-criteria"></a>
## F3 Success Criteria Registry

**Total F3 criteria: 18**

| ID | Criterion | Verification |
|----|-----------|-------------|
| F3-HIM-001 | Hardware Intelligence Matrix maps hardware → optimal model | Unit test |
| F3-HIM-002 | At least 10 hardware profiles in HIM database | Data inspection |
| F3-HIM-003 | CPU-only profile has valid model recommendation | Test on CPU-only machine |
| F3-HIM-004 | Fallback cascade: try Q4_K_M → Q3_K → smaller model → error | Test |
| F3-DL-001 | Model download from HuggingFace with progress bar | Manual test |
| F3-DL-002 | Download resume after interruption | Kill/resume test |
| F3-DL-003 | Disk space check before download | Low-disk test |
| F3-DL-004 | SHA256 verification after download | Hash check |
| F3-DL-005 | Storage location: `~/.llmhosts/models/` | Path check |
| F3-AUTO-001 | `llmhosts serve` (zero args) auto-detects hardware, recommends model, prompts download | Manual test |
| F3-AUTO-002 | Auto-selection completes in <5s | Timed test |
| F3-SWAP-001 | Hot-swap protocol: <100ms interruption | Timed test |
| F3-SWAP-002 | Zero dropped requests during swap (drain → activate) | Load test |
| F3-AGENT-001 | Model Agent scans for better models periodically | Agent test |
| F3-AGENT-002 | Notification when better model available | Notification test |
| F3-DASH-001 | Dashboard shows LLMHosts Core model info | E2E test |
| F3-DASH-002 | Dashboard shows inference metrics (tok/s, TTFT) | E2E test |
| F3-DASH-003 | Dashboard shows GPU utilization graph | E2E test |

---

<a id="f4-success-criteria"></a>
## F4 Success Criteria Registry

**Total F4 criteria: 12**
(J4-SC-015 through J4-SC-022, J4-SC-026, plus below)

| ID | Criterion | Verification |
|----|-----------|-------------|
| F4-TP-001 | 2-GPU tensor parallelism works | Multi-GPU test |
| F4-TP-002 | 4-GPU tensor parallelism works | Multi-GPU test |
| F4-TP-003 | 8-GPU tensor parallelism works (datacenter) | Datacenter test |
| F4-PS-001 | Phase-split serving reduces cost by >40% | Cost comparison |

---

<a id="f5-success-criteria"></a>
## F5 Success Criteria Registry

**Total F5 criteria: 15**
(J1-SC-008, J1-SC-009, J4-SC-002, J4-SC-003, plus below)

| ID | Criterion | Verification |
|----|-----------|-------------|
| F5-WIN-001 | Windows installer (MSI) | Install test |
| F5-WIN-002 | Windows system tray app | UI test |
| F5-WIN-003 | Windows service mode | Service test |
| F5-MAC-001 | Homebrew formula | `brew install llmhosts` |
| F5-MAC-002 | macOS menu bar app | UI test |
| F5-LINUX-001 | apt/deb package | Package test |
| F5-LINUX-002 | systemd service file | Service test |
| F5-AMD-001 | AMD GPU inference via CubeCL/ROCm | Benchmark on AMD |
| F5-APPLE-001 | Apple Silicon inference via CubeCL/Metal | Benchmark on Mac |
| F5-INTEL-001 | Intel GPU inference via CubeCL/WGPU | Benchmark on Intel |
| F5-UX-001 | One-click install → working inference flow | Fresh machine test |

---

## Cross-Cutting Success Criteria

These apply to ALL phases and must never regress.

| ID | Criterion | Verification |
|----|-----------|-------------|
| CC-001 | Zero Forge references in codebase, docs, or GitHub (LLMHosts branding only) | grep audit |
| CC-002 | All Python tests pass (`pytest tests/ -v`) | CI |
| CC-003 | All frontend tests pass (`npm run test` in frontend/) | CI |
| CC-004 | All E2E tests pass (`npm run test:e2e` in frontend/) | CI |
| CC-005 | Ruff lint clean | CI |
| CC-006 | Ruff format clean | CI |
| CC-007 | MyPy type check passes | CI |
| CC-008 | Cargo clippy clean | CI |
| CC-009 | Cargo test passes | CI |
| CC-010 | Docker build succeeds | CI |
| CC-011 | No console.* in frontend/src/app/api (use logger) | Lint rule |
| CC-012 | WCAG 2.2 AA accessibility | axe-core |
| CC-013 | Branch protection: PR + CI required for main | GitHub ruleset |
| CC-014 | Conventional commits | commitlint |
| CC-015 | No secrets in code (no .env committed) | Secret scan |
| CC-016 | Pydantic v2 for all request/response validation | Code review |
| CC-017 | async/await everywhere in FastAPI paths | Code review |
| CC-018 | Type hints on all functions | MyPy |
| CC-019 | Docstrings on public functions | Code review |
| CC-020 | GitHub project board updated after each PR | Process check |

---

<a id="dogfooding-protocol"></a>
## Dogfooding Protocol — Alienware Laptop

### Hardware Profile (to be confirmed on first test)

```
GPU: NVIDIA (model TBD — detect with nvidia-smi)
RAM: TBD
CPU: TBD
OS: Windows (primary) or Linux (WSL2)
Disk: SSD (for model loading speed)
```

### Pre-Test Setup

1. Clone repo: `git clone https://github.com/LookNoHandsMom/LLMHosts.com.git`
2. Install: `pip install -e ".[dev]"`
3. Build Rust: `cd llmhosts_gpu && maturin develop && cd .. && cd llmhosts_core && maturin develop --release && cd ..`
4. Download test model: `huggingface-cli download Qwen/Qwen2.5-0.5B-Instruct-GGUF qwen2.5-0.5b-instruct-q4_k_m.gguf --local-dir ~/models/`

### Test Sequence (ALL must pass)

| # | Test | Command | Expected |
|---|------|---------|----------|
| 1 | Rust unit tests | `cargo test --workspace` | All pass, >60 tests |
| 2 | Rust lint | `cargo clippy --workspace` | Zero warnings |
| 3 | Python tests | `pytest tests/ -v --timeout=120` | All pass, zero regressions |
| 4 | Python lint | `ruff check . && ruff format --check .` | Clean |
| 5 | GPU detection | `python -c "from llmhosts.inference.gpu import detect_gpus; import json; print(json.dumps(detect_gpus(), indent=2))"` | Shows Alienware GPU with accurate VRAM |
| 6 | Synthetic E2E | `pytest tests/test_llmhosts_core.py -v` | All pass including E2E with generated model |
| 7 | Real model load | `python -c "from llmhosts.inference.core import LLMHostsInferenceEngine; e = LLMHostsInferenceEngine(); e.load_model('~/models/qwen2.5-0.5b-instruct-q4_k_m.gguf'); print('Loaded!')"` | Loads without error |
| 8 | Real model inference | `python -c "from llmhosts.inference.core import LLMHostsInferenceEngine; e = LLMHostsInferenceEngine(); e.load_model('~/models/qwen2.5-0.5b-instruct-q4_k_m.gguf'); r = e.generate('Hello! How are you?', max_tokens=32); print(r['text'])"` | Prints coherent text |
| 9 | API server | `llmhosts serve --model ~/models/qwen2.5-0.5b-instruct-q4_k_m.gguf --no-tui` (in background) | Server starts on port 4000 |
| 10 | API test | `curl -s http://localhost:4000/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"auto","messages":[{"role":"user","content":"What is 2+2?"}]}' \| python -m json.tool` | Valid OpenAI response with answer |
| 11 | Models list | `curl -s http://localhost:4000/v1/models \| python -m json.tool` | Shows LLMHosts Core model |
| 12 | Benchmark | `python scripts/benchmark_core.py ~/models/qwen2.5-0.5b-instruct-q4_k_m.gguf` | Prints tok/s (expect >3 on CPU) |
| 13 | Health check | `curl -s http://localhost:4000/health` | 200 OK with backend status |
| 14 | Full test suite | `pytest tests/ -v --cov=llmhosts --cov-report=term-missing` | Coverage report, all pass |

### Success = ALL 14 tests pass with zero failures, zero warnings, zero errors.

### Failure Protocol

If any test fails:
1. Do NOT proceed to the next test
2. Diagnose the root cause
3. Fix the issue
4. Re-run ALL tests from the beginning
5. Repeat until 14/14 pass

---

## Score Card

| Phase | Total Criteria | Status |
|-------|---------------|--------|
| F1 | 112 | IN PROGRESS |
| F2 | 28 | NOT STARTED |
| F3 | 18 | NOT STARTED |
| F4 | 12 | NOT STARTED |
| F5 | 15 | NOT STARTED |
| Cross-Cutting | 20 | ONGOING |
| Existing (already built) | ~200+ | PASSING |
| **TOTAL** | **~405** | — |

**Launch readiness = 100% of phase criteria pass + 100% cross-cutting + 14/14 dogfood tests.**
