# LLMHosts Core-First Wheel Packaging & Complete Product Design

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Ship LLMHosts as a single `pip install llmhosts` wheel with pre-compiled Rust extensions (inference engine, GPU detection, tunnel, router) — no Ollama, no external dependencies, GPU-first inference on every platform.

**Architecture:** Maturin replaces Hatchling as build backend. A unified Rust bridge crate (`_llmhosts`) re-exports all 4 Rust crates as submodules. CubeCL provides write-once GPU kernels compiled to CUDA/ROCm/Metal/Vulkan. CPU fallback always available. Core-first `llmhosts up` pipeline replaces Ollama-based flow.

**Tech Stack:** Rust (pyo3 0.28 + maturin 1.8+), CubeCL (GPU kernels), cudarc (NVIDIA hot paths), Python 3.10+ (FastAPI, Click CLI), HuggingFace Hub (model downloads)

---

## Table of Contents

1. [Wheel Packaging](#1-wheel-packaging)
2. [Bridge Crate Architecture](#2-bridge-crate-architecture)
3. [GPU Compute — CubeCL + cudarc](#3-gpu-compute)
4. [CPU Fallback](#4-cpu-fallback)
5. [NPU Detection & Future Support](#5-npu-support)
6. [Core-First Pipeline](#6-core-first-pipeline)
7. [Model Registry & Download](#7-model-registry)
8. [Dispatcher Rewrite](#8-dispatcher-rewrite)
9. [Legacy Cleanup](#9-legacy-cleanup)
10. [Uninstall Command](#10-uninstall-command)
11. [CI/CD Platform Matrix](#11-cicd-platform-matrix)
12. [Platform Support Matrix](#12-platform-support-matrix)
13. [Performance Targets](#13-performance-targets)
14. [Dogfooding Protocol](#14-dogfooding-protocol)
15. [Phasing](#15-phasing)

---

## 1. Wheel Packaging

### Decision: Maturin as sole build backend

Replace Hatchling with maturin. One `pip install llmhosts` delivers everything: Python control plane + compiled Rust extensions.

**Root `pyproject.toml`:**

```toml
[build-system]
requires = ["maturin>=1.8"]
build-backend = "maturin"

[project]
name = "llmhosts"
requires-python = ">=3.10"
# ... existing dependencies, extras, scripts unchanged

[tool.maturin]
features = ["pyo3/extension-module"]
module-name = "_llmhosts"
python-packages = ["llmhosts", "hive"]
```

**What ships in the wheel:**
- `llmhosts/` — Python control plane (proxy, config, CLI, dashboard, metrics)
- `hive/` — Agent system
- `_llmhosts.so` (Linux) / `_llmhosts.pyd` (Windows) / `_llmhosts.dylib` (macOS) — compiled Rust containing inference engine, GPU detection, tunnel client, router

**No sdist published.** Only platform-specific wheels. Source stays in private GitHub repo.

**Individual Rust crate `pyproject.toml` files** remain for `maturin develop` during local development.

---

## 2. Bridge Crate Architecture

### Problem: Maturin builds one extension per wheel. We have 4 Rust crates.

### Solution: Single bridge crate re-exports all 4 as submodules.

**New file: `src/lib.rs`** (root of project):

```rust
use pyo3::prelude::*;

#[pymodule]
fn _llmhosts(m: &Bound<'_, PyModule>) -> PyResult<()> {
    let core = PyModule::new(m.py(), "core")?;
    llmhosts_core::register_module(&core)?;
    m.add_submodule(&core)?;

    let gpu = PyModule::new(m.py(), "gpu")?;
    llmhosts_gpu::register_module(&gpu)?;
    m.add_submodule(&gpu)?;

    let relay = PyModule::new(m.py(), "relay")?;
    relay_core::register_module(&relay)?;
    m.add_submodule(&relay)?;

    let router = PyModule::new(m.py(), "router")?;
    router_core::register_module(&router)?;
    m.add_submodule(&router)?;

    m.add("__version__", env!("CARGO_PKG_VERSION"))?;
    Ok(())
}
```

**Root `Cargo.toml` becomes the bridge crate + workspace:**

```toml
[package]
name = "llmhosts"
version = "0.5.0"
edition = "2024"

[lib]
name = "_llmhosts"
crate-type = ["cdylib"]

[dependencies]
pyo3 = { version = "0.28", features = ["extension-module"] }
llmhosts_core = { path = "llmhosts_core" }
llmhosts_gpu = { path = "llmhosts_gpu" }
relay_core = { path = "relay_core" }
router_core = { path = "router_core" }

[workspace]
members = ["llmhosts_core", "llmhosts_gpu", "relay_core", "router_core"]
```

**Each Rust crate** adds a `register_module` function that exposes its pyclasses/pyfunctions to the parent module. Example for `llmhosts_core`:

```rust
pub fn register_module(m: &Bound<'_, PyModule>) -> PyResult<()> {
    m.add("LLMHOSTS_CORE_VERSION", LLMHOSTS_CORE_VERSION)?;
    m.add_class::<LLMHostsEngine>()?;
    m.add_function(wrap_pyfunction!(generate_test_model, m)?)?;
    Ok(())
}
```

**Python import changes:**

```python
# Before:
import _llmhosts_core
engine = _llmhosts_core.LLMHostsEngine()

# After:
from _llmhosts import core
engine = core.LLMHostsEngine()
```

**pyo3 version alignment:** All crates use pyo3 0.28. `router_core` currently uses 0.25 — must be upgraded.

---

## 3. GPU Compute — CubeCL + cudarc

### The main event. GPU-first inference.

**Architecture:**

```
llmhosts_core/src/
├── compute/
│   ├── mod.rs          # ComputeBackend trait
│   ├── cuda.rs         # CubeCL → CUDA kernels (NVIDIA)
│   ├── rocm.rs         # CubeCL → ROCm/HIP kernels (AMD)
│   ├── metal.rs        # CubeCL → WGPU/Metal kernels (Apple)
│   ├── vulkan.rs       # CubeCL → WGPU/Vulkan kernels (Intel, generic)
│   ├── cpu.rs          # CPU fallback (SIMD-optimized Rust)
│   └── auto.rs         # Auto-detect best backend at startup
├── kernels/
│   ├── matmul.rs       # CubeCL matrix multiply
│   ├── softmax.rs      # CubeCL softmax (numerically stable)
│   ├── rope.rs         # CubeCL RoPE (fix precision bug)
│   ├── rmsnorm.rs      # CubeCL RMS normalization
│   ├── silu.rs         # CubeCL SiLU/SwiGLU activation
│   ├── attention.rs    # CubeCL grouped-query attention
│   └── flash.rs        # cudarc Flash Attention (NVIDIA-specific hot path)
├── memory/
│   ├── allocator.rs    # VRAM allocation, OOM prevention
│   ├── paged_kv.rs     # PagedAttention KV cache (F2)
│   └── fp8_cache.rs    # FP8 KV cache on Hopper+ (F2)
```

### ComputeBackend trait

```rust
pub trait ComputeBackend: Send + Sync {
    fn name(&self) -> &str;
    fn matmul(&self, a: &Tensor, b: &Tensor, out: &mut Tensor);
    fn softmax(&self, x: &mut Tensor, dim: usize);
    fn rope(&self, q: &mut Tensor, k: &mut Tensor, freqs: &Tensor, start_pos: usize);
    fn rmsnorm(&self, x: &mut Tensor, weight: &Tensor, eps: f32);
    fn silu_elementwise_mul(&self, gate: &mut Tensor, up: &Tensor);
    fn attention(&self, q: &Tensor, k: &Tensor, v: &Tensor, mask: Option<&Tensor>) -> Tensor;
}
```

All 6 backends implement this trait. The model code calls trait methods — never vendor-specific APIs directly. `auto.rs` selects the best backend at startup:

```rust
pub fn detect_best_backend() -> Box<dyn ComputeBackend> {
    if let Some(cuda) = CudaBackend::try_new() { return Box::new(cuda); }
    if let Some(rocm) = RocmBackend::try_new() { return Box::new(rocm); }
    if let Some(metal) = MetalBackend::try_new() { return Box::new(metal); }
    if let Some(vulkan) = VulkanBackend::try_new() { return Box::new(vulkan); }
    Box::new(CpuBackend::new()) // Always works
}
```

### CubeCL kernel example (matmul — write once, runs everywhere)

```rust
#[cube(launch)]
fn matmul_kernel<F: Float>(
    a: &Tensor<F>,
    b: &Tensor<F>,
    out: &mut Tensor<F>,
    #[comptime] block_size: u32,
) {
    // Tiled matrix multiply — compiles to CUDA PTX, ROCm HIP, WGPU shaders
    let row = ABSOLUTE_POS_X;
    let col = ABSOLUTE_POS_Y;
    let mut sum = F::new(0.0);
    for k in 0..a.shape(1) {
        sum += a[row * a.stride(0) + k] * b[k * b.stride(0) + col];
    }
    out[row * out.stride(0) + col] = sum;
}
```

### cudarc hot paths (NVIDIA-specific, maximum performance)

Used only where vendor-specific code delivers >20% speedup over CubeCL:
- **Flash Attention v3** — fused kernel, O(N) memory instead of O(N²)
- **FP8 GEMM on Hopper+** — halves memory for KV cache on RTX 4090, H100
- **cuBLAS SGEMM** — NVIDIA's tuned matmul for large matrices

```rust
impl CudaBackend {
    fn attention(&self, q: &Tensor, k: &Tensor, v: &Tensor, mask: Option<&Tensor>) -> Tensor {
        if self.has_flash_attention() {
            // cudarc: launch Flash Attention v3 kernel
            self.flash_attention_v3(q, k, v, mask)
        } else {
            // CubeCL: generic attention kernel (still fast, just not fused)
            self.cubecl_attention(q, k, v, mask)
        }
    }
}
```

### GPU library loading

**All GPU vendor libraries loaded dynamically** via `libloading`. No compile-time CUDA/ROCm/Metal dependencies. The binary works on every platform — vendor modules return "not available" when drivers aren't installed.

```rust
// Runtime detection — no compile-time dependency
let cuda_available = libloading::Library::new("libcuda.so").is_ok()
    || libloading::Library::new("nvcuda.dll").is_ok();
```

---

## 4. CPU Fallback

Always works. Every platform. No GPU required.

| Operation | CPU Implementation | Performance |
|-----------|-------------------|-------------|
| matmul | `matrixmultiply::sgemm` (SIMD AVX2/NEON) | 10-20x naive |
| softmax | Pure Rust, auto-vectorized | Adequate |
| RoPE | Pure Rust, precomputed frequency tables | **Fix precision bug** |
| RMSNorm | Pure Rust, auto-vectorized | Adequate |
| SiLU | Pure Rust | Adequate |
| attention | Pure Rust with causal masking | Adequate |

**The RoPE precision bug** (garbled output) is the #1 fix. Likely cause: f32 precision loss in frequency computation for large `rope_theta` values (Qwen uses 1,000,000 vs Llama's 10,000). Fix: use f64 for frequency table computation, downcast to f32 for kernel input.

---

## 5. NPU Detection & Future Support

**F1: Detect and report.** `_llmhosts.gpu.detect_gpus()` already probes:
- Intel NPU via Level Zero / sysfs
- Apple Neural Engine via IOKit
- Qualcomm NPU (future — no standard API yet)

**F5+: Inference on NPU.** Different compilation story — NPUs use fixed-function operators, not programmable shaders. Requires per-vendor SDK integration (OpenVINO, Core ML, QNN). Deferred to F5+.

**What ships now:** Detection returns NPU info in the hardware profile. HIM uses it for model selection (smaller quantized models run well on NPUs). Inference stays on GPU/CPU until F5+.

---

## 6. Core-First Pipeline

### `llmhosts up` — Complete Rewrite

**Phase 1: Detect Hardware**
```
$ llmhosts up

[Phase 1] Scanning hardware...
  GPU: NVIDIA RTX 4090 (24 GB VRAM, 23.1 GB free) — CUDA 12.4
  CPU: AMD Ryzen 9 7945HX (16 cores), 32 GB RAM
  NPU: None detected
  Compute: CUDA (GPU-accelerated inference)
  Recommended: Qwen2.5-7B-Instruct Q4_K_M (uses ~5 GB VRAM)
```

- `_llmhosts.gpu.detect_gpus()` — returns GPU info, VRAM, driver version
- HIM selects optimal model for detected hardware
- No external engine scanning. No HTTP probes. No port scanning.

**Phase 2: Provision Model**
```
[Phase 2] Downloading Qwen2.5-7B-Instruct Q4_K_M...
  Source: huggingface.co/Qwen/Qwen2.5-7B-Instruct-GGUF
  Progress: ████████████████████ 100% (4.4 GB, 2m 14s)
  Tokenizer: auto-downloaded
  SHA256: verified ✓
  Saved to: ~/.llmhosts/models/qwen2.5-7b-instruct-q4_k_m.gguf
```

- Downloads GGUF from HuggingFace via `huggingface_hub`
- Tokenizer auto-downloaded alongside
- SHA256 verification
- Resume on interruption
- Skips if model already present in `~/.llmhosts/models/`

**Phase 3: Load Engine**
```
[Phase 3] Loading into LLMHosts Core...
  Backend: CUDA (RTX 4090)
  VRAM allocated: 5.2 GB / 24 GB
  KV cache: PagedAttention (18.8 GB available)
  Ready in 2.1s
```

- `_llmhosts.core.LLMHostsEngine.load_model()` → auto-detects GPU → loads weights to VRAM
- Tokenizer loaded automatically
- PagedAttention KV cache pre-allocated in remaining VRAM

**Phase 4: Start Proxy + Tunnel**
```
[Phase 4] Starting proxy...
  API: http://localhost:11400/v1/chat/completions
  Models: http://localhost:11400/v1/models
  Dashboard: http://localhost:11400/dashboard
  Health: http://localhost:11400/health
  Tunnel: connecting to relay.llmhosts.com...
  Remote URL: https://abc123.relay.llmhosts.com
```

- FastAPI proxy with Core as sole local backend
- Tunnel via `_llmhosts.relay`
- Device registration via device code auth (RFC 8628)

**Phase 5: Serve**
```
[Phase 5] LLMHosts is ready.
  Model: qwen2.5-7b-instruct (Q4_K_M, CUDA)
  Speed: ~80 tok/s
  VRAM: 5.2 / 24 GB
  Press Ctrl+C to stop
```

- Heartbeat to LLMHosts.com every 30s
- Filesystem watcher on `~/.llmhosts/models/` for new models
- Savings tracking (local vs cloud cost)

---

## 7. Model Registry & Download

### Model Registry

```python
# llmhosts/models/registry.py
MODELS = {
    "qwen2.5:0.5b": {
        "repo": "Qwen/Qwen2.5-0.5B-Instruct-GGUF",
        "file": "qwen2.5-0.5b-instruct-q4_k_m.gguf",
        "tokenizer_repo": "Qwen/Qwen2.5-0.5B-Instruct",
        "size_gb": 0.4,
        "min_vram_gb": 1,
        "architecture": "llama",
        "context_length": 32768,
    },
    "qwen2.5:3b": { ... },
    "qwen2.5:7b": { ... },
    "qwen2.5:14b": { ... },
    "llama3.1:8b": { ... },
    "llama3.3:70b": { ... },
    "mistral:7b": { ... },
    "codellama:7b": { ... },
    "deepseek-r1:7b": { ... },
}
```

### Hardware Intelligence Matrix (HIM)

Maps GPU VRAM → optimal model:

| Available VRAM | Recommended Model | Quant | Expected tok/s |
|---------------|-------------------|-------|----------------|
| <2 GB | qwen2.5:0.5b | Q4_K_M | >100 |
| 2-4 GB | qwen2.5:3b | Q4_K_M | ~60 |
| 4-8 GB | qwen2.5:7b | Q4_K_M | ~80 |
| 8-12 GB | qwen2.5:14b | Q4_K_M | ~40 |
| 12-24 GB | llama3.3:70b | Q4_K_M | ~18 (partial) |
| 24-48 GB | llama3.3:70b | Q5_K_M | ~25 |
| CPU only | qwen2.5:0.5b | Q4_K_M | >3 |

### `llmhosts pull` Command

```bash
# Pull by friendly name (uses registry)
llmhosts pull qwen2.5:7b

# Pull by HuggingFace repo
llmhosts pull Qwen/Qwen2.5-7B-Instruct-GGUF

# Pull recommended model for your hardware
llmhosts pull --auto

# List available models
llmhosts models list

# List downloaded models
llmhosts models list --local
```

### Storage

```
~/.llmhosts/
├── models/
│   ├── qwen2.5-7b-instruct-q4_k_m.gguf
│   ├── qwen2.5-7b-instruct-tokenizer.json
│   └── registry.json  (cached model metadata)
├── config.toml
├── keys/  (encrypted API keys)
├── cache/
└── audit.db
```

---

## 8. Dispatcher Rewrite

### Core is the default. No Ollama.

**Before (current — Ollama primary, Core optional):**
```python
if self._is_core_available() and backend.backend_type in ("ollama", "litellm"):
    # Core is preferred over Ollama when loaded
    body, headers = await self._dispatch_core(...)
elif backend.backend_type == "ollama":
    response = await self._dispatch_ollama(...)
```

**After (Core primary, cloud fallback):**
```python
if self._core_engine.is_loaded:
    # Core IS the local engine — always use it
    body, headers = await self._dispatch_core(...)
elif backend.backend_type == "cloud":
    # Cloud BYOK fallback (OpenAI, Anthropic, etc.)
    response = await self._dispatch_cloud(...)
elif backend.backend_type == "remote":
    # Marketplace: forward to another LLMHosts node
    response = await self._dispatch_remote(...)
else:
    raise BackendUnavailableError("No backend available")
```

**Routing priority:**
1. **LLMHosts Core** — local GPU/CPU inference (always first)
2. **Remote device** — marketplace nodes via tunnel
3. **Cloud API** — BYOK keys via LiteLLM (last resort)

No Ollama. No vLLM. No llama.cpp. No external engine HTTP probes.

---

## 9. Legacy Cleanup

### Remove (Ollama-specific, no longer needed)

| File/Module | What It Does | Action |
|---|---|---|
| `discovery/scanner.py` — OllamaProbe | HTTP probe to localhost:11434 | **Delete** |
| `discovery/scanner.py` — VLLMProbe | HTTP probe to vLLM | **Delete** |
| `discovery/scanner.py` — LlamaCppProbe | HTTP probe to llama.cpp | **Delete** |
| `discovery/scanner.py` — LMStudioProbe | HTTP probe to LM Studio | **Delete** |
| `discovery/scanner.py` — LocalAIProbe | HTTP probe to LocalAI | **Delete** |
| `discovery/scanner.py` — TGIProbe | HTTP probe to TGI | **Delete** |
| `discovery/scanner.py` — GenericOpenAIProbe | HTTP probe to generic servers | **Delete** |
| `discovery/provisioner.py` — `install_ollama()` | curl/winget Ollama install | **Delete** |
| `discovery/watcher.py` — Ollama polling | Poll Ollama API every 5s | **Replace** with `~/.llmhosts/models/` filesystem watcher |
| `proxy/dispatcher.py` — `_dispatch_ollama()` | Route to Ollama via HTTP | **Delete** |
| `proxy/dispatcher.py` — `_is_core_available()` | "Is Core loaded?" guard | **Delete** — Core is always the engine |
| `proxy/dispatcher.py` — Ollama backend entries | BackendEntry for Ollama | **Delete** |
| `backends/` — Ollama model registry | Ollama model list | **Delete** |
| `up.py` Phase 2 — Ollama install/pull | Install Ollama + ollama pull | **Replace** with GGUF download |

### Keep

| Component | Reason |
|---|---|
| Cloud API routing (LiteLLM) | BYOK keys → OpenAI/Anthropic/Google (valid fallback) |
| Remote device routing | Phase 5 marketplace (forward to other LLMHosts nodes) |
| Privacy routing (IslandRun) | PII detection before external routing |
| Semantic cache (vCache) | Cache shared across all backends |
| Three-tier router | Classifies requests, selects backend tier |
| Audit logging | Every request logged |
| Metrics/Prometheus | Observability |
| TUI dashboard | `llmhosts serve` display |

---

## 10. Uninstall Command

### Two tiers following GPU Respect Principle

**Tier 1: `llmhosts uninstall`** — removes LLMHosts data, keeps models

```
$ llmhosts uninstall

This will remove:
  ✓ Config        ~/.config/llmhosts/       (12 KB)
  ✓ Cache         ~/.llmhosts/cache/        (45 MB)
  ✓ Audit logs    ~/.llmhosts/audit.db      (2 MB)
  ✓ API keys      ~/.config/llmhosts/keys/  (encrypted)

This will NOT remove:
  ✗ Models        ~/.llmhosts/models/       (4.4 GB)
  ✗ Package       (use: pip uninstall llmhosts)

Continue? [y/N]
```

**Tier 2: `llmhosts uninstall --all`** — removes everything including models

```
$ llmhosts uninstall --all

This will remove everything above, PLUS:
  ✓ Models        ~/.llmhosts/models/       (4.4 GB)

Continue? [y/N]
```

**Then:** `pip uninstall llmhosts` removes the package + compiled engine.

**Implementation:**
- New file: `llmhosts/uninstall.py`
- New CLI command: `@cli.command("uninstall")` in `cli.py`
- Always interactive (confirmation prompt) unless `--yes` flag
- Prints summary of what was removed and what remains
- Windows: uses `%APPDATA%\llmhosts\` instead of `~/.config/llmhosts/`
- Cross-platform path resolution via `platformdirs`

---

## 11. CI/CD Platform Matrix

### Release Workflow

```yaml
# .github/workflows/release.yml
name: Release

on:
  workflow_dispatch:
    inputs:
      version:
        description: "Version to release (e.g., 0.5.0)"
        required: true

jobs:
  build-wheels:
    strategy:
      matrix:
        include:
          # F1: Linux + Windows
          - os: ubuntu-latest
            target: x86_64-unknown-linux-gnu
            manylinux: auto
          - os: windows-latest
            target: x86_64-pc-windows-msvc
          # F5: macOS + ARM
          # - os: macos-14
          #   target: aarch64-apple-darwin
          # - os: ubuntu-latest
          #   target: aarch64-unknown-linux-gnu
          # - os: windows-latest
          #   target: aarch64-pc-windows-msvc
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v4
      - uses: PyO3/maturin-action@v1
        with:
          target: ${{ matrix.target }}
          args: --release --out dist
          manylinux: ${{ matrix.manylinux || 'auto' }}
      - uses: actions/upload-artifact@v4
        with:
          name: wheel-${{ matrix.target }}
          path: dist/*.whl

  publish:
    needs: build-wheels
    runs-on: ubuntu-latest
    environment: pypi
    permissions:
      id-token: write  # Trusted publishing
    steps:
      - uses: actions/download-artifact@v4
        with:
          pattern: wheel-*
          merge-multiple: true
          path: dist/
      - uses: PyO3/maturin-action@v1
        with:
          command: upload
          args: --non-interactive --skip-existing dist/*.whl
```

### What this produces

| Wheel | Size | Platform |
|-------|------|----------|
| `llmhosts-0.5.0-cp310-abi3-manylinux_2_17_x86_64.whl` | ~15 MB | Linux x86_64 |
| `llmhosts-0.5.0-cp310-abi3-win_amd64.whl` | ~12 MB | Windows x86_64 |
| _(F5)_ `llmhosts-0.5.0-cp310-abi3-macosx_11_0_arm64.whl` | ~14 MB | macOS Apple Silicon |
| _(F5)_ `llmhosts-0.5.0-cp310-abi3-manylinux_2_17_aarch64.whl` | ~13 MB | Linux ARM64 |
| _(F5)_ `llmhosts-0.5.0-cp310-abi3-win_arm64.whl` | ~11 MB | Windows ARM64 |

`cp310-abi3` = Python 3.10+ stable ABI. One wheel per platform covers all Python versions.

### No sdist

`maturin upload` only sends wheel files. No source distribution published to PyPI.

---

## 12. Platform Support Matrix

### GPU Support (CubeCL write-once kernels)

| Vendor | API | CubeCL Target | Phase | Status |
|--------|-----|---------------|-------|--------|
| **NVIDIA** | CUDA | `cubecl-cuda` | **F2** | Priority 1 (RTX 4090) |
| **AMD** | ROCm/HIP | `cubecl-hip` | F5 | Same kernels |
| **Apple** | Metal | `cubecl-wgpu` → Metal | F5 | Same kernels |
| **Intel** | Vulkan | `cubecl-wgpu` → Vulkan | F5 | Same kernels |
| **Generic** | Vulkan/DX12 | `cubecl-wgpu` | F5 | Any Vulkan-capable GPU |

### cudarc NVIDIA Hot Paths

| Feature | Benefit | Requirement |
|---------|---------|-------------|
| Flash Attention v3 | O(N) memory, fused kernel | CUDA compute 7.0+ |
| FP8 GEMM | 2x KV cache capacity | Hopper+ (RTX 4090, H100) |
| cuBLAS SGEMM | Vendor-tuned matmul | Any CUDA GPU |

### NPU Support

| Vendor | Detection | Inference | Phase |
|--------|-----------|-----------|-------|
| Intel (Meteor Lake+) | `_llmhosts.gpu` via Level Zero / sysfs | OpenVINO integration | Detect: F1, Inference: F5+ |
| Apple Neural Engine | `_llmhosts.gpu` via IOKit | Core ML integration | Detect: F1, Inference: F5+ |
| Qualcomm (Snapdragon X) | `_llmhosts.gpu` via QNN SDK | QNN integration | Detect: F5+, Inference: F5+ |

### OS Support

| OS | Architecture | Phase | Wheel Tag |
|----|-------------|-------|-----------|
| **Linux x86_64** | amd64 | **F1** | `manylinux_2_17_x86_64` |
| **Windows x86_64** | amd64 | **F1** | `win_amd64` |
| macOS arm64 | Apple Silicon | F5 | `macosx_11_0_arm64` |
| Linux aarch64 | ARM servers | F5 | `manylinux_2_17_aarch64` |
| Windows arm64 | Snapdragon X | F5 | `win_arm64` |
| macOS x86_64 | Intel Mac | F5 | `macosx_10_12_x86_64` |

### CPU Fallback (every platform)

| Architecture | SIMD | Performance |
|-------------|------|-------------|
| x86_64 | AVX2 / SSE4.2 | 10-20x naive via matrixmultiply |
| aarch64 | NEON | 10-15x naive via matrixmultiply |
| Generic | Auto-vectorized | 5-8x naive |

---

## 13. Performance Targets

### F1 — CPU Baseline

| Metric | Target | Model |
|--------|--------|-------|
| tok/s (decode) | >3 | Qwen2.5-0.5B Q4_K_M |
| TTFT (50-token prompt) | <10s | Qwen2.5-0.5B Q4_K_M |
| Model load | <30s | Qwen2.5-0.5B Q4_K_M |
| Memory | <3 GB | Qwen2.5-0.5B (dequantized f32) |
| Binary size | <50 MB | All Rust extensions combined |

### F2 — Single GPU (NVIDIA CUDA)

| Metric | Target | Model |
|--------|--------|-------|
| tok/s (decode) | >80 | Qwen2.5-7B Q4_K_M |
| TTFT | <200 ms | Qwen2.5-7B Q4_K_M |
| vs Ollama (single-stream) | Within 20% | Qwen2.5-7B |
| vs Ollama (10 concurrent) | >2x | Qwen2.5-7B |
| PagedAttention throughput | >24x vs simple KV | Batch workload |
| Continuous batching | >23x vs sequential | 10 concurrent |
| GPU memory utilization | >80% | Under load |

### F4 — Multi-GPU & Datacenter

| Metric | Target | Model |
|--------|--------|-------|
| 10 concurrent requests | >200 agg tok/s | Qwen2.5-7B |
| Phase-split cost reduction | >40% | vs uniform serving |
| P99 latency (short request during long-context) | <500 ms | Mixed workload |
| Tensor parallelism | 2/4/8 GPU | Llama-3.3-70B |

---

## 14. Dogfooding Protocol — Alienware Laptop

### Hardware: Alienware, Windows 11, RTX 4090 (24 GB), AMD Ryzen 9

### 14-test sequence (ALL must pass)

| # | Test | Command | Expected |
|---|------|---------|----------|
| 1 | Rust tests | `cargo test --workspace` | >60 tests pass |
| 2 | Rust lint | `cargo clippy --workspace -- -D warnings` | Zero warnings |
| 3 | Python tests | `pytest tests/ -v` | Zero failures |
| 4 | Python lint | `ruff check . && ruff format --check .` | Clean |
| 5 | GPU detection | `python -c "from _llmhosts import gpu; print(gpu.detect_gpus())"` | RTX 4090, 24 GB VRAM |
| 6 | Synthetic E2E | `pytest tests/test_llmhosts_core.py -v` | All pass |
| 7 | Install from wheel | `pip install llmhosts` (from built wheel) | Installs in <60s |
| 8 | Auto model pull | `llmhosts pull --auto` | Downloads recommended GGUF |
| 9 | llmhosts up | `llmhosts up` | Detects GPU → loads model → starts proxy |
| 10 | API response | `curl localhost:11400/v1/chat/completions -d '{"model":"auto","messages":[{"role":"user","content":"Hello"}]}'` | Coherent response, <500ms TTFT |
| 11 | Models list | `curl localhost:11400/v1/models` | Shows loaded model |
| 12 | Streaming | Same as #10 with `"stream": true` | SSE chunks, ends with `[DONE]` |
| 13 | Benchmark | `llmhosts benchmark` | >80 tok/s on 7B model |
| 14 | Health | `curl localhost:11400/health` | 200 with GPU backend status |

**Success = 14/14 with zero failures, zero warnings, zero errors.**

---

## 15. Phasing

### What ships when

| Phase | Scope | Platforms | GPU |
|-------|-------|-----------|-----|
| **F1 (current)** | Wheel packaging, bridge crate, Core-first pipeline, model registry, RoPE fix, CPU inference, legacy cleanup, uninstall | Linux x86_64 + Windows x86_64 | CPU only |
| **F2** | CubeCL CUDA kernels, PagedAttention, continuous batching, Flash Attention, FP8 cache, benchmark CLI | Same | **NVIDIA CUDA** |
| **F3** | HIM auto-selection, model download UX, hot-swap, model agent, dashboard GPU metrics | Same | Same |
| **F4** | Tensor parallelism, phase-split serving, multi-GPU, cluster scheduling | Same + Linux aarch64 | Multi-GPU NVIDIA |
| **F5** | macOS/Windows ARM wheels, AMD ROCm, Apple Metal, Intel Vulkan, NPU inference, standalone installers | All platforms | All GPUs + NPU |

### F1 + F2 = Alienware dogfooding ready

F1 delivers the packaging, pipeline, and CPU baseline. F2 adds GPU inference. Together they make the Alienware laptop demo work at production quality.

---

## Decisions Summary

| Decision | Choice | Rationale |
|----------|--------|-----------|
| Build backend | Maturin (replaces Hatchling) | Industry standard for Rust+Python (Polars, Pydantic v2, tiktoken) |
| Extension structure | Single bridge crate `_llmhosts` | Maturin = one cdylib per wheel |
| Source protection | Rust compiled, Python readable | Competitive moat is in Rust. Python is commodity glue. |
| GPU kernels | CubeCL (cross-platform) + cudarc (NVIDIA hot paths) | Write once, compile everywhere. Vendor-specific where it matters. |
| GPU library loading | Dynamic (`libloading`) | No compile-time CUDA/ROCm dependency. Binary works everywhere. |
| Local inference | LLMHosts Core only | No Ollama, no external engine dependencies |
| Cloud fallback | Keep LiteLLM | BYOK keys to OpenAI/Anthropic remain valid |
| Model format | GGUF primary | Zero-copy mmap, community standard, pre-quantized |
| Model source | HuggingFace Hub | Largest model repository, `hf_hub_download` with resume + verification |
| Model storage | `~/.llmhosts/models/` | User-visible, easy to manage, survives uninstall by default |
| Uninstall | Two-tier (`llmhosts uninstall` + `--all`) | GPU Respect: never delete models without explicit consent |
| Platform wheels (F1) | Linux x86_64 + Windows x86_64 | Covers VPS + Alienware. More platforms = more CI matrix entries. |
| No sdist | Wheels only on PyPI | Source stays in private GitHub repo |

---

## Related Documents

- **ADR-010**: LLMHosts Core inference engine decision
- **ADR-009**: Self-hosted Rust relay tunnel
- **ADR-008**: Tiered pip install strategy
- **User Journeys**: `docs/plans/2026-03-09-llmhosts-core-user-journeys.md` (405 success criteria)
- **Engine Design**: `docs/plans/2026-03-09-llmhosts-core-inference-engine-design.md`
- **F1 Foundation**: `docs/plans/2026-03-09-llmhosts-core-f1-foundation.md`
- **Research**: `docs/knowledge-base/llmhosts-core-inference-research-2026.md`
