# LLMHosts Core — Inference Engine Design Document

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Replace all external inference dependencies (Ollama, vLLM) with a vertically-integrated, Rust-native inference engine that auto-discovers hardware, curates optimal models, and delivers market-leading performance from a Mac Mini to an 8x H100 datacenter rack.

**Architecture:** Full custom Rust engine (CubeCL + cudarc) with PagedAttention, continuous batching, phase-split serving, cross-platform GPU abstraction, and AI-powered model curation. Ships as a single binary inside the pip wheel via maturin.

**Tech Stack:** Rust (CubeCL, cudarc, tokio, hyper), Python (FastAPI control plane), Next.js (dashboard)

**Working name:** **LLMHosts Core** — we turn raw GPU silicon into intelligence.

---

## 1. Product Philosophy

### The Three Promises

1. **You plug it in. We handle everything.** Install is one command. No model selection, no config files, no quantization decisions. LLMHosts Core detects your hardware, selects the optimal model, downloads it, and starts serving.

2. **It's always the best it can be.** An AI research agent continuously scans the model landscape. When a better model exists for your hardware profile, you get a notification. One tap to upgrade. Zero-downtime hot-swap.

3. **Your hardware, maximized.** We maintain a Hardware Intelligence Matrix (HIM) — a living database mapping every GPU + RAM + CPU combination to its optimal model, quantization, context length, and batch size.

### What We Don't Do

Let users load arbitrary models. Our value prop is curation — we guarantee the best experience on your hardware. We control the model, the quantization, the context length, the batch size. Like Apple controls the hardware-software integration, we control the hardware-model integration.

---

## 2. Architecture Overview

```
+------------------------------------------------------------------+
|                    LLMHosts Platform                              |
|                                                                   |
|  +--------------+  +--------------+  +-------------------------+ |
|  |  Core        |  |  Relay       |  |  Control Plane          | |
|  |  (Rust)      |  |  (Rust)      |  |  (Python + Next.js)     | |
|  |              |  |              |  |                         | |
|  |  Inference   |  |  WSS Tunnel  |  |  Dashboard              | |
|  |  Engine      |  |  Noise NK    |  |  Model Agent            | |
|  |  GPU Manager |  |  yamux       |  |  API Proxy              | |
|  |  Model Loader|  |  HTTP fwd    |  |  Router                 | |
|  |  KV Cache    |  |              |  |  Cache                  | |
|  |  Scheduler   |  |              |  |  Metrics                | |
|  +------+-------+  +------+------+  +----------+--------------+ |
|         |                  |                    |                 |
|         +------------------+--------------------+                 |
|                            |                                      |
|                     Single Binary                                 |
|                     (~15-20MB)                                    |
+------------------------------------------------------------------+
          |                  |                    |
     +----v----+       +----v----+          +----v----+
     | NVIDIA  |       |  AMD    |          | Apple   |
     | CUDA    |       | ROCm   |          | Metal   |
     | cudarc  |       | CubeCL |          | CubeCL  |
     +---------+       +---------+          +---------+
```

### Three Rust Crates

| Crate | Role | Ships as |
|-------|------|----------|
| `llmhosts_core` | Inference: model loading, tensor ops, KV cache, scheduling | Shared lib (.so/.dylib/.dll) |
| `relay_core` | Tunnel: WSS + Noise NK + yamux (already started in M1) | Part of llmhosts-core binary |
| `llmhosts_gpu` | GPU detection, memory management, cross-platform abstraction | Part of llmhosts-core binary |

### Python Control Plane (existing LLMHosts codebase)

- HTTP API proxy (OpenAI/Anthropic compatible)
- Intelligent routing & privacy (three-tier router)
- Dashboard (Textual TUI + Next.js web)
- Model Agent (AI-powered model research & selection)
- Cloud spillover (LiteLLM for overflow)
- Marketplace (Phase 5)

**The Rust binary does the heavy lifting. Python does the orchestration.**

---

## 3. LLMHosts Core Engine — Inference Core

### 3.1 Compute Foundation

**CubeCL** (by Tracel.ai/Burn team) for cross-platform GPU kernels. Write once in Rust, compiles to:
- CUDA (NVIDIA)
- ROCm/HIP (AMD)
- WGPU -> Vulkan/Metal/DX12 (everything else)

**cudarc** for NVIDIA-specific hot paths where maximum performance matters (Flash Attention fused kernels, FP8 KV cache on Hopper+).

### 3.2 Model Loader

- **GGUF** (primary): Zero-copy mmap via `gguf-rs-lib`. Model weights stay on disk and are paged into GPU memory on demand.
- **SafeTensors** (secondary): For larger models and fine-tunes. HuggingFace's `safetensors` Rust crate (MIT).
- **Tokenizers**: HuggingFace's `tokenizers` crate (Rust-native, MIT, supports every tokenizer family).

### 3.3 Attention & Tensor Operations

- **Flash Attention**: Via `candle-flash-attn` CUDA bindings for NVIDIA; custom CubeCL implementation for AMD/Apple.
- **Grouped-Query Attention (GQA)**: First-class support (most modern models use this).
- **Fused operations**: RoPE, SwiGLU, RMSNorm fused into single kernels to minimize memory round-trips.

### 3.4 KV Cache Manager (PagedAttention)

- Allocates KV cache in fixed-size pages (like virtual memory).
- Eliminates internal fragmentation (vLLM showed 24x throughput improvement from this alone).
- **FP8 KV cache** on Hopper+ GPUs (halves memory vs FP16).
- **RadixAttention** for prefix caching: multi-turn conversations share cached prefixes (75-90% cache hit rate, per SGLang research).
- **Dynamic eviction**: LRU pages evicted to CPU RAM when GPU VRAM pressure exceeds threshold.

### 3.5 Scheduler (Continuous Batching + Phase-Split)

- **Continuous batching**: New requests enter the batch mid-generation. 23x throughput vs naive sequential (Anyscale benchmark).
- **Phase-split serving** (when multiple GPUs available):
  - Prefill (compute-bound): runs on the most powerful GPU
  - Decode (memory-bound): runs on any GPU with enough VRAM
  - 40% cost reduction without latency penalties (research consensus)
- **Priority scheduling**: Interactive (low latency) vs batch (high throughput).
- **Adaptive batch size**: Auto-tunes based on available VRAM and current load.

### 3.6 Model Architecture Support (Phased)

| Phase | Architectures | Models Covered |
|-------|---------------|---------------|
| P1 | LLMHostsTransformer (Llama-family) | Llama 3.x, CodeLlama, Mistral, Mixtral, Yi, Qwen2.5 |
| P2 | LLMHostsTransformerV2 (Phi/Gemma) | Phi-3/4, Gemma 2 |
| P3 | LLMHostsMoE (DeepSeek-family) | DeepSeek-V3/R1 (Mixture of Experts) |
| P4 | Vision encoders | Multimodal (Llava, Qwen-VL) |

Phase 1 covers ~80% of local inference usage — the Llama-family architecture (which Mistral, Qwen, Yi all share) dominates open-source models.

---

## 4. Hardware Intelligence Matrix (HIM)

A continuously-updated database mapping hardware profiles to optimal model configurations:

```
(GPU model, VRAM, system RAM, CPU cores, platform) ->
  (recommended model, quantization, context length, max batch size, expected tok/s)
```

### Example Entries

| Hardware Profile | Model | Quant | Context | Expected tok/s |
|-----------------|-------|-------|---------|----------------|
| Mac Mini M4 (16GB unified) | Qwen2.5-7B | Q4_K_M | 8K | ~35 |
| Mac Mini M4 Pro (36GB) | Qwen2.5-14B | Q5_K_M | 16K | ~28 |
| RTX 4090 (24GB) | Llama-3.3-70B | Q4_K_M | 8K | ~18 (partial offload) |
| RTX 4090 (24GB) + 64GB RAM | Llama-3.3-70B | Q4_K_M | 16K | ~15 (CPU offload) |
| 2x RTX 4090 (48GB total) | Llama-3.3-70B | Q5_K_M | 32K | ~25 (tensor parallel) |
| NVIDIA Jetson Orin (16GB) | Qwen2.5-3B | Q5_K_M | 4K | ~50 |
| 8x H100 (640GB total) | Llama-3.1-405B | FP8 | 128K | ~800 |
| AMD MI300X (192GB) | DeepSeek-R1-671B | FP8 | 64K | ~200 |

### Model Selection Algorithm

```
1. Detect hardware -> look up HIM profile
2. Select model that maximizes: quality_score * (1 / memory_pressure)
   where quality_score = benchmark performance on standard eval
   and memory_pressure = model_vram / available_vram
3. If multiple viable: prefer highest throughput
4. Always leave 10% VRAM headroom for KV cache growth
5. If no model fits at Q4_K_M: try Q3_K, then smaller model
```

### HIM Maintenance

1. Our benchmark runs on actual hardware (we build a benchmark suite)
2. Community telemetry (anonymized, opt-in performance data from deployed nodes)
3. AI-driven analysis of new model releases (papers, HuggingFace activity)

---

## 5. Model Agent — AI-Powered Curation

Background service running on LLMHosts cloud infrastructure:

1. **Monitors model releases**: Scans HuggingFace, arXiv, model leaderboards daily
2. **Evaluates candidates**: Runs standardized benchmarks (MMLU, HumanEval, MT-Bench) on reference hardware
3. **Updates the HIM**: When a new model outperforms the current recommendation, updates the matrix
4. **Notifies users**: Dashboard push notification with performance comparison
5. **Handles upgrades**: Download -> verify -> hot-swap (zero downtime)

### Hot-Swap Protocol (Zero-Downtime Model Updates)

```
1. Download new model to disk (background, doesn't affect serving)
2. Pre-load into GPU alongside current model (if VRAM allows)
   -- OR queue for swap after current batch completes
3. Drain: stop accepting new requests on old model
4. Wait for in-flight requests to complete (timeout: 30s)
5. Activate new model
6. Unload old model, free VRAM
7. Resume accepting requests
Total interruption: <100ms (or 0ms if dual-load fits in VRAM)
```

### Future: LLMHosts Fine-Tuned Models

- Optimized for LLMHosts Core engine's strengths (specific attention patterns, batch sizes)
- Instruction-tuned for common use cases (coding, chat, reasoning)
- Quantized with our own calibration data for maximum quality retention
- Distributed via our own model registry (no HuggingFace dependency for distribution)

---

## 6. GPU Discovery & Management (llmhosts_gpu)

Standalone Rust crate — the definitive cross-platform GPU detection library. This doesn't exist in the ecosystem; we build it and open-source it.

### Detection Matrix

| Vendor | Method | Library | Capabilities |
|--------|--------|---------|-------------|
| NVIDIA | NVML | `nvml-wrapper` (runtime loaded) | VRAM, utilization, temp, power, driver, compute capability |
| AMD | sysfs (Linux), WMI (Windows) | Custom + `amdgpu-sysfs` | VRAM, utilization, temp |
| Apple | IOKit + Metal capabilities | `objc2-metal` | Unified memory, GPU cores, ANE cores |
| Intel | sysfs (Linux), Level Zero API | Custom | Xe cores, local memory |

### Runtime Loading (Critical)

All GPU vendor libraries loaded dynamically via `libloading`. No compile-time GPU dependencies. The binary works on every platform — vendor modules gracefully return "not available" when drivers aren't installed.

### Multi-GPU Topology Detection

```rust
pub struct GpuTopology {
    gpus: Vec<GpuDevice>,
    interconnects: Vec<GpuInterconnect>,  // NVLink, PCIe, etc.
    total_vram_mb: u64,
    can_tensor_parallel: bool,
    recommended_strategy: InferenceStrategy,  // Single, TensorParallel, PhaseSplit
}
```

### Mixed GPU Handling

When a machine has mixed GPUs (e.g., RTX 4090 + RTX 3080):
- Detect each GPU's capabilities independently
- If models are small enough: run separate model on each GPU, route by capability
- If model needs both GPUs: phase-split (4090 for prefill, 3080 for decode)
- Never tensor-parallel across mismatched GPUs (performance cliff)

---

## 7. Cross-Platform Delivery

| Platform | Install Method | GPU Support | Notes |
|----------|---------------|-------------|-------|
| Linux x86_64 | `pip install llmhosts` | NVIDIA, AMD, Intel | Primary platform |
| Linux aarch64 | Same pip install | NVIDIA Jetson, Ampere | ARM datacenter + edge |
| macOS Apple Silicon | `brew install llmhosts` or pip | Metal (M1-M4) | Unified memory |
| macOS Intel | Same | CPU-only | Fallback for older Macs |
| Windows x86_64 | `pip install llmhosts` or MSI | NVIDIA, AMD, Intel | Service + tray app |
| Docker | `docker pull llmhosts/llmhosts` | NVIDIA, AMD (ROCm) | Datacenter deployment |

Rust binary ships inside the pip wheel via maturin (same pattern as `relay_core`).

---

## 8. Cloud Spillover (Adaptive Overflow)

### Overflow Logic

```
Load Monitor (continuous):
  current_utilization = gpu_vram_used / gpu_vram_total
  pending_requests = scheduler.queue_depth

  if utilization > 85% AND pending > batch_size * 2:
    -> Route new requests to cloud (LiteLLM -> cheapest provider)
    -> Header: X-LLMHosts-Overflow: true

  if utilization < 70% AND cloud_requests_active > 0:
    -> Route new requests back to local
    -> Drain cloud requests (let them finish)
    -> Clear overflow flag
```

### Smart Provider Selection

When overflowing to cloud, pick the provider that:
1. Has the closest model to what's running locally (preserve quality)
2. Has the lowest cost per token
3. Has the lowest latency from this region
4. Has available capacity (check rate limits)

### Transparency

Users always know when requests go to cloud vs local. Dashboard shows real-time split. Cost tracking separates local (free) from cloud (paid).

---

## 9. Datacenter Mode

### Multi-Node Coordination

LLMHosts Core instances discover each other via:
1. mDNS on local network (already built)
2. Relay registration (cross-network)
3. Manual node list (air-gapped environments)

### Cluster-Aware Scheduling

```
Request arrives at any node ->
  Cluster scheduler checks all nodes:
    - Which node has the model loaded?
    - Which node has the most free VRAM?
    - Which node has the lowest queue depth?
    - Network latency between nodes?
  Route to optimal node ->
  If no capacity anywhere: cloud spillover
```

### Phase-Split Across Nodes

Prefill on H100 node, decode on consumer GPU node. HTTP forwarding via relay mesh (encrypted, multiplexed).

### GPU Partitioning

On multi-GPU nodes (8x H100), LLMHosts Core can run different models on different GPU subsets. Scheduler treats each GPU group as an independent endpoint.

---

## 10. Self-Healing & Resilience

### Connection Resilience (Relay)

- Exponential backoff reconnection (1s -> 2s -> 4s -> 8s -> max 30s)
- Background health ping every 10s
- Connection migration: if relay server fails, try backup relay
- Local-only mode: if all connectivity is lost, keep serving local requests

### Model Resilience

- Model loading fails: fall back to smaller model from HIM
- GPU crash (CUDA error): restart engine, reload model, replay queued requests
- OOM: evict KV cache pages to CPU, reduce batch size, continue serving
- Watchdog monitors LLMHosts Core engine, restarts within 2s on crash

### Data Resilience

- Request queue persisted to disk (SQLite WAL)
- In-flight requests replayed after crash recovery
- Model files verified by SHA-256 before loading

---

## 11. Performance Targets

| Metric | Target | Benchmark |
|--------|--------|-----------|
| Single-stream latency (7B Q4) | <20ms TTFT | vs Ollama ~25ms |
| Throughput (7B, single GPU) | >100 tok/s | vs Ollama ~40 tok/s |
| Throughput (7B, concurrent) | >2000 tok/s | vs Ollama ~160 tok/s (4 parallel) |
| Memory overhead (engine) | <50MB RSS | vs Ollama ~150MB + Python ~80MB |
| Binary size | <20MB | vs Ollama ~150MB |
| Cold start (model load) | <5s (mmap) | vs Ollama ~10-30s |
| Hot-swap model update | <100ms interruption | Novel capability |
| GPU utilization | >80% under load | vs typical 40-60% |
| Relay latency overhead | <50ms | Already targeted in ADR-009 |

---

## 12. Competitive Positioning

| Capability | LLMHosts Core | Ollama | vLLM | GPUStack |
|------------|---------------|--------|------|----------|
| One-command install | Yes | Yes | No (complex) | Moderate |
| Auto GPU detection | All vendors | NVIDIA/Apple | NVIDIA | All vendors |
| Auto model selection | Yes (HIM) | No (manual) | No (manual) | No (manual) |
| Continuous batching | Yes | No | Yes | Depends on backend |
| PagedAttention | Yes | No | Yes | Depends on backend |
| Phase-split serving | Yes | No | No | No |
| Mixed GPU support | Yes | No | No | Basic |
| Built-in tunnel | Yes (relay) | No | No | No |
| Cloud spillover | Yes (auto) | No | No | No |
| Model hot-swap | Yes (zero-DT) | No | No | No |
| Self-healing | Yes | Basic | No | Basic |
| Memory footprint | ~50MB | ~150MB | ~2GB+ | Varies |
| Cross-platform | All | macOS/Linux | Linux | All |
| Language | Rust | Go/C++ | Python/C++ | Python |

---

## 13. Implementation Phases

### Phase F1: Foundation (Months 1-2)
- `llmhosts_gpu` crate: cross-platform GPU detection
- `llmhosts_core` crate: GGUF loader, basic tensor ops, single-model inference
- LlamaForCausalLM architecture (covers ~80% of models)
- CPU fallback path (no GPU required for development)
- Python bridge via pyo3 (same pattern as relay_core)

### Phase F2: Performance (Months 2-4)
- PagedAttention KV cache manager
- Continuous batching scheduler
- Flash Attention integration (CUDA via candle-flash-attn)
- CubeCL kernels for AMD/Apple
- Benchmark suite against Ollama

### Phase F3: Intelligence (Months 3-5)
- Hardware Intelligence Matrix (static initial version)
- Model Agent (automatic model selection based on hardware)
- Model download & verification
- Hot-swap protocol
- Dashboard integration (model status, recommendations)

### Phase F4: Scale (Months 4-6)
- Phase-split serving (prefill/decode separation)
- Multi-GPU tensor parallelism (NVLink-aware)
- Cluster scheduling (multi-node coordination)
- Datacenter mode (GPU partitioning, node discovery)

### Phase F5: Polish (Months 5-7)
- Windows service + tray app
- MSI installer
- Homebrew formula
- Community telemetry (opt-in)
- Fine-tuned model distribution
- Model Agent AI research automation

---

## 14. Key Decisions Made

| Decision | Choice | Rationale |
|----------|--------|-----------|
| Inference engine | Full custom (CubeCL + cudarc) | Maximum control, smallest footprint, cross-platform |
| Model experience | Curated (we pick, user approves) | Simplicity, guaranteed quality, differentiation |
| GPU abstraction | CubeCL write-once kernels | NVIDIA + AMD + Apple from one codebase |
| KV cache | PagedAttention + FP8 | 24x throughput, 2x memory savings on Hopper+ |
| Scheduling | Continuous batching + phase-split | Maximum throughput + heterogeneous GPU support |
| Model format | GGUF primary, SafeTensors secondary | Community standard, zero-copy mmap, wide model support |
| Cross-platform | Runtime GPU library loading | Single binary works everywhere |
| Tokenization | HuggingFace tokenizers crate (MIT) | Battle-tested, every tokenizer, not worth rebuilding |
| Vendor dependencies | Zero external inference deps | Full control, no Ollama/vLLM/llama.cpp |

---

## 15. Related Documents

- **Research archive**: `docs/knowledge-base/llmhosts-core-inference-research-2026.md`
- **ADR**: `docs/architecture/adr-010-llmhosts-core-inference-engine.md`
- **Relay tunnel**: `docs/architecture/adr-009-relay-tunnel.md`
- **Implementation plan**: `docs/plans/2026-03-09-llmhosts-core-f1-foundation.md`
