# ADR-010: LLMHosts Core — Custom Rust Inference Engine

**Status:** Accepted
**Date:** 2026-03-09
**Deciders:** Product owner, Claude (architecture)
**Supersedes:** ADR-002 (Ollama-primary inference engine strategy)

---

## Context

LLMHosts currently depends on Ollama for local inference. This creates several strategic and technical problems:

1. **Vendor dependency**: Ollama controls the inference layer. Breaking changes, licensing shifts, or feature gaps directly impact our product.
2. **Performance ceiling**: Ollama lacks continuous batching, PagedAttention, and phase-split serving. Our throughput is capped by their architecture decisions.
3. **No cross-platform GPU support**: Ollama supports NVIDIA and Apple Silicon. AMD ROCm support is experimental. Intel is absent.
4. **No model curation**: Users must manually select, download, and configure models. This contradicts our "it just works" promise.
5. **Memory overhead**: Ollama's Go runtime + llama.cpp layer consumes ~260MB before any model loads. On 16GB devices, this matters.
6. **Datacenter gap**: Ollama has no multi-GPU scheduling, no tensor parallelism, no cluster coordination. It's a single-machine tool.

The product vision demands a vertically-integrated inference stack: one command to install, automatic hardware detection, AI-curated model selection, and performance from a Mac Mini to an 8x H100 rack. No existing tool provides this.

---

## Decision

Build **LLMHosts Core**, a custom Rust-native inference engine that replaces all external inference dependencies.

### Architecture

Three Rust crates shipping inside the pip wheel via maturin:

| Crate | Purpose |
|-------|---------|
| `llmhosts_core` | Inference: model loading, tensor ops, KV cache (PagedAttention), scheduler (continuous batching + phase-split) |
| `llmhosts_gpu` | Cross-platform GPU detection and memory management (NVML, sysfs, IOKit, Level Zero) |
| `relay_core` | Tunnel connectivity (already in progress per ADR-009) |

### Compute Foundation

- **CubeCL**: Write-once GPU kernels compiling to CUDA, ROCm/HIP, and WGPU (Metal/Vulkan/DX12). Provides cross-platform coverage from a single codebase.
- **cudarc**: NVIDIA-specific optimizations for hot paths (Flash Attention fused kernels, FP8 KV cache on Hopper+) where maximum performance justifies vendor-specific code.

### Model Strategy

- **Curated, not open**: The Hardware Intelligence Matrix (HIM) maps hardware profiles to optimal models. Users don't pick models — they approve recommendations.
- **Model Agent**: AI-powered background service that monitors the model landscape and updates recommendations when better options exist.
- **Hot-swap protocol**: Zero-downtime model updates (download → pre-load → drain → activate → unload, <100ms interruption).
- **GGUF primary format**: Zero-copy mmap, community standard, pre-quantized. SafeTensors as secondary for larger models.

### Python Control Plane (unchanged)

The existing LLMHosts Python codebase retains: HTTP API proxy, intelligent routing, dashboard, marketplace, metrics. LLMHosts Core replaces only the inference layer.

---

## Alternatives Considered

### Option A: Continue with Ollama (Status Quo)

**Pros:** Already working, large community, actively developed.
**Cons:** Vendor dependency, no batching, no PagedAttention, no AMD support, ~260MB overhead, no datacenter features. Our product quality is forever bounded by their roadmap.

**Rejected:** Contradicts our core value proposition of owning the entire stack.

### Option B: Candle-based (HuggingFace Rust Framework)

**Pros:** Mature Rust ML framework (16K stars), good CUDA support, SafeTensors integration.
**Cons:** Framework not product — no scheduler, no batching, no auto-config. Candle provides tensor operations and model implementations but we'd still need to build PagedAttention, continuous batching, GPU detection, and model management on top. The abstraction layer doesn't buy enough.

**Rejected:** We'd build most of the inference engine anyway. The Candle layer adds dependency without sufficient value.

### Option C: llama.cpp FFI (Rust Bindings)

**Pros:** Fastest path to working inference. llama.cpp has the most model support and widest platform coverage.
**Cons:** C++ dependency chain, build complexity (CMake + CUDA toolkit), llama.cpp's API surface changes frequently, no PagedAttention (PR #1), limited batching. We'd be wrapping someone else's code with no control over the hot path.

**Rejected:** Same vendor dependency problem as Ollama, just at a lower level.

### Option D: Hybrid (CubeCL + cudarc) — SELECTED

**Pros:** Full control over inference hot path. CubeCL provides true cross-platform GPU kernels (write once, compile to CUDA/ROCm/Metal). cudarc provides NVIDIA-specific optimizations where they matter most. We own every line of the inference pipeline.

**Cons:** Highest engineering investment. CubeCL is bleeding-edge (v0.4.x). We must implement model architectures ourselves.

**Selected because:** Maximum control, smallest footprint, cross-platform from day one, and no ceiling on performance optimizations. The engineering investment is front-loaded but gives us permanent competitive advantage.

---

## Consequences

### Positive

- **Zero vendor dependencies** for inference. We control the entire path from GPU to API response.
- **Cross-platform from one codebase**: CubeCL compiles to CUDA, ROCm, and Metal.
- **~50MB binary** (vs ~260MB Ollama, ~2GB vLLM). More VRAM for models.
- **Market-leading features**: PagedAttention + continuous batching + phase-split in a single-binary tool. No competitor offers this combination in a consumer-friendly package.
- **Datacenter ready**: Multi-GPU tensor parallelism, cluster scheduling, GPU partitioning — features Ollama will never have.
- **Model curation**: HIM + Model Agent deliver the "it just works" promise.

### Negative

- **5–7 month engineering timeline** for full feature parity with Ollama on basic inference.
- **CubeCL risk**: Library is young. We may need to contribute fixes upstream or work around gaps.
- **Model coverage**: Phase 1 covers LlamaForCausalLM (~80% of models). Other architectures phased in over time.
- **Maintenance burden**: We own the inference code. Bugs in tensor operations, attention, or scheduling are our responsibility.

### Mitigations

- Phased delivery: F1 (foundation, 2 months) delivers basic inference. Value accrues incrementally.
- `llmhosts_gpu` open-sourced early to build community and get battle-testing.
- Ollama remains as a fallback until LLMHosts Core reaches parity (LLMHosts already has the abstraction layer).
- cudarc escape hatch: when CubeCL can't express a needed operation, drop to raw CUDA.

---

## Related

- **Design document**: `docs/plans/2026-03-09-llmhosts-core-inference-engine-design.md`
- **Research archive**: `docs/knowledge-base/llmhosts-core-inference-research-2026.md`
- **Relay tunnel ADR**: `docs/architecture/adr-009-relay-tunnel.md`
- **Previous inference strategy**: ADR-002 (now superseded)
