# ADR-002: Inference Engine Selection

## STATUS
SUPERSEDED (2026-03-09) -- replaces SUPERSEDED (2026-02-14) -- replaces PROPOSED (2026-02-13)

## CHANGE LOG
- 2026-02-13: Original proposal -- three-engine strategy (vLLM + exo + Ollama)
- 2026-02-14: **SUPERSEDED** -- simplified to Ollama-primary with vLLM opt-in. exo restricted to LAN-only. WAN sharded inference killed. Based on Debate-001 Q3 Holes 6, 7 + technical research proving exo over WAN is a dead end.
- 2026-03-09: **SUPERSEDED by ADR-010** -- replaced by custom Rust inference engine (LLMHosts Core). Ollama dependency eliminated. Full-stack ownership: CubeCL + cudarc, PagedAttention, continuous batching, phase-split serving, AI-curated models. See [ADR-010](adr-010-llmhosts-core-inference-engine.md).

## CONTEXT
LLMHosts serves models on user hardware ranging from consumer laptops to multi-GPU workstations. The pivoted architecture uses "Ollama Inside" -- wrapping and enhancing the user's existing Ollama installation rather than competing with it.

### Why Three-Engine Was Killed
The original ADR proposed vLLM + exo + Ollama with auto-detection. Debate-001 identified fatal problems:
1. **exo over WAN is dead**: UDP multicast discovery is LAN-only. Pipeline parallelism across 50ms WAN adds ~200ms overhead PER TOKEN for a 4-stage pipeline. Petals proved WAN inference is limited (4-6 tok/s for 70B) and the project went inactive.
2. **Three-engine complexity**: 3 engines x N models = massive testing matrix for a solopreneur
3. **Model distribution**: Auto-provisioning weights means LLMHosts faces legal/bandwidth costs
4. **Ollama already won**: Most popular consumer LLM runner. Users already have it. Don't fight it.

### The "Ollama Inside" Insight
Instead of replacing Ollama, LLMHosts WRAPS it. Discovers existing Ollama installations and models, routes to them, adds smart routing + caching + cost dashboard on top. Zero re-download. Zero new model management. Users keep their familiar `ollama pull` workflow.

## OPTIONS EVALUATED (REVISED)

### Option A: Ollama-Primary ("Ollama Inside") -- SELECTED
- **Pros**: Users already have it, zero re-download of existing models, simple setup, broad hardware support, OpenAI-compatible API, active ecosystem
- **Cons**: Lower throughput than vLLM for production loads
- **Best For**: 90% of LLMHosts users (single-device, consumer hardware)
- **Strategy**: LLMHosts discovers existing Ollama, wraps it, adds routing + caching + dashboard

### Option B: vLLM -- OPT-IN FOR POWER USERS
- **Pros**: Industry-leading throughput, prefix caching (10x savings), parallelism
- **Cons**: Requires NVIDIA GPU, heavier setup, less consumer-friendly
- **Best For**: Power users with dedicated GPU(s) wanting maximum throughput
- **Strategy**: `llmhosts config set engine vllm` for users who want it

### Option C: exo -- LAN CLUSTERS ONLY
- **Pros**: P2P auto-discovery, dynamic partitioning across devices on same LAN
- **Cons**: WAN latency makes it unusable across the internet
- **Best For**: Multiple GPUs in same house/office (same subnet, <1ms latency)
- **Strategy**: Auto-detected when multiple LLMHosts nodes on same LAN. NOT used for Hive Mode across WAN.

### Option D: Cloud API Fallback (BYOK) -- ALWAYS AVAILABLE
- **Pros**: Instant value before local models are ready, handles frontier model requests
- **Cons**: Costs money (user's API keys), not private
- **Best For**: Cold start, frontier models not available locally, overflow capacity

## DECISION
**Ollama-primary** with **vLLM opt-in** for power users, **exo for LAN-only** clusters, and **cloud API fallback** via BYOK keys.

### Routing Hierarchy
```
Request arrives at LLMHosts proxy
  → Semantic cache check (vCache) → if HIT, return cached response
  → kNN router classifies request
  → Route decision:
      1. LOCAL OLLAMA: if suitable model available, fastest, free, private
      2. LOCAL VLLM: if user opted in, high-throughput workload
      3. LAN CLUSTER (exo): if multiple LAN nodes, model needs more GPU than one node has
      4. HIVE NODE (WAN): if Hive Mode enabled, safe request, remote node has better model
      5. CLOUD API: if no local option sufficient, use user's BYOK keys (cheapest provider)
  → Privacy gate: sensitive requests NEVER go to options 4 or 5
```

### Model Distribution Strategy: "Don't Distribute"
- LLMHosts discovers what models users already have installed via Ollama API
- Recommends models to pull based on hardware profile: `llmhosts suggest-models`
- User runs `ollama pull <model>` themselves (user accepts license, not LLMHosts)
- LLMHosts pays $0 in bandwidth, has $0 legal exposure
- For vLLM users: same pattern via HuggingFace direct download

### WAN Routing (Hive Mode) Architecture
```
Hive Mode does NOT shard models across WAN.
Instead:
  - Each node runs WHOLE models locally
  - Router picks which NODE handles each request (based on model availability + latency)
  - 50ms WAN overhead per REQUEST is acceptable
  - 50ms WAN overhead per TOKEN (what sharded inference does) is NOT acceptable
  - This means: if a friend has Llama-3-70B running, your request goes to them WHOLE
  - Not: your request gets split across 4 friends' GPUs and reassembled
```

## RATIONALE
- "Ollama Inside" leverages existing ecosystem instead of competing with it
- Zero model distribution cost/risk (legal + bandwidth)
- Dramatically simpler testing matrix (Ollama covers 90% of users)
- vLLM opt-in serves power users without burdening casual users
- exo LAN-only preserves the multi-GPU home cluster value without WAN latency problem
- Cloud fallback ensures instant value from Minute 0 (before local models are ready)
- Whole-model WAN routing is architecturally simple and proven (it's just HTTP request forwarding)

## CONSEQUENCES
- Primary dependency on Ollama project health (mitigated: it's the most active OSS LLM project)
- Users without Ollama need to install it first (mitigated: `llmhosts serve` auto-installs Ollama in Docker)
- vLLM users are a smaller audience, may get less testing attention
- LAN cluster detection requires mDNS/UDP broadcast on same subnet
- Hive Mode WAN routing limited to models that fit on a single node (mitigated: quantization makes most models fit on consumer GPUs)
