# LLMHosts Core Inference Engine — Research Archive (March 2026)

> Research grounding for the LLMHosts Core inference engine design decision. All sources dated or verified as of March 2026.

---

## 1. Market Context

### Total Addressable Market (TAM)

- **Self-hosted AI inference**: $3–5B estimated market by 2028 (Gartner, IDC projections)
- **Growth**: 30% CAGR in on-premise AI deployments
- **Enterprise preference**: 46–57% of organizations prefer on-premise inference for data sovereignty (Gartner 2025 survey, McKinsey 2025 AI adoption report)
- **GPU ownership expansion**: Consumer GPU shipments with 16GB+ VRAM growing 40% YoY (NVIDIA/AMD quarterly reports)

### Competitive Landscape (March 2026)

| Tool | Language | Stars | Focus | Weakness |
|------|----------|-------|-------|----------|
| Ollama | Go + C++ (llama.cpp) | ~120K | Simplicity, macOS/Linux | No batching, no PagedAttention, single-model only, NVIDIA+Apple only |
| vLLM | Python + C++ | ~45K | Production throughput | Complex install, Linux-only, NVIDIA-only (ROCm experimental), >2GB footprint |
| llama.cpp | C++ | ~75K | Universal platform support | Library not product, no scheduling, no auto-config, manual everything |
| GPUStack | Python | ~8K | Multi-GPU management | Wraps other backends, adds layer but not engine, limited differentiation |
| Candle | Rust | ~16K | Rust ML framework | Framework not product, no inference scheduler, no GPU auto-detection |
| exo | Python | ~20K | LAN clustering | P2P only, no cloud, no tunnel, limited model support |

### Key Gap in Market

No existing tool provides: **one-command install + auto GPU detection + auto model selection + continuous batching + PagedAttention + cross-platform + built-in tunnel + cloud spillover** in a single product. Each existing tool has 2–3 of these; none has all.

---

## 2. Rust Inference Performance Data

### Cloudflare Infire (Internal Benchmark, 2025)

Cloudflare rebuilt their inference stack in Rust (internal project "Infire") to replace vLLM:

- **7% faster throughput** on equivalent hardware (H100 SXM)
- **82% less CPU overhead** during inference serving
- **Smaller memory footprint**: Rust binary ~40MB vs vLLM Python stack ~2GB+
- **Operational benefit**: Single binary deploys, no Python dependency chain, faster cold starts

Source: Cloudflare engineering blog post (mid-2025) and AI inference infrastructure talk (re:Invent 2025 equivalent).

### Why Rust for Inference

1. **Zero-cost abstractions**: Trait dispatch compiles to direct function calls. No vtable overhead for hot paths.
2. **Memory safety without GC**: No stop-the-world pauses during inference. Deterministic memory management.
3. **Fearless concurrency**: Ownership system prevents data races at compile time. Critical for multi-GPU scheduling.
4. **SIMD intrinsics**: First-class support for AVX-512, NEON. CPU fallback path is competitive.
5. **Small binary**: 15–20MB static binary vs 150MB+ Go (Ollama) or multi-GB Python (vLLM).
6. **pyo3 bridge**: Zero-overhead Python interop via maturin. Ships inside pip wheel.

---

## 3. GPU Compute Abstraction — CubeCL

### What is CubeCL?

- **Author**: Tracel.ai (same team behind the Burn ML framework)
- **License**: MIT + Apache 2.0 dual license
- **Purpose**: Write GPU compute kernels once in Rust, compile to multiple backends
- **Repository**: github.com/tracel-ai/cubecl

### Compilation Targets

| Frontend | Backend | GPU Vendor |
|----------|---------|------------|
| Rust (CubeCL macros) | CUDA PTX | NVIDIA (all compute capabilities) |
| Rust (CubeCL macros) | ROCm/HIP | AMD (RDNA 2+, CDNA) |
| Rust (CubeCL macros) | WGPU → Vulkan | Cross-platform fallback |
| Rust (CubeCL macros) | WGPU → Metal | Apple Silicon |
| Rust (CubeCL macros) | WGPU → DX12 | Windows (Intel, NVIDIA, AMD) |

### Key Properties

- **Write-once semantics**: Same kernel code targets all backends. No `#ifdef CUDA` / `#ifdef HIP`.
- **Performance**: Within 5-10% of hand-written CUDA for standard operations (matmul, attention). The gap narrows with each release.
- **Composability**: Kernels compose via traits. Higher-order operations (attention = matmul + softmax + matmul) built from primitives.
- **JIT compilation**: Kernels compiled at runtime for the detected GPU. No pre-compilation needed for different architectures.

### Limitations

- **Bleeding edge**: As of March 2026, CubeCL is v0.4.x. API still evolving.
- **Not all CUDA features exposed**: Some vendor-specific features (Tensor Cores warp-level primitives) require drop-down to raw CUDA via cudarc.
- **Documentation**: Sparse. Community is small but active.

### Risk Mitigation

- cudarc provides escape hatch for NVIDIA-specific optimizations (Flash Attention fused kernels, FP8 on Hopper)
- CubeCL only used for cross-platform paths (AMD, Apple, Intel)
- We contribute upstream fixes — aligns with open-source-first tenet

---

## 4. NVIDIA-Specific — cudarc

### What is cudarc?

- **Author**: coreylowman (Rust CUDA ecosystem)
- **License**: MIT + Apache 2.0 dual license
- **Purpose**: Safe Rust bindings for CUDA driver API, runtime API, cuBLAS, cuDNN, cuRAND
- **Key feature**: Runtime library loading. Binary compiles without CUDA SDK. Loads `libcuda.so` at runtime.

### Why cudarc + CubeCL (not just one)

| Scenario | Use | Reason |
|----------|-----|--------|
| Cross-platform kernels (matmul, softmax, RoPE) | CubeCL | Write once, runs on NVIDIA + AMD + Apple |
| NVIDIA-only hot paths (Flash Attention v3, FP8 GEMM) | cudarc | Maximum performance, vendor-specific features |
| GPU detection, memory management | cudarc (NVIDIA) + sysfs (AMD) + IOKit (Apple) | Each vendor has unique APIs |

---

## 5. PagedAttention (KV Cache Management)

### Origin

- **Paper**: "Efficient Memory Management for Large Language Model Serving with PagedAttention" (Kwon et al., UC Berkeley, 2023)
- **Implementation**: vLLM project
- **Impact**: 24x throughput improvement over naive sequential serving

### How It Works

Traditional KV cache allocates contiguous memory per request. This causes:
- Internal fragmentation (allocated > used)
- External fragmentation (gaps between allocations)
- Worst case: 60–80% of GPU memory wasted on fragmentation

PagedAttention fixes this by:
1. Splitting KV cache into fixed-size **pages** (like virtual memory pages)
2. Using a **page table** to map logical positions to physical pages
3. Pages allocated on demand, freed immediately when no longer needed
4. Non-contiguous physical layout transparent to attention computation

### Performance Impact

| Metric | Before (naive) | After (PagedAttention) |
|--------|----------------|----------------------|
| Max concurrent requests (13B model, 40GB) | ~4 | ~100 |
| GPU memory utilization | 20-40% | 90%+ |
| Throughput (req/s) | 1.2 | 28.8 (24x) |

### FP8 KV Cache (Hopper+)

NVIDIA H100/H200 support FP8 (8-bit float) natively. Using FP8 for KV cache:
- Halves KV cache memory (vs FP16)
- Doubles effective context length for same VRAM
- Quality loss negligible (<0.5% on benchmarks)
- Requires Hopper architecture (compute capability 9.0+)

---

## 6. Continuous Batching

### Origin

- **Paper**: "Orca: A Distributed Serving System for Transformer-Based Generative Models" (Yu et al., 2022)
- **Key insight**: In autoregressive generation, different requests finish at different times. Static batching wastes GPU cycles waiting for the longest request.

### Mechanism

Traditional (static) batching:
```
Batch starts → all requests processed → batch ends → new batch starts
If request A finishes early, GPU idles until batch completes
```

Continuous batching:
```
Batch runs → request A finishes → slot freed → new request C enters immediately
GPU never idles. Every cycle is utilized.
```

### Performance Impact

- **Anyscale benchmark (2023)**: 23x throughput improvement over sequential serving
- **vLLM benchmark (2024)**: 2-5x throughput over static batching, depending on request length variance
- **Latency benefit**: New requests don't wait for a batch boundary. Lower time-to-first-token.

---

## 7. Phase-Split Serving (Prefill/Decode Separation)

### Background

LLM inference has two distinct phases:
1. **Prefill**: Process all input tokens at once. Compute-bound. Benefits from high FLOPS.
2. **Decode**: Generate one token at a time. Memory-bandwidth-bound. Benefits from high memory bandwidth.

### Insight

These phases have different hardware requirements:
- Prefill: Needs high compute (FLOPS). H100 ideal.
- Decode: Needs high memory bandwidth. Even consumer GPUs work well.

### Research

- **Splitwise** (Microsoft Research, 2024): Split prefill and decode across different GPU pools. 40% cost reduction, no latency penalty.
- **DistServe** (PKU/MSRA, 2024): Similar approach. 80% cost savings in large deployments.
- **SGLang** (LMSYS/UC Berkeley, 2024): RadixAttention + phase-split for multi-turn conversations. 5x throughput on chat workloads.

### Application to LLMHosts

Phase-split is transformative for mixed hardware:
- User has RTX 4090 (high FLOPS) + RTX 3080 (decent bandwidth): prefill on 4090, decode on 3080
- Datacenter: H100 pool for prefill, consumer GPU fleet for decode
- Cost arbitrage: decode on cheap hardware, only pay premium for prefill

---

## 8. RadixAttention (Prefix Caching)

### Origin

- **Paper**: SGLang project (LMSYS/UC Berkeley, 2024)
- **Key insight**: Multi-turn conversations share long prefixes (system prompt + conversation history). Computing attention over the shared prefix repeatedly is wasteful.

### Mechanism

RadixAttention stores computed KV cache entries in a radix tree indexed by token sequences. When a new request shares a prefix with a cached request, the cached KV entries are reused.

### Performance

- **Cache hit rate**: 75–90% for multi-turn conversations
- **Throughput improvement**: 5x on chat workloads (SGLang benchmark)
- **Latency improvement**: TTFT drops from O(prefix_length) to O(new_tokens)

### Application

Perfect for LLMHosts where users have ongoing conversations via the proxy. The system prompt + conversation history is a long shared prefix. RadixAttention means the first message pays the full prefill cost; subsequent messages only pay for the new user input.

---

## 9. Model Format: GGUF

### What is GGUF?

- **Author**: Georgi Gerganov (creator of llama.cpp)
- **Format**: Binary container for quantized model weights
- **Library**: `gguf-rs-lib` (Rust, MIT license)

### Properties

- **Memory-mapped (mmap)**: Model files loaded via OS page cache. No copy into application memory. Startup time proportional to first-access pages, not total model size.
- **Quantization built-in**: GGUF files contain pre-quantized weights (Q2_K through Q8_0, plus K-quant variants).
- **Single file**: Entire model (weights + metadata + tokenizer) in one file. No scattered config files.
- **Community standard**: ~90% of locally-run models on HuggingFace have GGUF variants (as of March 2026).

### Quantization Quality

| Quant | Bits/weight | Quality retention (MMLU) | Memory savings vs FP16 |
|-------|-------------|------------------------|----------------------|
| Q8_0 | 8 | 99.5% | 50% |
| Q6_K | 6.6 | 99% | 59% |
| Q5_K_M | 5.5 | 98% | 66% |
| Q4_K_M | 4.5 | 96% | 72% |
| Q3_K_M | 3.4 | 92% | 79% |
| Q2_K | 2.6 | 85% | 84% |

We default to Q4_K_M as the sweet spot: 72% memory savings with 96% quality retention.

---

## 10. Tokenization: HuggingFace Tokenizers

### Library

- **Crate**: `tokenizers` (Rust-native, MIT license, by HuggingFace)
- **Performance**: Tokenizes 1GB of text in <20 seconds (single core)
- **Coverage**: BPE, WordPiece, Unigram, SentencePiece — covers every major model family

### Why Not Build Our Own

Tokenization is a solved problem. The HuggingFace crate is:
- MIT licensed (no vendor lock-in)
- Rust-native (no FFI overhead)
- Battle-tested (used by every major framework)
- Actively maintained
- Supports all tokenizer families we'd ever need

Building our own would add months of work with zero differentiation.

---

## 11. Flash Attention

### Origin

- **Paper**: "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness" (Dao et al., Stanford, 2022)
- **Flash Attention 2** (Dao, 2023): 2x faster via better work partitioning
- **Flash Attention 3** (Dao et al., 2024): Hopper-specific optimizations (TMA, warp specialization)

### Mechanism

Standard attention: O(N²) memory for attention matrix.
Flash Attention: Fuses attention computation into a single kernel pass using tiling. Never materializes the full N×N attention matrix.

### Performance

- **2-4x faster** than standard attention
- **5-20x less memory** (no materialized attention matrix)
- **Exact** — not an approximation. Bit-identical results.

### Implementation Strategy for LLMHosts Core

| Platform | Implementation |
|----------|---------------|
| NVIDIA (Ampere+) | `candle-flash-attn` Rust bindings (MIT) |
| NVIDIA (older) | CubeCL tiled attention (custom) |
| AMD | CubeCL tiled attention (custom, ROCm target) |
| Apple Silicon | CubeCL tiled attention (custom, Metal target) |

NVIDIA gets the fastest path via vendor-specific Flash Attention. Other platforms get a CubeCL implementation that's still significantly faster than naive attention.

---

## 12. GPU Detection Ecosystem Gap

### The Problem

No existing Rust crate provides unified cross-platform GPU detection:

| Crate | NVIDIA | AMD | Apple | Intel | Status |
|-------|--------|-----|-------|-------|--------|
| `nvml-wrapper` | Yes | No | No | No | Maintained |
| `amdgpu-sysfs` | No | Linux only | No | No | Minimal |
| `metal` / `objc2-metal` | No | No | Yes | No | Maintained |
| `wgpu` | Partial | Partial | Partial | Partial | Adapter info only, not metrics |

### Our Opportunity: llmhosts_gpu

Build the definitive GPU detection crate. Open-source it. Become the standard:

```rust
let topology = llmhosts_gpu::detect().await;
// Returns: Vec<GpuDevice> with vendor, vram, utilization, temperature, driver, compute capability
// Works on: NVIDIA (NVML), AMD (sysfs/WMI), Apple (IOKit/Metal), Intel (Level Zero/sysfs)
// All vendor libs loaded at runtime via libloading — no compile-time deps
```

This becomes our first open-source contribution from LLMHosts Core and seeds ecosystem adoption.

---

## 13. Memory Footprint Comparison

| Component | Ollama | vLLM | LLMHosts Core (Target) |
|-----------|--------|------|----------------|
| Runtime | Go runtime ~30MB | Python interpreter ~40MB | None (Rust static binary) |
| Binary | ~150MB | ~200MB+ | ~15-20MB |
| Framework overhead | llama.cpp ~80MB | PyTorch ~1.5GB | Minimal (custom kernels) |
| Total (without model) | ~260MB | ~1.8GB+ | ~50MB |

### Why This Matters

Every MB of overhead is VRAM not available for the model. On a 16GB Mac Mini, 260MB overhead means the model gets 15.74GB. With 50MB overhead, the model gets 15.95GB. That extra 210MB could mean:
- Running 7B at Q5_K_M instead of Q4_K_M (better quality)
- Fitting 2K more context length
- Supporting 3 more concurrent requests via larger KV cache

At 24GB (RTX 4090), the difference is even more pronounced for 70B models that barely fit.

---

## 14. Cross-Platform Delivery via Maturin

### How It Works

1. Rust crate compiled to platform-specific shared library (.so/.dylib/.dll)
2. maturin packages the shared library inside a Python wheel
3. `pip install llmhosts` downloads the correct platform wheel
4. Python imports the Rust extension via pyo3

### Proof of Concept

Already implemented for `relay_core`:
- `relay_core/Cargo.toml`: pyo3 + maturin config
- `llmhosts/tunnel/relay.py`: Graceful import with `RELAY_AVAILABLE` flag
- CI: maturin builds wheels for linux-x86_64, linux-aarch64, macos-arm64, macos-x86_64, windows-x86_64

Same pattern scales to `llmhosts_core` and `llmhosts_gpu`.

---

## 15. Sources & References

### Papers

- Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention", SOSP 2023
- Yu et al., "Orca: A Distributed Serving System for Transformer-Based Generative Models", OSDI 2022
- Dao et al., "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness", NeurIPS 2022
- Dao, "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning", 2023
- Dao et al., "FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision", 2024
- Patel et al., "Splitwise: Efficient Generative LLM Inference Using Phase Splitting", ISCA 2024
- Zhong et al., "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving", OSDI 2024
- Zheng et al., "SGLang: Efficient Execution of Structured Language Model Programs", 2024

### Projects & Libraries

- CubeCL: github.com/tracel-ai/cubecl (MIT + Apache 2.0)
- cudarc: github.com/coreylowman/cudarc (MIT + Apache 2.0)
- Candle: github.com/huggingface/candle (MIT + Apache 2.0)
- tokenizers: github.com/huggingface/tokenizers (MIT)
- gguf-rs-lib: Rust GGUF parser (MIT)
- nvml-wrapper: Rust NVML bindings (MIT)
- pyo3: github.com/PyO3/pyo3 (MIT + Apache 2.0)
- maturin: github.com/PyO3/maturin (MIT + Apache 2.0)

### Industry Data

- Cloudflare engineering blog: Rust inference infrastructure benchmarks (2025)
- Gartner: "Predicts 2025: AI Infrastructure" — on-premise preference data
- McKinsey: "The State of AI in 2025" — enterprise deployment patterns
- NVIDIA quarterly earnings (FY2026 Q1-Q3): GPU shipment trends
- AMD quarterly earnings (FY2026 Q1-Q3): ROCm adoption metrics

### Benchmarks Referenced

- Anyscale: Continuous batching 23x throughput (2023)
- vLLM: PagedAttention 24x throughput (SOSP 2023)
- SGLang: RadixAttention 5x throughput on chat (2024)
- Splitwise: 40% cost reduction phase-split (ISCA 2024)
- Cloudflare Infire: Rust 7% faster + 82% less CPU vs vLLM (2025)
