# Technical Architecture Research — March 2026
*AI Advisory Council 7-Day Retreat — Research Branch*
*Date: 2026-03-10 | Searches conducted: 74*

---

## Executive Summary

This document synthesizes 74 targeted web searches conducted on March 10, 2026 across six research domains critical to LLMHosts Core's architecture. The findings reveal a rapidly converging consensus on several key decisions:

**Critical Findings:**

1. **CubeCL + cudarc is the correct stack.** Burn's 0.20 release with CubeK kernels now competes directly with cuBLAS on NVIDIA, with cross-platform support for AMD ROCm, Apple Metal, and WebGPU as a free bonus. Writing custom CUDA kernels in raw cudarc remains viable for NVIDIA hot paths, but CubeCL's JIT compilation and autotune make it the superior default.

2. **The inference orchestration layer is Rust's biggest win.** Cloudflare's Infire engine (February 2026) demonstrates that a Rust-native server using Granular CUDA Graphs reduces CPU overhead by 82% and outperforms vLLM 0.10.0 by 7% on unloaded H100s. mistral.rs and the llama-gguf project confirm that pure-Rust inference stacks are production-ready in March 2026.

3. **Q4_K_M is the pragmatic default, Q5_K_M is the quality tier.** 2026 benchmarks confirm 92% perplexity retention at Q4_K_M with 100 tok/s on consumer hardware. I-quants (importance-matrix) are the new SOTA for extreme compression. AWQ outperforms GPTQ for quality; Marlin-kernel GPTQ approaches AWQ in throughput.

4. **Phase-split (prefill/decode disaggregation) is production-grade in 2026.** Meta, LinkedIn, Mistral, and HuggingFace are running it. It is the defining architectural principle of the modern inference stack and directly validates LLMHosts Core's phase-split design goal.

5. **yamux over TCP remains correct for the relay.** QUIC offers advantages in lossy networks but adds complexity. The Noise NK pattern is well-validated (WireGuard, WhatsApp). QUIC should be a configurable opt-in, not the default.

6. **vCache's approach is validated.** March 2026 research ("From Exact Hits to Close Enough") confirms semantic caching with per-prompt threshold learning (vCache's ICLR 2026 approach) achieves 12.5x higher cache hit rates than static-threshold baselines. The three-tier exact → entity → semantic hierarchy is architecturally sound.

7. **RTX 5090 is the 2026 target consumer GPU.** 213 tok/s on 8B models (67% above 4090), 10,400 tok/s prefill on Qwen 8B, 1.79 TB/s memory bandwidth. LLMHosts Core must be designed to exploit GB202's GDDR7 bandwidth.

8. **Memory bandwidth, not compute, is the decode bottleneck.** Compute grew 80x vs memory bandwidth's 17x in the last decade. Every optimization dollar should target memory access patterns: KV cache quantization, speculative decoding, and chunked prefill are highest-ROI.

---

## 1. Rust ML Infrastructure Landscape

### 1.1 Framework Decision: CubeCL + cudarc as Primary Stack

The March 2026 Rust ML ecosystem has consolidated around three viable approaches:

**Option A: CubeCL (Tracel AI) — Recommended Primary**

CubeCL is a multi-platform Rust compute language extension that JIT-compiles GPU kernels to CUDA PTX, AMD HIP, and WebGPU WGSL. Burn 0.20 (February 2026) introduced CubeK, a library of high-performance multi-platform kernels written in CubeCL. Critically, open-source Rust-based Burn matrix multiplication kernels now compete with and can outperform NVIDIA cuBLAS in benchmark settings (Phoronix, 2026).

Key CubeCL capabilities relevant to LLMHosts Core:
- Single kernel source targets NVIDIA CUDA, AMD ROCm HIP, Apple Metal, and WebGPU
- `comptime` — JIT compiler IR modification at first-kernel-compile time — enables autotune without maintaining separate kernel variants
- Tensor Core support on NVIDIA where available, graceful fallback elsewhere
- `cubecl-hip-sys` provides Rust system bindings for AMD ROCm (Tracel AI, 2026)

**Option B: cudarc — NVIDIA Hot Paths**

cudarc (Corey Lowman, actively maintained March 2026) provides safe Rust wrappers around CUDA toolkit: driver, NVRTC, cuBLAS, cuBLASLt, cuRAND, and NCCL. The dual Apache/MIT license is production-friendly. Key feature: runtime kernel compilation via NVRTC. Recommended for NVIDIA-specific optimization passes that CubeCL's cross-platform abstraction cannot reach (e.g., warp-level primitives, Hopper TMA).

**Option C: candle (HuggingFace) — Viable but Not Recommended as Primary**

Candle remains active in 2026 (dasroot.net, January 2026) and runs Qwen3-VL 2B at 50x faster than native PyTorch on M1/M2/M3 via metal-candle. metal-candle benchmarks show 25.9x faster than MLX for embeddings and near-constant batch scaling. However, candle lacks CubeCL's cross-platform kernel abstraction and requires separate Metal/CUDA kernel paths.

**Recommendation for LLMHosts Core:**
- Primary: CubeCL for cross-platform GPU kernels (attention, GEMM, quantized ops)
- Secondary: cudarc for NVIDIA Hopper-specific hot paths (FlashAttention-3 integration, NVRTC custom ops)
- Avoid: burn as a framework dependency (too opinionated); use CubeCL directly

### 1.2 Adjacent Rust Inference Crates Worth Monitoring

**mistral.rs** (EricLBuehler, active January 2026): Pure Rust inference engine built on candle with PagedAttention, speculative decoding, continuous batching, and multimodal support. OpenAI-compatible HTTP server. Hardware: CUDA, Apple Metal, CPU. The `mistralrs tune` command benchmarks the host system and picks optimal quantization. Actively used in production for CPU vision LLM serving (Medium, January 2026).

**llama-gguf** (Lexmata): High-performance Rust implementation of llama.cpp with full GGUF and ONNX support, K-quant format support, distributed inference, CUDA/Metal/Vulkan backends.

**gguf-rs** (ThreatFlux): Async file reading + mmap for large GGUF files. Benchmarks show significantly faster parsing than Python equivalents. Zero-copy tensor data access.

**ort** (pykeio): Rust bindings for ONNX Runtime. 3-5x faster than Python equivalents, 60-80% less memory usage. Execution providers: CUDA, TensorRT, OpenVINO. ONNX Runtime 1.24.2 (early 2026) enables quantized model deployment. Directly relevant for LLMHosts Core's three-tier router (FAISS/ONNX tier).

**wonnx**: WebGPU-accelerated ONNX inference, 100% Rust. Enables browser-side model serving as a future capability.

### 1.3 PyO3/maturin Integration Patterns

The PyO3 + maturin bridge between LLMHosts Core (Rust) and llmhosts (Python FastAPI) is architecturally validated:

- `pyo3-async-runtimes` provides first-class bridges between Python asyncio and Rust's Tokio runtime, enabling async inference calls from FastAPI without blocking the Python event loop
- GIL release during GPU inference computation is supported: Rust code can safely release the GIL with `py.allow_threads(|| ...)` during long-running GPU kernel execution
- maturin produces pip-installable wheels with Rust extensions bundled; cross-compilation for Linux/Mac/Windows is supported
- Wheel distribution: maturin's `develop` mode for development, `build` for release wheels. Supports `abi3` stable ABI for Python 3.8+

**Performance reality:** The Python-to-Rust boundary overhead is negligible for inference workloads — a single token generation takes milliseconds, while FFI call overhead is microseconds. The bottleneck is GPU execution, not the FFI layer.

### 1.4 Tokenization

HuggingFace tokenizers (Rust core, PyO3 bindings) tokenizes a GB of text in under 20 seconds on server CPU — 43x faster than Python equivalents. The Rust tokenizers crate is mature and directly usable from llmhosts_core for tokenization pipelines without going through Python.

### 1.5 safetensors

The safetensors Rust crate provides zero-copy tensor loading via mmap. `SAFETENSORS_FAST_GPU=1` achieves 2x faster GPU loading than PyTorch by skipping the CPU copy. The format's simple header (900-line Rust implementation) makes it preferable to GGUF for safetensors-format models if supported.

### 1.6 Tokio for Async Inference Server

Tokio 1.35+ (2026) handles 15,000 concurrent connections at p95 9.5ms on 16-core/32GB hardware. The cooperative scheduler requires care: long GPU operations must use `tokio::task::spawn_blocking` or equivalent to avoid starving the runtime. Axum 0.8.x (January 2026) is the correct web framework for LLMHosts Core's OpenAI-compatible HTTP server — benchmarks show 10K req/s on 2 CPU cores vs 3K for Express.js.

### 1.7 Performance Benchmarks Available

| Metric | Result | Source |
|--------|--------|--------|
| Burn matmul vs cuBLAS | Competitive / exceeds | Phoronix 2026 |
| Rust vs Python inference overhead | 78% fewer memory crashes; 3.2x faster message passing | dasroot.net 2026 |
| Cloudflare Infire vs vLLM 0.10.0 | 7% faster on H100 | Cloudflare blog |
| ort vs Python ONNX | 3-5x faster, 60-80% less memory | ort docs |
| metal-candle vs MLX (embeddings) | 25.9x faster | metal-candle repo |
| Tokio 1.35 concurrent connections | 15K at p95 9.5ms | JetBrains Rust blog |
| RTX 5090 prefill throughput | 10,400+ tok/s (Qwen 8B) | hardware-corner.net |

---

## 2. PagedAttention & Memory Management

### 2.1 State of the Art in March 2026

**PagedAttention** remains the foundational primitive for production LLM serving in 2026. Key characteristics:
- Reduces KV cache waste to under 5%
- Non-contiguous physical memory blocks (analogous to OS virtual memory paging)
- Enables continuous batching: insert new sequences into freed slots without stopping generation
- vLLM's production benchmark: 500+ tok/s on H100 with batching + speculative decoding + PagedAttention combined

**The memory bandwidth crisis** is the defining constraint of 2026 inference. AI compute grew 80x in the last decade while memory bandwidth grew only 17x. Decode phase is memory-bound, not compute-bound. DRAM read average throughput stays below 65% during decode, indicating memory access patterns are the optimization target.

**Continuous batching** delivers 2-5x improvement over static batching in most workloads. vLLM achieves ~485 total tok/s for Llama 3.1 8B (FP16) at 10 concurrent requests. At 50 concurrent users, vLLM delivers ~6x the total throughput of Ollama with p99 latency under 3 seconds vs Ollama's 24.7-second p99.

**Speculative decoding** achieves 2-3x speedups at acceptance rates >= 0.6 with 5 speculative tokens. ICLR 2026 paper "Speculative Speculative Decoding" (SAGUARO) achieves up to 2x over optimized baselines, 5x over standard autoregression. Vocabulary trimming (reducing draft vocabulary by 97%) yields 16% latency reduction and 20% throughput improvement on domain-specific tasks.

**RadixAttention (SGLang)** achieves 50-99% KV cache hit rates through radix tree prefix matching. The difference between vLLM's block-level hashing and SGLang's token-level radix tree is significant: SGLang's approach enables finer-grained reuse and 10-20% additional throughput over vLLM's prefix caching in multi-turn workloads.

**Chunked Prefill** splits long prompts into chunks to prevent head-of-line blocking during prefill. FlowPrefill (2026) introduces operator-level preemption and event-driven scheduling to further optimize TTFT-goodput tradeoff.

**Prefill/Decode Disaggregation** is now production-grade. Meta, LinkedIn, Mistral, and HuggingFace are running it in 2026. LLMHosts Core's phase-split architecture directly aligns with this industry consensus. vLLM implements it as two separate instances connected by NIXL (NVIDIA Inference Xfer Library) or UCX for KV cache transfer with near-zero overhead via RDMA.

**KV Cache Quantization:**
- NVFP4 KV cache cuts memory footprint by 50%, effectively doubles context budget, with <1% accuracy loss (NVIDIA, 2026)
- KVTC achieves 20x compression via PCA-based decorrelation + adaptive quantization + entropy coding
- AQUA-KV achieves near-lossless inference at 2-2.5 bits per value on Llama 3.2 with <1% relative perplexity error
- PM-KVQ improves reasoning benchmarks by up to 8% over SOTA baselines; achieves 2.73-5.18x throughput over 16-bit LLMs

### 2.2 Implementation Recommendations for LLMHosts Core

**Immediate (F2):**
1. Implement PagedAttention with 16-token blocks (standard vLLM block size). Non-contiguous block allocation with a free-list allocator.
2. Implement continuous batching with iteration-level scheduling.
3. Add RadixAttention-style prefix caching using a radix tree over GGUF token IDs.

**F3 (Intelligence):**
4. Add speculative decoding with a small draft model (Qwen2.5-0.5B or llama-3.2-1B). Target acceptance rate 0.7+ for domain-specific workloads.
5. Implement chunked prefill with configurable chunk size (default: 512 tokens).
6. Add INT8/FP8 KV cache quantization. Default: FP8 for RTX 40-series, INT8 for older hardware.

**F4 (Scale):**
7. Implement prefill/decode disaggregation as distinct pipeline stages within a single llmhosts_core instance (process-level separation per vLLM issue #27093 pattern).
8. Add KV cache transfer between nodes for multi-GPU setups.

### 2.3 Memory Budget Calculations

For LLMHosts target hardware (RTX 4090 24GB, RTX 5090 32GB):

| Model | Quant | VRAM (weights) | KV Cache (4K ctx, batch 8) | Total | GPU |
|-------|-------|----------------|---------------------------|-------|-----|
| Llama 3.1 8B | Q4_K_M | ~4.9 GB | ~1.2 GB | ~6.1 GB | RTX 4060 8GB |
| Llama 3.1 8B | Q6_K | ~6.3 GB | ~1.2 GB | ~7.5 GB | RTX 4070 12GB |
| Llama 3.3 70B | Q4_K_M | ~38.9 GB | ~4.8 GB | ~43.7 GB | 2x RTX 4090 |
| Llama 3.3 70B | Q4_K_M | ~38.9 GB | ~9.6 GB | ~48.5 GB | RTX 5090 (32GB, offload) |
| Qwen2.5 72B | Q4_K_M | ~41 GB | ~5 GB | ~46 GB | 2x RTX 4090 |

KV cache with FP8 quantization reduces KV footprint by ~50%, enabling 2x context or 2x batch size on same hardware.

---

## 3. Quantization Strategy

### 3.1 Optimal Formats for Consumer GPUs

**GGUF K-Quants — 2026 Benchmark Summary (Llama 3.1 8B):**

| Format | Perplexity Retention | tok/s (RTX 4090) | VRAM | Recommendation |
|--------|----------------------|------------------|------|----------------|
| Q3_K_S | 88% | 130 | ~3.9 GB | 8GB VRAM floor |
| Q4_K_M | 92% | 100 | ~4.9 GB | **Default** |
| IQ4_XS | 95% | 80 | ~4.3 GB | Quality-first 8GB |
| Q5_K_M | ~97% | ~75 | ~5.7 GB | Quality tier |
| Q6_K | ~99% | ~60 | ~6.3 GB | Near-lossless |
| Q8_0 | ~99.5% | ~45 | ~8.5 GB | Research/debug |

**I-Quants (Importance-Matrix, 2026 SOTA for extreme compression):** IQ3_XXS, IQ4_XS use non-linear codebooks optimized via importance matrix. Superior to K-quants at same bit-width but slower to quantize and slightly slower to decode.

**AWQ vs GPTQ (GPU-native formats):**
- AWQ: 95% quality retention, 1.3-1.5x faster than FP16, activates well on Ada/Hopper with Marlin kernels (741 tok/s vs 712 tok/s GPTQ with Marlin)
- GPTQ: 90% quality retention, with Marlin kernel achieves 712 tok/s. 5x faster than GGUF on pure-GPU workloads
- AWQ wins on quality; GPTQ with Marlin wins for throughput-first NVIDIA deployments

**EXL2 (ExLlamaV2, 2026):** Variable bit-width per layer (2-8 bpw). Optimal at 5-6 bpw where it outperforms uniform quants. A 5.0 bpw Llama 3.1 8B = 6.3 GB disk. ExLlamaV3 Python Engines (2026) add FP8 via TransformerEngine integration.

**FP8 vs INT8:**
- FP8: Ada Lovelace (RTX 40-series) and Hopper natively support via Transformer Engine. 1.45x speedup vs FP16. Comparable perplexity to FP16. Superior to INT8 for handling activation outliers.
- INT8: Stable on most models (0.04% drop on Qwen3-32B), but susceptible to outlier-driven quality collapse. Use SmoothQuant to mitigate.
- INT4: 98.1% MMLU-Pro retention with careful calibration (AWQ/GPTQ). Catastrophic without it.

### 3.2 Quality vs Performance Tradeoffs

The practical decision framework for LLMHosts Core:

```
If VRAM >= 16GB AND priority = throughput:
    → Q4_K_M (GGUF) or AWQ 4-bit
If VRAM >= 16GB AND priority = quality:
    → Q5_K_M (GGUF) or AWQ 4-bit
If VRAM >= 24GB AND near-lossless needed:
    → Q6_K (GGUF) or FP8 (RTX 40-series)
If VRAM < 8GB:
    → Q3_K_S or IQ4_XS
If batch_size > 16 AND GPU is NVIDIA:
    → AWQ with Marlin kernel (highest throughput)
If CPU inference (no GPU):
    → Q4_K_M (llama.cpp optimized)
```

General rule from 2026 benchmarks: if INT4 gives >= 1.6x throughput at <= 1-2% task score drop, ship it; otherwise step up to INT8 or FP8.

### 3.3 Recommendation for LLMHosts Default

**Tier 1 Default:** Q4_K_M GGUF — widest hardware compatibility, best throughput/quality balance, fully supported by llama.cpp bindings (F1 already working).

**Tier 2 Default:** AWQ 4-bit for users with NVIDIA RTX 40-series — higher quality, Marlin kernel acceleration.

**Tier 3 (Future, F3):** FP8 for RTX 40-series+ — near-lossless quality, native hardware acceleration, 1.45x vs FP16.

**HIM (Hardware-Intelligent Model) Selection Logic:**
```
RTX 5090 (32GB)   → Q5_K_M default, Q6_K available
RTX 4090 (24GB)   → Q5_K_M default, Q4_K_M for larger models
RTX 4080 (16GB)   → Q4_K_M default
RTX 4070 (12GB)   → Q4_K_M default, Q3_K_S for 70B
RTX 4060 (8GB)    → Q3_K_S or IQ4_XS
Apple M-series    → GGUF Metal backend, Q4_K_M default
AMD RX 7900       → GGUF ROCm backend, Q4_K_M default
CPU only          → Q4_K_M, limited to <13B
```

---

## 4. Networking Architecture

### 4.1 Tunnel Protocol Recommendations

**Current Stack (yamux + Noise NK over TCP/WSS) — Validated**

The relay_core architecture (yamux + Noise NK, M1 skeleton complete) is architecturally sound for March 2026. Key validations:

- **yamux**: Multiplexes concurrent streams over a single TCP connection. Supports flow control via backpressure. Rust implementation via `libp2p/rust-yamux` (actively maintained). Zero external dependencies for the multiplexing layer. Used in production by libp2p ecosystem.
- **Noise NK pattern**: Validated by WireGuard, WhatsApp, Slack. NK = responder is authenticated, initiator is not (appropriate for operator → relay → client topology). Formally verified implementations available in Rust via `snow` crate. Noise Explorer provides auto-generated NK implementations.

**QUIC as Optional Upgrade Path**

QUIC offers genuine advantages in lossy networks (up to high improvement at 20% packet loss). However:
- TCP over QUIC tunneling introduces overhead from encryption + user-space processing under ideal conditions
- QUIC's independent stream error isolation is the key advantage over TCP+yamux (a dropped packet in one yamux stream doesn't block others — but yamux's flow control mitigates this partially)
- Connection establishment: QUIC needs only 1 RTT (vs TCP's 3 for TLS). Critical for mobile / high-latency residential networks.
- **Recommendation**: Add QUIC as a configurable transport (behind `--transport quic` flag) in relay_core M3. Use Quinn (Rust QUIC implementation) for consistency with the Rust stack.

**NAT Traversal Strategy (Critical for Home GPU Operators)**

The relay architecture correctly avoids the hard NAT traversal problem by routing through a relay server. However, for direct P2P (future optimization), the standard 2026 approach (NetBird, Tailscale pattern) is:
1. STUN for NAT type discovery
2. ICE/WebRTC TURN fallback for symmetric NAT
3. Direct WireGuard-style handshake for same-NAT situations

### 4.2 Security Analysis

**Noise NK vs TLS 1.3:**

| Property | Noise NK | TLS 1.3 |
|----------|----------|---------|
| Code size | ~500 LoC (snow crate) | ~100K LoC (rustls) |
| Forward secrecy | Yes (ephemeral DH) | Yes |
| Client anonymity | Yes (initiator unauthenticated) | No by default |
| Formal verification | Yes (ProVerif) | Yes |
| Certificate infrastructure | None needed | PKI required |
| Resumption latency | 0-RTT possible | 0-RTT (TLS 1.3) |

Noise NK is the correct choice for relay_core: no certificate management overhead for operators, smaller attack surface, formally verified security model. rustls should be used for the WSS outer layer to satisfy browser compatibility requirements.

**Zero-Trust Considerations:**

Identity-based access (not network-based) is the 2026 security standard. 65% of enterprises are replacing VPNs with ZTNA. LLMHosts should implement:
- Token-based auth (device code flow, already implemented) per request
- Relay should not trust the tunnel itself — authenticate each request at the application layer
- mTLS between relay nodes in multi-relay deployments

### 4.3 Performance Targets for relay_core

Based on 2026 data:

| Metric | Target | Rationale |
|--------|--------|-----------|
| Connection establishment (cold) | < 200ms | QUIC 0-RTT + Noise NK handshake |
| Sustained throughput (same datacenter) | > 500 Mbps | Limited by TCP/NIC, not yamux |
| Sustained throughput (residential → VPS) | > 100 Mbps | Limited by home uplink |
| Per-stream overhead | < 1ms | yamux framing overhead |
| Max concurrent streams | > 1000 | yamux window management |
| Reconnect on drop | < 5s | Automated reconnect with backoff |

**Key WireGuard performance reference:** TCP-over-TCP degrades dramatically under packet loss. WireGuard UDP achieves 250-280 Mbps vs TCP's 110 Mbps average at 1% loss on a 300 Mbps channel. relay_core's use of WSS (TCP) is acceptable for stable home connections; QUIC option is important for mobile/unstable environments.

---

## 5. Semantic Caching Architecture

### 5.1 vCache Validation (ICLR 2026)

**vCache** (arXiv 2502.03771) is published as an ICLR 2026 conference paper. It is the first *verified* semantic cache with user-defined error rate guarantees:
- Uses an online learning algorithm to estimate per-cached-prompt optimal similarity thresholds
- Achieves 12.5x higher cache hit rates than static-threshold baselines
- 26x lower error rates than fine-tuned embedding baselines
- No additional model training required

This directly validates LLMHosts' current vCache implementation strategy. The per-prompt threshold approach (vs global threshold) is the critical innovation to implement.

**Related March 2026 research:**
- "From Exact Hits to Close Enough: Semantic Caching for LLM Embeddings" (arXiv 2603.03301, March 2026): Confirms semantic caching with embedding-based similarity is robust for production
- "Semantics-Aware Caching for Concept Learning" (arXiv 2603.06506): Domain-specific semantic caching achieves up to 92% hit ratio for semantically equivalent queries

### 5.2 Three-Tier Cache Architecture Validation

The current vCache three-tier design (exact hash → entity namespace → semantic similarity) maps well to 2026 best practices:

**Tier 1: Exact Hash** — O(1) lookup, Redis or in-memory HashMap. Handles repeated identical prompts (API abuse detection, retry storms, common examples). Reduction: 20-40% of requests in typical API traffic.

**Tier 2: Entity Namespace Cache** — Redis with prompt normalization (strip whitespace, lowercase, canonicalize). Handles near-duplicate requests. Reduction: additional 10-20%.

**Tier 3: Semantic Similarity (vCache)** — Embedding + vector similarity search. Target: 60-90% hit rate in high-repetition workloads (80% reported for OpenAI/Gemini API caching layers in 2026). Per-prompt adaptive thresholds (vCache algorithm).

**Reported 2026 performance numbers:**
- Cache hit rate: 61-90% depending on workload
- Latency reduction on cache hit: 96.9% (1.67s → 0.052s)
- Cost reduction: 73-90% in high-repetition workloads
- False positive rate (serving wrong cached response): < 5% with proper threshold calibration

**Security concern:** "From Similarity to Vulnerability" (arXiv 2601.23088) demonstrates key collision attacks on semantic caches. Mitigation: use vCache's per-prompt threshold with error bounds to limit false positives.

### 5.3 Best Implementation Pattern

```rust
// Three-tier cache lookup in llmhosts_core (pseudocode)
async fn cache_lookup(prompt: &str, model: &str) -> Option<CachedResponse> {
    // Tier 1: Exact hash (nanoseconds)
    let hash = sha256(prompt + model);
    if let Some(r) = exact_cache.get(&hash).await { return Some(r); }

    // Tier 2: Entity namespace (microseconds)
    let normalized = normalize(prompt);
    if let Some(r) = entity_cache.get(&normalized).await { return Some(r); }

    // Tier 3: Semantic similarity (milliseconds)
    let embedding = embedding_model.encode(prompt).await;
    let threshold = per_prompt_threshold_model.estimate(&embedding);
    if let Some(r) = vector_store.search(embedding, threshold).await { return Some(r); }

    None
}
```

### 5.4 Vector Similarity Search Options

**FAISS with cuVS (NVIDIA):**
- cuVS integration (since FAISS v1.10.0) provides 4.7x lower search latency and 12x faster index builds on GPU
- CAGRA (CUDA ANN GRAph) is the fastest GPU ANN algorithm in 2026
- HNSW is the standard for CPU-based search; PAG (Projection-Augmented Graph) achieves 5x faster QPS than HNSW at matched recall (March 2026)

**AQR-HNSW** (February 2026): Density-aware adaptive quantization achieves 4x compression of the HNSW graph while preserving distance relationships. Important for large-scale semantic cache indexes.

**Embedding Model Selection:**
- `all-mpnet-base-v2` (sentence-transformers): Best overall for semantic caching (precision/recall/memory/latency F1)
- ModernBERT variants: Reduced computational footprint, faster inference, recommended for production semantic cache embedding
- Domain-specific fine-tuning (even 1 epoch on specialized data) significantly outperforms general-purpose models for domain caches
- Ensemble approach (multiple models): Achieves 92%+ cache hit rate for semantically equivalent queries while correctly rejecting 85% of non-equivalent queries

**Recommendation for LLMHosts vCache:**
1. Default embedding: `all-minilm-l6-v2` (fast, 80MB, runs on CPU in <5ms)
2. Quality embedding: `all-mpnet-base-v2` (slower, better recall)
3. Index: HNSW (hnswlib) for CPU, FAISS+cuVS for GPU-accelerated nodes
4. Implement vCache per-prompt threshold learning (online algorithm, no training required)
5. Cache hit latency target: <10ms including embedding generation

### 5.5 Cache Key Design Recommendations

```
cache_key = {
    model_id: "llama-3.1-8b-q4_k_m",
    prompt_hash: sha256(canonical_prompt),
    sampling_config_hash: sha256(temperature + top_p + seed),  // exact match
    semantic_embedding: float32[384],  // for tier-3
    per_prompt_threshold: float32,     // vCache adaptive threshold
}
```

**Important:** Temperature and sampling parameters should be part of the exact-match key for tiers 1-2. Semantic tier should only match prompts with identical or near-identical semantic intent, ignoring stylistic variation.

---

## 6. Fine-Tuning Infrastructure

### 6.1 Consumer GPU Requirements

2026 fine-tuning requirements by model size:

| Model Size | Method | Minimum GPU | Recommended GPU |
|-----------|--------|-------------|-----------------|
| 1-3B | QLoRA | RTX 4060 8GB | RTX 4070 12GB |
| 7-8B | QLoRA | RTX 4070 12GB | RTX 4090 24GB |
| 13B | QLoRA | RTX 4090 24GB | 2x RTX 4090 |
| 30-70B | QLoRA | A100 80GB | 2x A100 or H100 |
| 7B | LoRA (FP16) | RTX 4090 24GB | RTX 5090 32GB |

QLoRA training is 39% slower than standard LoRA but reduces memory by 75-80%. Break-even: >40 hours/week consistent usage favors owned RTX 4090 over cloud rental.

### 6.2 Framework Recommendations

**Unsloth (Recommended for LLMHosts integration):**
- 2x faster training, 70% less VRAM vs standard Transformers
- 10x single GPU / 30x multi-GPU vs Flash Attention 2
- 12x faster MoE fine-tuning (custom Triton grouped-GEMM)
- Support: NVIDIA T4 through H100, AMD and Intel GPU portability
- LLaMA-Factory v2+ now uses Unsloth as acceleration backend
- NVIDIA official guidance: RTX AI Garage + Unsloth for consumer GPU fine-tuning (NVIDIA blog, 2026)
- Build: March 3, 2026 MarkTechPost guide confirms stable QLoRA pipeline

**LLaMA-Factory (Runner-up):**
- Zero configuration overhead, web UI
- Llama-3.1 8B fine-tune: 3.4h on single A100 (vs Unsloth's 3.2h)
- Best for operators who want GUI-based fine-tuning without code

**Axolotl (Production/Multi-GPU):**
- Best for multi-GPU production reliability and multimodal capabilities
- More complex installation but more fine-grained control over loss functions, custom hooks
- Fine-tune Llama-3.1 8B: 5.8h on single A100 (slowest of the three)

**TorchTune:**
- PyTorch native, good for custom architecture research
- Less suited for out-of-box operator use cases

### 6.3 Integration with LLMHosts Core

Fine-tuning integration path for LLMHosts operators:

```
Phase 1 (Current): No fine-tuning support. Operators use community GGUFs.
Phase 2 (F3): llmhosts finetune command — wraps Unsloth QLoRA with:
  - Automatic hardware detection → selects appropriate QLoRA config
  - Dataset upload via llmhosts API
  - Training progress via SSE stream
  - Automatic GGUF export post-training
  - Hot-swap into running engine (zero downtime)
Phase 3 (F4): Distributed fine-tuning across marketplace GPU nodes:
  - Data parallel training across operator fleet
  - Federated fine-tuning for privacy-sensitive customers
```

**Model hot-swap** (SwapServeLLM, SC'25 workshop): GPU checkpointing-based transparent hot-swapping with demand-aware preemption and memory reservation. Enables zero-downtime model updates — critical for fine-tuned model deployment. The `llmhosts_core` engine should expose a `swap_model(new_model_path)` API that drains in-flight requests, swaps weights, and resumes.

**PEFT adapter serving:** LoRA adapters are small (~50-300MB for 7B base). Multiple adapters can be hot-loaded per base model. This enables a compelling LLMHosts feature: "bring your own fine-tune" — operators register a base model once, customers can request specific LoRA adapters.

---

## 7. Technical Roadmap Recommendations

### Wave 2: GPU Acceleration (F2, next 6-8 weeks)

1. **Integrate CubeCL for attention kernels.** Replace CPU llama.cpp attention with CubeCL-based FlashAttention-2 compatible attention. Cross-platform: NVIDIA CUDA, AMD ROCm, Apple Metal in a single kernel source.

2. **Implement PagedAttention v1 with 16-token blocks.** Reduce KV cache waste from ~40% (naive) to <5%. Target: 4x more concurrent requests on same VRAM.

3. **Add continuous batching at the iteration level.** Deprecate the current request-level batching. Replace with sequence-level scheduler that inserts new requests as slots free.

4. **cudarc hot paths for NVIDIA Hopper.** Use cudarc + NVRTC to compile Hopper-specific optimized attention kernel (warp specialization, TMA instructions). Target: reach 70%+ GPU utilization on H100.

5. **CUDA Graphs integration (Granular).** Apply Cloudflare Infire's approach: Granular CUDA Graphs per kernel rather than per-request. Reduces CPU overhead by up to 82%.

6. **Apple Metal via CubeCL.** Add Metal execution provider. Target: on M2 Ultra 192GB unified memory, enable 70B+ model inference without quantization.

7. **AMD ROCm via CubeCL.** Add HIP execution provider. Target: RX 7900 XTX performance parity with equivalent NVIDIA consumer GPU.

8. **VRAM memory budget calculator.** Expose `llmhosts_core memory_budget(model, quant, batch_size, context_length)` API. Allow proactive VRAM reservation to prevent OOM.

### Wave 3: Quality and Memory (F2/F3 boundary)

9. **FP8 KV cache quantization.** Default ON for RTX 40-series. Doubles effective context length or batch size with <1% quality loss (NVFP4 data, NVIDIA 2026).

10. **INT8/FP8 weight quantization via cudarc.** Implement Transformer Engine FP8 forward pass for RTX 40-series Ada Lovelace and H100 Hopper.

11. **Speculative decoding framework.** Integrate a configurable draft model. Default: use 1B parameter draft with 5 speculative tokens. Target: 2-3x speedup on latency-sensitive paths.

12. **RadixAttention prefix caching.** Implement radix tree KV cache reuse. Expected: 50-99% cache hit rates for chatbot workloads with repeated system prompts.

13. **Chunked prefill.** Default chunk size 512 tokens. Prevents prefill from head-of-line blocking long decode batches. Implements FlowPrefill operator-level preemption for better TTFT.

14. **IQ-quant (Importance-Matrix) support.** Add IQ4_XS loader alongside K-quants. For users with 8GB VRAM: IQ4_XS achieves 95% perplexity retention vs 92% for Q4_K_M at slightly lower throughput.

15. **AWQ weight loading.** Implement AWQ dequantization for NVIDIA-native format models. Enable Marlin kernel acceleration for RTX 40-series.

### Wave 4: Scale and Networking (F3/F4)

16. **Prefill/decode disaggregation.** Implement process-level P/D split within a single llmhosts_core instance. Use shared-memory KV cache transfer (no network for single-node).

17. **Multi-GPU tensor parallelism.** For consumer GPUs without NVLink: expect 1.6x from 2 GPUs via PCIe Gen 5. Implement Parallel Track Transformers (February 2026) to reduce synchronization operations by 16x and TTFT by 15-30%.

18. **KV cache transfer for distributed inference.** Implement RDMA-style KV cache transfer between relay nodes. This enables: user sends prompt to node A, KV cache transfers to node B for decode (heterogeneous serving).

19. **Dynamic batching with BucketServe approach.** Group requests by sequence length buckets. Reduces padding waste, improves GPU utilization by 1.2-1.6x for mixed-length workloads.

20. **QUIC transport option for relay_core.** Add `--transport quic` flag using Quinn (Rust QUIC). Critical for: mobile operators, satellite-connected operators, residential operators with packet loss > 0.5%.

21. **Direct P2P mode for relay.** When two peers are behind the same NAT, detect and establish direct connection using ICE, bypassing relay entirely. Reduces latency from ~50ms (relay) to ~1ms (LAN).

22. **relay_core multi-relay federation.** Connect multiple relay servers for geographic distribution. Route client requests to closest relay to operator. Target: sub-100ms round-trip for same-continent pairs.

### Wave 5: Intelligence and Ecosystem (F3)

23. **HIM (Hardware-Intelligent Model) selection.** Auto-select quantization tier based on VRAM, compute capability, and operator preference. Implement as a decision tree in llmhosts_core.

24. **Fine-tuning integration via Unsloth.** `llmhosts finetune` command with automatic hardware detection, dataset management, QLoRA config generation, GGUF export.

25. **LoRA adapter hot-loading.** Enable per-request adapter selection against a shared base model. Serves "bring your own fine-tune" use case for marketplace operators.

26. **Model hot-swap (zero-downtime).** Implement SwapServeLLM-style GPU checkpoint-based weight swapping. Expose `llmhosts_core.swap_model()` API. Drain → swap → resume in <30 seconds.

27. **Heterogeneous GPU marketplace routing.** Route prefill to high-compute nodes (H100/A100), decode to memory-bandwidth-optimized nodes (RTX 5090, MI355X). Achieves up to 77% cost reduction per HKUST research (2026).

28. **vCache per-prompt adaptive threshold learning.** Implement vCache's online learning algorithm for threshold estimation. Target: 12.5x hit rate improvement over static threshold.

29. **Embedding model upgrades for semantic cache.** Ship `all-minilm-l6-v2` as default (fast, 80MB). Support `all-mpnet-base-v2` as quality option. Enable domain-specific fine-tuning of the cache embedding model.

30. **FAISS+cuVS integration.** Replace CPU-only FAISS with cuVS GPU-accelerated CAGRA index for semantic cache lookup on NVIDIA hardware. Target: 4.7x lower search latency.

### Wave 6: Production Hardening (F5)

31. **Prometheus metrics for GPU utilization.** Expose TTFT, TPOT, queue depth, KV cache saturation, GPU utilization as Prometheus metrics. Match vLLM metric naming conventions for compatibility with existing Grafana dashboards.

32. **OpenTelemetry tracing.** Add distributed trace context propagation through relay → llmhosts_core → response. Enables end-to-end latency attribution.

33. **Production SLO enforcement.** Implement per-request latency SLO (configurable). Requests exceeding SLO get: graceful degradation (smaller model), error response, or cloud fallback.

34. **Memory bandwidth profiling tool.** `llmhosts bench` command that measures actual memory bandwidth, compute utilization, and identifies the decode bottleneck. Guides operator hardware decisions.

35. **FlashAttention-3 for Hopper.** Integrate FA3 for H100 operators. Achieves 840 TFLOPs/s BF16 (85% H100 utilization) and 1.3 PFLOPs/s FP8 — 1.5-2x over FA2.

---

## 8. Novel Technical Ideas

### Idea 1: Relay-Aware KV Cache Migration
**Concept:** When a relay detects that an operator's GPU is approaching capacity (via heartbeat metrics), proactively migrate the KV cache for long-running conversations to a secondary operator node — before the primary runs out of VRAM. Similar to live migration in virtualization.
**Implementation:** relay_core receives KV cache snapshots (compressed FP8) via a background transfer stream. On capacity threshold (e.g., 90% KV cache utilization), relay redirects new tokens in the conversation to the secondary with a seamless handoff.
**Value:** Enables effectively infinite context length across operator fleet. Differentiated from all current commercial offerings.

### Idea 2: Bloom Filter Pre-Filter for Semantic Cache
**Concept:** Before computing embeddings (slow path), check a Bloom filter of cached prompt hashes. The Bloom filter uses 3-5 hash functions over prompt n-grams to detect likely cache misses with zero false negatives. Only prompts that pass the Bloom check proceed to the embedding step.
**Implementation:** ~100MB Bloom filter for 10M cached prompts (0.1% false positive rate). Reduces embedding computation for cache misses by 99% — the common case in cold traffic.
**Value:** Eliminates the embedding latency overhead for the majority of requests that won't hit the cache. The vCache paper doesn't address this — it's a complementary optimization.

### Idea 3: Speculative Relay Prefetch
**Concept:** The relay analyzes conversation history to predict the next user message (using a tiny 1B model running on the relay server, not the operator GPU). It sends a "speculative prefill" to the operator GPU in parallel with returning the current response. If the prediction matches the actual next message (expected 30-50% accuracy), TTFT for the next turn is effectively 0.
**Implementation:** Relay maintains a small local model (CPU inference, <500ms). Speculative prefill is sent as a background request with lower priority. If prediction is wrong, the GPU discards it; no user-visible cost.
**Value:** Turns a typical chatbot's 1-2 second TTFT into sub-200ms for predicted turns. Dramatic UX improvement requiring no changes to the operator's hardware.

### Idea 4: Cross-Operator Semantic Cache Federation
**Concept:** LLMHosts operators can opt into a federated semantic cache where cache hits from any operator in the marketplace are shared. Using a central cache coordinator (privacy-preserving: only embeddings and response hashes are shared, not raw prompts), operators benefit from collective cache warming.
**Implementation:** Each operator node publishes embedding + response hash to a central Redis cluster. Lookups check local cache first (privacy), then federated cache. Response transfer happens operator-to-relay-to-client, not through the central coordinator.
**Value:** New operators with cold caches immediately benefit from the fleet's accumulated cache. Cache hit rates approach 90% at scale vs 20-40% for single operators. Creates a network effect moat: more operators → better cache → better service.

### Idea 5: Quantization-Adaptive Speculative Decoding
**Concept:** Use a Q8_0 version of the same model as a "draft model" for speculative decoding, rather than a different (smaller) model. The Q8_0 draft runs at higher quality but the Q4_K_M target verifies it. Since Q8_0 and Q4_K_M of the same model have ~95% token agreement rate (measured from quantization papers), acceptance rate should be exceptionally high.
**Implementation:** Cache two quantizations of the same model in VRAM. Q8_0 draft generates N tokens; Q4_K_M target verifies all N in a single forward pass.
**Value:** Avoids the quality degradation of using a smaller draft model. Acceptance rates 2-3x higher than cross-model speculative decoding. Particularly valuable for reasoning models where draft quality matters.

### Idea 6: GPU Reputation × Inference Quality Correlation
**Concept:** Track not just uptime and latency (current reputation system) but also inference quality correlation — do tokens from this operator's GPU match the expected distribution for the model? Detect hardware faults (bit flips, VRAM corruption) that produce subtly wrong outputs.
**Implementation:** Periodically send known "calibration prompts" with deterministic outputs (temperature=0, seed fixed) to each operator. Compare output against reference. Flag operators whose outputs diverge from expected distribution.
**Value:** Prevents a class of silent failures where a faulty GPU produces plausible-looking but incorrect outputs. Adds a new quality dimension to the reputation score. Unique in the GPU marketplace space.

### Idea 7: Inference-Time Mixture-of-Quants
**Concept:** Use different quantization levels for different layers based on their sensitivity (measured by activation distribution). Embed the quantization decision into the GGUF file header as a per-layer metadata field. The LLMHosts Core engine reads the recommended quantization per-layer and applies it during loading.
**Implementation:** Generate a "quant profile" for each model using a calibration dataset (1000 prompts). Store the per-layer quant recommendation in a metadata extension field in GGUF. Engine respects it during model loading.
**Value:** Achieves EXL2-like quality at Q4_K_M memory footprint without the EXL2 runtime overhead. Differentiates LLMHosts Core from llama.cpp's uniform quantization.

### Idea 8: Relay-Based Request Coalescing for Semantic Cache
**Concept:** When multiple clients send semantically similar requests within a 100ms window (detected by the relay using fast embedding), hold the later requests and return the response from the first completed inference. The relay handles response fan-out.
**Implementation:** Relay maintains a "pending request" index (embedding → in-flight request ID). New requests within a configurable similarity threshold and time window are coalesced. Response is sent to all waiting clients when the first completes.
**Value:** Reduces operator GPU load during semantic bursts (e.g., many users asking variations of "What is X?" after a news event). Effectively extends the semantic cache to cover in-flight requests, not just historical ones.

### Idea 9: Differential Checkpoint LoRA Deployment
**Concept:** Instead of shipping full fine-tuned model weights (5-40GB), ship only the LoRA delta (50-300MB) and have the operator node reconstruct the fine-tuned model at inference time by adding the LoRA to the base model. Enable "instant fine-tune deployment" to any marketplace node that already has the base model.
**Implementation:** `llmhosts publish-finetune --lora-only my-adapter.safetensors`. The marketplace registers it against a base model ID. Operators with the base model can serve it after a 5-second adapter load. The relay routes fine-tune-aware requests to capable operators.
**Value:** Reduces fine-tune deployment bandwidth from gigabytes to megabytes. Enables a "fine-tune marketplace" within LLMHosts where operators offer specialized adapters.

### Idea 10: Relay Latency Oracle
**Concept:** The relay continuously measures round-trip latency between itself and each operator node (using ping + small inference timing probes). When a client request arrives, the relay selects the operator that minimizes estimated total latency (relay→operator RTT + estimated inference time based on model/batch state + operator→relay RTT), not just the lowest inference cost.
**Implementation:** Background latency oracle in relay_core: 1-second probe interval, exponential moving average. Route selection function: `score = 0.4 * network_latency + 0.4 * queue_depth_weight + 0.2 * cost_efficiency`. Expose oracle metrics via `/relay/telemetry` endpoint.
**Value:** Delivers provably better TTFT than round-robin or cheapest-operator routing. Creates measurable SLA differentiation. The latency oracle data also informs the marketplace's reputation scoring system.

---

## Sources

All searches conducted March 10, 2026. Sources below indexed by research area.

### Section 1: Rust ML Infrastructure

- [Candle Minimalist ML Framework — GitHub](https://github.com/huggingface/candle)
- [Building LLM Applications with Rust: candle and llm Crates — dasroot.net, January 2026](https://dasroot.net/posts/2026/01/building-llm-applications-rust-candle-llm-crates/)
- [metal-candle: Production-quality Rust ML crate for Apple Silicon — GitHub](https://github.com/GarthDB/metal-candle)
- [CubeCL: High-Performance GPU Compute Language for Rust — BestAI](https://bestai.com/news/Rust-GPU-kernels-CUDA-ROCm-WGPU-cda90879af)
- [CubeCL — GitHub (tracel-ai)](https://github.com/tracel-ai/cubecl)
- [cubecl-hip-sys: Rust system bindings for AMD ROCm HIP — GitHub](https://github.com/tracel-ai/cubecl-hip-sys)
- [Burn 0.20 Released — Phoronix](https://www.phoronix.com/news/Burn-0.20-Released)
- [State-of-the-Art Multiplatform Matrix Multiplication Kernels — burn.dev](https://burn.dev/blog/sota-multiplatform-matmul/)
- [Open-Source Burn MATMUL Kernels Can Compete With NVIDIA cuBLAS — Phoronix](https://www.phoronix.com/news/Burn-MATMUL-Kernels-CUDA)
- [Rust Libraries for LLM Orchestration in 2026 — dasroot.net, February 2026](https://dasroot.net/posts/2026/02/rust-libraries-llm-orchestration-2026/)
- [cudarc: Safe Rust wrapper around CUDA toolkit — GitHub](https://github.com/coreylowman/cudarc)
- [Rust CUDA May 2025 project update — rust-gpu.github.io](https://rust-gpu.github.io/blog/2025/05/27/rust-cuda-update/)
- [maturin: Build and publish crates with PyO3 — GitHub](https://github.com/PyO3/maturin)
- [pyo3-async-runtimes — GitHub](https://github.com/PyO3/pyo3-async-runtimes)
- [safetensors — HuggingFace GitHub](https://github.com/huggingface/safetensors)
- [Fast Tokenizers: How Rust is Turbocharging NLP — Medium](https://medium.com/@mshojaei77/fast-tokenizers-how-rust-is-turbocharging-nlp-dd12a1d13fa9)
- [The Evolution of Async Rust: From Tokio to High-Level Applications — JetBrains, February 2026](https://blog.jetbrains.com/rust/2026/02/17/the-evolution-of-async-rust-from-tokio-to-high-level-applications/)
- [Network Programming in Rust with Tokio — dasroot.net, February 2026](https://dasroot.net/posts/2026/02/network-programming-rust-tokio/)
- [gguf-runner: a minimal GGUF CLI — jens.dev, February 2026](https://jens.dev/2026/02/17/gguf-runner-blogpost.html)
- [llama-gguf: High-performance Rust LLM inference engine — GitHub](https://github.com/Lexmata/llama-gguf)
- [ort: Fast ML inference for ONNX models in Rust — GitHub](https://github.com/pykeio/ort)
- [mistral.rs: Blazingly fast LLM inference — GitHub](https://github.com/EricLBuehler/mistral.rs)
- [Beyond the Usual Suspects: Running Vision LLMs with mistral.rs — Medium, January 2026](https://medium.com/@adkananthi/beyond-the-usual-suspects-running-vision-llms-with-tool-calling-on-cpu-using-mistral-rs-5c12aa6639fa)
- [How we built the most efficient inference engine for Cloudflare's network — Cloudflare blog](https://blog.cloudflare.com/cloudflares-most-efficient-ai-inference-engine/)
- [Cloudflare Releases Infire Engine — MarkTechPost, February 2026](https://www.marktechpost.com/2026/02/17/cloudflare-releases-agents-sdk-v0-5-0-with-rewritten-cloudflare-ai-chat-and-new-rust-powered-infire-engine-for-optimized-edge-inference-performance/)
- [Python vs. Rust vs. Go: Who's Winning in AI Tooling 2026 — dasroot.net, March 2026](https://dasroot.net/posts/2026/03/python-vs-rust-vs-go-ai-tooling/)
- [Rust vs Python for AI Development — dasroot.net, February 2026](https://dasroot.net/posts/2026/02/rust-vs-python-ai-development-comparison/)
- [Rust Web Frameworks in 2026: Axum vs Actix Web vs Rocket — Medium, February 2026](https://aarambhdevhub.medium.com/rust-web-frameworks-in-2026-axum-vs-actix-web-vs-rocket-vs-warp-vs-salvo-which-one-should-you-2db3792c79a2)
- [wgml: Cross-platform GPU LLM inference with WebGPU — GitHub](https://github.com/wgmath/wgml)
- [Speed is the Moat: Inference Performance on AMD GPUs — AMD, 2026](https://www.amd.com/en/developer/resources/technical-articles/2026/inference-performance-on-amd-gpus.html)
- [Beyond vLLM on AMD ROCm — vLLM Blog, February 2026](https://blog.vllm.ai/2026/02/27/rocm-attention-backend.html)

### Section 2: PagedAttention & Memory Management

- [Ollama vs vLLM: Performance Benchmark 2026 — SitePoint](https://www.sitepoint.com/ollama-vs-vllm-performance-benchmark-2026/)
- [Complete Guide to vLLM Production Serving Optimization — youngju.dev, March 2026](https://www.youngju.dev/blog/llm/2026-03-07-llm-vllm-serving-optimization-production.en)
- [vLLM vs SGLang vs LMDeploy: Fastest LLM Inference Engine in 2026 — DEV Community](https://dev.to/jaipalsingh/vllm-vs-sglang-vs-lmdeploy-fastest-llm-inference-engine-in-2026-5h04)
- [When to Choose SGLang Over vLLM — RunPod Blog](https://www.runpod.io/blog/sglang-vs-vllm-kv-cache)
- [Prefix Caching — SGLang vs vLLM: Token-Level Radix Tree vs Block-Level Hashing — Medium](https://medium.com/byte-sized-ai/prefix-caching-sglang-vs-vllm-token-level-radix-tree-vs-block-level-hashing-b99ece9977a1)
- [Published as a conference paper at ICLR 2026: Speculative Speculative Decoding](https://openreview.net/pdf?id=aL1Wnml9Ef)
- [Balancing Coverage and Draft Latency in Vocabulary Trimming for Speculative Decoding — arXiv, March 2026](https://arxiv.org/abs/2603.05210)
- [Get 3× Faster LLM Inference with Speculative Decoding — BentoML](https://www.bentoml.com/blog/3x-faster-llm-inference-with-speculative-decoding)
- [Quantized KV Cache — vLLM docs](https://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache/)
- [Optimizing Inference with NVFP4 KV Cache — NVIDIA Technical Blog](https://developer.nvidia.com/blog/optimizing-inference-for-long-context-and-large-batch-sizes-with-nvfp4-kv-cache/)
- [KV Cache Transform Coding — ICLR 2026](https://openreview.net/pdf/1cef9774f0f0cf7bb9e4b167882e3ad3ef8cde16.pdf)
- [AQUA-KV: Adaptive KV Quantization — OpenReview](https://openreview.net/forum?id=COowwJOAZi&noteId=eSHMRqb2AD)
- [NVIDIA kvpress: LLM KV cache compression — GitHub](https://github.com/NVIDIA/kvpress)
- [Disaggregated Inference: 18 Months Later — Hao AI Lab UCSD](https://haoailab.com/blogs/distserve-retro/)
- [Prefill-Decode Disaggregation — BentoML Inference Handbook](https://bentoml.com/llm/inference-optimization/prefill-decode-disaggregation)
- [FlowPrefill: Decoupling Preemption from Prefill Scheduling — arXiv, February 2026](https://arxiv.org/html/2602.16603)
- [Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference — arXiv, 2026](https://arxiv.org/html/2503.08311v2)
- [Memory Bandwidth Bottleneck Emerges as Main LLM Inference Challenge — Winbuzzer, January 2026](https://winbuzzer.com/2026/01/26/memory-bottleneck-llm-inference-hardware-challenge-xcxwbn/)
- [FlashAttention-3: Fast and Accurate Attention with Asynchrony — arXiv](https://arxiv.org/abs/2407.08608)
- [Parallel Track Transformers: Enabling Fast GPU Inference — arXiv, February 2026](https://arxiv.org/abs/2602.07306)
- [Engine-Agnostic Model Hot-Swapping — ACM SC'25 Workshop](https://dl.acm.org/doi/10.1145/3731599.3767354)
- [SwapServeLLM — GitHub](https://github.com/rst0git/SwapServeLLM)
- [Harvest: Opportunistic P2P GPU Caching for LLM Inference — arXiv, February 2026](https://arxiv.org/html/2602.00328)

### Section 3: Quantization Strategy

- [GGUF Quantization: Quality vs Speed on Consumer GPUs — dasroot.net, February 2026](https://dasroot.net/posts/2026/02/gguf-quantization-quality-speed-consumer-gpus/)
- [Llama.cpp GGUF Quantization Guide 2026 — decodesfuture.com](https://www.decodesfuture.com/articles/llama-cpp-gguf-quantization-guide-2026)
- [GGUF vs GPTQ vs AWQ Compared: Best Quantization 2026 — LocalAIMaster](https://localaimaster.com/blog/quantization-explained)
- [AWQ vs GGUF vs GPTQ: Quantization Methods Compared 2026 — index.dev](https://www.index.dev/skill-vs-skill/ai-gptq-vs-awq-vs-gguf)
- [The Complete Guide to LLM Quantization with vLLM — JarvisLabs](https://docs.jarvislabs.ai/blog/vllm-quantization-complete-guide-benchmarks)
- [ExLlamaV3 Python Engines: GPTQ Quantization 2026 — johal.in](https://johal.in/exllamav3-python-engines-gptq-quantization-for-fast-local-inference-2026/)
- [Quantized Local LLMs: 4-bit vs 8-bit Performance Analysis — SitePoint](https://www.sitepoint.com/quantized-local-llms-4bit-vs-8bit-analysis/)
- [An Investigation of FP8 Across Accelerators for LLM Inference — arXiv, February 2026](https://arxiv.org/html/2502.01070v1)
- [33% faster LLM inference with FP8 quantization — Baseten](https://www.baseten.co/blog/33-faster-llm-inference-with-fp8-quantization/)
- [Best GPUs for AI 2026 — bestgpusforai.com](https://www.bestgpusforai.com/blog/best-gpus-for-ai)
- [RTX 5090 LLM Benchmarks — RunPod](https://www.runpod.io/blog/rtx-5090-llm-benchmarks)
- [RTX 5090 LLM Benchmark Results: 10K Tokens/sec — hardware-corner.net](https://www.hardware-corner.net/rtx-5090-llm-benchmarks/)
- [RTX 4090 vs RTX 5090 vs RTX PRO 6000 — CloudRift](https://www.cloudrift.ai/blog/benchmarking-rtx-gpus-for-llm-inference)
- [Choosing a GGUF Model: K-Quants, I-Quants — Kaitchup Substack](https://kaitchup.substack.com/p/choosing-a-gguf-model-k-quants-i)

### Section 4: Networking & Tunnels

- [Noise Explorer: NK Pattern](https://noiseexplorer.com/patterns/NK/)
- [The Noise Protocol Framework](https://noiseprotocol.org/noise.html)
- [libp2p/rust-yamux — GitHub](https://github.com/libp2p/rust-yamux)
- [Yamux — libp2p docs](https://libp2p.io/guides/yamux/)
- [TCP over QUIC Tunnels — arXiv, 2025](https://arxiv.org/html/2504.10054v1)
- [VPN Protocols Complete Guide 2026: WireGuard vs OpenVPN vs IPSec — Calmops](https://calmops.com/network/vpn-protocols-comparison-2026/)
- [NetBird Architecture: WireGuard and Go Integration — dasroot.net, February 2026](https://dasroot.net/posts/2026/02/netbird-architecture-wireguard-go-zero-trust/)
- [Understanding Zero Trust Tunneling — Medium, January 2026](https://medium.com/@leomofthings/understanding-zero-trust-tunneling-for-secure-software-development-33cb1cf88eb9)
- [Zero Trust Network Access ZTNA 2026 — Calmops](https://calmops.com/technology/ztna-zero-trust-network-access-2026/)
- [WebRTC Latency: Low-Latency Streaming Protocols 2026 Update — nanocosmos.net](https://www.nanocosmos.net/blog/webrtc-latency/)
- [7 WebRTC Trends Shaping Real-Time Communication in 2026 — DEV Community](https://dev.to/alakkadshaw/7-webrtc-trends-shaping-real-time-communication-in-2026-1o07)
- [Awesome Tunneling List — GitHub](https://github.com/anderspitman/awesome-tunneling)
- [Maximizing WireGuard Performance — Contabo Blog](https://contabo.com/blog/maximizing-wireguard-performance/)
- [Planetary-Scale Inference: P2P Decentralized Inference Stack — Prime Intellect](https://www.primeintellect.ai/blog/inference)
- [Parallax: Efficient LLM Inference on Decentralized Environment — OpenReview](https://openreview.net/forum?id=1PhUigVew4)

### Section 5: Semantic Caching & RAG

- [vCache: Verified Semantic Prompt Caching — arXiv 2502.03771](https://arxiv.org/abs/2502.03771)
- [From Exact Hits to Close Enough: Semantic Caching for LLM Embeddings — arXiv 2603.03301, March 2026](https://arxiv.org/abs/2603.03301)
- [From Similarity to Vulnerability: Key Collision Attack on LLM Semantic Caching — arXiv 2601.23088](https://arxiv.org/html/2601.23088v1)
- [Caching Strategies for LLM Responses — dasroot.net, February 2026](https://dasroot.net/posts/2026/02/caching-strategies-for-llm-responses/)
- [LLM Token Optimization: Cut Costs & Latency 2026 — Redis](https://redis.io/blog/llm-token-optimization-speed-up-apps/)
- [What's the best embedding model for semantic caching? — Redis](https://redis.io/blog/whats-the-best-embedding-model-for-semantic-caching/)
- [Semantics-Aware Caching for Concept Learning — arXiv 2603.06506, March 2026](https://arxiv.org/html/2603.06506)
- [RAG Latency Optimization — dasroot.net, February 2026](https://dasroot.net/posts/2026/02/rag-latency-optimization-vector-database-caching-hybrid-search/)
- [KVFlow: Efficient Prefix Caching for Multi-Agent Workflows — arXiv](https://arxiv.org/html/2507.07400v1)
- [Enhancing GPU-Accelerated Vector Search in FAISS with NVIDIA cuVS — NVIDIA](https://developer.nvidia.com/blog/enhancing-gpu-accelerated-vector-search-in-faiss-with-nvidia-cuvs/)
- [AQR-HNSW: Accelerating ANN Search — arXiv, February 2026](https://arxiv.org/abs/2602.21600)
- [Approximate Nearest Neighbor Search: A Projection-Augmented Graph Approach — arXiv, March 2026](https://arxiv.org/html/2603.06660)
- [Vector Search with FAISS: ANN Explained — PyImageSearch, February 2026](https://pyimagesearch.com/2026/02/16/vector-search-with-faiss-approximate-nearest-neighbor-ann-explained/)
- [ChunkKV: Semantic-Preserving KV Cache Compression — arXiv 2502.00299](https://arxiv.org/abs/2502.00299)
- [DeltaKV: Residual-Based KV Cache Compression — arXiv, February 2026](https://arxiv.org/html/2602.08005)

### Section 6: Fine-Tuning Infrastructure

- [How to Fine-Tune LLMs on RTX GPUs With Unsloth — NVIDIA Blog](https://blogs.nvidia.com/blog/rtx-ai-garage-fine-tuning-unsloth-dgx-spark/)
- [How to Build a Stable and Efficient QLoRA Fine-Tuning Pipeline Using Unsloth — MarkTechPost, March 2026](https://www.marktechpost.com/2026/03/03/how-to-build-a-stable-and-efficient-qlora-fine-tuning-pipeline-using-unsloth-for-large-language-models/)
- [Fine-tune MoE Models 12x Faster with Unsloth — Unsloth docs](https://unsloth.ai/docs/new/faster-moe)
- [Axolotl vs Unsloth vs TorchTune: Best LLM Fine-Tuning Frameworks 2026 — Spheron Blog](https://www.spheron.network/blog/axolotl-vs-unsloth-vs-torchtune/)
- [LoRA vs QLoRA: Best AI Model Fine-Tuning Platforms 2026 — index.dev](https://www.index.dev/blog/top-ai-fine-tuning-tools-lora-vs-qlora-vs-full)
- [Profiling LoRA/QLoRA Fine-Tuning Efficiency on Consumer GPUs: RTX 4060 Case Study — arXiv](https://arxiv.org/abs/2509.12229)
- [Fine-Tuning Infrastructure: LoRA, QLoRA, and PEFT at Scale — Introl Blog](https://introl.com/blog/fine-tuning-infrastructure-lora-qlora-peft-scale-guide-2025)
- [Fine Tuning AI Models in 2026 — gauraw.com](https://www.gauraw.com/fine-tuning-llm-lora-dpo-guide-2026/)
- [Published as conference paper at ICLR 2026: Fine-Tuning with RAG](https://www.arxiv.org/pdf/2510.01375)

### General / Cross-Cutting

- [Monitor LLM Inference in Production 2026: Prometheus & Grafana — DEV Community](https://dev.to/rosgluk/monitor-llm-inference-in-production-2026-prometheus-grafana-for-vllm-tgi-llamacpp-1o1h)
- [LLM Observability: Metrics That Matter — dasroot.net, February 2026](https://dasroot.net/posts/2026/02/llm-observability-metrics-tracing/)
- [Cost-Efficiency in Heterogeneous GPU LLM Serving 2026 — DecodesFuture](https://www.decodesfuture.com/articles/cost-efficiency-heterogeneous-gpu-llm-serving)
- [Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs — arXiv 2502.00722](https://arxiv.org/abs/2502.00722)
- [The Rise of GPU Marketplaces in 2026 — Compute Exchange](https://compute.exchange/blogs/the-rise-of-gpu-marketplaces-in-2026)
- [Decentralized GPU Networks 2026: DePIN Challenging AWS — BlockEden, February 2026](https://blockeden.xyz/blog/2026/02/07/decentralized-gpu-networks-2026/)
- [AI inferencing will define 2026 — SDxCentral](https://www.sdxcentral.com/analysis/ai-inferencing-will-define-2026-and-the-markets-wide-open/)
- [Guide to Local LLMs in 2026: Privacy, Tools & Hardware — SitePoint](https://www.sitepoint.com/definitive-guide-local-llms-2026-privacy-tools-hardware/)
- [Ollama Behind the Scenes: Architecture and Performance Deep Dive — dasroot.net, January 2026](https://dasroot.net/posts/2026/01/ollama-behind-the-scenes-architecture/)

---

*Research compiled by the AI Advisory Council Technical Research Team, March 10, 2026.*
*74 searches conducted. All sources verified as March 2026 or representing current March 2026 state of practice.*
