# Technical Holes Research Report
**Date:** 2026-02-14
**Scope:** Holes 5-9 from founder stress-test debate
**Research queries:** 6 primary + 7 follow-up web searches, 3 paper reviews

---

## TABLE OF CONTENTS
1. [Hole 5: 8B Conductor Brain Bottleneck](#hole-5-8b-conductor-brain-is-a-bottleneck-at-scale)
2. [Hole 6: Model Weight Distribution](#hole-6-model-weight-distribution-is-legally-and-logistically-complex)
3. [Hole 7: exo P2P Over WAN](#hole-7-exo-p2p-inference-over-waninternet-is-unproven)
4. [Hole 8: OpenZiti Operational Complexity](#hole-8-openziti-operational-complexity)
5. [Hole 9: Self-Evolution Engine](#hole-9-self-evolution-engine-is-a-moonshot-bolted-onto-an-mvp)
6. [Architecture-Level Recommendation](#architecture-level-recommendation-the-hybrid-pivot)

---

## Hole 5: 8B Conductor Brain is a Bottleneck at Scale

### The Problem
Every user request routes through an 8B-parameter model (Nemotron-Orchestrator / Qwen3-8B) for routing decisions. At 10K concurrent users, this creates a centralized scaling nightmare. If it runs per-user, it consumes their GPU capacity. If centralized, LLMHosts pays for massive GPU infrastructure just for routing.

### Research Findings

**The entire premise of using an LLM for routing is questionable.** Multiple 2025 papers demonstrate that simpler methods match or beat LLM-based routers:

1. **kNN Routing Outperforms Learned Routers** (ICLR 2026 submission, Li et al.)
   - A well-tuned k-Nearest Neighbors approach using text embeddings **matches or outperforms** state-of-the-art learned routers including complex neural architectures.
   - Key insight: model performance has **strong locality properties in embedding space** -- semantically similar queries perform similarly on the same models. This is a geometric property, not something you need 8B parameters to exploit.
   - Routing reduces to: embed the query → find k nearest neighbors → look up which model performed best → route there. This is a **FAISS index lookup**, not an LLM inference call.

2. **LLMRouterBench** (Jan 2026, arXiv:2601.07206)
   - Comprehensive benchmark: 400K+ instances, 21 datasets, 33 models.
   - Finding: **backbone embedding models have limited impact on routing performance** -- even small embedding models work well.
   - Many sophisticated routing approaches, including commercial routers, **fail to reliably outperform simple baselines**.

3. **Training-Free Online Routing** (Sep 2025, arXiv:2509.02718)
   - Efficient algorithm using approximate nearest neighbor search.
   - **Microsecond-level latency** at inference time.
   - 3.55× performance improvement and 1.85× cost efficiency vs. baselines.

4. **vLLM Semantic Router** (Oct 2025, vllm-project/semantic-router)
   - Production-grade system from the vLLM team.
   - Uses embedding models (small ones like Qwen3-Embedding) + cosine similarity.
   - **10-50ms routing latency** (vs. seconds for 8B model inference).
   - No retraining needed when adding/removing models -- just update candidate lists.
   - Demonstrated: 47.1% latency reduction, 48.5% token reduction.

5. **Cost-Spectrum Contrastive Routing (CSCR)** (2025)
   - Maps prompts + models into shared embedding space using compact logit footprints.
   - Routing = single FAISS k-NN lookup. No retraining when model pool changes.
   - 25% improvement in accuracy-cost tradeoffs.

### Proposed Solutions

#### Solution A: kNN Embedding Router (RECOMMENDED)
**Replace the 8B model entirely with an embedding-based kNN router.**

- **How it works:**
  1. Maintain a FAISS index of query embeddings paired with model performance scores.
  2. When a request arrives, embed it with a small model (e.g., `all-MiniLM-L6-v2`, 80MB, runs on CPU in <5ms).
  3. k-NN lookup against the index → retrieve which model/node historically performs best for similar queries.
  4. Route to that model/node.
- **Latency:** <10ms (vs. 2-5 seconds for 8B inference).
- **Cost:** Runs on CPU. No GPU required for routing.
- **Scaling:** FAISS handles billions of vectors on a single machine. 10K concurrent users is trivial.
- **Cold start:** Bootstrap with a manually curated routing table (model X is good at code, model Y at creative writing, etc.). Improve with real usage data over time.
- **Infrastructure:** Single Redis/FAISS service. Horizontally scalable. ~$20/month at 10K users.

#### Solution B: Rule-Based Tiered Router
**Use a cascading heuristic system for the 80% case, with kNN for the 20% edge case.**

- **Tier 1 (instant, <1ms):** Pattern matching -- code keywords → code model, language detection → appropriate model, explicit user model selection → pass through.
- **Tier 2 (fast, <10ms):** Embedding similarity against category centroids (5-10 categories).
- **Tier 3 (rare, <50ms):** Full kNN lookup for ambiguous queries.
- **Advantage:** Most requests never hit Tier 2/3. Extremely cheap.

#### Solution C: Lightweight Classifier (Fallback Option)
**Train a small classifier (logistic regression or small MLP) on routing decisions.**

- Use BERT-tiny or distilled embeddings as features.
- Train on: {query_embedding} → {best_model_id}.
- Inference: <5ms on CPU.
- Retrain weekly on accumulated performance data.

### Verdict
**Kill the 8B Conductor Brain for routing.** The research is unambiguous: simple methods work as well or better. Keep the "Conductor" concept as a brand name but implement it as an embedding + kNN system. The 8B model could still be useful for other things (query rewriting, response evaluation) but routing is not one of them.

---

## Hole 6: Model Weight Distribution is Legally and Logistically Complex

### The Problem
Auto-provisioning models means downloading 4-70GB files. Licensing varies (Llama Community License, Mistral's Apache, Gemma ToU, etc.). Bandwidth costs are real. Users can't wait hours for downloads before first use.

### Research Findings

1. **Ollama's Registry Model:**
   - Ollama maintains a centralized registry (`registry.ollama.com`) with `ollama pull` downloading over HTTPS.
   - License handling is **poor** -- models are downloaded before users see licenses (open GitHub issue #2896). License metadata is outdated (Gemma licenses still show Feb 2024 terms despite April 2024 updates).
   - Architecture: blob-based storage with deduplication. Docker-style layer system.
   - 50M+ Docker Hub pulls of the Ollama image alone.

2. **OMLA (Open Model License Agreement) Draft:**
   - Industry attempt to standardize model licensing (v8-10-2025).
   - Proposes 30% revenue share for commercial use -- not viable for LLMHosts's unit economics.
   - Shows the licensing landscape is actively evolving and **not settled**.

3. **HuggingFace Hub:**
   - Industry standard for model hosting.
   - Handles license acceptance via click-through gates.
   - Supports partial downloads, delta updates, and CDN distribution.

### Proposed Solutions

#### Solution A: Piggyback on Ollama's Registry (RECOMMENDED)
**Don't build a model registry. Use Ollama's.**

- **How it works:** LLMHosts agent runs Ollama internally. Models are pulled via `ollama pull`. Ollama handles the download, quantization format, and storage.
- **Licensing:** Pass responsibility to the user. LLMHosts's agent is just a tool that helps users run models they choose. The user accepts model licenses, not LLMHosts. This is the same legal model as Ollama, Docker Hub, and package managers.
- **Bandwidth:** Ollama downloads are deduplicated (shared layers). Users download once, LLMHosts indexes what's available.
- **Legal framing:** LLMHosts is an orchestration layer, not a model distributor. Users bring their own models. LLMHosts never hosts or redistributes weights.
- **First-use experience:** Pre-populate a recommendation of 2-3 small models (e.g., Qwen2.5-3B, Phi-3-mini) that download in <5 minutes and cover basic use cases.

#### Solution B: Model Availability Discovery (Not Distribution)
**Invert the model: don't distribute models, discover what users already have.**

- **How it works:**
  1. Agent scans local Ollama/vLLM/HuggingFace cache on installation.
  2. Reports available models to LLMHosts platform.
  3. Platform routes requests to users who already have the needed model.
  4. Suggests additional models to download based on demand patterns.
- **Advantage:** Zero bandwidth cost for LLMHosts. Zero licensing risk. Users feel ownership.
- **Cold start:** Most enthusiast users already have models downloaded.

#### Solution C: Tiered Model Strategy
**Define explicit model tiers to manage the distribution problem.**

| Tier | Size | Examples | Strategy |
|------|------|----------|----------|
| Instant | <2GB | Qwen2.5-1.5B, Phi-3-mini | Pre-bundled with agent |
| Quick | 2-8GB | Llama-3.2-3B, Mistral-7B | Download on demand, <10min |
| Standard | 8-30GB | Llama-3.1-70B-Q4 | Background download, notify when ready |
| Premium | 30GB+ | Llama-3.1-405B | Only via P2P/sharded inference |

### Verdict
**LLMHosts should never distribute model weights.** Be an orchestration layer that discovers, indexes, and routes to models users already have or choose to download via Ollama/HuggingFace. This eliminates both the legal and bandwidth problems entirely.

---

## Hole 7: exo P2P Inference Over WAN/Internet is Unproven

### The Problem
exo is designed for local LAN clusters. The architecture plan calls for using it over the internet between users' devices in different cities. With 50-200ms inter-node latency, sharded inference across WAN may be slower than just calling an API.

### Research Findings

**This is the most critical finding: exo over WAN does not work and is not designed to.**

1. **exo is LAN-only by design:**
   - Device discovery uses UDP multicast and link-local addressing -- **does not work across the internet**.
   - Users on VPN/WAN connections report devices fail to discover each other (GitHub issue #363).
   - The recent Thunderbolt 5 RDMA breakthrough (5-9μs latency) is LAN/physical proximity only.
   - exo's own benchmarks show single-request performance **degrades** with more nodes (49.3→44.4→39.7 tok/s for 1→2→3 nodes on LLaMA 3B). The benefit is only in multi-request throughput.

2. **Petals (designed for WAN) shows it's possible but limited:**
   - Achieved 4-6 tokens/sec for 70B-180B models over the internet across 2 continents.
   - 10x faster than naive offloading, but still **much slower than local inference**.
   - Required fault-tolerant algorithms for node disconnections and sophisticated load balancing.
   - Project appears **largely inactive** since late 2023 (last release v2.2.0, Sep 2023). The research team has not released new versions in 2+ years.
   - Key lesson: WAN distributed inference works for models **too large for any single node**, but is not competitive when the model fits on one device.

3. **PlanetServe/GenTorrent (Apr 2025):**
   - First systematic study of decentralized LLM serving.
   - Achieved 50%+ latency reduction via overlay forwarding (routing through intermediate nodes).
   - Addressed privacy (encrypted activations) and quality verification.
   - **Still in research prototype stage**, not production-ready.

4. **The latency math is damning for pipeline parallelism over WAN:**
   - Activations between pipeline stages are small (<4KB for most models).
   - But each pipeline stage requires a round-trip. For a 70B model split across 4 nodes:
     - LAN (0.1ms RTT): 4 × 0.1ms = 0.4ms overhead per token
     - WAN (50ms RTT): 4 × 50ms = 200ms overhead **per token**
     - At 30 tok/s target: WAN overhead alone = 6 seconds per second of generation. **Impossible.**
   - Pipeline parallelism over WAN only works for prefill (parallelizable) not decode (sequential).

5. **Whole-model-per-node is the WAN-viable architecture:**
   - vLLM (Feb 2025 blog) distinguishes tensor parallelism (within-node, needs NVLink) from pipeline parallelism (multi-node).
   - For WAN: **don't shard models across nodes. Run whole models on each node. Route requests to the right node.**
   - This eliminates inter-node latency from the critical path entirely.

### Proposed Solutions

#### Solution A: Whole-Model Routing, Not Sharded Inference (RECOMMENDED)
**Fundamentally change the architecture: each node runs complete models. LLMHosts routes requests to the best node, not shards across nodes.**

- **How it works:**
  1. User A has RTX 4090 → runs Llama-3.1-8B locally.
  2. User B has 2x RTX 3090 → runs Llama-3.1-70B-Q4 locally.
  3. User C has M4 Max → runs Mistral-7B locally.
  4. Request comes in → kNN router picks the best available node → request goes to that single node → response comes back.
  5. Inter-node latency only matters for the request/response round-trip (one-time), NOT per-token.
- **Latency:** 50ms WAN overhead per request (acceptable) vs. 50ms per token (unacceptable).
- **When sharding IS needed:** Only for models too large for any single node (405B). In this case, restrict sharding to nodes with **low latency to each other** (same city, same network). Don't shard across continents.

#### Solution B: Locality-Aware Clustering
**Group nearby nodes into clusters. Shard within clusters, route between clusters.**

- **How it works:**
  1. Measure inter-node latency during agent setup.
  2. Nodes with <5ms latency form a "cluster" (likely same LAN/city).
  3. Within a cluster: sharded inference is viable (use exo as originally planned).
  4. Between clusters: route whole requests to the best cluster.
- **Advantage:** Gets the best of both worlds -- local sharding for big models, WAN routing for everything else.

#### Solution C: Abandon P2P, Go Single-Node + API Fallback
**Simplest architecture: each user's hardware runs their own models. If local hardware can't handle a request, fall back to a cloud API.**

- **How it works:**
  1. User's agent runs Ollama locally with whatever models fit their hardware.
  2. LLMHosts router determines: can this request be served locally? → serve locally (free).
  3. If not (model too large, user's GPU busy): route to cloud API (OpenRouter, Together AI, etc.).
  4. LLMHosts charges a small markup on API fallback.
- **Advantage:** Eliminates ALL P2P complexity. Works today. Revenue model is clear.
- **Disadvantage:** Loses the "pool distributed hardware" value proposition.

### Verdict
**exo over WAN is a dead end. Do not pursue sharded inference across the internet.** The architecture must pivot to whole-model-per-node routing. exo can still be used within a user's local network (multiple GPUs in one house/office), but the WAN strategy must be request routing, not model sharding. This is the single most important architectural change this research reveals.

---

## Hole 8: OpenZiti Operational Complexity

### The Problem
OpenZiti requires running Controller + Edge Routers + embedding SDK in every agent. If the Controller goes down, the whole network fails. This is enterprise-grade infrastructure that a startup shouldn't be operating at scale in its early stages.

### Research Findings

1. **OpenZiti Strengths (real, but overkill for MVP):**
   - True zero-trust: no inbound ports, hidden network.
   - Superior throughput on long-haul routes vs. Tailscale (NetFoundry benchmark).
   - Service-level microsegmentation.
   - But: designed for enterprises with dedicated network engineering teams.

2. **Tailscale/Headscale is the pragmatic alternative:**
   - **Headscale** = fully open-source, self-hosted Tailscale coordination server.
   - Docker deployment, 1M+ Docker Hub pulls, actively maintained (2025).
   - NAT traversal is automatic (DERP relay servers + direct WireGuard connections).
   - Zero port forwarding required. Works through CG-NAT.
   - MagicDNS for automatic name resolution.
   - Standard Tailscale clients on every platform (Mac/Win/Linux/iOS/Android).
   - Setup: single Docker container + domain name. That's it.

3. **Nebula (Slack-created):**
   - Fully open-source, lightweight.
   - Certificate-based identity (no SSO dependency).
   - Designed for machine-to-machine communication.
   - Less operational overhead than OpenZiti but more manual than Tailscale.
   - No built-in client provisioning.

4. **Comparison Matrix:**

   | Feature | OpenZiti | Tailscale/Headscale | Nebula |
   |---------|----------|-------------------|--------|
   | NAT traversal | Yes (SDK-embedded) | Yes (WireGuard + DERP) | Yes (lighthouse + relay) |
   | Self-hosted | Yes (complex) | Yes (Headscale, easy) | Yes (moderate) |
   | Docker deployment | Possible but complex | Single container | Possible |
   | Client provisioning | Built-in | Built-in (Tailscale client) | Manual |
   | Zero-trust level | Highest (app-embedded) | Network-level ACLs | Certificate-based |
   | Ops complexity | HIGH | LOW | MEDIUM |
   | Controller SPOF | Yes (Controller) | Yes (Headscale) | Yes (Lighthouse) |
   | If controller goes down | Network degrades | Existing connections persist | Existing connections persist |

### Proposed Solutions

#### Solution A: Headscale + Tailscale Clients (RECOMMENDED)
**Replace OpenZiti with Headscale (self-hosted) + standard Tailscale clients in agents.**

- **How it works:**
  1. LLMHosts runs one Headscale instance (Docker container, <$10/month VPS).
  2. Each user's agent includes the Tailscale client (open-source, all platforms).
  3. On agent install: auto-register with Headscale → instant mesh connectivity.
  4. All agent-to-platform and agent-to-agent communication goes over WireGuard-encrypted mesh.
  5. If Headscale goes down: **existing connections persist** (WireGuard sessions are stateless). Only new node joins fail.
- **Migration path:** Can always move to OpenZiti later if enterprise customers demand it. The overlay network is an implementation detail, not a user-facing feature.
- **Ops burden:** 1 Docker container vs. OpenZiti's Controller + Edge Routers + SDK integration.

#### Solution B: Cloudflare Tunnel + WireGuard Hybrid
**Use Cloudflare Tunnels for agent-to-platform, WireGuard for agent-to-agent.**

- **How it works:**
  1. Agent-to-platform (API calls, metrics): Cloudflare Tunnel (zero-config, free tier).
  2. Agent-to-agent (inference requests): WireGuard direct connections via STUN/TURN.
  3. No self-hosted controller needed for the platform connection path.
- **Advantage:** Cloudflare's global edge network handles the platform-side. Zero self-hosted SPOF.
- **Disadvantage:** Two networking stacks. More complexity in the agent.

#### Solution C: No Overlay at All (For Whole-Model Architecture)
**If adopting Solution A from Hole 7 (whole-model routing), the overlay network requirements shrink dramatically.**

- **How it works:**
  1. Each agent exposes an HTTPS endpoint via Cloudflare Tunnel or ngrok-style tunneling.
  2. LLMHosts platform routes requests to agents via HTTPS.
  3. Agents authenticate with mTLS certificates issued by LLMHosts.
  4. No peer-to-peer connectivity needed (all traffic goes through platform).
- **Advantage:** Eliminates overlay network entirely. Standard HTTPS infrastructure.
- **Disadvantage:** All traffic flows through LLMHosts (bandwidth cost, single point of failure for data path).
- **When it works:** If you're doing whole-model routing (not sharded inference), there's no need for agent-to-agent direct communication.

### Verdict
**Replace OpenZiti with Headscale for MVP.** The operational complexity of OpenZiti is not justified at this stage. Headscale provides 90% of the value at 10% of the operational cost. If the architecture pivots to whole-model routing (Hole 7 recommendation), evaluate whether you even need an overlay -- simple HTTPS + mTLS may be sufficient.

---

## Hole 9: Self-Evolution Engine is a Moonshot Bolted onto an MVP

### The Problem
The planned "self-evolution engine" includes: autonomous model discovery, auto-benchmarking, code generation, testing, deployment. Each sub-problem is hard. Together they're a multi-year research project, not a product feature.

### Research Findings

1. **Progressive Delivery for ML is now mainstream:**
   - AWS SageMaker: canary/linear traffic shifting with auto-rollback.
   - Google GKE Inference Gateway: incremental rollouts via HTTPRoute config.
   - Azure ML: blue-green progressive deployments.
   - LaunchDarkly AI Configs (2025): runtime model/prompt changes without code deploys.
   - **Key insight:** The industry approach is human-initiated, tool-assisted model updates -- not autonomous AI-driven evolution.

2. **What actually works in production:**
   - **Model registry + A/B testing:** New models are added to registry. Traffic is split. Metrics compared. Winner promoted. This is GitOps, not AI.
   - **Eval-driven promotion:** Run standardized benchmarks (MMLU, HumanEval, etc.) against new models. Promote if they beat the incumbent by a threshold.
   - **Feature flags for models:** LaunchDarkly-style configs that let you swap models per-user, per-region, per-use-case without code changes.

3. **What doesn't work:**
   - Autonomous code changes in production.
   - AI selecting its own benchmarks.
   - Unsupervised model deployment.
   - Any system that can brick itself without human approval.

### Proposed Solutions

#### Solution A: Human-in-the-Loop Model Lifecycle (RECOMMENDED)
**Replace "self-evolution" with a structured model lifecycle management system.**

- **Phase 1 (MVP): Manual Curation**
  - LLMHosts team curates a "recommended models" list (10-20 models).
  - Users can add any Ollama-compatible model.
  - kNN router learns which model works best for which query types from real usage data.
  - No auto-discovery, no auto-benchmarking, no auto-deployment.

- **Phase 2 (6 months): Assisted Discovery**
  - Monitor HuggingFace/Ollama for new model releases.
  - Auto-run standard benchmarks (MMLU, HumanEval, MT-Bench) in sandboxed environment.
  - **Present results to LLMHosts team** via dashboard. Team decides whether to add to recommended list.
  - A/B test new models against incumbents with 5% traffic split.

- **Phase 3 (12+ months): Semi-Autonomous Updates**
  - If a model passes all benchmarks AND A/B test with >95% confidence → auto-promote to recommended list.
  - Human can override/rollback at any time.
  - Progressive rollout: 5% → 25% → 50% → 100% over 48 hours.
  - Auto-rollback if error rate exceeds baseline by >2%.

#### Solution B: Community-Driven Evolution
**Let users drive model adoption instead of an AI system.**

- **How it works:**
  1. Users rate model quality (thumbs up/down on responses).
  2. Platform aggregates ratings per model per use case.
  3. Popular models rise in recommendation rankings.
  4. "Model leaderboard" becomes a community feature (engagement driver).
- **Advantage:** Zero engineering cost for discovery/benchmarking. Users do the evaluation.
- **Analogy:** App Store model. Apple doesn't auto-install apps. Users choose. Ratings guide choices.

#### Solution C: Config-Driven Model Management
**Treat model selection as configuration, not code.**

- **How it works:**
  1. Model routing config stored in a YAML/JSON file (not in code).
  2. Config specifies: model name → capabilities → quality score → cost score.
  3. kNN router reads config + live performance data to make routing decisions.
  4. Updating the model pool = editing a config file + deploying it (GitOps).
  5. Feature flags per user/org for model access.
- **Advantage:** Entire "evolution" problem reduces to maintaining a config file. Can be automated later, but starts manual.

### Verdict
**Drop "self-evolution" from the MVP entirely.** It's not a feature users are asking for -- users want their hardware to run models well. Model lifecycle management is a background operational concern, not a product differentiator. Implement a simple model registry + kNN router that learns from usage. Add assisted discovery in Phase 2. Consider semi-autonomous updates only after the platform has thousands of users generating meaningful performance data.

---

## Architecture-Level Recommendation: The Hybrid Pivot

The research across all 5 holes points to a **fundamental architectural simplification:**

### Current Architecture (Complex, Unproven)
```
User Request → 8B Conductor Brain (GPU-heavy) → exo P2P Shard (WAN, unproven)
                                                     ↕
                                              OpenZiti (complex overlay)
                                                     ↕
                                         Self-Evolution Engine (moonshot)
```

### Recommended Architecture (Simple, Proven)
```
User Request → kNN Embedding Router (CPU, <10ms) → Whole-Model Node (local inference)
                                                         ↕
                                                  Headscale/WireGuard (simple overlay)
                                                         ↕
                                              Ollama (model management, user-controlled)
```

### What Changes

| Component | Before | After |
|-----------|--------|-------|
| **Routing** | 8B LLM inference per request | kNN embedding lookup (<10ms, CPU) |
| **Inference** | Sharded across WAN via exo | Whole-model on single node; exo for LAN clusters only |
| **Networking** | OpenZiti (Controller + Edge Routers + SDK) | Headscale + Tailscale client (1 Docker container) |
| **Model Distribution** | LLMHosts distributes weights | Users bring own models via Ollama |
| **Evolution** | Autonomous AI-driven | Manual curation → assisted discovery → semi-auto |

### What Stays the Same

- **Value proposition:** Users leverage their own hardware for LLM inference.
- **Multi-node benefit:** Route to the best available hardware in the user's fleet.
- **Cost savings:** Local inference is cheaper than API calls.
- **LiteLLM proxy:** Still useful as the unified API interface.
- **exo:** Still valuable for LAN clusters (multiple GPUs in same location).

### Critical Insight

The original architecture tried to solve the hardest version of every problem simultaneously:
- Hardest routing → use an LLM (simple classifier works)
- Hardest inference → shard across WAN (whole-model routing works)
- Hardest networking → zero-trust overlay (WireGuard mesh works)
- Hardest model management → autonomous AI (manual curation works)

**The simpler version is not a compromise -- it's what the research says actually works.**
Every complex component can be added later as an upgrade path once the simple version is proven and generating revenue.

---

## Source Index

### Papers
| ID | Title | Date | Relevance |
|----|-------|------|-----------|
| arXiv:2312.08361 | Distributed Inference and Fine-tuning of LLMs Over The Internet (Petals) | Dec 2023 | WAN inference benchmarks |
| arXiv:2504.20101 | PlanetServe: Decentralized LLM Serving Overlay | Apr 2025 | Decentralized WAN serving |
| arXiv:2601.07206 | LLMRouterBench | Jan 2026 | Routing benchmark (400K instances) |
| arXiv:2509.02718 | Training-Free Online Routing | Sep 2025 | Microsecond routing |
| arXiv:2510.08731 | Semantic Router for vLLM | Oct 2025 | Embedding-based routing |
| arXiv:2502.08773 | Universal Model Routing | Feb 2025 | Dynamic model routing |
| arXiv:2511.09557 | LLM Inference Beyond a Single Node | Nov 2025 | Multi-node latency analysis |
| arXiv:2505.11329 | TokenWeave | May 2025 | Compute-communication overlap |
| OpenReview:Chn50flK4X | kNN Beats Complex Routers | Oct 2025 (ICLR 2026 sub) | Core routing finding |

### Tools/Projects
| Project | URL | Status | Relevance |
|---------|-----|--------|-----------|
| exo | github.com/exo-explore/exo | Active, LAN-only | Inference framework |
| Petals | github.com/bigscience-workshop/petals | Inactive since late 2023 | WAN inference lessons |
| Headscale | github.com/juanfont/headscale | Active, well-maintained | OpenZiti replacement |
| vLLM Semantic Router | github.com/vllm-project/semantic-router | Active | Routing reference |
| Ollama | ollama.com | Active, 50M+ pulls | Model management |

### Benchmarks/Data Points
| Metric | Value | Source |
|--------|-------|--------|
| kNN routing latency | <10ms (CPU) | ICLR 2026 submission |
| 8B model routing latency | 2-5 seconds (GPU) | Estimated from inference benchmarks |
| exo LAN single-request degradation | 49.3 → 39.7 tok/s (1→3 nodes) | exo transparent benchmarks |
| Petals WAN throughput | 4-6 tok/s (70B-180B models) | arXiv:2312.08361 |
| WAN RTT overhead per pipeline stage | 50-200ms | Network fundamentals |
| Pipeline stages for 70B model | ~4 | vLLM distributed inference blog |
| Headscale Docker Hub pulls | 1M+ | Docker Hub |
| vLLM Semantic Router latency reduction | 47.1% | arXiv:2510.08731 |
