# Router Architecture Research: Finding the Right Middle Ground
**Date**: 2026-02-14
**Context**: LLMHosts pivoted from 8B Nemotron-Orchestrator to pure kNN routing (ADR-001). This research evaluates whether a hybrid approach can combine kNN speed with deeper task analysis -- running ON the user's GPU without impacting inference.

---

## Executive Summary

**The answer is yes, and it's not even close on VRAM.** A three-tier hybrid router (kNN + classifier + optional small LLM) fits in **0.5-2.5 GB of VRAM** and runs in **<50ms for 95%+ of requests**. On a 24GB GPU with a 13B Q4 model loaded (~8-10GB), this leaves 12-14GB free -- more than enough.

The research strongly validates the pivot away from Nemotron-8B, but reveals that pure kNN has a known ceiling. The state of the art (Feb 2026) is **tiered routing** -- fast path for obvious cases, classifier for medium cases, and optional deep analysis for edge cases.

---

## Research Findings by Search

### 1. Small Language Models for Routing Classification

**Key Finding**: Sub-1B models are now viable for classification tasks.

| Model | Params | VRAM (Q4) | VRAM (FP16) | Speed (tok/s) | Classification Quality |
|-------|--------|-----------|-------------|---------------|----------------------|
| **Qwen2.5-0.5B-Instruct** | 500M | ~400MB | ~1GB | 260-311 (vLLM, A100) | MMLU 29.7% (Q4) / 34.4% (BF16) |
| **Qwen2.5-1.5B-Instruct** | 1.5B | ~1GB | ~3GB | 183-217 (vLLM, A100) | MMLU 45.0% (Q4) / 48.4% (BF16) |
| **Phi-3-mini** | 3.8B | ~2.5GB | ~7.6GB | ~97% MMLU retention at Q4 | Strong general reasoning |
| **ModernBERT-base** | 149M | N/A (CPU) | ~87MB (FP32) | Sub-10ms classification | SOTA encoder for classification tasks |
| **all-MiniLM-L6-v2** | 22M | N/A (CPU) | ~87MB (FP32) | ~5ms embedding | Embedding only (for kNN) |

**Verdict**: For *routing classification* (not general reasoning), a 0.5B-1.5B model is massive overkill. ModernBERT (149M params, ~87MB) is purpose-built for classification and runs in single-digit milliseconds. For the rare case where you need actual language understanding of a prompt, Qwen2.5-0.5B at 400MB Q4 is sufficient.

### 2. Hybrid kNN + Classifier Approaches

**Key Finding**: ICLR 2026 confirms kNN is surprisingly strong, but the best systems layer classifiers on top.

**Paper: "Rethinking Predictive LLM Routing" (ICLR 2026)**
- kNN outperforms complex learned routers due to "strong locality properties of model performance in embedding space"
- Validated on instruction-following, QA, reasoning, AND multi-modal routing
- This is the strongest validation of LLMHosts's kNN pivot

**CSCR -- Cost-Spectrum Contrastive Routing (NeurIPS 2025 Spotlight)**
- Maps prompts AND models into shared embedding space
- Routes via single FAISS kNN lookup with **microsecond latency**
- 25% improvement in accuracy-cost tradeoff over baselines
- Generalizes to unseen LLMs without retraining
- Uses "logit footprints" for open-source models, "perplexity fingerprints" for APIs
- **This is the closest to what LLMHosts needs**: cost-aware routing that learns

**MixLLM (NAACL 2025)**
- Contextual bandit + lightweight predictor
- 97.25% of GPT-4 quality at 24.18% of cost
- Continual learning from feedback
- Enhances embeddings with "query tags" (task type metadata)

**LLMRouterBench (January 2026)**
- 400K+ instances, 21 datasets, 33 models, 10 routing baselines
- **Critical finding**: Many sophisticated routers fail to outperform simple baselines under unified evaluation. Even commercial routers struggle.
- Validates LLMHosts's "don't over-engineer" instinct

### 3. Nemotron-Orchestrator-8B Quantized

**Key Finding**: Technically feasible, but confirmed overkill.

| Variant | File Size | VRAM (est.) | Available |
|---------|-----------|-------------|-----------|
| Full precision | 32.8 GB | ~33 GB | Yes (HF) |
| **Q4_K_M GGUF** | **~5.03 GB** | **~5.5 GB** | Yes -- bartowski, jmelahman, MaziyarPanahi on HF |
| Q8_0 GGUF | ~8.5 GB | ~9 GB | Yes |

- Works with Ollama, llama.cpp, LM Studio
- Q4_K_M is "almost no perceptible quality loss" (it's Ollama's default quant)
- On a 24GB GPU with 13B model (~10GB): **5.5GB for Nemotron-8B fits**, leaving ~8.5GB for KV cache
- BUT: 8B model routing takes 1-3 seconds per decision vs <50ms for tiered approach
- **Verdict**: Could be offered as an opt-in "deep routing" mode for power users, but should NOT be the default

### 4. RouteLLM / Routing Benchmarks State of the Art

**Key Finding**: The field is maturing rapidly; simple methods are holding up.

**RouterEval (2025)**: 200M+ performance records across 8,500+ LLMs. Discovered "model-level scaling up" -- good routers improve as candidate pool grows (network effect, relevant for Hive).

**LLMRouterBench (Jan 2026)**: Most rigorous benchmark yet. Key takeaway: **many recent routing approaches, including commercial routers, fail to reliably outperform simple baselines.** This is a strong signal to start simple and only complexify when data proves it's needed.

**ICLR 2026 submission**: Critiques existing benchmarks for limited task diversity. Proposes 33,337 queries across 68 categories and 85 complementary models. Important for future LLMHosts evaluation.

### 5. Running Small Model Alongside Main Model on Same GPU

**Key Finding**: Ollama natively supports this. vLLM has speculative decoding.

**Ollama Multi-Model Support**:
- `OLLAMA_MAX_LOADED_MODELS` controls simultaneous model count
- Different models CAN be loaded simultaneously, sharing GPU VRAM
- No special configuration needed -- Ollama scheduler handles it
- GPU grouping optimization merged in 2025 (PR #10678) for better VRAM management

**vLLM Speculative Decoding**:
- Small draft model + large target model on same GPU
- Up to 2.8x inference speedup
- The **Speculators** library (2025) standardizes draft model training/deployment
- Relevant if LLMHosts adds vLLM opt-in support

**vLLM Semantic Router (IBM, NeurIPS 2025)**:
- ModernBERT classifier decides routing paths
- Rust implementation (Candle) for zero-copy, high-concurrency
- 6 signal types: keyword, embedding, domain classification, fact-check requirements, user feedback, preference matching
- Plugin chain: semantic caching, jailbreak detection, PII protection
- Results: +10.2% accuracy, -47.1% latency, -48.5% token consumption
- **This is the production reference architecture for what LLMHosts should build**

---

## The Key Question Answered

> Can we run an intelligent routing model ON THE USER'S OWN GPU without significantly impacting inference? If they have a 24GB GPU with a 13B model loaded (using 10GB), can a quantized 1-3B router model fit in the remaining VRAM and run in <500ms?

### VRAM Budget on 24GB GPU with 13B Q4 Model

| Component | VRAM | Latency | Purpose |
|-----------|------|---------|---------|
| 13B Q4 main model | ~8-10 GB | N/A | Actual inference |
| KV cache (main model) | ~2-4 GB | N/A | Context window |
| all-MiniLM-L6-v2 (embeddings) | ~90 MB | ~5ms | kNN embedding generation |
| FAISS index (10K entries) | ~6 MB | ~3ms | kNN lookup |
| ModernBERT-base (classifier) | ~150 MB | ~5-10ms | Task classification |
| Qwen2.5-0.5B Q4 (deep analysis) | ~400 MB | ~200-500ms | Edge case analysis |
| **Routing total** | **~650 MB** | **<15ms (95%), <500ms (5%)** | |
| **GPU total** | **~11-15 GB** | | **9-13 GB free** |

**Answer: YES.** Even the most aggressive configuration (all tiers loaded) uses <1GB for routing. A 1.5B model (Qwen2.5-1.5B at ~1GB Q4) also fits easily. You could even load the 5GB Nemotron-8B Q4 alongside the 13B main model on a 24GB card, though there's no need.

### What About 8GB GPUs?

| Component | VRAM | Notes |
|-----------|------|-------|
| 7B Q4 main model | ~4-5 GB | Typical small model |
| KV cache | ~1.5 GB | |
| Tier 1+2 routing (kNN + ModernBERT) | ~250 MB | |
| **Total** | **~6-7 GB** | **Fits on 8GB card** |

Tier 3 (0.5B LLM) would need to be CPU-offloaded on 8GB cards, adding ~1-2s latency for edge cases only.

---

## Recommended Architecture: Three-Tier Hybrid Router

### Tier 1: Fast Path -- kNN (handles ~75% of requests)
- **How**: Embed request with all-MiniLM-L6-v2 → FAISS kNN lookup → score candidates
- **VRAM**: ~96 MB (embedding model + FAISS index)
- **Latency**: <15ms on CPU (no GPU needed at all)
- **When**: kNN confidence > threshold (e.g., >0.85 cosine similarity to known routing decisions)
- **Quality**: Matches or beats learned routers per ICLR 2026
- **Already in ADR-001**: Yes, this is the current plan

### Tier 2: Signal Fusion -- ModernBERT Classifier (handles ~20% of requests)
- **How**: When kNN confidence is low, run ModernBERT classifier to determine:
  - Task type (code, creative, reasoning, factual, multi-step)
  - Complexity level (simple lookup vs. deep reasoning)
  - Domain (science, law, medical, general)
  - Privacy sensitivity (PII detection)
  - Tool-use requirements
- **VRAM**: ~150 MB (can also run on CPU in ~20-50ms)
- **Latency**: 5-10ms GPU, 20-50ms CPU
- **When**: kNN confidence < 0.85 but > 0.50
- **Quality**: IBM's vLLM Semantic Router proves this works: +10.2% accuracy, -47.1% latency
- **Architecture reference**: vLLM semantic-router (Rust + Candle + ModernBERT + signal fusion)
- **Status in ADR-001**: Listed as "Option D: Upgrade Path" -- this research says do it in Phase 2

### Tier 3: Deep Analysis -- Small LLM (handles ~5% of requests)
- **How**: For genuinely ambiguous requests, run a quantized 0.5B-1.5B model to understand:
  - "Is this a coding task that needs tool use?"
  - "Does this need multi-step reasoning or a single-shot answer?"
  - "Should this use a specialized model (math, code, medical)?"
- **Model options**:
  - **Qwen2.5-0.5B Q4**: 400MB VRAM, 260+ tok/s, good enough for classification prompts
  - **Qwen2.5-1.5B Q4**: 1GB VRAM, 180+ tok/s, significantly better reasoning
  - **Phi-3-mini Q4**: 2.5GB VRAM, strongest reasoning, but may be excessive
- **Latency**: 200-500ms (acceptable because rare, and still faster than Nemotron-8B's 1-3s)
- **When**: kNN confidence < 0.50, or request has novel characteristics not in training data
- **Status**: Not in current ADR-001. This research justifies adding it as Phase 3+ enhancement.

### Optional Tier 4: Power User Mode -- Nemotron-8B Q4 (opt-in)
- **How**: For users who explicitly want the most intelligent routing
- **VRAM**: ~5.5 GB (Q4_K_M GGUF, available on HuggingFace)
- **Latency**: 1-3 seconds
- **When**: User enables `llmhosts config set router.deep=true`
- **Viability**: Fits on 24GB GPU alongside 7B model, NOT alongside 13B+ model
- **Status**: Available as opt-in, not recommended as default

---

## Concrete Implementation Recommendations

### Phase 1 (Weeks 1-6): Rule-Based + Embedding Collection
- **No change from current plan**
- Rule-based router (CB-002)
- Collect embeddings with all-MiniLM-L6-v2 (CB-004)
- Collect task-type labels from response metadata for future classifier training

### Phase 2 (Weeks 7-14): kNN + Signal Fusion
- Deploy kNN router (CB-005) -- already planned
- **NEW**: Add ModernBERT classifier as Tier 2
  - Fine-tune ModernBERT-base on collected Phase 1 routing data
  - Classify: task_type, complexity, domain, privacy_sensitivity
  - Run on CPU (20-50ms) to avoid any GPU impact
  - Trigger only when kNN confidence < threshold
- **NEW**: Implement CSCR-style cost-aware embeddings
  - Map models into same embedding space as prompts
  - Cost-banded contrastive learning from Phase 1 cost data
  - Enables microsecond cost-optimal routing

### Phase 3+ (Weeks 15+): Optional Deep Analysis
- If Phase 2 classifier still misroutes >5% of edge cases:
  - Add Qwen2.5-0.5B Q4 as Tier 3 for ambiguous requests
  - Load via Ollama alongside main model (`OLLAMA_MAX_LOADED_MODELS=2`)
  - Only invoked for <5% of requests
- Evaluate based on user feedback data from Phase 1-2

---

## Key Academic References

| Paper | Venue | Key Takeaway for LLMHosts |
|-------|-------|--------------------------|
| "Rethinking Predictive LLM Routing: When Simple KNN Beats Complex Learned Routers" | ICLR 2026 | kNN is the right default. Don't over-engineer. |
| CSCR (Cost-Spectrum Contrastive Routing) | NeurIPS 2025 Spotlight | Cost-aware embeddings + FAISS = microsecond routing. Adopt this. |
| "When to Reason: Semantic Router for vLLM" (IBM) | NeurIPS 2025 | ModernBERT + signal fusion is production-proven. Reference architecture. |
| MixLLM | NAACL 2025 | Query tags + contextual bandits = continual learning router. |
| LLMRouterBench | arXiv Jan 2026 | Most fancy routers fail to beat simple baselines. Stay humble. |
| RouterEval | EMNLP 2025 | More candidate models = better routing (network effect). Validates Hive. |
| Conformal Risk-Controlled Routing | ICLR 2026 sub | Contrastive learning for fine-grained small model capability prediction. |
| UniRoute | 2025 | Feature-vector model representation enables routing to unseen models. |

---

## Bottom Line

1. **The kNN pivot was correct.** ICLR 2026 validates it. Don't second-guess.
2. **kNN alone has a ceiling.** ~75% of requests are obvious; ~25% benefit from classification; ~5% need deeper analysis.
3. **ModernBERT (149M params, ~150MB, <10ms) fills the gap** between kNN and an 8B model. It's what IBM uses in production for vLLM.
4. **A 0.5B-1.5B model is the maximum you'd ever need** for routing classification. It fits in <1GB VRAM.
5. **Nemotron-8B Q4 is technically feasible** (~5GB) but unnecessary. Keep it as an opt-in power-user feature.
6. **The recommended path**: kNN (Tier 1) + ModernBERT classifier (Tier 2) + optional small LLM (Tier 3). Total routing VRAM: <1GB. Total routing latency: <50ms for 95%+ of requests.
7. **CSCR's cost-aware contrastive embeddings** should be adopted -- they make the kNN index inherently cost-optimizing.
8. **All of this runs on CPU except Tier 3**, meaning zero GPU impact for the vast majority of requests.
