# ADR-001: Orchestrator Model Selection

## STATUS
REVISED (2026-02-14) -- course correction restoring intelligent routing

## CHANGE LOG
- 2026-02-13: Original proposal -- Nemotron-Orchestrator-8B with rule-based Phase 1 fallback
- 2026-02-14 (pivot): SUPERSEDED -- pivoted to pure kNN embedding router. Threw out the baby with the bathwater.
- 2026-02-14 (course correction): **REVISED** -- three-tier hybrid router that RESTORES intelligent fit-for-purpose routing WITHOUT the scaling/VRAM problems. Combines kNN speed with classifier intelligence and small-model reasoning for edge cases. All tiers fit in <1GB VRAM alongside actual inference models.

## CONTEXT
LLMHosts needs a "Conductor" -- a routing layer that analyzes incoming requests and decides which backend model/node to route to. The router must understand WHAT a request needs and match it to the right fit-for-purpose model. A kNN lookup alone can't reason about "this is a complex multi-file refactoring task that needs a 70B model" vs "this is a quick translation a 7B can handle."

### Why Pure kNN Was Insufficient
The pivot to pure kNN (Debate-001) solved the scaling problem but killed the intelligence:
1. **No task understanding**: kNN classifies by surface similarity, not by what the task NEEDS
2. **Can't assess complexity**: "Write hello world" and "Refactor this 500-line module" both look like "code requests" to embeddings
3. **Cold-start bias**: kNN trained on rule-based routing data just replicates rule-based decisions
4. **Domain blindness**: all-MiniLM-L6-v2 embeddings perform poorly on code and domain-specific queries

### The Insight: The User's GPU IS the Compute
The original 8B model was killed because "it eats GPU that should do actual inference." But research shows: a three-tier hybrid using a 150MB classifier + optional 400MB tiny model fits alongside a 13B inference model on a 24GB GPU with 9-11GB to spare. The router runs ON the user's hardware -- which is the entire point of LLMHosts.

## DECISION
**Three-Tier Hybrid Router** that escalates through increasingly intelligent (and expensive) tiers based on confidence.

### Architecture
```
Request arrives
  │
  ▼
┌─────────────────────────────────────────┐
│ TIER 1: kNN Embedding Lookup            │
│ Handles: ~75% of requests               │
│ Latency: <15ms                          │
│ Resources: CPU only, ~96MB RAM (FAISS)  │
│ How: Embed query → FAISS nearest-K →    │
│      score on [model_fit, latency,      │
│      cost, privacy, cache_prob]         │
│ If confidence > 0.85 → ROUTE            │
│ If confidence ≤ 0.85 → escalate         │
└────────────┬────────────────────────────┘
             │ low confidence
             ▼
┌─────────────────────────────────────────┐
│ TIER 2: ModernBERT Classifier           │
│ Handles: ~20% of requests               │
│ Latency: <50ms (GPU) / <100ms (CPU)    │
│ Resources: ~150MB VRAM or CPU           │
│ How: 6 signal analysis (keyword,        │
│      embedding, domain, complexity,     │
│      PII, user preference)              │
│ Determines: task type, complexity,      │
│      required capability level          │
│ If classification confident → ROUTE     │
│ If ambiguous → escalate                 │
└────────────┬────────────────────────────┘
             │ ambiguous
             ▼
┌─────────────────────────────────────────┐
│ TIER 3: Qwen2.5-0.5B Q4 Reasoning      │
│ Handles: ~5% of requests                │
│ Latency: <500ms                         │
│ Resources: ~400MB VRAM (via Ollama)     │
│ How: Small LLM reasons about request    │
│      intent, complexity, and which      │
│      available model is best fit        │
│ Always produces a routing decision      │
│ Falls back to CPU if no VRAM available  │
└─────────────────────────────────────────┘
```

### VRAM Budget (Runs ON User Hardware)

**24GB GPU (e.g., RTX 4090, 3090)**:
| Component | VRAM |
|-----------|------|
| 13B Q4 main inference model + KV cache | ~12-14 GB |
| Tier 1: kNN (FAISS index) | ~96 MB |
| Tier 2: ModernBERT classifier | ~150 MB |
| Tier 3: Qwen2.5-0.5B Q4 | ~400 MB |
| **Total router overhead** | **~650 MB** |
| **Free VRAM remaining** | **9-11 GB** |

**12GB GPU (e.g., RTX 4070)**:
| Component | VRAM |
|-----------|------|
| 7B Q4 main model + KV cache | ~6-8 GB |
| Tiers 1+2 | ~250 MB |
| Tier 3 | ~400 MB |
| **Free VRAM remaining** | **3-5 GB** |

**8GB GPU (e.g., RTX 4060)**:
| Component | VRAM |
|-----------|------|
| 7B Q4 main model + KV cache | ~6-7 GB |
| Tiers 1+2 | ~250 MB |
| Tier 3: falls back to CPU | 0 |
| **Free VRAM remaining** | **~1 GB** |

### Phased Deployment
1. **Phase 1 (Weeks 1-6)**: Rule-based router (Tier 0) + kNN embedding collection. ModernBERT classifier prototype. Deliberate 5% exploration routing for counterfactual training data.
2. **Phase 2 (Weeks 7-14)**: Full three-tier deployment. kNN trained on Phase 1 data. ModernBERT trained on IBM production data + LLMHosts-specific fine-tuning. Qwen-0.5B loaded via Ollama (`OLLAMA_MAX_LOADED_MODELS=2`).
3. **Phase 3+ (Weeks 15+)**: Global router intelligence from aggregate anonymized data. Domain-specific embedding fine-tuning.

### Key Models
- **Embedding**: all-MiniLM-L6-v2 (22M params, 384-dim, Apache 2.0) -- Phase 1. ONNX Runtime for lightweight install.
- **Classifier**: ModernBERT (149M params) -- IBM production-proven, 6-signal analysis. +10.2% accuracy, -47.1% latency vs alternatives.
- **Reasoning**: Qwen2.5-0.5B-Instruct Q4_K_M -- smallest competent reasoning model. Runs alongside main model in Ollama.

## OPTIONS EVALUATED (REVISED)

### Option A: Nemotron-Orchestrator-8B -- AVAILABLE AS OPT-IN
- Available as Q4_K_M GGUF (~5GB) for power users with 24GB+ GPUs
- `llmhosts config set router nemotron` for maximum routing intelligence
- Not default because most users don't have the VRAM headroom

### Option B: Pure kNN -- INSUFFICIENT (previous pivot decision)
- Fast but dumb. Can't assess task complexity or reason about fit-for-purpose.
- Kept as Tier 1 of the hybrid for the 75% of requests it handles well.

### Option C: Three-Tier Hybrid -- SELECTED
- Combines speed (kNN), intelligence (ModernBERT), and reasoning (Qwen-0.5B)
- <1GB total VRAM overhead
- 95% of requests resolved in <50ms, 100% in <500ms

### Option D: Rule-Based Only -- BOOTSTRAP
- Day-1 fallback before ML tiers have data
- Stays as Tier 0 for deterministic cases (user preferences, explicit model selection)

## RATIONALE
- The router runs ON the user's hardware -- their GPU has room for <1GB of routing intelligence
- Three tiers mean 75% of requests get kNN speed, only 5% pay the 500ms cost of reasoning
- ModernBERT's 6-signal analysis provides the fit-for-purpose intelligence that pure kNN lacks
- Deliberate exploration (5% of Phase 1 requests) generates counterfactual training data, avoiding cold-start bias
- The HONEST routing story: the router knows WHEN local is good enough and WHEN to route to cloud

## CONSEQUENCES
- Three-tier adds implementation complexity vs pure kNN
- ModernBERT needs training data (mitigated: IBM production data as baseline)
- Qwen-0.5B requires Ollama multi-model loading (mitigated: `OLLAMA_MAX_LOADED_MODELS=2` is standard)
- `pip install llmhosts[smart]` needed for Tier 2+ (core install uses rule-based + kNN only)
- Router intelligence improves over time as data accumulates -- early users get rule-based, later users get full hybrid
