# Epic: Conductor (Three-Tier Hybrid Router)

## DESCRIPTION
Intelligent routing layer that decides which backend handles each request using a three-tier hybrid: kNN for speed (75%), ModernBERT for intelligence (20%), Qwen-0.5B for reasoning on edge cases (5%). Speaks both OpenAI and Anthropic API dialects. All tiers run on user's hardware in <1GB VRAM.

**COURSE CORRECTION (2026-02-14)**: Restored intelligent fit-for-purpose routing that the pure-kNN pivot lost. The router now UNDERSTANDS what a request needs and matches it to the right model -- "this is a complex refactoring task, needs 70B" vs "this is a quick translation, 7B is fine." See ADR-001 (REVISED).

## BACKLOG ITEMS

### CB-001: OpenAI + Anthropic Compatible Proxy
- **Priority**: P0
- **Phase**: 1
- **Status**: BACKLOG
- **Description**: FastAPI proxy on localhost:4000. Speaks TWO API dialects: OpenAI format on `/v1/chat/completions` (90%+ of tools) and Anthropic format on `/v1/messages` (Claude Code and Anthropic-native tools). Both route through the same Conductor internally.
- **Acceptance**: Proxy accepts both OpenAI and Anthropic API formats. Streaming works. Compatible with Cursor, Claude Code, Aider, Continue.dev, Open WebUI, LangChain, any OpenAI-compatible client.
- **Dependencies**: None

### CB-002: Rule-Based Router v1 (Tier 0)
- **Priority**: P0
- **Phase**: 1
- **Status**: BACKLOG
- **Description**: Deterministic routing for clear-cut cases: user preferences ("always use local for code"), explicit model selection, availability check, cost heuristics, latency awareness. Always runs first, handles explicit routing before ML tiers engage.
- **Acceptance**: Respects user preferences. Routes to available backends. Latency-aware: interactive → low-latency, batch → cheapest. Handles 100% of Phase 1 requests before ML tiers are ready.
- **Dependencies**: CB-001, Ollama discovery, BYOK config

### CB-003: Cost Tracking Per Request
- **Priority**: P0
- **Phase**: 1
- **Status**: BACKLOG
- **Description**: Track per-request costs with HONEST comparisons. Local inference = $0. Cloud API = actual token cost. Show the ROUTING DECISION, not just cost: "Routed to local Llama-3.2:8b (quality: sufficient, cost: $0.00)" or "Routed to Claude via BYOK (quality: required, cost: $0.04)".
- **Acceptance**: Every response includes cost + routing decision metadata. Running total tracks real baseline (not strawman). Dashboard shows monthly actual vs estimated-without-LLMHosts.
- **Dependencies**: CB-001

### CB-004: Embedding Pipeline (Tier 1 Prep)
- **Priority**: P1
- **Phase**: 1
- **Status**: BACKLOG
- **Description**: Generate embeddings for all routed requests using ONNX Runtime + all-MiniLM-L6-v2 (lightweight, no PyTorch needed for `llmhosts` core install). Store embeddings + routing outcomes + user feedback for kNN training. Include 5% deliberate exploration routing for counterfactual training data.
- **Acceptance**: Embeddings generated in <5ms per request via ONNX. 5% of requests deliberately routed to alternative models. Quality feedback collected.
- **Dependencies**: CB-001

### CB-005: kNN Router - Tier 1
- **Priority**: P0
- **Phase**: 1 (prototype), 2 (production)
- **Status**: BACKLOG
- **Description**: kNN embedding router over FAISS index. Fast classification for obvious cases (~75% of requests). Embed → K nearest lookup → score on [model_fit, node_latency, cost, privacy_tier, cache_probability]. If confidence >0.85, route directly. If <0.85, escalate to Tier 2.
- **Acceptance**: <15ms routing latency. Zero GPU. Handles 75% of requests without escalation. Confidence threshold tunable.
- **Dependencies**: CB-004, production routing data

### CB-015: ModernBERT Classifier - Tier 2
- **Priority**: P0
- **Phase**: 2
- **Status**: BACKLOG
- **Description**: ModernBERT (149M params, ~150MB) classifier for requests where kNN confidence is low. Analyzes 6 signals: keyword patterns, embedding similarity, domain classification, task complexity assessment, PII detection, user preference matching. Determines fit-for-purpose: "complex multi-file refactoring → needs 70B frontier" vs "code completion → local 7B is fine."
- **Acceptance**: +10% routing accuracy vs kNN alone. <50ms on GPU, <100ms on CPU. ~150MB VRAM. Correctly identifies task complexity and routes fit-for-purpose.
- **Dependencies**: CB-005, training data from Phase 1 exploration

### CB-016: Qwen-0.5B Reasoning Router - Tier 3
- **Priority**: P1
- **Phase**: 2
- **Status**: BACKLOG
- **Description**: Qwen2.5-0.5B-Instruct Q4_K_M (~400MB VRAM) for the ~5% of requests that are genuinely ambiguous. Runs alongside main model via Ollama (`OLLAMA_MAX_LOADED_MODELS=2`). Small LLM reasons about request intent, required capabilities, and available models to make fit-for-purpose decision. Falls back to CPU on 8GB GPUs.
- **Acceptance**: <500ms routing decision. ~400MB VRAM. Handles ambiguous edge cases that kNN+ModernBERT can't resolve. Falls back to CPU gracefully on low-VRAM hardware.
- **Dependencies**: CB-015, Ollama multi-model support

### CB-006: User Preference Engine
- **Priority**: P1
- **Phase**: 2
- **Status**: BACKLOG
- **Description**: Store and apply user preference vectors. "Always use local for code" → Tier 0 handles before ML tiers engage.
- **Acceptance**: User preferences respected as Tier 0 rule-based overrides.
- **Dependencies**: CB-002, user accounts

### CB-007: Multi-Turn Context Management
- **Priority**: P0
- **Phase**: 1
- **Status**: BACKLOG
- **Description**: Maintain conversation context across routing decisions. If model switches mid-conversation, context transfers seamlessly.
- **Acceptance**: Multi-turn conversation spans different backends without context loss.
- **Dependencies**: CB-001

### CB-008: Fallback Chain & Graceful Degradation
- **Priority**: P0
- **Phase**: 1
- **Status**: RESEARCH
- **Description**: Automatic fallback when primary target fails. Priority: local Ollama → other local → LAN cluster → Hive node (Phase 3) → cloud API (BYOK). Detect failure within 3 seconds, reroute invisibly.
- **Acceptance**: Zero user-visible errors from node failures. Health monitored via heartbeat.
- **Dependencies**: CB-002, Ollama discovery, BYOK config

### CB-009: Latency-Aware Routing
- **Priority**: P0
- **Phase**: 1
- **Status**: BACKLOG
- **Description**: Continuously measure round-trip latency per backend. Factor into routing: interactive → lowest latency; batch/agentic → cheapest.
- **Acceptance**: Per-backend latency tracked (p50, p95). Dashboard shows latency per backend.
- **Dependencies**: CB-002, CB-003

### CB-010: Privacy Classification Router
- **Priority**: P0
- **Phase**: 2
- **Status**: BACKLOG
- **Description**: Classify request sensitivity (PII detection via ModernBERT Tier 2 signals). PRIVATE → local only. STANDARD → eligible for Hive/cloud.
- **Acceptance**: PII requests never leave user's hardware. Dashboard shows routing privacy decisions.
- **Dependencies**: CB-015

### CB-011: Routing Explanation
- **Priority**: P1
- **Phase**: 2
- **Status**: BACKLOG
- **Description**: Human-readable explanation: "Routed to local Llama-3.2:8b. Reason: code completion, complexity: low, local model sufficient. Confidence: 0.92 (Tier 1 kNN)."
- **Acceptance**: Every response includes routing explanation in metadata. Expandable in dashboard.
- **Dependencies**: CB-005

### CB-012: Metrics Export
- **Priority**: P1
- **Phase**: 1
- **Status**: BACKLOG
- **Description**: Export routing decisions, tier used, latency, cost, quality metrics for dashboard and router training.
- **Acceptance**: All metrics available for dashboard and kNN/ModernBERT retraining.
- **Dependencies**: CB-003

### CB-013: Global Router Intelligence (Hive Intelligence)
- **Priority**: P1
- **Phase**: 4
- **Status**: BACKLOG
- **Description**: Anonymized aggregate routing data from all opt-in users → global kNN index + ModernBERT fine-tuning. Paid tier feature.
- **Acceptance**: Global router improves quality for new users (cold-start solved). Privacy via aggregation.
- **Dependencies**: CB-005, CB-015, user base at scale

### CB-014: GPU-Less Smart Routing
- **Priority**: P0
- **Phase**: 1
- **Status**: BACKLOG
- **Description**: For users without local models: intelligent routing across BYOK cloud APIs. Three-tier router picks cheapest model that's GOOD ENOUGH for the task.
- **Acceptance**: GPU-less user → router picks optimal cloud API → measurable savings vs naive hardcoded choice.
- **Dependencies**: CB-002, BYOK config
