# ADR-003: API Proxy Layer

## STATUS
PROPOSED (2026-02-13)

## CONTEXT
LLMHosts routes requests across local models (vLLM, exo, Ollama), external APIs (OpenAI, Anthropic via BYOK), and the Conductor Brain. Need a unified proxy that handles routing, fallback, rate limiting, cost tracking, and provider abstraction.

## OPTIONS EVALUATED

### Option A: LiteLLM
- **Pros**: 100+ providers, OpenAI-compatible API, 6 routing strategies (simple-shuffle, latency-based, cost-based, least-busy, usage-based, custom), fallback chains, rate limiting (multi-level), Redis-backed distributed state, traffic mirroring for A/B testing, built-in cost tracking, Docker deployment, active community
- **Cons**: Python-based (potential performance ceiling at extreme scale), some advanced routing strategies not recommended for production
- **Best For**: Multi-provider proxy with intelligent routing
- **Evidence**: Industry standard for LLM proxy/gateway

### Option B: Custom Proxy (FastAPI/Go)
- **Pros**: Full control, optimized for LLMHosts-specific needs
- **Cons**: Massive development effort to replicate LiteLLM features, no provider integrations built-in
- **Best For**: If LiteLLM proves insufficient

### Option C: Kong/Envoy + Custom Plugin
- **Pros**: Production-grade API gateway, high performance
- **Cons**: Not LLM-aware, no built-in cost tracking, provider integration must be built
- **Best For**: General API gateway needs

## DECISION
**LiteLLM** as primary API proxy with **custom ConductorRoutingStrategy** plugin.

### Integration Design
```
User Request → LiteLLM Proxy
  → Auth (Virtual Key validation)
  → ConductorRoutingStrategy.async_get_available_deployment()
    → Query Conductor Brain for routing decision
    → Conductor considers: task type, user preferences, compute availability, cost, cache state
    → Return selected deployment
  → LiteLLM executes request with fallback/retry
  → Response + cost metadata back to user
```

## RATIONALE
- LiteLLM's `CustomRoutingStrategyBase` allows injecting Conductor Brain decisions into routing
- Built-in fallback chains handle failure gracefully (Conductor picks primary, LiteLLM handles failover)
- Built-in cost tracking feeds directly into savings dashboard
- 100+ provider integrations mean BYOK support is trivial
- Redis-backed state enables horizontal scaling
- Traffic mirroring enables safe A/B testing of Conductor routing changes

## CONSEQUENCES
- Dependency on LiteLLM project (MIT license, actively maintained)
- Need to implement `ConductorRoutingStrategy` class
- Redis required for distributed deployments
- LiteLLM config YAML must be dynamically updated as compute nodes join/leave
