# LiteLLM -- Unified LLM Proxy and Router

## META
- repo: https://github.com/BerriAI/litellm
- docs: https://docs.litellm.ai
- license: MIT
- relevance: PRIMARY -- API proxy layer for multi-provider routing, BYOK support, and unified interface
- last_verified: 2026-02-13

## SUMMARY

LiteLLM provides a unified proxy server that abstracts 100+ LLM providers through a standardized OpenAI API format. Handles load balancing, failover, rate limiting, and cost tracking across multiple deployments. Critical component for LLMHosts's Tier 3 (external API) and unified routing layer.

## CORE ARCHITECTURE

### Request Flow
```
Client Request → Virtual Key Auth + Rate Limiting
    → LiteLLM Router (load balancing + fallbacks)
        → litellm.completion() / embedding() call
            → Provider API (OpenAI, Anthropic, Azure, self-hosted, etc.)
    → Response back to client
```

### Proxy Server
- Runs as standalone server: `litellm --config /path/to/config.yaml`
- OpenAI-compatible endpoint: `POST /v1/chat/completions`
- Supports streaming and non-streaming modes
- Virtual key management for multi-tenant access
- Redis-backed distributed state for multi-instance deployments

## ROUTING STRATEGIES

### Available Strategies
1. **simple-shuffle** (DEFAULT, RECOMMENDED for production)
   - Weighted random based on RPM/TPM or explicit weights
   - Minimal latency overhead
   - Best for high-traffic scenarios

2. **latency-based-routing**
   - Routes to fastest-responding deployment
   - Caches and updates response times per deployment
   - Configurable TTL and lowest-latency buffer
   - Buffer prevents overloading single fastest endpoint

3. **usage-based-routing** (async, NOT recommended for production)
   - Routes to deployment with lowest TPM usage
   - Redis-tracked across instances
   - Filters out deployments exceeding TPM/RPM limits

4. **least-busy**
   - Routes to deployment with fewest ongoing requests

5. **cost-based-routing** (async)
   - Routes to lowest-cost deployment
   - Uses litellm_model_cost_map for pricing
   - Supports custom input/output cost per token

6. **Custom Routing Strategy**
   - Extend `CustomRoutingStrategyBase`
   - Implement `async_get_available_deployment()`
   - Full access to model_list, messages, request kwargs

## FALLBACK & RETRY ARCHITECTURE

### Three-Layer Exception Handling
```
function_with_fallbacks (model group fallbacks)
    └── function_with_retries (within-group retries)
        └── litellm.completion() (actual API call)
```

### Fallback Configuration
- Model group fallbacks: `fallbacks=[{"gpt-3.5-turbo": ["gpt-4", "claude-3"]}]`
- Content policy fallbacks for moderation rejections
- Context window fallbacks for token limit errors
- Provider-specific fallbacks

### Retry Configuration
- Fixed or exponential backoff
- Configurable max retries per deployment
- Cooldown periods for failed deployments
- Timeout settings (per request and connection)

## RATE LIMITING

### Multi-Level Enforcement
- **Deployment level**: RPM/TPM per deployment
- **Virtual key level**: Per API key limits
- **User level**: Per end-user limits
- **Team level**: Per organization limits

### Pre-Call Validation
- `enable_pre_call_checks=True` validates limits before sending request
- TPM: best-effort (checked before request)
- RPM: strictly enforced at exact threshold
- Redis-backed for distributed deployments

## PROVIDER SUPPORT (100+)

### Major Providers
- OpenAI (GPT-4, GPT-5, etc.)
- Anthropic (Claude family)
- Azure OpenAI
- Google (Gemini, PaLM)
- AWS Bedrock
- Cohere, Mistral, Meta
- DeepSeek, Together AI
- Self-hosted: vLLM, Ollama, TGI, OpenAI-compatible

### BYOK (Bring Your Own Key)
- Users provide their own API keys
- LiteLLM routes using user's credentials
- No markup on inference costs
- Key management and rotation support

## ADVANCED FEATURES

### Traffic Mirroring / A/B Testing
- Mirror production traffic to secondary model
- Silent evaluation without affecting primary response
- Useful for comparing model performance

### Deployment Priority
- `order` parameter in litellm_params
- Lower values = higher priority
- Same-order deployments use routing strategy

### Caching
- Built-in caching with multiple backends
- Redis, in-memory, S3, Qdrant (semantic)
- Configurable TTL and cache key strategies

### Observability
- Logging to Langfuse, Helicone, Datadog, etc.
- Cost tracking per request/user/team
- Latency metrics
- Usage analytics

## CONFIGURATION EXAMPLE

```yaml
model_list:
  - model_name: gpt-4
    litellm_params:
      model: azure/gpt-4-deployment
      api_key: os.environ/AZURE_KEY
      api_base: os.environ/AZURE_BASE
      rpm: 1000
      tpm: 100000
  - model_name: gpt-4
    litellm_params:
      model: openai/gpt-4
      api_key: os.environ/OPENAI_KEY
      rpm: 500
  - model_name: local-llama
    litellm_params:
      model: openai/llama-3-70b
      api_base: http://localhost:8000/v1
      api_key: dummy

router_settings:
  routing_strategy: simple-shuffle
  redis_host: redis
  redis_port: 6379
  enable_pre_call_check: true

general_settings:
  master_key: sk-llmhost-master
```

## RELEVANCE TO LLMHOST

### Direct Integration Points
1. **Unified API Proxy**: LiteLLM serves as the API gateway between Conductor and all compute tiers
2. **BYOK Support**: Users add their own API keys for external providers
3. **Cost-Based Routing**: Built-in lowest-cost routing aligns with LLMHosts cost optimization
4. **Custom Routing**: Extend with Conductor-informed routing strategy
5. **Rate Limiting**: Multi-tenant rate limiting for shared platform
6. **Observability**: Built-in cost tracking feeds into savings dashboard

### Custom Integration Plan
1. Create `ConductorRoutingStrategy(CustomRoutingStrategyBase)` that queries Conductor brain
2. Register local exo/vLLM endpoints as LiteLLM deployments
3. Register user's BYOK keys as additional deployments
4. Conductor decides routing, LiteLLM executes with fallback/retry
5. Feed LiteLLM's cost/latency metrics back to Conductor's RL training loop

### Deployment
- Docker: `docker run -p 4000:4000 ghcr.io/berriai/litellm:main-latest`
- Config via YAML file or environment variables
- Redis for distributed state
- Scales horizontally behind load balancer
