# vLLM -- High-Throughput LLM Serving Engine

## META
- repo: https://github.com/vllm-project/vllm
- docs: https://docs.vllm.ai
- license: Apache 2.0
- relevance: PRIMARY -- production model serving engine for local GPU inference with prefix caching
- last_verified: 2026-02-13

## SUMMARY

vLLM is the leading production-grade LLM inference engine optimized for high throughput and low latency. Features automatic prefix caching, PagedAttention, tensor/pipeline parallelism, and OpenAI-compatible API server. Critical for LLMHosts's local model serving layer.

## CORE FEATURES

### PagedAttention
- Efficient KV cache management using paging (inspired by OS virtual memory)
- Near-zero memory waste for KV cache
- Enables higher batch sizes and throughput

### Automatic Prefix Caching (APC)
- Hash-based approach using SHA256 (default) for collision-free block identification
- Each KV cache block hashed by: parent hash + block tokens + extra hashes (LoRA ID, multimodal hash, cache salt)
- Full blocks only are cached
- LRU eviction policy for cached blocks

### Prefix Caching Impact
- **10x cost difference** between cached and uncached tokens
- Critical for system prompts, repeated prefixes, multi-turn conversations
- Automatic -- no manual configuration needed

### Hash Algorithm Options
- `sha256` (default): Pickle serialization, collision-free
- `sha256_cbor`: CBOR serialization, cross-environment reproducible
- `xxhash`: Faster non-cryptographic hashing (128-bit)
- `xxhash_cbor`: CBOR + xxHash combination

### Cache Isolation (Multi-Tenant Security)
- `cache_salt` parameter per request
- Only requests with same salt can reuse cached blocks
- Prevents timing-based cache inference attacks
- Essential for LLMHosts's multi-tenant environment

## KV CACHE ARCHITECTURE (v1)

### Data Structure: KVCacheBlock
```python
class KVCacheBlock:
    block_id: int          # Immutable block ID
    block_hash: BlockHash  # Assigned when block is full
    ref_cnt: int           # Number of requests using this block
    prev_free_block: KVCacheBlock | None  # Doubly linked list
    next_free_block: KVCacheBlock | None
```

### Components
- **Block Pool**: Pre-allocated list of all KVCacheBlocks (avoids Python object creation overhead)
- **Free Block Queue**: Doubly-linked list (O(1) move operations)
- **Cache Blocks**: Hash → block_id mapping
- **Request Blocks**: request_id → allocated block_ids

### Operations
1. **Allocation**: Hash prompt tokens → lookup cached blocks → touch (increase ref count) → allocate new blocks from free queue head
2. **Free**: Decrease ref count → add to free queue tail in reverse order (last blocks evicted first)
3. **Eviction**: Pop LRU block from free queue head → remove from cache

## OPENAI-COMPATIBLE API SERVER

### Endpoints
- `POST /v1/chat/completions` -- Chat completions
- `POST /v1/completions` -- Text completions
- `POST /v1/embeddings` -- Embeddings
- `GET /v1/models` -- List available models
- `GET /health` -- Health check

### Usage
```bash
vllm serve meta-llama/Llama-3.1-70B --host 0.0.0.0 --port 8000
```

### Client Usage (drop-in OpenAI replacement)
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-70B",
    messages=[{"role": "user", "content": "Hello!"}]
)
```

## DISTRIBUTED SERVING

### Tensor Parallelism (within single node)
- Split model layers across multiple GPUs
- `--tensor-parallel-size N`
- Best for latency-sensitive workloads

### Pipeline Parallelism (across nodes)
- Split model stages across nodes
- `--pipeline-parallel-size N`
- Best for throughput and memory-constrained setups

### Multi-Node Setup
```bash
# Head node
vllm serve model --tensor-parallel-size 4 --pipeline-parallel-size 2

# Worker nodes connect via Ray cluster
ray start --address=<head-node-ip>:6379
```

## ADVANCED FEATURES

### Speculative Decoding
- Draft model generates candidate tokens
- Target model verifies in parallel
- Reduces latency for autoregressive generation

### LoRA Serving
- Serve multiple LoRA adapters on single base model
- Dynamic adapter loading/unloading
- Efficient memory sharing across adapters

### Quantization Support
- AWQ, GPTQ, SqueezeLLM, FP8
- Reduces memory footprint and increases throughput
- Minimal quality degradation

### Structured Output
- JSON schema enforcement
- Regex-guided generation
- Grammar-constrained decoding

## PERFORMANCE BENCHMARKS
- Industry-leading throughput for production deployments
- Full tool calling support
- Excellent GPU utilization
- Continuous batching for optimal throughput

## RELEVANCE TO LLMHOST

### Direct Integration Points
1. **Model Serving**: vLLM serves models on user's GPU hardware
2. **Prefix Caching**: Automatic cost reduction for repeated prompts/system prompts
3. **Cache Isolation**: Multi-tenant security via cache_salt
4. **OpenAI API**: Drop-in compatible with LiteLLM proxy and Conductor
5. **Distributed**: Tensor/pipeline parallelism for multi-GPU setups

### Integration with LLMHosts Stack
```
Conductor Brain → LiteLLM Router → vLLM Server (local GPU)
                                  → exo Cluster (distributed home devices)
                                  → External API (BYOK)
```

### KV Cache Optimization Strategy for LLMHosts
1. Enable APC by default on all vLLM instances
2. Use cache_salt per tenant for isolation
3. Implement llm-d style cache-aware scheduling (route requests with common prefixes to same vLLM instance)
4. Monitor cache hit rates as KPI in savings dashboard
5. Use KVShare concepts for cross-instance cache reuse

### Key Config for LLMHosts
```bash
vllm serve <model> \
  --host 0.0.0.0 \
  --port 8000 \
  --enable-prefix-caching \
  --prefix-caching-hash-algo sha256 \
  --tensor-parallel-size <num_gpus> \
  --max-model-len <context_length> \
  --gpu-memory-utilization 0.9
```
