# Caching Strategies -- Cost Reduction Engine

## META
- sources: vCache, KVShare (arXiv:2503.16525), llm-d, vLLM APC, vCache
- relevance: PRIMARY -- caching is the #1 mechanism for cost reduction
- last_verified: 2026-02-13

## SUMMARY

Multi-layered caching strategy combining verified semantic caching (vCache, ICLR 2026) with KV cache optimization (reuse computation for shared prefixes). vCache uses per-prompt adaptive thresholds (not static similarity), formally guarantees <1% false positives, and uses entity-aware namespacing. Three-tier cache pipeline: exact hash → entity namespace → vCache adaptive semantic. Combined impact: 50-70% cost reduction possible through cache hits + prefix reuse.

## LAYER 1: SEMANTIC CACHING (vCache)

### Architecture (Three-Tier Cache Pipeline)
```
User Query →
  Tier 1: Exact hash match → if HIT, return cached response (fastest, zero risk)
  Tier 2: Entity namespace lookup → if HIT in same entity scope, return cached response
  Tier 3: vCache adaptive semantic match → per-prompt adaptive threshold
    → Hit (score >= adaptive threshold): Return cached response
    → Miss: Forward to LLM → Cache response + embedding + entity tags
```

### Core Components
1. **Exact Hash Layer**: SHA256 hash of normalized query for identical-query dedup (zero false positives)
2. **Entity Namespace Layer**: Entity-aware namespacing -- groups cached responses by detected entities (people, products, topics) to reduce search space and improve precision
3. **vCache Adaptive Semantic Layer**: Per-prompt adaptive thresholds (ICLR 2026) -- NOT static similarity. Online learning estimates optimal threshold for each cached prompt. Formally guarantees <1% false positive rate.
4. **Vector Store**: FAISS for similarity search (all-MiniLM-L6-v2 via ONNX embeddings)
5. **Cache Store**: Stores query-response pairs + entity tags (Redis, SQLite, etc.)
6. **Eviction Manager**: LRU, TTL, or custom eviction policies

### Performance Metrics
- API call reduction: up to 68.8%
- Cache hit rates: 61.6-68.8% (across query categories)
- Speed improvement: 2-10x on cache hits
- Accuracy on hits: >97%
- Cost reduction: up to 10x on cached queries

### Threshold Optimization (vCache Approach)
- Static thresholds cause false positives (wrong cached answers) or false negatives (missed cache hits)
- vCache: Online learning algorithm estimates optimal threshold per cached prompt
- Provides user-defined error rate guarantees
- No additional training required

### LLMHosts Integration
```
User Request → Conductor Brain
  → Semantic Cache Check (vCache layer)
    → HIT: Return cached response (zero cost)
    → MISS: Route to model via LiteLLM
      → Cache response for future queries
```

## LAYER 2: KV CACHE / PREFIX CACHING (vLLM APC)

### How It Works
- Hash each KV cache block by: parent hash + block tokens + extra hashes
- Default: SHA256 for collision-free identification
- Full blocks only are cached
- LRU eviction for cached blocks

### Impact
- **10x cost difference** between cached and uncached tokens
- Critical for: system prompts, multi-turn conversations, repeated prefixes
- Automatic -- no manual configuration needed

### Cache Isolation for Multi-Tenant
- `cache_salt` per request prevents cross-tenant cache inference
- Essential for LLMHosts's shared infrastructure
- Configurable per user/team

### Data Structure (KVCacheBlock)
- Pre-allocated block pool (avoids Python object creation overhead)
- Doubly-linked list free queue (O(1) operations)
- Hash → block_id mapping for cache lookups
- Request → block_ids mapping for allocation tracking

## LAYER 3: DISTRIBUTED CACHE (KVShare)

### Architecture
- Multi-tenant KV cache reuse system
- Dual-Stage High Deviation algorithm
- Selectively recomputes small portions during prefill and decode

### Performance
- TTFT reduction: up to 9.39x
- Throughput increase: 1.2x
- Accuracy improvement: 20.38% over naive methods

### LLMHosts Application
- Share KV cache across user requests with similar system prompts
- Conductor-level cache coordination across vLLM instances
- Particularly effective for: shared system prompts, RAG with common documents

## LAYER 4: CACHE-AWARE SCHEDULING (llm-d)

### Problem
Standard load balancers scatter related requests across instances, destroying cache locality

### Solution
- Prefix-cache-aware request routing
- Maintain cache locality across distributed deployments
- Route requests with common prefixes to same vLLM instance

### Performance
- 57x faster response times
- 2x throughput on identical hardware
- Critical for production deployments at scale

### LLMHosts Integration
- Conductor considers cache state when routing
- Requests with same system prompt → same vLLM instance
- Multi-turn conversations → same instance (preserve KV cache)
- Fallback to any instance if preferred is overloaded

## COMPARISON: SEMANTIC VS TOKEN-LEVEL CACHING

| Aspect | Semantic (vCache) | Token-Level (vLLM APC) |
|--------|-------------------|----------------------|
| Granularity | Full response | KV cache blocks |
| Match Type | Fuzzy (embedding similarity) | Exact (hash match) |
| Cost Savings | 100% on hit (zero API cost) | 10x on prefix match |
| Risk | Wrong answer on false positive | None (exact match) |
| Latency Savings | Eliminates inference entirely | Eliminates prefix computation |
| Best For | Repeated/similar queries | System prompts, multi-turn |
| Storage | Response text + embeddings | GPU memory (KV blocks) |

### Recommendation: Use ALL Layers
1. Exact hash as first layer (zero false positive risk, fastest)
2. Entity namespace as second layer (scoped semantic search, high precision)
3. vCache adaptive semantic as third layer (highest savings potential, <1% false positives guaranteed)
4. KV prefix cache as fourth layer (no accuracy risk, prefix reuse)
5. Cache-aware scheduling as fifth layer (infrastructure optimization)
6. KVShare for cross-instance reuse (advanced optimization)

## CACHE METRICS FOR SAVINGS DASHBOARD

### KPIs to Track
- Semantic cache hit rate (% of requests served from cache)
- KV prefix cache hit rate (% of tokens reused from cache)
- Estimated cost saved per cache hit
- Cumulative savings over time
- Cache efficiency (hits per GB of cache storage)
- Average response latency: cached vs uncached

### Gamification Opportunities
- "Cache Champion" badge: >80% cache hit rate
- "Savings Streak": Consecutive days with >50% cache utilization
- Progress bar: "You've saved $X through intelligent caching"
