# Epic: Caching Layer (vCache Verified Semantic Caching)

## DESCRIPTION
Multi-tier caching with FORMAL guarantees on false positive rates. Core cost-reduction mechanism: cache hit = $0 cost, instant response. Uses vCache (ICLR 2026, UC Berkeley/Stanford) for per-prompt adaptive thresholds instead of static similarity thresholds.

**COURSE CORRECTION (2026-02-14)**: Replaced GPTCache static threshold with vCache verified semantic caching. Added entity-aware namespace filtering to structurally prevent cross-domain false positives ("sort in Python" vs "sort in JavaScript"). Three-tier architecture: exact hash -> entity namespace -> vCache verified semantic. Formally guaranteed <1% false positive rate.

## CACHE ARCHITECTURE
```
Request arrives
  |
  v
Tier 0: Exact Hash Match (SHA-256 of normalized prompt)
  Hit -> return cached response (0% false positives, 5-15% hit rate)
  Miss -> continue
  |
  v
Tier 1: Entity-Aware Namespace Filter
  Extract key entities (language names, frameworks, locations, etc.)
  Route to namespace-specific cache bucket
  "Python sort" and "JavaScript sort" go to DIFFERENT buckets
  Prevents cross-domain false positives structurally
  |
  v
Tier 2: vCache Verified Semantic Match
  Per-prompt adaptive similarity threshold (NOT one global threshold)
  Online logistic regression learns separate threshold per cached embedding
  User sets max error rate (delta=0.01 = 1%)
  MATHEMATICALLY GUARANTEES false positive rate stays below bound
  20-40% hit rate that improves over time
  |
  v
Miss -> forward to router
```

**Result**: 25-50% overall cache hit rate with <1% false positive rate. Formally guaranteed.

## BACKLOG ITEMS

### CACHE-001: Exact Hash Cache (Tier 0)
- **Priority**: P0
- **Phase**: 1
- **Status**: BACKLOG
- **Description**: SHA-256 hash of normalized prompt (lowercased, whitespace-normalized, system prompt removed). Exact match returns cached response. Local SQLite storage. Zero false positives. Handles repeated identical queries.
- **Acceptance**: Identical prompts return cached response in <5ms. 5-15% hit rate. 0% false positives. Cache stored in local SQLite.
- **Dependencies**: None

### CACHE-009: Entity-Aware Namespace Filter (Tier 1)
- **Priority**: P0
- **Phase**: 1
- **Status**: BACKLOG
- **Description**: Extract key entities from prompts using lightweight NER: programming languages, frameworks, file types, locations, domain keywords. Each entity combination maps to a cache namespace. "Python sorting algorithm" and "JavaScript sorting algorithm" go to different namespace buckets. Prevents the #1 source of semantic cache false positives: cross-domain queries that look similar in embedding space but need different answers.
- **Acceptance**: Entity extraction in <10ms. Cross-domain queries routed to separate namespaces. Measurable false positive reduction vs global cache.
- **Dependencies**: CACHE-001

### CACHE-010: vCache Verified Semantic Match (Tier 2)
- **Priority**: P0
- **Phase**: 1
- **Status**: BACKLOG
- **Description**: vCache (ICLR 2026) implementation. Instead of one global similarity threshold (GPTCache's approach), vCache learns a SEPARATE threshold per cached embedding using online logistic regression. User sets maximum error rate (delta parameter, default 0.01 = 1%). The algorithm mathematically guarantees the actual false positive rate stays below delta. Thresholds adapt: frequently-hit embeddings with clear semantic boundaries get tighter thresholds; rare edge-case embeddings get looser ones. Uses same embedding pipeline as Tier 1 kNN router (all-MiniLM-L6-v2 via ONNX).
- **Acceptance**: <1% false positive rate (formally verified). 20-40% hit rate. Per-embedding adaptive thresholds. <50ms lookup. delta parameter user-configurable.
- **Dependencies**: CB-004 (embedding pipeline), CACHE-009

### CACHE-002: Cache Metrics Dashboard
- **Priority**: P1
- **Phase**: 1
- **Status**: BACKLOG
- **Description**: Track and display: cache hit rates per tier (exact/namespace/semantic), savings from cache, false positive rate monitoring, cache efficiency. Visible in TUI and web dashboard.
- **Acceptance**: Users see "X% served from cache (Y% exact, Z% semantic), saving $A. False positive rate: B%."
- **Dependencies**: CACHE-001, CACHE-010, FE-001

### CACHE-003: Cache Management CLI
- **Priority**: P1
- **Phase**: 1
- **Status**: BACKLOG
- **Description**: `llmhosts cache stats` (per-tier hit rates, false positive rate). `llmhosts cache clear [--namespace X]`. `llmhosts cache config delta 0.01` (set max false positive rate). `llmhosts cache config max-size 1G`.
- **Acceptance**: Users can view per-tier stats, clear by namespace, and configure delta threshold from CLI.
- **Dependencies**: CACHE-001, CACHE-010

### CACHE-004: Intelligent Cache Eviction
- **Priority**: P1
- **Phase**: 2
- **Status**: BACKLOG
- **Description**: Intelligent eviction: LRU + frequency + cost-awareness. High-value entries (expensive cloud API responses) preserved over cheap local inference results.
- **Acceptance**: Cache eviction prioritizes removing low-value entries. High-cost responses retained longer.
- **Dependencies**: CACHE-001, CB-003

### CACHE-005: DEPRECATED -- Merged into CACHE-010
- **Priority**: N/A
- **Phase**: N/A
- **Status**: DEPRECATED
- **Description**: Original "adaptive similarity threshold" item. Superseded by vCache (CACHE-010) which provides formally verified adaptive thresholds per embedding.

### CACHE-006: vLLM Prefix Caching (Opt-In)
- **Priority**: P1
- **Phase**: 2
- **Status**: BACKLOG
- **Description**: For vLLM users: enable APC (Automatic Prefix Caching). System prompts and multi-turn prefixes cached. 10x savings on repetitive prompt patterns (agentic workflows, coding assistants). Dashboard shows: "Prefix cache saved X tokens."
- **Acceptance**: vLLM users see prefix caching active. Measurable throughput improvement. Works alongside semantic cache.
- **Dependencies**: CF-009 (vLLM integration)

### CACHE-007: Cache Isolation (Multi-User)
- **Priority**: P1
- **Phase**: 2
- **Status**: BACKLOG
- **Description**: Cache isolated per user when accounts exist (Phase 2+). Prevent cross-user cache inference via timing attacks.
- **Acceptance**: Users cannot infer other users' cached content.
- **Dependencies**: CACHE-001, FE-012 (auth)

### CACHE-008: Cross-User Semantic Dedup (Phase 4)
- **Priority**: P2
- **Phase**: 4
- **Status**: BACKLOG
- **Description**: For non-sensitive, general-knowledge queries: shared cache across users (opt-in). vCache thresholds applied per-user even in shared mode.
- **Acceptance**: Opt-in shared cache. Privacy gate prevents sensitive queries from entering shared pool.
- **Dependencies**: CACHE-007, CB-010 (privacy router)

### CACHE-011: GenCache for Agentic Workflows
- **Priority**: P2
- **Phase**: 3
- **Status**: BACKLOG
- **Description**: Structural matching for agentic workflows (tool call sequences, multi-step chains). Caches intermediate results when tool call patterns match structurally, even if specific parameters differ. Complements semantic cache for non-conversational use cases.
- **Acceptance**: Repeated agentic workflow patterns hit cache. Structural matching works across parameter variations.
- **Dependencies**: CACHE-010, CB-007 (multi-turn context)
