# System Architecture -- LLMHosts.com

## META
- format: C4-inspired (Context, Container, Component)
- created: 2026-02-14
- synced_with: ADR-001 through ADR-008, all epics

---

## 1. SYSTEM CONTEXT (C4 Level 1)

```
                            EXTERNAL SYSTEMS
    +-----------------------------------------------------------+
    |  Ollama API        Cloud LLM APIs       Tailscale/CF      |
    |  (localhost:11434) (OpenAI, Anthropic)   (tunnel infra)    |
    +-----------------------------------------------------------+
                              |    |    |
                              v    v    v
    +-----------------------------------------------------------+
    |                    LLMHosts System                          |
    |                                                           |
    |  Proxy -> Cache -> Router -> Backend Selection -> Response |
    |  Dashboard (TUI + Web) <- Metrics <- Request Logs         |
    +-----------------------------------------------------------+
                              ^    ^
                              |    |
    +-----------------------------------------------------------+
    |                      USERS                                |
    |  Developer (Cursor, Claude Code, Aider, CLI, any tool)    |
    |  Accessed via: localhost:4000 OR tunnel HTTPS URL          |
    +-----------------------------------------------------------+
```

### Actors
| Actor | Description | Interface |
|-------|------------|-----------|
| Developer (local) | Uses AI tools on same machine | `localhost:4000` |
| Developer (remote) | Uses AI tools from laptop/phone/CI | Tunnel HTTPS URL |
| Ollama | Local inference runtime | `localhost:11434` REST API |
| Cloud LLM APIs | OpenAI, Anthropic, etc. | HTTPS + API keys (BYOK) |
| Tailscale/Cloudflare | Tunnel infrastructure | CLI tools on user machine |
| LLMHosts Cloud | Dashboard hosting, router intelligence | HTTPS API |

---

## 2. CONTAINER DIAGRAM (C4 Level 2)

```
+------------------------------------------------------------------+
|                     llmhosts Python Package                        |
|                                                                   |
|  +------------------+    +------------------+   +--------------+  |
|  |  FastAPI Proxy   |    |  Three-Tier      |   |   vCache     |  |
|  |                  |--->|  Router           |-->|   Semantic   |  |
|  |  /v1/chat/compl  |    |                  |   |   Cache      |  |
|  |  /v1/messages    |    |  T0: Rules       |   |              |  |
|  |  /v1/embeddings  |    |  T1: kNN(FAISS)  |   |  T0: Hash    |  |
|  +--------+---------+    |  T2: ModernBERT  |   |  T1: Entity  |  |
|           |              |  T3: Qwen-0.5B   |   |  T2: Adaptive|  |
|           v              +--------+---------+   +------+-------+  |
|  +------------------+             |                    |          |
|  |  LiteLLM Backend |<-----------+                    |          |
|  |                  |                                  |          |
|  |  Ollama adapter  |    +------------------+          |          |
|  |  OpenAI adapter  |    |  Metrics Engine  |<---------+          |
|  |  Anthropic adapt |    |                  |                     |
|  +------------------+    |  SQLite store    |                     |
|                          |  Cost tracking   |                     |
|  +------------------+    |  Routing logs    |                     |
|  |  Discovery       |    +--------+---------+                     |
|  |                  |             |                                |
|  |  Ollama detect   |    +--------+---------+                     |
|  |  Hardware detect  |    |  Dashboard      |                     |
|  |  Tunnel detect   |    |                  |                     |
|  +------------------+    |  TUI (textual)   |                     |
|                          |  Web (React SPA) |                     |
|  +------------------+    +------------------+                     |
|  |  Tunnel Manager  |                                             |
|  |                  |    +------------------+                     |
|  |  Tailscale detect|    |  CLI             |                     |
|  |  Cloudflare det  |    |                  |                     |
|  |  Auto-configure  |    |  serve, tunnel,  |                     |
|  +------------------+    |  keys, doctor,   |                     |
|                          |  cache, config   |                     |
+------------------------------------------------------------------+
```

### Container Inventory
| Container | Technology | Port | Purpose |
|-----------|-----------|------|---------|
| FastAPI Proxy | Python, FastAPI, uvicorn | 4000 | OpenAI + Anthropic API proxy |
| Three-Tier Router | FAISS, ONNX, Ollama | internal | Fit-for-purpose model selection |
| vCache | SQLite, ONNX embeddings | internal | Verified semantic caching |
| LiteLLM Backend | litellm library | internal | Unified backend routing |
| Metrics Engine | SQLite | internal | Cost tracking, routing logs |
| Discovery | Python stdlib + httpx | internal | Ollama, hardware, tunnel detection |
| Tunnel Manager | subprocess (tailscale/cloudflared) | varies | Remote access orchestration |
| TUI Dashboard | textual/rich | terminal | Real-time terminal UI |
| Web Dashboard | React SPA (bundled) | 4000/dashboard | Real-time web UI |
| CLI | click/typer | terminal | User commands |

---

## 3. COMPONENT DIAGRAM (C4 Level 3)

### 3.1 FastAPI Proxy (Component Detail)
```
Request IN (OpenAI or Anthropic format)
  |
  v
[Format Detector] -- determines if OpenAI or Anthropic dialect
  |
  v
[Request Normalizer] -- converts to internal CanonicalRequest format
  |
  v
[Auth Middleware] -- API key validation (Phase 2+)
  |
  v
[Rate Limiter] -- per-key limits (Phase 2+, tunnel public mode)
  |
  v
[Cache Check] -- vCache three-tier lookup
  |     |
  | hit | miss
  v     v
[Return] [Router] -- three-tier routing decision
            |
            v
         [LiteLLM] -- backend call (Ollama, cloud API)
            |
            v
         [Response Normalizer] -- convert to original dialect
            |
            v
         [Metrics Logger] -- log cost, routing decision, latency
            |
            v
         [Cache Store] -- store response for future cache hits
            |
            v
         Response OUT (streaming or complete)
```

### 3.2 Three-Tier Router (Component Detail)
```
CanonicalRequest IN
  |
  v
[Tier 0: Rule Engine]
  | explicit model requested? user preference match? -> ROUTE
  | no explicit match -> continue
  v
[Tier 1: kNN Embedding]
  | embed query (all-MiniLM-L6-v2 via ONNX, <5ms)
  | FAISS nearest-K lookup (<10ms)
  | score: [model_fit, latency, cost, privacy, cache_prob]
  | confidence > 0.85 -> ROUTE
  | confidence <= 0.85 -> escalate
  v
[Tier 2: ModernBERT Classifier]
  | 6-signal analysis (<50ms GPU, <100ms CPU)
  |   - keyword patterns
  |   - embedding similarity
  |   - domain classification
  |   - complexity assessment
  |   - PII detection
  |   - user preference matching
  | classification confident -> ROUTE
  | ambiguous -> escalate
  v
[Tier 3: Qwen-0.5B Reasoning] (if installed)
  | small LLM reasons about intent + available models (<500ms)
  | always produces routing decision
  | falls back to CPU if no VRAM
  v
RoutingDecision OUT:
  {backend, model, reason, confidence, tier_used, alternatives}
```

### 3.3 vCache (Component Detail)
```
Query IN
  |
  v
[Normalizer] -- lowercase, strip whitespace, remove system prompt
  |
  v
[Tier 0: Exact Hash]
  | SHA-256(normalized_query)
  | lookup in hash_cache table
  | hit -> return cached response (0% FP)
  | miss -> continue
  v
[Tier 1: Entity Namespace]
  | extract entities via lightweight NER
  |   (languages, frameworks, locations, domains)
  | compute namespace key: "python|sorting" or "javascript|react"
  | restrict search to matching namespace only
  v
[Tier 2: vCache Adaptive]
  | embed query (shared embedding with router Tier 1)
  | search within namespace for nearest cached embeddings
  | per-embedding adaptive threshold (online logistic regression)
  | if similarity > learned_threshold AND confidence > (1 - delta)
  |   -> hit (return cached, formally guaranteed <1% FP)
  | else -> miss
  v
Cache Miss -> forward to Router
Cache Hit -> return response + log hit metrics
```

---

## 4. API CONTRACTS

### 4.1 OpenAI Chat Completions (Primary)
```
POST /v1/chat/completions
Content-Type: application/json
Authorization: Bearer <optional-llmhosts-api-key>

Request:
{
  "model": "string",              // model name or "auto" for router
  "messages": [
    {"role": "system"|"user"|"assistant", "content": "string"}
  ],
  "stream": bool,                 // default false
  "temperature": float,           // 0.0-2.0
  "max_tokens": int,
  "top_p": float,
  "stop": string | [string],
  "presence_penalty": float,
  "frequency_penalty": float
}

Response (non-streaming):
{
  "id": "chatcmpl-xxx",
  "object": "chat.completion",
  "created": unix_timestamp,
  "model": "string",              // actual model used
  "choices": [{
    "index": 0,
    "message": {"role": "assistant", "content": "string"},
    "finish_reason": "stop"|"length"
  }],
  "usage": {
    "prompt_tokens": int,
    "completion_tokens": int,
    "total_tokens": int
  },
  "llmhost": {                    // LLMHosts extension metadata
    "routing": {
      "tier": "rule"|"knn"|"modernbert"|"qwen",
      "confidence": float,
      "reason": "string",
      "backend": "ollama"|"openai"|"anthropic"|"vllm"|"hive",
      "model_requested": "string",
      "model_used": "string",
      "alternatives_considered": ["string"]
    },
    "cost": {
      "actual": float,            // what this request cost
      "estimated_cloud": float,   // what it would cost on cloud
      "saved": float,             // difference
      "currency": "USD"
    },
    "cache": {
      "hit": bool,
      "tier": "exact"|"namespace"|"semantic"|null,
      "confidence": float|null
    },
    "latency": {
      "routing_ms": float,
      "inference_ms": float,
      "total_ms": float
    }
  }
}

Response (streaming): SSE with data: chunks, same structure per chunk
```

### 4.2 Anthropic Messages
```
POST /v1/messages
Content-Type: application/json
x-api-key: <optional-llmhosts-api-key>
anthropic-version: 2023-06-01

Request:
{
  "model": "string",
  "messages": [
    {"role": "user"|"assistant", "content": "string"|[content_block]}
  ],
  "system": "string",
  "max_tokens": int,
  "stream": bool,
  "temperature": float,
  "top_p": float,
  "stop_sequences": ["string"]
}

Response: Standard Anthropic messages response format
  + llmhosts extension metadata (same as OpenAI contract above)
```

### 4.3 Internal APIs
```
GET  /v1/models                    -- list available models (local + cloud)
GET  /health                       -- proxy health check
GET  /metrics                      -- Prometheus-compatible metrics
GET  /dashboard                    -- web dashboard SPA
WS   /ws/dashboard                 -- WebSocket for real-time dashboard updates

# CLI-triggered internal endpoints
POST /internal/keys/add            -- add BYOK key
POST /internal/keys/remove         -- remove key
GET  /internal/keys/list           -- list configured keys
GET  /internal/doctor              -- system health check
GET  /internal/cache/stats         -- per-tier cache statistics
POST /internal/cache/clear         -- clear cache (optionally by namespace)
GET  /internal/tunnel/status       -- tunnel connection status
POST /internal/tunnel/start        -- start tunnel
POST /internal/tunnel/stop         -- stop tunnel
```

---

## 5. DATA MODEL

### 5.1 Entity Inventory
| Entity | Owner | Storage | Description |
|--------|-------|---------|-------------|
| Request | Metrics Engine | SQLite | Every proxied request |
| RoutingDecision | Router | SQLite | Router's decision per request |
| CacheEntry | vCache | SQLite | Cached responses |
| CacheEmbedding | vCache | FAISS + SQLite | Embedding vectors for semantic cache |
| EntityNamespace | vCache | SQLite | Extracted entities per cached query |
| RouterIndex | Router | FAISS file | kNN index for Tier 1 |
| BYOKKey | Discovery | SQLite (encrypted) | Cloud API keys |
| HardwareProfile | Discovery | SQLite | GPU, CPU, RAM, disk info |
| UserPreference | Router | SQLite | Routing preferences |
| UserBaseline | Metrics | SQLite | Pre-LLMHosts spending baseline |

### 5.2 Schema Definition

```sql
-- Core request logging
CREATE TABLE requests (
    id TEXT PRIMARY KEY,                    -- UUID
    created_at REAL NOT NULL,              -- unix timestamp
    dialect TEXT NOT NULL,                  -- 'openai' | 'anthropic'
    model_requested TEXT,                   -- what user asked for
    model_used TEXT NOT NULL,               -- what was actually used
    backend TEXT NOT NULL,                  -- 'ollama' | 'openai' | 'anthropic' | 'vllm' | 'hive'
    prompt_tokens INTEGER NOT NULL,
    completion_tokens INTEGER NOT NULL,
    total_tokens INTEGER NOT NULL,
    cost_actual REAL NOT NULL DEFAULT 0.0,  -- USD
    cost_estimated_cloud REAL NOT NULL DEFAULT 0.0,
    cost_saved REAL NOT NULL DEFAULT 0.0,
    latency_routing_ms REAL NOT NULL,
    latency_inference_ms REAL NOT NULL,
    latency_total_ms REAL NOT NULL,
    cache_hit INTEGER NOT NULL DEFAULT 0,   -- 0/1
    cache_tier TEXT,                         -- 'exact' | 'namespace' | 'semantic' | null
    stream INTEGER NOT NULL DEFAULT 0,      -- 0/1
    error TEXT                              -- null if success
);
CREATE INDEX idx_requests_created ON requests(created_at);
CREATE INDEX idx_requests_model ON requests(model_used);
CREATE INDEX idx_requests_backend ON requests(backend);

-- Routing decisions (1:1 with requests)
CREATE TABLE routing_decisions (
    request_id TEXT PRIMARY KEY REFERENCES requests(id),
    tier TEXT NOT NULL,                     -- 'rule' | 'knn' | 'modernbert' | 'qwen'
    confidence REAL NOT NULL,
    reason TEXT NOT NULL,                   -- human-readable explanation
    alternatives_json TEXT,                 -- JSON array of considered alternatives
    knn_distance REAL,                     -- if Tier 1 was used
    modernbert_scores_json TEXT,           -- if Tier 2 was used (6 signals)
    feedback_score INTEGER                 -- user feedback: -1, 0, 1
);

-- vCache: exact hash tier
CREATE TABLE cache_hash (
    hash TEXT PRIMARY KEY,                 -- SHA-256 of normalized query
    response_json TEXT NOT NULL,           -- cached response
    model TEXT NOT NULL,                   -- model that generated it
    created_at REAL NOT NULL,
    hit_count INTEGER NOT NULL DEFAULT 0,
    last_hit_at REAL
);

-- vCache: entity namespace
CREATE TABLE cache_namespaces (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    query_hash TEXT NOT NULL,              -- links to original query
    namespace_key TEXT NOT NULL,           -- e.g., "python|sorting"
    entities_json TEXT NOT NULL,           -- extracted entities
    embedding_id INTEGER REFERENCES cache_embeddings(id)
);
CREATE INDEX idx_cache_ns_key ON cache_namespaces(namespace_key);

-- vCache: adaptive semantic tier
CREATE TABLE cache_embeddings (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    namespace_key TEXT NOT NULL,
    embedding BLOB NOT NULL,               -- 384-dim float32 vector
    response_json TEXT NOT NULL,
    model TEXT NOT NULL,
    learned_threshold REAL NOT NULL DEFAULT 0.85,  -- adaptive per-embedding
    logistic_weights BLOB,                 -- online logistic regression state
    hit_count INTEGER NOT NULL DEFAULT 0,
    false_positive_count INTEGER NOT NULL DEFAULT 0,
    created_at REAL NOT NULL,
    last_hit_at REAL
);
CREATE INDEX idx_cache_emb_ns ON cache_embeddings(namespace_key);

-- BYOK API keys (encrypted)
CREATE TABLE byok_keys (
    provider TEXT PRIMARY KEY,             -- 'openai' | 'anthropic' | etc.
    encrypted_key BLOB NOT NULL,           -- Fernet-encrypted
    validated INTEGER NOT NULL DEFAULT 0,  -- 0/1
    added_at REAL NOT NULL,
    last_used_at REAL
);

-- Hardware profile (single row, updated on serve start)
CREATE TABLE hardware_profile (
    id INTEGER PRIMARY KEY CHECK (id = 1),
    gpu_name TEXT,
    gpu_vram_mb INTEGER,
    gpu_driver TEXT,
    cpu_model TEXT,
    cpu_cores INTEGER,
    ram_mb INTEGER,
    disk_free_mb INTEGER,
    ollama_detected INTEGER NOT NULL DEFAULT 0,
    ollama_models_json TEXT,               -- JSON array of model info
    tailscale_installed INTEGER NOT NULL DEFAULT 0,
    cloudflared_installed INTEGER NOT NULL DEFAULT 0,
    updated_at REAL NOT NULL
);

-- User preferences
CREATE TABLE user_preferences (
    key TEXT PRIMARY KEY,                  -- e.g., 'always_local_for_code'
    value TEXT NOT NULL,                   -- preference value
    updated_at REAL NOT NULL
);

-- User baseline (for honest savings comparison)
CREATE TABLE user_baseline (
    id INTEGER PRIMARY KEY CHECK (id = 1),
    monthly_ai_spend_usd REAL,            -- self-reported
    captured_at REAL,
    source TEXT                            -- 'onboarding' | 'manual' | 'api_import'
);

-- Router training data (for kNN/ModernBERT improvement)
CREATE TABLE router_training_data (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    request_id TEXT REFERENCES requests(id),
    embedding BLOB NOT NULL,               -- 384-dim query embedding
    routing_decision TEXT NOT NULL,         -- what was decided
    was_exploration INTEGER NOT NULL DEFAULT 0,  -- deliberate 5% exploration
    feedback_score INTEGER,                -- user feedback
    created_at REAL NOT NULL
);
CREATE INDEX idx_rtd_created ON router_training_data(created_at);
```

### 5.3 Storage Estimates
| Scale | Requests/day | DB Size/month | FAISS Index | Total |
|-------|-------------|--------------|-------------|-------|
| 1 user | 100 | ~5 MB | ~1 MB | ~6 MB |
| 10 users | 1,000 | ~50 MB | ~10 MB | ~60 MB |
| 100 users (self-hosted) | 10,000 | ~500 MB | ~100 MB | ~600 MB |

### 5.4 Migration Strategy
- Tool: Alembic (SQLAlchemy migrations)
- All schema changes via versioned migration files
- `llmhosts db upgrade` CLI command
- Backward-compatible changes only (add columns, never remove)
- SQLite for Phase 1 (single-user, local). PostgreSQL migration path for Phase 2+ cloud dashboard.

---

## 6. DEPLOYMENT TOPOLOGY

### Phase 1: Single Machine
```
User's Machine:
  Docker OR native Python:
    llmhosts serve (port 4000)
      -> Ollama (port 11434, user-managed)
      -> Cloud APIs (HTTPS, BYOK keys)
    SQLite DB: ~/.llmhosts/data/llmhosts.db
    FAISS index: ~/.llmhosts/data/router.faiss
    Config: ~/.llmhosts/config.toml
```

### Phase 2: Multi-Device + Tunnel
```
Home Machine (GPU):                    Laptop (coffee shop):
  llmhosts serve (port 4000)              export OPENAI_API_BASE=https://xxx.ts.net/v1
  llmhosts tunnel                         Cursor/Claude Code routes through tunnel
    -> Tailscale Funnel                    -> encrypted -> home machine -> Ollama
       OR Cloudflare Tunnel                -> response streams back
    -> HTTPS URL generated

LLMHosts Cloud:
  Cloud dashboard (Next.js + PostgreSQL)
  Global router intelligence API
  Tunnel coordination metadata
```

### Phase 3: Hive Mode
```
User A (Home):                         User B (Home):
  llmhosts serve                          llmhosts serve
  llmhosts hive join ABC123               llmhosts hive join ABC123
    -> Headscale mesh                      -> Headscale mesh
    -> WireGuard tunnel to User B          -> WireGuard tunnel to User A
    -> Karma: contributed 40h, used 20h    -> Karma: contributed 20h, used 35h

LLMHosts Cloud:
  Headscale coordination server
  ACL policy management
  Karma tracking
```

---

## 7. SECURITY ARCHITECTURE

### Data Classification
| Data | Classification | Storage | Encryption |
|------|---------------|---------|-----------|
| User prompts/responses | SENSITIVE | Local SQLite only | At rest (SQLite encryption optional) |
| BYOK API keys | SECRET | Local SQLite | Fernet encryption at rest |
| Routing decisions | INTERNAL | Local SQLite | None (not sensitive) |
| Cache entries | SENSITIVE | Local SQLite | At rest (mirrors prompt data) |
| Hardware profile | INTERNAL | Local SQLite | None |
| User baseline spend | PRIVATE | Local SQLite | None |
| Tunnel URLs | INTERNAL | Memory only | TLS in transit |

### Encryption Layers
- **At rest**: BYOK keys encrypted via Fernet (cryptography library). Key derived from machine-specific secret.
- **In transit (local)**: localhost, no encryption needed
- **In transit (tunnel)**: TLS via Tailscale (WireGuard) or Cloudflare (HTTPS)
- **In transit (Hive)**: WireGuard (kernel-level encryption)
- **In transit (cloud APIs)**: HTTPS (provider TLS)

### PII Protection
- ModernBERT Tier 2 includes PII signal detection
- Requests classified PRIVATE: names, emails, phone numbers, SSN, medical terms
- PRIVATE requests -> local-only routing (never cloud, never Hive)
- Dashboard shows: "Stayed on YOUR machine" confirmation

### Threat Model (STRIDE Summary)
| Threat | Mitigation |
|--------|-----------|
| Spoofing (fake requests) | API key auth (Phase 2+), rate limiting |
| Tampering (modify in transit) | TLS for all external communication |
| Repudiation (deny actions) | Full request audit log in SQLite |
| Information Disclosure (data leak) | PII routing, encryption at rest, no telemetry without consent |
| Denial of Service | Rate limiting, graceful degradation, fallback chain |
| Elevation of Privilege | Minimal permissions, no root required, sandboxed Docker |
