# GPU-as-API: Technical Reference

> Turn any home GPU into a production cloud API endpoint with a single command.

This document is the complete technical reference for the LLMHosts GPU-as-API system. It covers every component, data flow, API surface, configuration option, error state, and performance target involved in making `llmhosts up` work end-to-end.

---

## Table of Contents

1. [Architecture Overview](#architecture-overview)
2. [System Data Flow](#system-data-flow)
3. [Component Reference](#component-reference)
4. [API Reference](#api-reference)
5. [Security Model](#security-model)
6. [Configuration Reference](#configuration-reference)
7. [Error Codes and Troubleshooting](#error-codes-and-troubleshooting)
8. [Performance Targets](#performance-targets)
9. [Database Schema](#database-schema)
10. [Wire Protocols](#wire-protocols)

---

## Architecture Overview

The GPU-as-API system connects three domains: the **device** (home GPU running `llmhosts`), the **server** (LLMHosts.com cloud relay), and the **client** (any application that speaks the OpenAI or Anthropic API).

```
 DEVICE SIDE                         SERVER SIDE                          CLIENT SIDE
 (Home Network)                      (LLMHosts.com)                       (Anywhere)

 +------------------+                +---------------------------+        +------------------+
 |  llmhosts up      |                |  LLMHosts.com Server      |        |  Any OpenAI      |
 |                  |                |                           |        |  Compatible App  |
 |  +-----------+   |   outbound     |  +--------------------+  |        |                  |
 |  | Ollama    |   |   tunnel       |  | Next.js Frontend   |  |        |  curl, Python,   |
 |  | (inference|<--+-- (CF/TS) -----+->| /api/devices/*     |  |        |  LangChain,      |
 |  |  engine)  |   |                |  | /api/auth/*        |  |        |  Continue, etc.  |
 |  +-----------+   |                |  +--------------------+  |        |                  |
 |       ^          |                |           |              |        +--------+---------+
 |       |          |                |           v              |                 |
 |  +-----------+   |                |  +--------------------+  |                 |
 |  | FastAPI   |   |   heartbeat    |  | Prisma/SQLite DB   |  |    POST /v1/chat/completions
 |  | Proxy     |<--+-- every 30s ---+->| - Users            |  |    Authorization: Bearer lh_sk_...
 |  | :4000     |   |                |  | - Devices          |  |                 |
 |  +-----------+   |                |  | - ApiKeys          |  |                 v
 |       ^          |                |  | - DeviceCodes      |  |        +--------+---------+
 |       |          |                |  +--------------------+  |        |  API Gateway     |
 |  +-----------+   |                |           |              |        |  /v1/*           |
 |  | Three-Tier|   |                |           v              |        |                  |
 |  | Router    |   |                |  +--------------------+  |        |  1. Validate key |
 |  +-----------+   |                |  | Relay Layer        |  |<-------+  2. Find devices |
 |       ^          |                |  |                    |  |        |  3. Forward req  |
 |       |          |                |  | key -> user        |  |------->|  4. Stream resp  |
 |  +-----------+   |                |  | user -> devices    |  |        |                  |
 |  | Three-Tier|   |                |  | device -> tunnel   |  |        +------------------+
 |  | Cache     |   |                |  | tunnel -> forward  |  |
 |  +-----------+   |                |  +--------------------+  |
 |                  |                |                           |
 +------------------+                +---------------------------+
```

### The Three Domains

**Device Side** -- The user's machine with a GPU. Runs `llmhosts up` which:
- Starts Ollama if not already running (inference engine)
- Starts a FastAPI proxy on port 4000 (request handling, routing, caching)
- Opens an outbound tunnel via Cloudflare Quick Tunnel or Tailscale Funnel
- Authenticates with LLMHosts.com via RFC 8628 Device Code Flow
- Registers the device (tunnel URL, GPU info, available models)
- Starts a 30-second heartbeat loop

**Server Side** -- LLMHosts.com (Next.js + Prisma/SQLite). Provides:
- Device Code authentication endpoints
- Device registration and health tracking
- API key management (keys prefixed `lh_sk_`)
- Relay layer that forwards incoming API requests to the user's device
- Web dashboard for monitoring devices, usage, and billing

**Client Side** -- Any application that speaks the OpenAI API. Clients send requests to `https://llmhosts.com/v1/chat/completions` with the user's API key. They do not need to know anything about tunnels, devices, or home networks.

---

## System Data Flow

### Flow 1: Device Startup (`llmhosts up`)

```
User runs: llmhosts up

Step 1: Start Ollama
  |-- Check if Ollama is running (GET http://127.0.0.1:11434/api/tags)
  |-- If not running, start it (subprocess: ollama serve)
  |-- Wait for health check to pass (up to 30s)
  |-- Enumerate available models
  v

Step 2: Start Local Proxy
  |-- Load config from ~/.llmhosts/config.toml
  |-- Create FastAPI app with:
  |     - POST /v1/chat/completions  (OpenAI compatible)
  |     - POST /v1/messages          (Anthropic compatible)
  |     - GET  /v1/models            (Model listing)
  |     - GET  /health               (Health check)
  |-- Initialize three-tier router (rule-based + kNN + ModernBERT)
  |-- Initialize three-tier cache (exact hash + namespace + vCache)
  |-- Start uvicorn on 127.0.0.1:4000
  v

Step 3: Open Tunnel
  |-- TunnelDetector.detect() probes for available providers:
  |     a) Check for cloudflared binary -> CloudflareInfo
  |     b) Check for tailscale binary + status -> TailscaleInfo
  |-- Select best provider (Cloudflare preferred for zero-config)
  |-- Start tunnel:
  |     - Cloudflare: cloudflared tunnel --url http://localhost:4000
  |       -> Parse stderr for https://*.trycloudflare.com URL
  |     - Tailscale: tailscale funnel 4000
  |       -> Use DNS name from tailscale status
  |-- Wait up to 30s for tunnel URL
  v

Step 4: Authenticate with LLMHosts.com (RFC 8628 Device Code Flow)
  |-- POST https://llmhosts.com/api/auth/device-code
  |     Request:  { "client_id": "llmhosts-cli" }
  |     Response: { "device_code": "...", "user_code": "ABCD-1234",
  |                 "verification_uri": "https://llmhosts.com/activate",
  |                 "expires_in": 900, "interval": 5 }
  |-- Print to terminal:
  |     "Visit https://llmhosts.com/activate and enter code: ABCD-1234"
  |-- Open browser to verification_uri (best effort)
  |-- Poll POST https://llmhosts.com/api/auth/device-code/poll
  |     Request:  { "device_code": "..." }
  |     Response: { "status": "pending" }  -- keep polling
  |     Response: { "status": "approved", "access_token": "...",
  |                 "api_key": "lh_sk_..." }  -- done
  |-- Store access_token + api_key in ~/.llmhosts/auth.json
  v

Step 5: Register Device
  |-- POST https://llmhosts.com/api/devices/register
  |     Headers: Authorization: Bearer <access_token>
  |     Body: {
  |       "name": "<hostname>",
  |       "tunnel_url": "https://abc123.trycloudflare.com",
  |       "models": ["llama3.1:70b", "codestral:latest", ...],
  |       "hardware": "NVIDIA RTX 4090 (24.0GB), 64GB RAM",
  |       "latency_ms": null
  |     }
  |     Response: { "id": "clxyz...", "status": "online" }
  |-- DeviceManager stores device locally in ~/.llmhosts/devices.json
  v

Step 6: Start Heartbeat Loop
  |-- Every 30 seconds:
  |     POST https://llmhosts.com/api/devices/heartbeat
  |     Headers: Authorization: Bearer <access_token>
  |     Body: {
  |       "device_id": "clxyz...",
  |       "status": "online",
  |       "models": ["llama3.1:70b", "codestral:latest"],
  |       "latency_ms": 2.3,
  |       "tunnel_url": "https://abc123.trycloudflare.com"
  |     }
  |-- Server updates last_seen timestamp
  |-- If 3 consecutive heartbeats missed -> device marked offline
  v

Step 7: Print Summary
  +-------------------------------------------------------+
  |  LLMHosts -- Your GPU is live!                        |
  |                                                       |
  |  Endpoint: https://llmhosts.com/v1/chat/completions   |
  |  API Key:  lh_sk_Ab...Xy                              |
  |  Models:   llama3.1:70b, codestral:latest             |
  |  Tunnel:   Cloudflare (abc123.trycloudflare.com)      |
  |  GPU:      NVIDIA RTX 4090 (24.0GB)                   |
  |                                                       |
  |  Test it:                                             |
  |  curl https://llmhosts.com/v1/chat/completions \      |
  |    -H "Authorization: Bearer lh_sk_Ab...Xy" \         |
  |    -d '{"model":"llama3.1:70b",                       |
  |         "messages":[{"role":"user",                    |
  |                      "content":"Hello"}]}'             |
  +-------------------------------------------------------+
```

### Flow 2: API Request Relay (Client -> Device)

```
Client sends:
  POST https://llmhosts.com/v1/chat/completions
  Headers:
    Authorization: Bearer lh_sk_AbCdEfGh...
    Content-Type: application/json
  Body:
    {
      "model": "llama3.1:70b",
      "messages": [{"role": "user", "content": "Explain quantum computing"}],
      "stream": true
    }

Server-side processing:
  |
  v
Step 1: API Key Validation
  |-- Query: SELECT * FROM ApiKey WHERE key = 'lh_sk_AbCdEfGh...'
  |     AND revokedAt IS NULL
  |     AND (expiresAt IS NULL OR expiresAt > now())
  |-- If not found -> 401 Unauthorized
  |-- Resolve userId from the key
  |-- Update lastUsed timestamp
  v

Step 2: Device Lookup
  |-- Query: SELECT * FROM Device WHERE userId = '<user_id>'
  |     AND status = 'online'
  |-- If no online devices -> 503 Service Unavailable
  |-- If multiple devices: select best based on:
  |     a) Model availability (does device have requested model?)
  |     b) Latency (lower latency_ms preferred)
  |     c) Last seen (more recent preferred)
  |-- Selected device has tunnel_url = 'https://abc123.trycloudflare.com'
  v

Step 3: Request Forwarding
  |-- Forward to: POST https://abc123.trycloudflare.com/v1/chat/completions
  |     Headers: (pass through Content-Type, add X-Request-ID)
  |     Body: (pass through as-is)
  |-- Timeout: 120s (matches BACKEND_TIMEOUT constant)
  v

Step 4: Response Streaming (SSE)
  |-- Device proxy receives request
  |     -> Three-tier cache check (exact hash -> namespace -> vCache)
  |     -> If cache miss: three-tier router selects backend
  |     -> Privacy check (PII detection -> IslandRun enforcement)
  |     -> Forward to local Ollama /api/chat
  |-- Ollama streams NDJSON response
  |-- Device proxy converts to OpenAI SSE format:
  |     data: {"id":"chatcmpl-...","choices":[{"delta":{"content":"Quantum"}}]}
  |     data: {"id":"chatcmpl-...","choices":[{"delta":{"content":" computing"}}]}
  |     ...
  |     data: [DONE]
  |-- SSE streams through Cloudflare tunnel back to LLMHosts.com relay
  |-- LLMHosts.com relay streams to the client
  v

Client receives: streaming SSE response identical to OpenAI API format
```

### Flow 3: Device Goes Offline

```
Scenario: User closes laptop lid or loses internet

  |-- Heartbeat fails (POST /api/devices/heartbeat times out)
  |-- Server increments missed_beats counter for this device
  |-- After 3 missed beats (90 seconds):
  |     UPDATE Device SET status = 'offline' WHERE id = '<device_id>'
  |-- Any API requests for this user now get:
  |     503 { "error": "No online devices available" }
  |     (or routed to a different online device if user has multiple)

When device comes back:
  |-- llmhosts up detects stale registration
  |-- Re-opens tunnel (gets new URL if Cloudflare Quick Tunnel)
  |-- POST /api/devices/register with new tunnel_url
  |-- Device marked online again
  |-- Heartbeat loop resumes
```

---

## Component Reference

### `llmhosts/up.py` -- Unified Startup Orchestrator

The `llmhosts up` command is the primary entry point. It orchestrates all other components into a single, seamless startup flow.

| Responsibility | Delegates To |
|---|---|
| Start Ollama if needed | `llmhosts/discovery/ollama.py` |
| Start local proxy | `llmhosts/server.py` -> `LLMHostsServer` |
| Open tunnel | `llmhosts/tunnel/manager.py` -> `TunnelManager` |
| Authenticate | `llmhosts/auth/device_code.py` (new) |
| Register device | `llmhosts/mesh/manager.py` -> `DeviceManager` |
| Heartbeat loop | `llmhosts/mesh/heartbeat.py` (new) |

**CLI Flags:**

| Flag | Type | Default | Description |
|---|---|---|---|
| `--name` | `str` | `platform.node()` | Device name for registration |
| `--port` | `int` | `4000` | Local proxy port |
| `--tunnel` | `str` | `auto` | Tunnel provider: `auto`, `cloudflare`, `tailscale` |
| `--no-browser` | `bool` | `false` | Don't auto-open browser for auth |
| `--config` | `Path` | `~/.llmhosts/config.toml` | Config file path |

### `llmhosts/tunnel/manager.py` -- Tunnel Lifecycle Orchestrator

Manages the full tunnel lifecycle: detection, startup, health monitoring, and shutdown.

**Class: `TunnelManager`**

| Method | Signature | Description |
|---|---|---|
| `__init__` | `(config_dir: Path \| None)` | Initialize with optional config directory |
| `start` | `async (config: TunnelConfig \| None) -> TunnelStatus` | Auto-detect and start best available tunnel |
| `stop` | `async () -> None` | Stop active tunnel and clean up |
| `status` | `async () -> TunnelStatus` | Query active provider for current status |
| `restart` | `async (config: TunnelConfig \| None) -> TunnelStatus` | Stop + start |

**Provider Selection Order:**
1. Explicit `config.provider` if set (not `NONE`)
2. `TunnelDetector.detect()` returns available providers ordered by preference
3. Cloudflare Quick Tunnel is preferred (zero account needed, works behind any NAT)
4. Tailscale Funnel as fallback (requires Tailscale account, but stable URL)

**Models:**

```python
class TunnelProvider(str, Enum):
    TAILSCALE = "tailscale"
    CLOUDFLARE = "cloudflare"
    NONE = "none"

class TunnelConfig(BaseModel):
    provider: TunnelProvider | None = None  # None = auto-detect
    port: int = 4000                        # 1-65535
    hostname: str | None = None             # Custom hostname for CF named tunnel
    funnel: bool = False                    # Tailscale Funnel (public) vs Serve (mesh)
    auth_required: bool = True              # Require API key for tunnel access

class TunnelStatus(BaseModel):
    provider: TunnelProvider = TunnelProvider.NONE
    active: bool = False
    url: str | None = None
    local_port: int = 4000
    started_at: datetime | None = None
    error: str | None = None
    pid: int | None = None
```

### `llmhosts/tunnel/cloudflare.py` -- Cloudflare Quick Tunnel

Manages the `cloudflared` subprocess for zero-config tunneling.

**How Quick Tunnel Works:**
1. Runs `cloudflared tunnel --url http://localhost:4000` as a subprocess
2. Monitors stderr for the `https://*.trycloudflare.com` URL (regex: `https://[a-zA-Z0-9-]+\.trycloudflare\.com`)
3. URL typically appears within 5-10 seconds
4. Timeout: 30 seconds (`_URL_WAIT_TIMEOUT`)
5. Process termination: SIGTERM first, SIGKILL after 5 seconds (`_PROCESS_KILL_TIMEOUT`)

**Key Detail:** Quick Tunnel URLs are ephemeral. Each restart of `cloudflared` generates a new random URL. This is why the device re-registers on each startup.

### `llmhosts/tunnel/tailscale.py` -- Tailscale Funnel

Uses Tailscale Funnel for stable, persistent tunnel URLs.

**Advantages over Cloudflare:**
- Stable URL tied to your tailnet DNS name
- End-to-end WireGuard encryption
- No dependency on Cloudflare infrastructure

**Disadvantages:**
- Requires a Tailscale account (free tier available)
- Funnel must be explicitly enabled in the Tailscale admin console

### `llmhosts/tunnel/auth.py` -- Tunnel Authentication

Generates and manages tunnel-level API keys (prefixed `lh-`).

**Key Format:** `lh-<32-byte-urlsafe-random-token>` (e.g., `lh-aBcDeFgHiJkLmNoPqRsTuVwXyZaBcDeFgHiJk`)

**Storage:** `~/.llmhosts/tunnel_key.json` with `0o600` file permissions (owner-only read/write).

**Verification:** Uses `secrets.compare_digest()` for constant-time comparison (timing-attack resistant).

**Note:** These `lh-` prefixed keys are the local tunnel authentication keys. They are distinct from the `lh_sk_*` API keys issued by the LLMHosts.com server for the relay system.

### `llmhosts/mesh/manager.py` -- Device Manager

Manages device registration, discovery, persistence, and health checks.

**Class: `DeviceManager`**

| Method | Signature | Description |
|---|---|---|
| `initialize` | `async () -> None` | Load device list from `~/.llmhosts/devices.json` |
| `register` | `async (DeviceRegistration) -> Device` | Register new device (checks limits, detects hardware) |
| `remove` | `async (device_id: str) -> bool` | Remove device by ID or name |
| `list_devices` | `async () -> list[Device]` | Return all registered devices |
| `get_device` | `async (device_id: str) -> Device \| None` | Get single device by ID or name |
| `get_local_device` | `async () -> Device \| None` | Return the local device entry |
| `health_check` | `async () -> MeshStatus` | Check health of all devices |
| `get_all_models` | `async () -> dict[str, list[str]]` | Model availability across all devices |
| `save` | `async () -> None` | Persist device list to disk as JSON |

**Device Limits:**

| Tier | Max Devices |
|---|---|
| Free | 1 |
| Pro ($19/mo) | 5 |
| Team ($49/seat) | 10 per seat |
| Enterprise ($50/user) | Unlimited |

**Health Check Mechanism:**
- Local devices: always considered healthy, latency = 0ms
- Remote devices: HTTP GET `{tunnel_url}/health` with 10s timeout
- Response updates: `healthy` flag, `latency_ms` measurement, `last_seen` timestamp
- If `/health` returns 200 with a `models` array, the device's model list is updated

### `llmhosts/mesh/models.py` -- Device Data Models

```python
class DeviceRole(str, Enum):
    COMPUTE = "compute"  # Has GPU, serves inference
    CLIENT = "client"    # Consumes inference only
    HYBRID = "hybrid"    # Both serves and consumes

class Device(BaseModel):
    id: str                         # UUID
    name: str                       # User-friendly (e.g., "home-desktop")
    hostname: str                   # Network hostname
    role: DeviceRole
    tunnel_url: str | None = None   # Tailscale/CF URL
    hardware_summary: str = ""      # "RTX 4090 (24GB), 64GB RAM"
    models: list[str] = []          # Ollama models available
    last_seen: datetime | None = None
    is_local: bool = False          # True if this is the current device
    healthy: bool = True
    latency_ms: float | None = None # Measured round-trip

class DeviceRegistration(BaseModel):
    name: str
    role: DeviceRole = DeviceRole.HYBRID
    tunnel_url: str | None = None

class MeshStatus(BaseModel):
    local_device: Device | None = None
    remote_devices: list[Device] = []
    total_models: int = 0
    total_gpu_vram_gb: float = 0.0
    mesh_healthy: bool = True
```

### `llmhosts/proxy/dispatcher.py` -- Backend Dispatcher

The dispatcher is the central routing engine that selects a backend and forwards requests.

**Backend Selection Hierarchy:**
1. Privacy enforcement (IslandRun PII detection -> force local if sensitive)
2. Karma-weighted ordering (Hive members with higher karma preferred)
3. Exact model match on local backends (priority 0)
4. LAN node match (priority 5, discovered via mDNS)
5. Cloud provider match via LiteLLM (priority 10)
6. Any healthy local backend as fallback

**Backend Types:**

| Type | Priority | Description |
|---|---|---|
| `ollama` (local) | 0 | Local Ollama instance at `http://127.0.0.1:11434` |
| `ollama` (LAN) | 5 | LAN nodes discovered via mDNS |
| `litellm` | 10 | Cloud APIs via LiteLLM (OpenAI, Anthropic, etc.) |

**Cloud Model Detection Prefixes:**

| Provider | Prefixes |
|---|---|
| OpenAI | `gpt-`, `o1-`, `o3-`, `chatgpt-`, `ft:gpt-`, `dall-e`, `tts-`, `whisper` |
| Anthropic | `claude-` |
| Google | `gemini-`, `gemini/` |
| Mistral | `mistral-`, `mixtral-`, `codestral-`, `open-mistral-`, `open-mixtral-` |
| Groq | `groq/` |
| Together | `together_ai/`, `together/` |
| DeepSeek | `deepseek-`, `deepseek/` |
| OpenRouter | `openrouter/` |

### `llmhosts/router/engine.py` -- Three-Tier Hybrid Router

The router evaluates requests through a prioritized rule pipeline with PII detection.

**Tier Architecture:**

| Tier | Name | Implementation | Requirements | Latency Target |
|---|---|---|---|---|
| 0 | Rule-based | `Router._default_rules()` | Always available | <1ms |
| 1 | kNN + FAISS | Embedding similarity search | `pip install llmhosts[smart]` | <15ms |
| 2 | ModernBERT | Transformer classifier | `pip install llmhosts[smart]` | <50ms |
| 3 | Qwen-0.5B | Reasoning LLM (Phase 4) | `pip install llmhosts[full]` | <500ms |

**Default Rule Priority:**

| Priority | Name | Condition | Behavior |
|---|---|---|---|
| 5 | `exact_model_match` | `model_exact` | Route to backend that has exact model |
| 10 | `prefer_local` | `prefer_local` | Prefer Ollama when model available locally |
| 20 | `cost_optimise` | `cost_optimize` | Choose cheapest option (local = $0) |
| 100 | `cloud_fallback` | `cloud_fallback` | Use cloud API when nothing local works |

**Privacy Tiers:**

| Tier | PII Types | Routing Constraint |
|---|---|---|
| `PUBLIC` | None detected | Any backend allowed |
| `PRIVATE` | Emails, phone numbers, names | Local-only routing |
| `SENSITIVE` | SSNs, credit cards, medical data | Encrypted local-only |

**Privacy Modes:**

| Mode | Behavior |
|---|---|
| `standard` | Scan for PII, enforce tiers as detected |
| `strict` | Force all requests to PRIVATE tier (local-only) |
| `paranoid` | Force all requests to SENSITIVE tier (encrypted local-only) |

### `llmhosts/cache/manager.py` -- Three-Tier Cache Manager

Orchestrates the cascade: exact hash -> namespace filter -> vCache semantic.

**Tier Architecture:**

| Tier | Name | Requirements | False Positive Rate | Latency |
|---|---|---|---|---|
| 0 | Exact Hash | Always available | 0% (exact SHA-256 match) | <1ms |
| 1 | Namespace Filter | `pip install llmhosts[smart]` | Configurable | <5ms |
| 2 | vCache Semantic | `pip install llmhosts[smart]` | delta-bounded (ICLR 2026) | <10ms |

**Cache Lookup Flow:**
1. Compute SHA-256 hash of `(model, normalized_messages)`
2. Check exact hash cache (SQLite-backed, LRU eviction)
3. If miss + smart tier available: compute embedding once
4. Check namespace filter (entity-based scoping)
5. If miss: check vCache (cosine similarity with adaptive threshold)
6. If all miss: forward to backend, then populate all available tiers

**Cache Configuration:**

| Parameter | Type | Default | Description |
|---|---|---|---|
| `enabled` | `bool` | `true` | Master cache switch |
| `max_size_mb` | `int` | `512` | Max size across all tiers |
| `ttl_seconds` | `int` | `3600` | Time-to-live for cache entries |
| `enable_namespace` | `bool` | `false` | Enable namespace filter tier |
| `enable_vcache` | `bool` | `false` | Enable vCache semantic tier |
| `delta` | `float` | `0.01` | Max false positive rate for vCache |
| `exact_max_entries` | `int` | `10,000` | Max entries in exact hash tier |
| `namespace_max_namespaces` | `int` | `500` | Max namespace buckets |
| `vcache_max_bytes` | `int` | `268,435,456` | 256 MB max for vCache embeddings |

### `llmhosts/config.py` -- Configuration System

TOML-based configuration with environment variable overrides.

**Priority (highest wins):** Environment variables > Config file > Defaults

**Config File Location:** `~/.llmhosts/config.toml`

**Environment Variable Format:** `LLMHOSTS_<SECTION>_<KEY>=<VALUE>`

Examples:
- `LLMHOST_SERVER_PORT=8080`
- `LLMHOST_ROUTER_PREFER_LOCAL=true`
- `LLMHOST_CACHE_TTL_SECONDS=7200`

### `llmhosts/proxy/models.py` -- Wire Protocol Models

The proxy supports two API dialects:

**OpenAI Dialect** (`POST /v1/chat/completions`):
- `ChatCompletionRequest` -> `UnifiedRequest` -> `UnifiedResponse` -> `ChatCompletionResponse`
- Streaming: `ChatCompletionChunk` via SSE

**Anthropic Dialect** (`POST /v1/messages`):
- `AnthropicRequest` -> `UnifiedRequest` -> `UnifiedResponse` -> `AnthropicResponse`
- Streaming: `AnthropicStreamEvent` via SSE

**Internal Unified Format:**

```python
class UnifiedRequest(BaseModel):
    model: str
    messages: list[UnifiedMessage]
    stream: bool = False
    temperature: float | None = None
    max_tokens: int | None = None
    top_p: float | None = None
    stop: list[str] | None = None
    tools: list[dict[str, Any]] | None = None
    extra: dict[str, Any] = {}

class UnifiedResponse(BaseModel):
    content: str = ""
    finish_reason: str = "stop"
    model: str = ""
    prompt_tokens: int = 0
    completion_tokens: int = 0
    total_tokens: int = 0
    tool_calls: list[dict[str, Any]] | None = None
```

---

## API Reference

### Device-Side Endpoints (Local Proxy)

These endpoints run on the user's machine at `http://127.0.0.1:4000` and are also accessible through the tunnel.

#### `POST /v1/chat/completions` -- OpenAI Chat Completion

```
Content-Type: application/json
Authorization: Bearer <tunnel_key or lh_sk_* key>

{
  "model": "llama3.1:70b",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is quantum computing?"}
  ],
  "stream": false,
  "temperature": 0.7,
  "max_tokens": 2048
}
```

**Response (non-streaming):**

```json
{
  "id": "chatcmpl-abc123def456...",
  "object": "chat.completion",
  "created": 1740268800,
  "model": "llama3.1:70b",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Quantum computing is a type of computation..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 150,
    "total_tokens": 175
  }
}
```

**Response Headers:**

| Header | Description |
|---|---|
| `X-Request-ID` | Unique request trace ID |
| `X-LLMHosts-Cost` | Estimated cost in USD (e.g., `0.000000` for local) |
| `X-LLMHosts-Model` | Actual model used (may differ from requested if aliased) |
| `X-LLMHosts-Cache` | Cache status: `HIT:exact`, `HIT:namespace`, `HIT:vcache`, or `MISS` |

#### `POST /v1/messages` -- Anthropic Messages

```
Content-Type: application/json
x-api-key: <key>
anthropic-version: 2023-06-01

{
  "model": "claude-3-opus-20240229",
  "max_tokens": 4096,
  "messages": [
    {"role": "user", "content": "Explain quantum computing."}
  ]
}
```

#### `GET /v1/models` -- List Available Models

```json
{
  "object": "list",
  "data": [
    {"id": "llama3.1:70b", "object": "model", "created": 1740268800, "owned_by": "ollama-local"},
    {"id": "codestral:latest", "object": "model", "created": 1740268800, "owned_by": "ollama-local"}
  ]
}
```

#### `GET /health` -- Health Check

```json
{
  "status": "healthy",
  "models": ["llama3.1:70b", "codestral:latest"],
  "uptime_seconds": 3600,
  "cache": {"tiers_active": ["exact", "namespace", "vcache"], "combined_hit_rate": 0.42},
  "router": {"tier": "rule-based"},
  "tunnel": {"provider": "cloudflare", "active": true}
}
```

### Server-Side Endpoints (LLMHosts.com)

These endpoints run on the LLMHosts.com server (Next.js API routes).

#### `POST /api/auth/device-code` -- Initiate Device Code Flow

See [device-code-auth.md](device-code-auth.md) for full specification.

```
Content-Type: application/json

{ "client_id": "llmhosts-cli" }
```

**Response:**

```json
{
  "device_code": "GmRh...xyz",
  "user_code": "ABCD-1234",
  "verification_uri": "https://llmhosts.com/activate",
  "verification_uri_complete": "https://llmhosts.com/activate?code=ABCD-1234",
  "expires_in": 900,
  "interval": 5
}
```

#### `POST /api/auth/device-code/poll` -- Poll for Approval

```
Content-Type: application/json

{ "device_code": "GmRh...xyz" }
```

**Responses:**

| Status | Body | Meaning |
|---|---|---|
| `200` | `{"status":"approved","access_token":"...","api_key":"lh_sk_..."}` | User approved; tokens issued |
| `200` | `{"status":"pending"}` | User hasn't acted yet; keep polling |
| `200` | `{"status":"expired"}` | Code expired; restart flow |
| `200` | `{"status":"denied"}` | User explicitly denied |
| `429` | `{"error":"slow_down"}` | Polling too fast; increase interval |

#### `POST /api/devices/register` -- Register a Device

```
Content-Type: application/json
Authorization: Bearer <access_token>

{
  "name": "home-desktop",
  "tunnel_url": "https://abc123.trycloudflare.com",
  "models": ["llama3.1:70b", "codestral:latest"],
  "hardware": "NVIDIA RTX 4090 (24.0GB), 64GB RAM",
  "latency_ms": null
}
```

**Response:**

```json
{
  "id": "clxyz123...",
  "name": "home-desktop",
  "tunnel_url": "https://abc123.trycloudflare.com",
  "models": ["llama3.1:70b", "codestral:latest"],
  "hardware": "NVIDIA RTX 4090 (24.0GB), 64GB RAM",
  "status": "online",
  "created_at": "2026-02-22T10:00:00Z"
}
```

**Error Responses:**

| Status | Cause |
|---|---|
| `401` | Missing or invalid access token |
| `409` | Device with same name already exists for this user |
| `422` | Validation error (missing required fields) |
| `429` | Device limit reached for current plan |

#### `POST /api/devices/heartbeat` -- Device Heartbeat

```
Content-Type: application/json
Authorization: Bearer <access_token>

{
  "device_id": "clxyz123...",
  "status": "online",
  "models": ["llama3.1:70b", "codestral:latest"],
  "latency_ms": 2.3,
  "tunnel_url": "https://abc123.trycloudflare.com"
}
```

**Response:**

```json
{
  "acknowledged": true,
  "server_time": "2026-02-22T10:00:30Z"
}
```

#### `DELETE /api/devices/:id` -- Unregister a Device

```
Authorization: Bearer <access_token>
```

**Response:** `204 No Content`

#### `GET /api/devices` -- List User's Devices

```
Authorization: Bearer <access_token>
```

**Response:**

```json
{
  "devices": [
    {
      "id": "clxyz123...",
      "name": "home-desktop",
      "tunnel_url": "https://abc123.trycloudflare.com",
      "models": ["llama3.1:70b", "codestral:latest"],
      "hardware": "NVIDIA RTX 4090 (24.0GB), 64GB RAM",
      "status": "online",
      "last_seen": "2026-02-22T10:00:30Z",
      "latency_ms": 2
    }
  ]
}
```

---

## Security Model

### Authentication Layers

The system has three distinct authentication layers:

```
Layer 1: Tunnel Auth (lh- prefix)
  Purpose: Protect the device's local proxy when exposed via tunnel
  Scope: Device-level, local to the user's machine
  Storage: ~/.llmhosts/tunnel_key.json (0o600 permissions)
  Verification: secrets.compare_digest() (constant-time)

Layer 2: Device Code Auth (access_token)
  Purpose: Authenticate the CLI with LLMHosts.com server
  Scope: Session-level, used for device registration and heartbeats
  Flow: RFC 8628 Device Authorization Grant
  Storage: ~/.llmhosts/auth.json (0o600 permissions)
  Lifetime: Configurable (default 24 hours, renewable)

Layer 3: API Key Auth (lh_sk_ prefix)
  Purpose: Authenticate external clients making inference requests
  Scope: User-level, used for the relay system
  Storage: Prisma/SQLite database on LLMHosts.com server
  Format: lh_sk_<32-byte-random> (e.g., lh_sk_AbCdEfGhIjKlMnOpQrStUvWxYz012345)
  Features: Named keys, expiration dates, revocation, last-used tracking
```

### Data Encryption

| Segment | Encryption | Protocol |
|---|---|---|
| Client -> LLMHosts.com | TLS 1.2/1.3 | HTTPS (Let's Encrypt) |
| LLMHosts.com -> Tunnel | TLS 1.2/1.3 | HTTPS (Cloudflare TLS or WireGuard) |
| Tunnel -> Local Proxy | Localhost | Not encrypted (loopback only) |
| Local Proxy -> Ollama | Localhost | Not encrypted (loopback only) |

**Cloudflare Tunnel encryption:** The Cloudflare Quick Tunnel creates a TLS-encrypted connection from the user's machine to Cloudflare's edge. The trycloudflare.com URL serves HTTPS.

**Tailscale encryption:** Tailscale uses WireGuard tunnels with per-connection keys, providing end-to-end encryption between the relay and the device.

### API Key Security

- Keys are generated using `secrets.token_urlsafe(32)` (cryptographically secure)
- Keys are stored hashed in the database (bcrypt)
- The full key is only shown once at creation time
- Keys can be scoped with expiration dates
- Keys can be revoked immediately
- Last-used timestamp tracked for audit
- Rate limiting per key: 60 RPM default, 10 burst

### PII Protection (IslandRun)

The router includes built-in PII detection that scans every request:

| PII Type | Detection Method | Action |
|---|---|---|
| Social Security Numbers | Regex `\d{3}-\d{2}-\d{4}` | Force SENSITIVE tier (local-only) |
| Credit Card Numbers | Luhn algorithm + regex | Force SENSITIVE tier (local-only) |
| Email Addresses | RFC 5322 regex | Force PRIVATE tier (local-only) |
| Phone Numbers | E.164 + common formats | Force PRIVATE tier (local-only) |
| Names (common) | NER heuristics | Force PRIVATE tier (local-only) |
| Medical Terms | Dictionary match | Force SENSITIVE tier (local-only) |

When PII is detected, the request is forced to a local backend regardless of model preference. If no local backend is available, the request is blocked with a clear error message explaining why.

---

## Configuration Reference

### Full `config.toml` Reference

```toml
# ~/.llmhosts/config.toml
# Env overrides: LLMHOSTS_<SECTION>_<KEY>=<VALUE>

[server]
host = "127.0.0.1"            # Bind address
port = 4000                    # Port number
workers = 1                    # Uvicorn workers
log_level = "info"             # debug, info, warning, error
cors_origins = ["*"]           # Allowed CORS origins

[ollama]
host = "http://127.0.0.1:11434"  # Ollama base URL
timeout = 120.0                    # Request timeout (seconds)
auto_discover = true               # Auto-discover Ollama on startup

[router]
prefer_local = true            # Prefer local backends over cloud
cost_optimize = true           # Optimize for cost (local = $0)
fallback_to_cloud = true       # Allow cloud fallback
enable_knn = false             # kNN tier (requires [smart])
enable_modernbert = false      # ModernBERT tier (requires [smart])
enable_qwen = false            # Qwen-0.5B tier (requires [full])

[cache]
enabled = true                 # Enable caching
max_size_mb = 512              # Max cache size
ttl_seconds = 3600             # Cache entry TTL
enable_namespace = false       # Namespace filter (requires [smart])
enable_vcache = false          # vCache semantic (requires [smart])
delta = 0.01                   # Max false positive rate
exact_max_entries = 10000      # Exact cache entry limit
namespace_max_namespaces = 500 # Namespace bucket limit
vcache_max_bytes = 268435456   # vCache max size (256 MB)

[dashboard]
enabled = true                 # Enable dashboard
tui = true                     # Enable TUI dashboard
web = true                     # Enable web dashboard

[engine]
type = "ollama"                # ollama | vllm | auto
vllm_host = "http://localhost:8000"

[security]
require_auth = false           # Require API key on /v1/* endpoints
rate_limit_rpm = 60            # Requests per minute per key/IP
rate_limit_burst = 10          # Burst allowance
cors_restrict = false          # Restrict CORS to cors_origins list

[observability]
sentry_dsn = ""                # Sentry DSN (empty = disabled)
log_format = "text"            # text | json
```

### Environment Variables

Every config option can be overridden via environment variables:

| Variable | Config Equivalent | Example |
|---|---|---|
| `LLMHOST_SERVER_HOST` | `server.host` | `0.0.0.0` |
| `LLMHOST_SERVER_PORT` | `server.port` | `8080` |
| `LLMHOST_SERVER_LOG_LEVEL` | `server.log_level` | `debug` |
| `LLMHOST_OLLAMA_HOST` | `ollama.host` | `http://192.168.1.100:11434` |
| `LLMHOST_OLLAMA_TIMEOUT` | `ollama.timeout` | `300.0` |
| `LLMHOST_ROUTER_PREFER_LOCAL` | `router.prefer_local` | `true` |
| `LLMHOST_ROUTER_ENABLE_KNN` | `router.enable_knn` | `true` |
| `LLMHOST_CACHE_ENABLED` | `cache.enabled` | `false` |
| `LLMHOST_CACHE_TTL_SECONDS` | `cache.ttl_seconds` | `7200` |
| `LLMHOST_SECURITY_REQUIRE_AUTH` | `security.require_auth` | `true` |
| `LLMHOST_SECURITY_RATE_LIMIT_RPM` | `security.rate_limit_rpm` | `120` |
| `LLMHOST_OBSERVABILITY_SENTRY_DSN` | `observability.sentry_dsn` | `https://...@sentry.io/...` |
| `LLMHOST_OBSERVABILITY_LOG_FORMAT` | `observability.log_format` | `json` |

**Boolean coercion:** `1`, `true`, `yes`, `on` -> `True`; `0`, `false`, `no`, `off` -> `False`

**List coercion:** Comma-separated values (e.g., `LLMHOST_SERVER_CORS_ORIGINS=http://localhost:3000,https://app.example.com`)

---

## Error Codes and Troubleshooting

### CLI Errors (`llmhosts up`)

| Error | Cause | Resolution |
|---|---|---|
| `No tunnel providers found` | Neither `cloudflared` nor `tailscale` is installed | Install Cloudflare: `brew install cloudflared` or download from https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/downloads/ |
| `Timed out waiting for tunnel URL` | `cloudflared` started but no URL appeared within 30s | Check firewall rules; try `cloudflared tunnel --url http://localhost:4000` manually |
| `cloudflared exited with code N` | `cloudflared` crashed | Check output for details; common cause is port already in use |
| `Ollama not reachable at http://127.0.0.1:11434` | Ollama is not running or is on a different port | Run `ollama serve` or set `LLMHOST_OLLAMA_HOST` |
| `Device limit reached (N)` | User has hit the max device count for their plan | Remove a device with `llmhosts devices remove <name>` or upgrade plan |
| `Device with name 'X' already exists` | Duplicate device name | Use `--name <unique_name>` flag or remove the existing device |
| `Failed to authenticate: code expired` | Device code expired before user approved | Restart `llmhosts up` to get a new code |

### API Errors (Client-Side)

| HTTP Status | Error Code | Cause | Resolution |
|---|---|---|---|
| `401` | `unauthorized` | Missing or invalid API key | Check `Authorization: Bearer lh_sk_...` header |
| `403` | `key_revoked` | API key has been revoked | Generate a new key in the dashboard |
| `403` | `key_expired` | API key has expired | Generate a new key or extend expiration |
| `404` | `model_not_found` | Requested model not available on any device | Check available models with `GET /v1/models` |
| `429` | `rate_limited` | Too many requests | Reduce request rate or upgrade plan |
| `500` | `internal_error` | Server-side error | Retry; check LLMHosts.com status page |
| `502` | `device_unreachable` | Tunnel to device is down | Check device status; device may have gone offline |
| `503` | `no_devices` | No online devices for this user | Run `llmhosts up` on your device |
| `504` | `device_timeout` | Device did not respond within 120s | Model may be loading; retry or check device health |

### Privacy-Related Errors

| Error | Cause | Resolution |
|---|---|---|
| `No local backend available ... contains sensitive-tier PII` | PII detected but no local Ollama available | Start Ollama locally or remove PII from request |
| `Cloud routing blocked` | Privacy mode is `strict` or `paranoid` but no local backend | Change privacy mode or start local backend |

### Heartbeat Errors

| Situation | Server Behavior | Resolution |
|---|---|---|
| 1 missed heartbeat | Device still shown as `online` | Transient; no action needed |
| 2 missed heartbeats | Device status changed to `degraded` | Check device connectivity |
| 3+ missed heartbeats | Device status changed to `offline` | Restart `llmhosts up`; check network |

---

## Performance Targets

### Latency Budget (Total: < 500ms for non-streaming first token)

| Hop | Target | Description |
|---|---|---|
| Client -> LLMHosts.com | <50ms | Client's internet to server |
| API key validation | <5ms | SQLite index lookup |
| Device lookup + selection | <10ms | SQLite query + routing logic |
| LLMHosts.com -> Tunnel entry | <50ms | Server to Cloudflare/Tailscale edge |
| Tunnel transit | <50ms | Through Cloudflare/WireGuard |
| Tunnel exit -> Local proxy | <1ms | Localhost |
| Cache lookup | <10ms | Three-tier cascade |
| Router decision | <15ms | Rule-based (Tier 0) |
| Ollama inference start | varies | Model-dependent (TTFT) |

**Total overhead (excluding inference):** < 200ms typical, < 500ms worst case

### Throughput Targets

| Metric | Target | Notes |
|---|---|---|
| Concurrent connections per device | 10 | Limited by Ollama concurrency |
| Requests per second (relay) | 100+ | Next.js API routes + SQLite |
| SSE stream throughput | Line-rate | No buffering in relay |
| Heartbeat frequency | 1 per 30s per device | Low server overhead |

### Cache Performance

| Tier | Lookup Latency | Hit Rate (steady state) |
|---|---|---|
| Exact Hash | <1ms | ~60% for repetitive workloads |
| Namespace Filter | <5ms | ~15% additional |
| vCache Semantic | <10ms | ~10% additional |
| **Combined** | **<10ms** | **~85% for typical workloads** |

### Router Performance

| Tier | Latency | VRAM Usage |
|---|---|---|
| Rule-based (Tier 0) | <1ms | 0 |
| kNN (Tier 1) | <15ms | ~50MB (FAISS index) |
| ModernBERT (Tier 2) | <50ms | ~400MB |
| Qwen-0.5B (Tier 3) | <500ms | ~500MB |
| **Total router VRAM budget** | | **<1GB** |

---

## Database Schema

The server-side database is SQLite managed by Prisma. Key tables for the GPU-as-API system:

### `Device` Table

| Column | Type | Description |
|---|---|---|
| `id` | `String` (cuid) | Primary key |
| `userId` | `String` | Foreign key to User |
| `name` | `String` | Device name (unique per user) |
| `tunnelUrl` | `String` | Tunnel URL for forwarding |
| `models` | `String` | JSON array of model names |
| `hardware` | `String` | JSON hardware summary |
| `status` | `String` | `online`, `offline`, `degraded` |
| `lastSeen` | `DateTime` | Last heartbeat timestamp |
| `latencyMs` | `Int?` | Measured latency |
| `createdAt` | `DateTime` | Registration time |
| `updatedAt` | `DateTime` | Last update time |

**Indexes:** `userId`, `status`, `UNIQUE(userId, name)`

### `DeviceCode` Table

| Column | Type | Description |
|---|---|---|
| `id` | `String` (cuid) | Primary key |
| `deviceCode` | `String` (unique) | Opaque code for CLI polling |
| `userCode` | `String` (unique) | Human-readable code (e.g., `ABCD-1234`) |
| `userId` | `String?` | Null until user approves |
| `approved` | `Boolean` | Whether user has approved |
| `apiKeyId` | `String?` | API key granted on approval |
| `expires` | `DateTime` | Code expiration time |
| `createdAt` | `DateTime` | Creation time |

**Indexes:** `deviceCode`, `userCode`

### `ApiKey` Table

| Column | Type | Description |
|---|---|---|
| `id` | `String` (cuid) | Primary key |
| `userId` | `String` | Foreign key to User |
| `name` | `String` | User-provided label |
| `key` | `String` (unique) | Full API key (`lh_sk_...`) |
| `prefix` | `String` | First 8 chars for display |
| `lastUsed` | `DateTime?` | Last usage timestamp |
| `expiresAt` | `DateTime?` | Expiration (null = never) |
| `createdAt` | `DateTime` | Creation time |
| `revokedAt` | `DateTime?` | Revocation time (null = active) |

**Indexes:** `userId`, `key`

---

## Wire Protocols

### OpenAI SSE Streaming Format

The relay preserves the exact OpenAI SSE wire format:

```
data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1740268800,"model":"llama3.1:70b","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1740268800,"model":"llama3.1:70b","choices":[{"index":0,"delta":{"content":"Quantum"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1740268800,"model":"llama3.1:70b","choices":[{"index":0,"delta":{"content":" computing"},"finish_reason":null}]}

...

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1740268800,"model":"llama3.1:70b","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]
```

### Heartbeat Protocol

```
Device -> Server (every 30 seconds):
  POST /api/devices/heartbeat
  {
    "device_id": "clxyz123...",
    "status": "online",
    "models": ["llama3.1:70b", "codestral:latest"],
    "latency_ms": 2.3,
    "tunnel_url": "https://abc123.trycloudflare.com"
  }

Server -> Device:
  200 OK
  { "acknowledged": true, "server_time": "2026-02-22T10:00:30Z" }

Failure handling:
  - Network timeout: increment missed_beats counter
  - 3 consecutive misses: mark device offline
  - On reconnect: device re-registers with fresh data
```

---

## Related Documentation

- [Device Code Authentication](device-code-auth.md) -- RFC 8628 implementation details
- [Relay Architecture](relay-architecture.md) -- Request forwarding and streaming
- [Quickstart Guide](quickstart.md) -- User-facing setup guide
- [Why LLMHosts](why-llmhosts.md) -- Value proposition and comparisons
