# GPU-as-API in 30 Seconds -- Design Document

## META
- created: 2026-02-22
- status: APPROVED
- scope: MVP E2E proof -- home GPU accessible via LLMHosts.com API
- success: `llmhosts up` on laptop with 4090 behind hotspot NAT, inference reachable from external client via LLMHosts.com API endpoint

## Problem

LLMHosts.com has a complete CLI proxy, three-tier router, three-tier cache, tunnel system, mesh/device models, Next.js dashboard with auth and billing, and Stripe live. But **the integration layer that connects a user's GPU to the cloud API is not wired**. Specifically:

1. `llmhosts serve` and `llmhosts tunnel` are separate commands -- no unified "just works" flow
2. Device registration exists (`llmhosts/mesh/manager.py`) but isn't exposed via server API routes
3. BackendDispatcher discovers local Ollama and LAN nodes but has no remote device routing
4. Frontend API keys (`lh_sk_*` in Prisma/SQLite) and proxy keys (`lmh_*` in JSON) are disconnected
5. No published PyPI package -- users can't `pip install llmhosts`

## Design

### The UX: One Command

```bash
pip install llmhosts
llmhosts up
```

That's it. `llmhosts up` does everything:

1. **Starts Ollama proxy** on `localhost:4000`
2. **Opens outbound tunnel** (auto-detects Cloudflare Quick Tunnel or Tailscale Funnel)
3. **Authenticates with LLMHosts.com** via OAuth 2.0 Device Code Flow (RFC 8628)
4. **Registers this device** with the server (tunnel URL, GPU info, available models)
5. **Prints the user's API endpoint and key**

### Architecture

```
[Laptop with 4090]                    [LLMHosts.com Server]              [Any Client]

llmhosts up
  |-> ollama serve (if needed)
  |-> llmhosts proxy on :4000
  |-> cloudflared tunnel            ------>  Device Registry
  |     (outbound, NAT-safe)                   |
  |-> POST /api/devices/register    ------>  Stores device record
  |     {tunnel_url, models, gpu}              |
  |                                            v
  |                                  API Gateway (/v1/chat/completions)
  |                                    |
  |                                    | User's API key -> lookup user
  |                                    | -> find user's devices
  |                                    | -> select best device
  |                                    | -> forward request via tunnel
  |                                    |
  |   <----- inference request --------|                    <---- POST /v1/chat/completions
  |   -----> inference response ------>|                    ----> response
```

### Key Flows

#### Flow 1: `llmhosts up` (Device Side)

```
1. Start Ollama if not running (auto-detect)
2. Start FastAPI proxy on :4000
3. Open tunnel:
   a. Try Cloudflare Quick Tunnel (zero account needed)
   b. Fallback to Tailscale Funnel
   c. Fallback to manual URL entry
4. Auth flow:
   a. POST /api/auth/device-code -> get device_code + user_code + verification_uri
   b. Print: "Visit https://llmhosts.com/activate and enter code: ABCD-1234"
   c. Poll /api/auth/device-code/poll until approved
   d. Receive JWT + API key
5. Register device:
   a. POST /api/devices/register with tunnel_url, models, hardware, auth token
   b. Server stores in DB, marks device as online
6. Heartbeat loop:
   a. Every 30s, POST /api/devices/heartbeat with health, latency, model list
   b. Server updates last_seen, marks offline if 3 missed beats
7. Print:
   "Your GPU is live at https://llmhosts.com/v1/chat/completions"
   "API Key: lh_sk_..."
   "Models: llama3.1:70b, codestral:latest"
```

#### Flow 2: API Request (Client Side)

```
1. Client POSTs to https://llmhosts.com/v1/chat/completions
   Headers: Authorization: Bearer lh_sk_...
2. API Gateway:
   a. Validate API key -> resolve to user_id
   b. Query user's online devices
   c. Router selects best device (model availability, latency, load)
   d. Forward request to device's tunnel_url/v1/chat/completions
   e. Stream response back to client
3. Device receives request on its local proxy:
   a. Routes to local Ollama
   b. Streams response back through tunnel
```

### What Makes This Transformative

This is NOT another tunnel product or Ollama wrapper. This is:

1. **Home GPU becomes a cloud API** -- same interface as OpenAI, zero infrastructure knowledge needed
2. **Works behind any NAT** -- cellphone hotspot, corporate firewall, double NAT -- doesn't matter
3. **Intelligent routing** -- three-tier hybrid router (rule-based + kNN + ModernBERT) picks the best model on the best device automatically
4. **Verified semantic cache** -- identical/similar prompts served from cache with formal correctness guarantees (ICLR 2026)
5. **Drop-in replacement** -- change one env var (`OPENAI_BASE_URL`) and every tool that speaks OpenAI API now uses your GPU
6. **Multi-device mesh** -- add more GPUs, they all register. Router load-balances across them.
7. **Privacy-first** -- IslandRun detects PII and routes to local-only. Your data never leaves unless you say so.

No product does all 7 simultaneously.

## MVP Build Items

| # | Item | What | Files | Estimate |
|---|------|------|-------|----------|
| 1 | `llmhosts up` command | Unified CLI: serve + tunnel + auth + register | `llmhosts/cli.py`, new `llmhosts/up.py` | Core |
| 2 | Device Code Auth Flow | RFC 8628 OAuth device flow for CLI auth | `frontend/src/app/api/auth/device-code/`, `llmhosts/auth/` | Core |
| 3 | Device Registration API | `POST /api/devices/register`, heartbeat, health | `frontend/src/app/api/devices/`, Prisma schema | Core |
| 4 | Dispatcher Remote Routing | BackendDispatcher forwards to remote devices via tunnel | `llmhosts/proxy/dispatcher.py`, `llmhosts/server.py` | Core |
| 5 | API Key Bridge | Proxy validates `lh_sk_*` keys against frontend DB | `llmhosts/proxy/auth.py` or server-side middleware | Core |

### Not in MVP (Phase 3+)
- Multi-device load balancing (MVP: single device per user)
- Embedded WebSocket relay (MVP: Cloudflare/Tailscale tunnels)
- Team device sharing (MVP: personal devices only)
- Karma system for GPU pooling

## Test Plan

1. Install `llmhosts` from local wheel on laptop with 4090
2. Connect laptop to cellphone hotspot (real-world NAT)
3. Run `llmhosts up`
4. From a completely separate machine/tool, call `https://llmhosts.com/v1/chat/completions` with the API key
5. Verify inference runs on the 4090 and response streams back
6. Verify cache works on repeated prompts
7. Verify device shows as online in web dashboard

## Success Criteria

- [ ] `pip install llmhosts && llmhosts up` works on fresh machine
- [ ] Device registers and shows online in dashboard
- [ ] External API call routes to home GPU through tunnel
- [ ] Streaming responses work end-to-end
- [ ] Works behind cellphone hotspot NAT
- [ ] Total time from install to working API: < 60 seconds
