# POC End-to-End Test Plan: Alienware 4090 + VPS

**Date**: 2026-02-23
**Status**: Draft
**Goal**: Validate the full LLMHosts GPU-as-API experience from zero to working, using an Alienware laptop with RTX 4090 Mobile on an isolated network, routing inference through llmhosts.com.

---

## Quick-Start Card

The minimum steps from zero to "it works":

```
1.  pip install llmhosts
2.  llmhosts up
3.  Browser opens llmhosts.com/activate — sign in, enter the code shown in terminal
4.  Copy API key from the success banner (or dashboard at llmhosts.com/dashboard/keys)
5.  Point any app at https://llmhosts.com/api/relay/v1/ with your API key
```

---

## Test Environment

| Component | Details |
|-----------|---------|
| **Laptop** | Alienware, NVIDIA RTX 4090 Mobile (18GB VRAM), Windows or Linux |
| **Network** | Neighbor's wifi (isolated from home network and VPS — proves NAT traversal) |
| **VPS** | Production server at llmhosts.com (Traefik + Next.js + Python proxy) |
| **Account** | brendonurban@gmail.com — existing middle-tier (pro) subscription |
| **Tunnel** | Cloudflare Quick Tunnel (zero-config, no account needed) or Tailscale Funnel |

### Network Topology

```
[Alienware on neighbor's wifi]
    |
    |-- outbound HTTPS --> Cloudflare Quick Tunnel (*.trycloudflare.com)
    |
    v
[Internet]
    |
    v
[llmhosts.com VPS]  <---  [Any client, anywhere]
    |                         |
    |  relay request          |  POST /api/relay/v1/chat/completions
    |  via tunnel URL ------->|
    |                         |
    v                         v
[Alienware GPU runs inference, streams response back]
```

---

## Phase 1: Laptop Preparation (User Steps)

### 1.1 Python Installation

**Windows (if Python not installed):**
```powershell
winget install Python.Python.3.12
# Close and reopen terminal
python --version  # Expect: Python 3.12.x
```

**Linux (Ubuntu/Debian):**
```bash
sudo apt update && sudo apt install python3.12 python3.12-venv python3-pip
python3 --version  # Expect: Python 3.12.x
```

**Verification:**
- [ ] `python --version` or `python3 --version` shows 3.10+
- [ ] `pip --version` or `pip3 --version` works

### 1.2 Install LLMHosts

```bash
pip install llmhosts
```

**Expected behavior:**
- Downloads ~50MB (core tier: FastAPI, httpx, LiteLLM, Click, Rich, Pydantic, etc.)
- No GPU-specific dependencies in core tier
- Completes in under 60 seconds on reasonable internet

**Verification:**
- [ ] `llmhosts --version` prints `llmhosts, version 0.1.0`
- [ ] `llmhosts --help` shows command group: `serve`, `up`, `doctor`, `keys`, `setup`, `suggest-models`

### 1.3 Install Cloudflare Tunnel Client

`llmhosts up` needs either `cloudflared` or `tailscale` for NAT traversal.

**Windows:**
```powershell
winget install Cloudflare.cloudflared
```

**Linux:**
```bash
# Debian/Ubuntu
curl -fsSL https://pkg.cloudflare.com/cloudflare-main.gpg | sudo tee /usr/share/keyrings/cloudflare-main.gpg >/dev/null
echo "deb [signed-by=/usr/share/keyrings/cloudflare-main.gpg] https://pkg.cloudflare.com/cloudflared $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/cloudflared.list
sudo apt update && sudo apt install cloudflared
```

**Verification:**
- [ ] `cloudflared --version` returns a version string

### 1.4 Install Ollama (if not present)

**Windows:**
```powershell
winget install Ollama.Ollama
```

**Linux:**
```bash
curl -fsSL https://ollama.com/install.sh | sh
```

**Verification:**
- [ ] `ollama --version` works
- [ ] `ollama serve` starts (or is already running as a service)
- [ ] `ollama list` shows available models (may be empty initially)

### 1.5 GPU Detection: `llmhosts doctor`

```bash
llmhosts doctor
```

**Expected output for RTX 4090 Mobile (18GB):**

```
LLMHosts Doctor Report
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Python version     ✓  Python 3.12.x
  Config file        !  Not found (using defaults)
  Ollama             ✓  Running at http://127.0.0.1:11434, N model(s): ...
  BYOK API keys      !  No keys configured
  Proxy port         ✓  127.0.0.1:4000 available
  Installed tier     ✓  Installed: [core] — rule-based router only
  Disk space         ✓  XX.X GB free / XX.X GB total
  Hardware           ✓  NVIDIA GeForce RTX 4090 Laptop GPU (18.0GB VRAM), XXGB RAM, XX CPU cores
  Database           !  Database not found (will be created on first start)
  Cache              ✓  ...
```

**Key checks:**
- [ ] GPU detected: `NVIDIA GeForce RTX 4090 Laptop GPU` with ~18432MB VRAM (~18.0GB)
- [ ] `gpu_type` = `nvidia`
- [ ] NVIDIA driver version shown
- [ ] Ollama shown as reachable (or warning if not started)
- [ ] Python 3.10+ confirmed
- [ ] Proxy port 4000 available
- [ ] `tunnel_tools` includes `cloudflared`

### 1.6 Pull a Model

With 18GB VRAM, the model catalog recommends models up to the "large" tier (12-24GB VRAM). The best fit is `qwen2.5:14b` (requires ~14GB VRAM, leaves 4GB headroom).

```bash
ollama pull qwen2.5:14b
```

**Verification:**
- [ ] Download completes (~8.2GB)
- [ ] `ollama list` shows `qwen2.5:14b`
- [ ] Quick smoke test: `ollama run qwen2.5:14b "Say hello in one word"`

**Alternative models for this VRAM tier:**
- `llama3.1:13b` — Meta's 13B (requires ~13GB VRAM)
- `codellama:13b` — Code-focused 13B
- `qwen2.5:7b` — Faster, smaller option if 14b is too tight

### 1.7 Model Suggestions (Optional Validation)

```bash
llmhosts suggest-models
```

**Expected:** Should recommend `qwen2.5:14b`, `llama3.1:13b`, `codellama:13b` and smaller models. Should NOT recommend 70B+ models (require 40GB+ VRAM). `qwen2.5:14b` should be marked `already_installed` if pulled in step 1.6.

---

## Phase 2: First Run — `llmhosts up`

### 2.1 Launch Command

```bash
llmhosts up
```

**Optional flags:**
```bash
llmhosts up --port 8080              # Custom local port
llmhosts up --server https://llmhosts.com  # Default, explicit
llmhosts up --name "alienware-4090"   # Custom device name
```

### 2.2 Expected 5-Phase Output

The `LLMHostUp.run()` orchestrator executes these phases in sequence:

**Phase A: Ollama Check**
```
LLMHosts — Your GPU, Everywhere

Checking Ollama...
  Ollama running — 1 model(s): qwen2.5:14b
```

If Ollama is NOT running:
```
  Ollama not detected. Please start Ollama first:
    ollama serve
```

**Phase B: Tunnel Establishment**
```
Starting tunnel...
  Tunnel active via Cloudflare: https://random-words.trycloudflare.com
```

The tunnel starts `cloudflared tunnel --url http://localhost:4000` as a subprocess. It parses stderr for the `*.trycloudflare.com` URL (up to 30 second timeout).

Possible issues:
- Firewall blocking outbound HTTPS on port 443
- `cloudflared` not installed (shown in doctor)
- Neither cloudflared nor tailscale detected: error with install instructions

**Phase C: Authentication (Device Code Flow — RFC 8628)**
```
Authenticate with LLMHosts.com
┌───────────────────────────────────────────────────────┐
│                                                       │
│  Open  https://llmhosts.com/activate                  │
│  Enter code:  ABCD-1234                               │
│                                                       │
└───────────────────────────────────────────────────────┘
Waiting for authorization...
```

The flow:
1. CLI calls `POST /api/auth/device-code` → gets `device_code`, `user_code`, `verification_uri`
2. Browser auto-opens `https://llmhosts.com/activate`
3. User signs in (if not already), enters the code (e.g. `ABCD-1234`)
4. Server calls `POST /api/auth/device-code/approve` linking device code to user
5. CLI polls `POST /api/auth/device-code/poll` every 5 seconds
6. On approval: server generates `lh_sk_*` API key, returns credentials
7. CLI saves credentials to `~/.llmhosts/credentials.json` (mode 0600)

If user already has saved credentials at `~/.llmhosts/credentials.json`:
```
  Authenticated as brendonurban@gmail.com (pro plan)
```
(Skips device code flow, validates existing API key by calling `GET /api/devices`)

**Phase D: Device Registration**
```
Registering device...
  Device registered as "HOSTNAME"
```

CLI calls `POST /api/devices` with:
```json
{
  "name": "alienware-4090",
  "tunnel_url": "https://random-words.trycloudflare.com",
  "models": ["qwen2.5:14b"],
  "hardware": "NVIDIA GeForce RTX 4090 Laptop GPU (18.0GB), 32GB RAM"
}
```

Server upserts device record (by userId + name). Device limit for pro plan: 5 devices.

**Phase E: Success Banner**
```
┌─────────────────────────────────────────────────────────┐
│                                                         │
│  Your GPU is live!                                      │
│                                                         │
│  API Endpoint:  https://llmhosts.com/api/relay/v1       │
│  API Key:       lh_sk_abc123def456...                   │
│  Device:        alienware-4090                          │
│  Tunnel:        https://random-words.trycloudflare.com  │
│                                                         │
│  Try it:                                                │
│    curl https://llmhosts.com/api/relay/v1/chat/...      │
│                                                         │
│  Or set environment variables:                          │
│    export OPENAI_BASE_URL=https://llmhosts.com/...      │
│    export OPENAI_API_KEY=lh_sk_...                      │
│                                                         │
└─────────────────────────────────────────────────────────┘

Press Ctrl+C to disconnect.
```

After success:
- Local FastAPI proxy runs on `0.0.0.0:4000` (bound to all interfaces for tunnel access)
- Heartbeat loop sends `POST /api/devices/heartbeat` every 30 seconds
- Heartbeat includes current model list (detects new pulls within ~30s)

### 2.3 Verification Checklist

- [ ] All 5 phases complete without error
- [ ] Tunnel URL is a valid `*.trycloudflare.com` URL (or Tailscale Funnel URL)
- [ ] Browser opened to `/activate` page
- [ ] Device code was accepted on the website
- [ ] Credentials saved to `~/.llmhosts/credentials.json`
- [ ] API key starts with `lh_sk_`
- [ ] Success banner shows correct API endpoint, device name, tunnel URL
- [ ] Process stays running (not exiting)
- [ ] Terminal shows no error output after success

---

## Phase 3: VPS-Side Validation (Remote Tests)

These tests run from any machine that is NOT the Alienware (the VPS, another computer, or even a phone). They prove the full relay path works.

### 3.1 Device Registration Verification

```bash
# From VPS or any machine with the API key
curl -s https://llmhosts.com/api/devices \
  -H "Authorization: Bearer lh_sk_YOUR_API_KEY" | python3 -m json.tool
```

**Expected response:**
```json
{
  "devices": [
    {
      "id": "...",
      "name": "alienware-4090",
      "tunnel_url": "https://random-words.trycloudflare.com",
      "models": ["qwen2.5:14b"],
      "hardware": "NVIDIA GeForce RTX 4090 Laptop GPU (18.0GB), 32GB RAM",
      "status": "online",
      "last_seen": "2026-02-23T...",
      "latency_ms": null
    }
  ]
}
```

**Checks:**
- [ ] Device appears with status `online`
- [ ] `tunnel_url` matches what was printed in the success banner
- [ ] `models` array includes `qwen2.5:14b`
- [ ] `last_seen` is within the last 60 seconds
- [ ] `hardware` string includes `4090` and `18` (GB)

### 3.2 Model List (OpenAI-Compatible)

```bash
curl -s https://llmhosts.com/api/relay/v1/models \
  -H "Authorization: Bearer lh_sk_YOUR_API_KEY" | python3 -m json.tool
```

**Expected response:**
```json
{
  "object": "list",
  "data": [
    {
      "id": "qwen2.5:14b",
      "object": "model",
      "created": 1740307200,
      "owned_by": "llmhosts-local"
    }
  ]
}
```

**Checks:**
- [ ] Returns OpenAI-compatible model list format
- [ ] `qwen2.5:14b` appears in the list
- [ ] `owned_by` is `llmhosts-local`

### 3.3 Non-Streaming Chat Completion

```bash
curl -s https://llmhosts.com/api/relay/v1/chat/completions \
  -H "Authorization: Bearer lh_sk_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:14b",
    "messages": [{"role": "user", "content": "What is 2+2? Answer in one word."}],
    "stream": false,
    "max_tokens": 10
  }' | python3 -m json.tool
```

**Expected response (OpenAI-compatible format):**
```json
{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "created": 1740307200,
  "model": "qwen2.5:14b",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Four"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": ...,
    "completion_tokens": ...,
    "total_tokens": ...
  }
}
```

**Checks:**
- [ ] HTTP 200 response
- [ ] Response contains a coherent answer
- [ ] Response is OpenAI-compatible JSON format
- [ ] `model` field matches requested model
- [ ] Response time < 10 seconds (first request may cold-load model into VRAM)
- [ ] Subsequent requests < 3 seconds (model already loaded)

### 3.4 Streaming Chat Completion

```bash
curl -N https://llmhosts.com/api/relay/v1/chat/completions \
  -H "Authorization: Bearer lh_sk_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:14b",
    "messages": [{"role": "user", "content": "Write a haiku about GPUs."}],
    "stream": true,
    "max_tokens": 50
  }'
```

**Expected: SSE stream of `data:` lines:**
```
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"delta":{"role":"assistant"},...}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"delta":{"content":"Silicon"},...}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"delta":{"content":" dreams"},...}]}

...

data: [DONE]
```

**Checks:**
- [ ] Tokens stream incrementally (not all at once)
- [ ] Content-Type header is `text/event-stream`
- [ ] Each chunk is valid JSON
- [ ] Stream ends with `data: [DONE]`
- [ ] `X-LLMHosts-Device` response header present
- [ ] `X-LLMHosts-Relay: true` header present

### 3.5 Response Headers

For any successful relay response:
- [ ] `X-LLMHosts-Device`: device name (e.g., `alienware-4090`)
- [ ] `X-LLMHosts-Relay`: `true`
- [ ] Content-Type: `application/json` (non-streaming) or `text/event-stream` (streaming)

### 3.6 Error Cases

**No API key:**
```bash
curl -s https://llmhosts.com/api/relay/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen2.5:14b", "messages": [{"role": "user", "content": "hi"}]}'
```
- [ ] Returns HTTP 401 with `authentication_error`

**Invalid API key:**
```bash
curl -s https://llmhosts.com/api/relay/v1/chat/completions \
  -H "Authorization: Bearer lh_sk_invalid_key_here" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen2.5:14b", "messages": [{"role": "user", "content": "hi"}]}'
```
- [ ] Returns HTTP 401 with `authentication_error`

**Model not available:**
```bash
curl -s https://llmhosts.com/api/relay/v1/chat/completions \
  -H "Authorization: Bearer lh_sk_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "hi"}]}'
```
- [ ] Returns error (device may still try; depends on model matching logic)

**Device offline (after Ctrl+C on laptop):**
```bash
curl -s https://llmhosts.com/api/relay/v1/chat/completions \
  -H "Authorization: Bearer lh_sk_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen2.5:14b", "messages": [{"role": "user", "content": "hi"}]}'
```
- [ ] Returns HTTP 503 with `device_error`: "No online devices" or "Device appears offline"

### 3.7 Direct Tunnel Access (Bypass Relay)

For debugging, test the tunnel directly:

```bash
# Hit the device's tunnel URL directly (only works if tunnel is public)
curl -s https://random-words.trycloudflare.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:14b",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": false
  }'
```

**Checks:**
- [ ] Tunnel URL is reachable from outside
- [ ] Returns inference response directly from the laptop's proxy
- [ ] Response includes `X-Request-ID` header (from local proxy)

---

## Phase 4: API Key Experience (OpenRouter-like)

### 4.1 Where to Get the API Key

Three sources:
1. **Success banner** — printed after `llmhosts up` completes
2. **Credentials file** — `~/.llmhosts/credentials.json` (on the laptop)
3. **Dashboard** — `https://llmhosts.com/dashboard/keys` (web UI)

### 4.2 Using with the OpenAI Python SDK

```python
from openai import OpenAI

client = OpenAI(
    base_url="https://llmhosts.com/api/relay/v1",
    api_key="lh_sk_YOUR_API_KEY",
)

response = client.chat.completions.create(
    model="qwen2.5:14b",
    messages=[{"role": "user", "content": "Explain quantum computing in 2 sentences."}],
    max_tokens=100,
)
print(response.choices[0].message.content)
```

**Checks:**
- [ ] OpenAI SDK works with zero code changes (just base_url + api_key)
- [ ] Response is a valid ChatCompletion object
- [ ] Streaming variant works: `stream=True`

### 4.3 Using with curl

```bash
export OPENAI_BASE_URL=https://llmhosts.com/api/relay/v1
export OPENAI_API_KEY=lh_sk_YOUR_API_KEY

curl "$OPENAI_BASE_URL/chat/completions" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen2.5:14b", "messages": [{"role": "user", "content": "Hi"}]}'
```

### 4.4 Using with JavaScript/TypeScript

```javascript
import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'https://llmhosts.com/api/relay/v1',
  apiKey: 'lh_sk_YOUR_API_KEY',
});

const response = await client.chat.completions.create({
  model: 'qwen2.5:14b',
  messages: [{ role: 'user', content: 'Hello from JS!' }],
});
console.log(response.choices[0].message.content);
```

### 4.5 Using with Continue.dev (VS Code)

In `~/.continue/config.yaml` or VS Code settings:

```yaml
models:
  - title: "LLMHosts (Qwen 14B)"
    provider: openai
    model: qwen2.5:14b
    apiBase: https://llmhosts.com/api/relay/v1
    apiKey: lh_sk_YOUR_API_KEY
```

**Checks:**
- [ ] Continue.dev connects successfully
- [ ] Code completions work
- [ ] Chat sidebar works

### 4.6 Using with Open WebUI

Set environment variables when launching Open WebUI:

```bash
OPENAI_API_BASE_URL=https://llmhosts.com/api/relay/v1 \
OPENAI_API_KEY=lh_sk_YOUR_API_KEY \
open-webui serve
```

Or configure in the Open WebUI admin panel under Connections:
- URL: `https://llmhosts.com/api/relay/v1`
- API Key: `lh_sk_YOUR_API_KEY`

**Checks:**
- [ ] Models list populates (calls `/models`)
- [ ] Chat works with streaming

---

## Phase 5: Resilience Tests

### 5.1 Kill Ollama While Running

```bash
# On the laptop:
# 1. With llmhosts up running, kill Ollama
pkill ollama                    # Linux
taskkill /im ollama.exe /f      # Windows

# 2. Wait 30 seconds (next heartbeat)
# 3. Check heartbeat from VPS
curl -s https://llmhosts.com/api/devices \
  -H "Authorization: Bearer lh_sk_YOUR_API_KEY"
```

**Expected:**
- [ ] Heartbeat still sends (tunnel stays up, proxy stays running)
- [ ] Model list in heartbeat becomes `[]` (Ollama is down)
- [ ] API requests return errors from the local proxy (Ollama connection refused)
- [ ] Device stays `online` on the server (proxy and tunnel are separate from Ollama)

```bash
# 4. Restart Ollama
ollama serve
# 5. Wait 30 seconds (next heartbeat re-discovers models)
# 6. API requests work again
```

- [ ] Model list repopulates after restart
- [ ] Inference works again without restarting `llmhosts up`

### 5.2 Pull a New Model While Running

```bash
# On the laptop, with llmhosts up running:
ollama pull llama3.2:8b
```

**Expected:**
- [ ] Within ~30 seconds (next heartbeat), the new model appears in device registration
- [ ] `GET /api/relay/v1/models` shows both `qwen2.5:14b` and `llama3.2:8b`
- [ ] Can immediately use the new model via relay:
  ```bash
  curl https://llmhosts.com/api/relay/v1/chat/completions \
    -H "Authorization: Bearer lh_sk_YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model": "llama3.2:8b", "messages": [{"role": "user", "content": "Hi"}]}'
  ```

### 5.3 Disconnect Wifi Briefly

```bash
# On the laptop:
# 1. Disconnect from wifi (pull the cable or disable adapter)
# 2. Wait 15 seconds
# 3. Reconnect to wifi
```

**Expected:**
- [ ] Cloudflare Quick Tunnel may drop (URL changes on reconnect)
- [ ] If tunnel survives (Tailscale is more resilient): requests resume automatically
- [ ] If tunnel dies: `llmhosts up` needs to be restarted (known limitation of Quick Tunnel)
- [ ] Server marks device as `offline` after 90 seconds without heartbeat

### 5.4 Laptop Sleep/Wake

```bash
# 1. Close laptop lid or trigger sleep
# 2. Wait 30 seconds
# 3. Open laptop lid / wake
```

**Expected:**
- [ ] Cloudflare tunnel subprocess dies or URL becomes stale on wake
- [ ] After wake: `llmhosts up` process may still be running but tunnel is dead
- [ ] Known behavior: user must `Ctrl+C` and re-run `llmhosts up` after sleep/wake
- [ ] Server marks device `offline` after 90 seconds of missed heartbeats

### 5.5 Ctrl+C Graceful Shutdown

```bash
# On the laptop: press Ctrl+C while llmhosts up is running
```

**Expected:**
```
Shutting down...
Disconnected. Your GPU is no longer accessible remotely.
```

**Checks:**
- [ ] Heartbeat loop cancels
- [ ] Device marked `offline` on server (calls `POST /api/devices/heartbeat` with `status: "offline"`)
- [ ] Tunnel process (`cloudflared`) is terminated (SIGTERM then SIGKILL if needed)
- [ ] Server-side device status changes to `offline` within seconds
- [ ] Subsequent API calls return 503 "No online devices"

### 5.6 Concurrent Requests

From the VPS, fire 5 requests simultaneously:

```bash
for i in $(seq 1 5); do
  curl -s https://llmhosts.com/api/relay/v1/chat/completions \
    -H "Authorization: Bearer lh_sk_YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model": "qwen2.5:14b", "messages": [{"role": "user", "content": "Count to 3"}], "max_tokens": 20}' &
done
wait
```

**Expected:**
- [ ] All 5 requests complete (Ollama handles queuing)
- [ ] Responses may be slower due to GPU contention
- [ ] No 502/503 errors (tunnel handles concurrent connections)

---

## Phase 6: Metrics & Dashboard

### 6.1 TUI Dashboard (Local)

While `llmhosts up` is running, the local proxy serves at `http://localhost:4000`.

Check endpoints:
- [ ] `http://localhost:4000/health` returns health status
- [ ] `http://localhost:4000/v1/models` lists models
- [ ] `http://localhost:4000/docs` shows FastAPI Swagger UI

### 6.2 Web Dashboard

Visit `https://llmhosts.com/dashboard` while signed in as brendonurban@gmail.com.

**Checks:**
- [ ] Dashboard loads without errors
- [ ] Spending page shows usage data (if any requests have been made)
- [ ] Keys page (`/dashboard/keys`) shows the `lh_sk_*` API key created during device auth
- [ ] Settings page loads with current plan info

### 6.3 Token Counting

After making several inference requests, verify:

```bash
# Make a request with a longer response
curl -s https://llmhosts.com/api/relay/v1/chat/completions \
  -H "Authorization: Bearer lh_sk_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:14b",
    "messages": [{"role": "user", "content": "Write a 100-word essay about neural networks."}],
    "max_tokens": 200
  }'
```

- [ ] Response includes `usage` object with `prompt_tokens`, `completion_tokens`, `total_tokens`
- [ ] Token counts are reasonable (not zero, not absurdly high)

---

## Troubleshooting Guide

### Firewall Blocking Tunnel

**Symptoms:** "Timed out waiting for tunnel URL" or "cloudflared exited with code 1"

**Fix:**
```bash
# Verify outbound HTTPS works
curl -s https://api.cloudflare.com/client/v4/ | head -1

# If blocked, try Tailscale instead
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up
llmhosts up  # Will auto-detect Tailscale
```

### Ollama Not Finding GPU

**Symptoms:** Doctor shows "No GPU detected" or Ollama runs on CPU only.

**Fix (NVIDIA):**
```bash
# Verify NVIDIA driver
nvidia-smi

# If nvidia-smi not found, install drivers:
# Ubuntu:
sudo apt install nvidia-driver-550
sudo reboot

# Verify Ollama sees GPU
ollama run qwen2.5:14b "Hi"
# Check: should show GPU utilization in nvidia-smi during inference
```

**Fix (Windows):**
```powershell
# Ensure latest NVIDIA drivers from geforce.com
# Ollama should auto-detect GPU on Windows
nvidia-smi.exe  # Verify in PowerShell
```

### Authentication Failures

**Symptoms:** "Auth failed" during `llmhosts up`

**Possible causes:**
1. **Not signed up**: Register at https://llmhosts.com/register (allowlisted emails only during POC)
2. **Email not verified**: Check inbox for verification email
3. **Code expired**: Device code expires after 15 minutes. Run `llmhosts up` again for a new code.
4. **Network issue**: `curl https://llmhosts.com/api/auth/device-code -X POST` should return 200
5. **Stale credentials**: Delete `~/.llmhosts/credentials.json` and re-authenticate

```bash
# Reset auth state
rm ~/.llmhosts/credentials.json
llmhosts up
```

### Tunnel Not Establishing

**Symptoms:** "No tunnel providers found" or "Failed to start cloudflared"

**Fix:**
```bash
# Verify cloudflared is installed and on PATH
which cloudflared    # Linux/macOS
where cloudflared    # Windows

# Test cloudflared directly
cloudflared tunnel --url http://localhost:4000
# Should print a *.trycloudflare.com URL within 10 seconds

# If using Tailscale:
tailscale status     # Should show "Online"
tailscale funnel 4000  # Test funnel directly
```

### Device Not Appearing Online

**Symptoms:** API returns "No online devices" even though `llmhosts up` is running

**Possible causes:**
1. **Heartbeat failing**: Check laptop terminal for heartbeat errors
2. **Stale device**: Device marked offline if no heartbeat in 90 seconds
3. **Different API key**: The dashboard key and CLI key must belong to the same user
4. **Clock skew**: Ensure laptop clock is accurate (`timedatectl status` on Linux)

```bash
# Debug: manually check heartbeat
curl -s https://llmhosts.com/api/devices \
  -H "Authorization: Bearer lh_sk_YOUR_API_KEY"
# Check last_seen timestamp — should be within 60 seconds
```

### Relay Returns 502 Bad Gateway

**Symptoms:** `Failed to reach device "...": connection refused` or timeout

**Possible causes:**
1. **Tunnel died**: Cloudflare Quick Tunnel URL changed or process crashed
2. **Local proxy not running**: `llmhosts up` exited or crashed
3. **Firewall on laptop**: Local firewall blocking inbound on port 4000 via tunnel

```bash
# On laptop: verify proxy is running
curl http://localhost:4000/health

# Verify tunnel is still active (from any machine)
curl https://random-words.trycloudflare.com/health
```

### Slow Inference

**Symptoms:** Requests take > 30 seconds

**Possible causes:**
1. **Cold model load**: First request loads model into VRAM (~5-15 seconds)
2. **CPU offload**: Model too large for VRAM, falling back to CPU/RAM
3. **Wrong model**: Using a 70B model on 18GB VRAM will be very slow
4. **Thermal throttling**: Laptop GPU throttling under sustained load

```bash
# On laptop: monitor GPU during inference
watch -n 1 nvidia-smi
# Check: GPU utilization should be high, temperature under 90C
# Check: memory used should match model size (~14GB for qwen2.5:14b)
```

---

## Success Criteria

The POC is successful when ALL of the following are true:

1. [ ] `pip install llmhosts && llmhosts up` completes the full flow without errors
2. [ ] GPU is detected as NVIDIA RTX 4090 with ~18GB VRAM
3. [ ] Tunnel establishes through NAT (neighbor's wifi, no port forwarding)
4. [ ] Device code authentication completes successfully
5. [ ] Device appears as `online` in the server's device registry
6. [ ] `POST /api/relay/v1/chat/completions` returns coherent inference from the laptop's GPU
7. [ ] Streaming responses work (SSE)
8. [ ] OpenAI Python SDK works as a drop-in replacement
9. [ ] Model list endpoint returns the laptop's models
10. [ ] Graceful shutdown marks device offline
11. [ ] New model pulls are detected within 30 seconds

---

## Test Execution Log

| # | Test | Result | Notes | Time |
|---|------|--------|-------|------|
| 1.1 | Python version | | | |
| 1.2 | pip install llmhosts | | | |
| 1.3 | cloudflared installed | | | |
| 1.4 | Ollama installed | | | |
| 1.5 | llmhosts doctor GPU detection | | | |
| 1.6 | ollama pull qwen2.5:14b | | | |
| 2.1 | llmhosts up completes | | | |
| 2.2 | All 5 phases pass | | | |
| 3.1 | Device registered (GET /api/devices) | | | |
| 3.2 | Model list (/api/relay/v1/models) | | | |
| 3.3 | Non-streaming completion | | | |
| 3.4 | Streaming completion | | | |
| 3.5 | Response headers correct | | | |
| 3.6 | Error cases (401, 503) | | | |
| 3.7 | Direct tunnel access | | | |
| 4.2 | OpenAI Python SDK | | | |
| 4.5 | Continue.dev | | | |
| 5.1 | Kill Ollama + recovery | | | |
| 5.2 | Pull new model detection | | | |
| 5.3 | Wifi disconnect/reconnect | | | |
| 5.5 | Ctrl+C graceful shutdown | | | |
| 5.6 | Concurrent requests | | | |
| 6.1 | Local proxy endpoints | | | |
| 6.2 | Web dashboard | | | |
| 6.3 | Token counting | | | |
