# P5 External Connectivity — Full Runbook (Scenario Reference)

**Master guide (single source of truth):** **`docs/testing/P5-full-test-guide.md`** — site readiness (Part A), full Alienware battery (Part B), recap and failure loop (Part C), canonical URLs and credentials. Use it for execution and for the datetime-stamped recap.

**This runbook** is an expanded scenario reference and codebase reference. It does **not** override the master guide. For step-by-step execution, credentials, and failure handling, follow the master guide.

**Source**: GitHub Issue #255 (P0-blocker).  
**Goal**: Test all scenarios so connectivity and inference work; validate in llmhosts.com; capture results with timestamps.  
**Runners**: AI or human. Execution order and record format: see master guide.

### How to use (AI vs human)

- **AI runner (e.g. Claude Coze)**: Execute steps in order. For "On laptop": use the machine you control (PC/laptop). For "In llmhosts.com": use the browser to open the given URLs and validate. **Record every step** in the results file with **timestamp (UTC)** and pass/fail. If something fails, capture the exact error and response in the results file so another agent can research and fix it.
- **Human backup**: Follow the same steps manually; fill the results file (copy from `docs/testing/P5-external-connectivity-results-template.md`). Timestamps are required so logs and failures can be correlated for debugging.

---

## 0. Before you start

### 0.1 Result capture (required)

- **Create a results file** for this run: copy `docs/testing/P5-external-connectivity-results-template.md` to `docs/testing/P5-external-connectivity-results-YYYY-MM-DD.md` (use today’s date).
- **For every step**: record **Step ID**, **Timestamp (UTC, ISO 8601)**, **Where** (Laptop | llmhosts.com | Server), **Action**, **Expected**, **Actual**, **Pass/Fail**, and **Notes** (or screenshot path). Example: `2026-02-25T14:03:22Z`.
- **If a step fails**: in the results file, under "Failures (detail for debugging)", paste the exact error, response body, and timestamp so another agent or human can research and fix it.
- Timestamps tie the run to server/proxy logs and make it possible to piece together and debug.

### 0.2 Roles and terms

| Term | Meaning |
|------|--------|
| **Server** | Machine running the LLMHosts proxy (and optionally Ollama/backends). Reachable via tunnel. |
| **Tunnel** | Tailscale or Cloudflare exposing the server (or the laptop when it’s the backend). Produces a **tunnel URL** (e.g. `https://myhost.tail1234.ts.net`). |
| **Laptop** | Test machine (e.g. Alienware). Can act as **client only** (nothing on it) or as **backend** (Ollama + `llmhosts serve`). |
| **llmhosts.com** | The SaaS site (browser). Dashboard at https://llmhosts.com/dashboard (login required). Uses the proxy for topology, backends, metrics. |

### 0.3 Codebase reference (for AI runners)

- Proxy **health**: `GET /health` → `{"status":"ok","version":"...","timestamp":<unix>}` (see `llmhosts/proxy/app.py`).
- Proxy **models**: `GET /v1/models` → OpenAI-format list from dispatcher.
- Proxy **completions**: `POST /v1/chat/completions` (OpenAI-compatible).
- Proxy **backends**: `GET /api/backends` → list of backends (type, url, healthy, models). Dashboard uses this via frontend proxy.
- Proxy **topology**: `GET /api/topology`, `POST /api/topology/ping` (see `llmhosts/proxy/topology_routes.py`, `dashboard_routes.py`).
- Frontend **topology**: https://llmhosts.com/dashboard/topology (calls `/api/topology` → proxy).
- Frontend **devices**: https://llmhosts.com/dashboard/devices (user’s devices; may show tunnel URL, status).
- Frontend **marketplace**: https://llmhosts.com/dashboard/marketplace (usage; relevant if using marketplace).
- Auth: proxy may require `Authorization: Bearer <api_key>`; dashboard requires session login.

---

## Scenario overview

| Scenario | Laptop has | What we test | Pass = |
|----------|------------|--------------|--------|
| **Smoke** | Not used | Server + tunnel only: health, models, optional dashboard | Proxy and tunnel work before using laptop |
| **A** | Nothing (curl/Python only; no Ollama) | Laptop as **client** → tunnel URL → server → inference | External client path works |
| **B** | Ollama + `llmhosts serve` + tunnel | Laptop as **backend**; server (or client) routes to laptop | External backend path works |
| **C** | Optional | Both: laptop as client and as backend in one flow | Full loop works |

Run **Smoke** first, then **A**, then **B**, then **C** (optional). Record every step in the results file with timestamps.

---

# Smoke — Server and tunnel only (no laptop yet)

**Purpose**: Confirm the server proxy and tunnel respond before testing the laptop.

## S1. Server: proxy and tunnel

- **Where**: Server (or machine that runs the proxy).
- **Action**: Start proxy (`llmhosts serve` or `llmhosts serve --no-tui`). Start tunnel (`llmhosts tunnel start`). Note the **tunnel URL** (e.g. `https://myhost.tail1234.ts.net`).
- **Validate**: Process stays up; tunnel prints a URL.
- **Record**: Step S1, timestamp, Pass/Fail, tunnel URL in notes.

## S2. Health via tunnel URL

- **Where**: Any machine that can reach the tunnel URL (can be server itself first: `curl -s <TUNNEL_URL>/health`).
- **Action**: `curl -s <TUNNEL_URL>/health`.
- **Expected**: HTTP 200, JSON with `"status":"ok"`, `"version"`, `"timestamp"`.
- **Record**: Step S2, timestamp, full response (or status code + body snippet), Pass/Fail.

## S3. Models via tunnel URL

- **Where**: Same machine.
- **Action**: `curl -s <TUNNEL_URL>/v1/models`. If proxy requires auth: `curl -s -H "Authorization: Bearer <API_KEY>" <TUNNEL_URL>/v1/models`.
- **Expected**: HTTP 200, JSON with `"data": [ ... ]` (model list, may be empty if no backends).
- **Record**: Step S3, timestamp, Pass/Fail, note if list is empty (expected if no Ollama on server yet).

## S4. llmhosts.com: login and dashboard

- **Where**: llmhosts.com (browser).
- **Action**: Open https://llmhosts.com, log in. Open **Dashboard → Overview** (or any dashboard page).
- **Expected**: Dashboard loads; no critical errors in console.
- **Record**: Step S4, timestamp, Pass/Fail.

## S5. llmhosts.com: Topology (proxy reachable from frontend)

- **Where**: llmhosts.com (browser).
- **Action**: Go to **Dashboard → Topology** (https://llmhosts.com/dashboard/topology). If the frontend is configured to use your proxy, topology may show nodes/backends.
- **Expected**: Page loads; may show "no nodes" or list of backends depending on config. No 5xx or network error.
- **Record**: Step S5, timestamp, what you see (e.g. "empty list" or "N nodes"), Pass/Fail.

**Smoke pass**: S1–S5 recorded and passed. Tunnel URL is known for Scenario A and B.

---

# Scenario A — Laptop as external client only (nothing on laptop)

**Laptop**: No Ollama, no `llmhosts serve`. Only what’s needed to **call** the API (curl and/or Python/OpenAI client).  
**Server**: Proxy + backends (Ollama) + tunnel. Laptop is on **external** network (e.g. neighbor WiFi / hotspot), different IP from server.

## A.1 Prerequisites

- [ ] Laptop on external network; different public IP than server (`curl -s ifconfig.me` on both).
- [ ] Laptop has curl and/or Python (optional: `pip install openai` or `requests`). Do **not** install Ollama or run `llmhosts serve` on laptop for Scenario A.
- [ ] Tunnel URL from Smoke (S2/S3) works from server; you will now call it **from the laptop**.

## A.2 Steps (record each in results file with timestamp)

### A.2.1 Connect laptop to external network

- **Where**: Laptop.
- **Action**: Connect to neighbor WiFi or phone hotspot. Run `curl -s ifconfig.me` on laptop and on server; confirm different IPs.
- **Record**: A.2.1, timestamp, Pass/Fail.

### A.2.2 Health from laptop via tunnel

- **Where**: Laptop.
- **Action**: `curl -s <TUNNEL_URL>/health`. Example: `curl -s https://myhost.tail1234.ts.net/health`.
- **Expected**: 200, `{"status":"ok", ...}`.
- **Record**: A.2.2, timestamp, status code and body snippet, Pass/Fail.

### A.2.3 Models from laptop via tunnel

- **Where**: Laptop.
- **Action**: `curl -s <TUNNEL_URL>/v1/models` (add `-H "Authorization: Bearer <KEY>"` if required).
- **Expected**: 200, JSON with model list (from server’s backends).
- **Record**: A.2.3, timestamp, Pass/Fail.

### A.2.4 Completion from laptop (non-streaming)

- **Where**: Laptop.
- **Action**: Send `POST <TUNNEL_URL>/v1/chat/completions` with a small prompt (curl or OpenAI client). Use a model from A.2.3. Include auth header if required.
- **Expected**: 200, JSON with `choices[].message.content`.
- **Record**: A.2.4, timestamp, Pass/Fail; if fail, paste full response and error.

### A.2.5 Completion from laptop (streaming)

- **Where**: Laptop.
- **Action**: Same as A.2.4 with `stream: true`; verify chunks arrive and stream completes.
- **Record**: A.2.5, timestamp, Pass/Fail.

### A.2.6 Latency

- **Where**: Laptop.
- **Action**: Short prompt; measure end-to-end time. Accept if < 10 s (tune threshold if needed).
- **Record**: A.2.6, timestamp, latency value, Pass/Fail.

### A.2.7 Disconnect / reconnect

- **Where**: Laptop.
- **Action**: Disconnect network (e.g. turn off hotspot); retry a request (expect failure). Reconnect; retry; expect success.
- **Record**: A.2.7, timestamp, Pass/Fail.

### A.2.8 Sequential and concurrent requests

- **Where**: Laptop.
- **Action**: 5 sequential completion requests; then 3 concurrent (e.g. 3 parallel curls or script). All should succeed.
- **Record**: A.2.8, timestamp, Pass/Fail.

### A.2.9 Large context

- **Where**: Laptop.
- **Action**: Send request with 2000+ tokens (prompt + completion); should complete without error.
- **Record**: A.2.9, timestamp, Pass/Fail.

### A.2.10 llmhosts.com: validate from dashboard (server side)

- **Where**: llmhosts.com (browser).
- **Action**: Open **Dashboard → Topology** (and/or **Devices** if you have devices). Confirm backend health / nodes are visible as expected (server’s backends; laptop in A is only client so may not appear as a device).
- **Record**: A.2.10, timestamp, what you see, Pass/Fail.

### A.2.11 Cost / usage (if enabled)

- **Where**: llmhosts.com (browser) or proxy.
- **Action**: If usage/cost is recorded (e.g. BillingStore, dashboard usage), confirm requests from laptop (client) appear (e.g. Dashboard → Marketplace usage or equivalent).
- **Record**: A.2.11, timestamp, Pass/Fail or N/A.

**Scenario A pass**: All A.2.x recorded and passed. Laptop with nothing (no Ollama) successfully uses the proxy through the tunnel.

---

# Scenario B — Laptop as backend (Ollama + serve on laptop)

**Laptop**: Runs Ollama + models + `llmhosts serve` + tunnel. So the **proxy and backends** are on the laptop; tunnel exposes the laptop.  
**Server or another client**: Sends requests **to** the laptop’s tunnel URL so inference runs on the laptop (e.g. 4090).

## B.1 Prerequisites

- [ ] Laptop on external network. Ollama installed and running; at least one model (e.g. `ollama pull llama3.2`). `pip install llmhosts` (or TestPyPI). Tunnel installed (Tailscale or Cloudflare).
- [ ] On laptop: `llmhosts serve` (or `llmhosts serve --no-tui`) and `llmhosts tunnel start`. Note the **laptop’s tunnel URL** (this is the backend URL that the server or a client will call).

## B.2 Steps (record each with timestamp)

### B.2.1 Ollama and models on laptop

- **Where**: Laptop.
- **Action**: `ollama serve` (or ensure running). `ollama list` shows at least one model.
- **Record**: B.2.1, timestamp, Pass/Fail.

### B.2.2 Doctor on laptop

- **Where**: Laptop.
- **Action**: `llmhosts doctor`. Expect Ollama PASS and hardware/GPU (e.g. 4090) reported.
- **Record**: B.2.2, timestamp, Pass/Fail.

### B.2.3 Serve and tunnel on laptop

- **Where**: Laptop.
- **Action**: `llmhosts serve` (or `llmhosts serve --no-tui`). Then `llmhosts tunnel start`. Note laptop’s tunnel URL.
- **Record**: B.2.3, timestamp, tunnel URL, Pass/Fail.

### B.2.4 Server (or client) sees laptop as backend

- **Where**: Server or config.
- **Action**: Ensure the laptop’s tunnel URL is registered as a backend (catalog, config, or manual registration) so requests can be routed to it.
- **Record**: B.2.4, timestamp, Pass/Fail.

### B.2.5 Inference routed to laptop

- **Where**: Server or another client (can be same laptop in a second role, or different machine).
- **Action**: Send completion request that is **routed to the laptop’s tunnel URL** (e.g. call that URL’s `/v1/chat/completions` directly, or use server that forwards to it). Inference should run on laptop’s GPU.
- **Expected**: 200, valid completion from laptop.
- **Record**: B.2.5, timestamp, Pass/Fail; if fail, full error and response.

### B.2.6 Streaming, latency, reconnect, concurrent, large context (to laptop)

- **Where**: Same as B.2.5.
- **Action**: Repeat streaming, latency check, disconnect/reconnect, sequential/concurrent, and large-context tests with the **laptop as backend** (requests sent to laptop’s tunnel URL).
- **Record**: B.2.6 (or split into B.2.6a–e), timestamp(s), Pass/Fail.

### B.2.7 llmhosts.com: backend health for laptop

- **Where**: llmhosts.com (browser).
- **Action**: If the server’s dashboard is configured to show backends (e.g. topology or devices), open **Dashboard → Topology** and/or **Devices**. Confirm the laptop (backend) appears and is healthy.
- **Record**: B.2.7, timestamp, what you see, Pass/Fail.

### B.2.8 Cost / usage for laptop backend (if applicable)

- **Where**: llmhosts.com or proxy.
- **Action**: If usage is recorded per backend, confirm usage for the laptop backend is recorded.
- **Record**: B.2.8, timestamp, Pass/Fail or N/A.

**Scenario B pass**: All B.2.x recorded and passed. Laptop as backend works through tunnel.

---

# Scenario C (optional) — Both client and backend

- **Where**: Laptop + server + llmhosts.com.
- **Action**: Use laptop as **client** (Scenario A style) to hit **server**; and use laptop as **backend** (Scenario B style) so server or client routes to laptop. Run at least one full loop: request from laptop (client) → server → backend on laptop → response. Confirm in llmhosts.com (topology/usage) if applicable.
- **Record**: C.1 (and sub-steps as needed), timestamps, Pass/Fail.

**Scenario C pass**: Both client and backend paths exercised and recorded.

---

# After the run

1. **Save the results file**: `docs/testing/P5-external-connectivity-results-YYYY-MM-DD.md` with every step, timestamp, and pass/fail filled in.
2. **If anything failed**: Under "Failures (detail for debugging)", paste:
   - Step ID and timestamp (UTC)
   - Exact command or browser action
   - Full error message or HTTP response body
   - Any proxy/server logs from that time window
   - Screenshot or UI state if relevant  
   So an agent or human can research and fix the issue.
3. **Summary**: Fill "What worked / what didn’t" and "Suggested next step for debugging" in the results file.
4. **GitHub**: Update issue #255 (e.g. "Run completed YYYY-MM-DD: Smoke [P/F], A [P/F], B [P/F], C [P/F]. Results in docs/testing/P5-external-connectivity-results-YYYY-MM-DD.md") and close or leave open for sign-off.

---

## Quick reference — endpoints and pages

| What | Where | Endpoint / URL |
|------|-------|-----------------|
| Proxy health | Laptop or server (curl) | `GET <TUNNEL_URL>/health` |
| Proxy models | Laptop or server (curl) | `GET <TUNNEL_URL>/v1/models` |
| Proxy completions | Laptop or server (curl/client) | `POST <TUNNEL_URL>/v1/chat/completions` |
| Proxy backends | Proxy (curl) | `GET <TUNNEL_URL>/api/backends` |
| Dashboard overview | llmhosts.com | https://llmhosts.com/dashboard |
| Topology | llmhosts.com | https://llmhosts.com/dashboard/topology |
| Devices | llmhosts.com | https://llmhosts.com/dashboard/devices |
| Marketplace usage | llmhosts.com | https://llmhosts.com/dashboard/marketplace |

---

## References (codebase)

- Health: `llmhosts/proxy/app.py` (`/health`).
- Models: `llmhosts/proxy/app.py` (`/v1/models`).
- Completions: `llmhosts/proxy/openai_routes.py` (`POST /v1/chat/completions`).
- Backends: `llmhosts/proxy/dashboard_routes.py` (`/api/backends`).
- Topology: `llmhosts/proxy/topology_routes.py`; frontend `frontend/src/app/dashboard/topology/page.tsx`, `frontend/src/app/api/topology/route.ts`.
- Tunnel: `llmhosts/tunnel/manager.py`, `llmhosts/tunnel/detector.py`.
