# ADR-009: Self-Hosted LLMHosts Relay Tunnel

## STATUS
ACCEPTED (2026-03-08) -- replaces Tailscale/Cloudflare dependency with self-hosted Rust relay

## CHANGE LOG
- 2026-03-08: Proposed and accepted. Supersedes ADR-004 Phase 2 (Tailscale/Cloudflare auto-detect) and Phase 3 (Headscale for Hive). Tailscale/Cloudflare retained as fallback providers only.

## CONTEXT

LLMHosts's transformative UX is `llmhosts up` turning a home GPU into production AI infrastructure accessible from anywhere. The tunnel layer is what makes "from anywhere" work.

### Why the Current Architecture is Untenable

ADR-004 designed a three-phase networking strategy:
- **Phase 1**: localhost only (complete)
- **Phase 2**: auto-detect Tailscale/Cloudflare (complete, shipping)
- **Phase 3**: Headscale for Hive Mode (deployed but deprecated by this ADR)

Three problems make this architecture unacceptable for a commercial product:

1. **Tailscale ToS** (Section 4.2) explicitly prohibits commercial exploitation of their coordination service. Building a paid marketplace on top of their free tier is a legal liability.
2. **Cloudflare dependency** means our paid customers' traffic routes through infrastructure we don't control. Outages, policy changes, or rate limiting by Cloudflare directly impact our SLA.
3. **Headscale** (BSD-3) was deployed for Hive Mode coordination but is overkill — it's a full mesh VPN when we only need a simple reverse proxy. Production HA is unproven, and SQLite scalability is a concern for multi-tenant.

### What User Journeys Require

From `docs/gpu-as-api/user-success-journeys.md`:

- **Journey 4 (Multi-Machine)**: "RTX 4090 at home, MacBook for travel, same API from both." The relay IS this journey. `https://your-device.llmhosts.com/v1` must work from anywhere.
- **Journey 5 (Team)**: GPU pooling across team members requires relay connections from every member's machine to the LLMHosts infrastructure.
- **Journey 6 (Graceful Degradation)**: "No internet → proxy still runs locally." Relay must be progressive enhancement, never a hard dependency.
- **Journey 7 (Marketplace Operator)**: Operators listing GPUs on the marketplace need persistent, reliable relay connections for buyer traffic routing.

### The Market Opportunity

Nobody has built "one install, your GPU works from anywhere" with zero third-party dependencies. Every competitor either requires manual port forwarding, depends on Tailscale/ngrok/Cloudflare, or only works on LAN. A branded, built-in relay that "just works" is a genuine moat.

## OPTIONS EVALUATED

### Approach 1: Raw TCP + Custom Framing
- Direct TCP tunnel with custom binary protocol
- Lower latency than WebSocket (no HTTP upgrade overhead)
- **Rejected**: Blocked by corporate proxies, hotel WiFi, and many NAT configurations. WebSocket traverses these transparently because it looks like HTTPS traffic.

### Approach 2: WebSocket + yamux + Noise Protocol (NK) ← CHOSEN
- WSS connection (NAT-friendly, proxy-friendly, firewall-friendly)
- yamux stream multiplexing (no head-of-line blocking for concurrent LLM streams)
- Noise NK pattern encryption (zero-knowledge relay — server can't read payloads)
- Ships as compiled Rust binary inside pip wheel via maturin

### Approach 3: QUIC + HTTP/3
- Modern transport with built-in multiplexing
- Better performance characteristics than WebSocket
- **Rejected (for now)**: QUIC uses UDP, which is blocked by many corporate firewalls and NATs. WebSocket over TCP is universally reachable. QUIC can be added as an upgrade path once relay is proven.

## DECISION

**Approach 2: WebSocket reverse tunnel with yamux multiplexing and Noise Protocol encryption, implemented in Rust, shipping inside the pip wheel via maturin.**

### Architecture Overview

```
Customer's Device                         LLMHosts VPS (llmhosts.com)
┌──────────────────────┐                 ┌───────────────────────────────┐
│                      │                 │                               │
│  llmhosts up         │                 │  Relay Server (Rust binary)   │
│  ┌────────────────┐  │   WSS + Noise   │  ┌─────────────────────────┐ │
│  │ relay_core     │──┼────────────────→│  │ WebSocket acceptor      │ │
│  │ (pyo3 + Rust)  │  │   NK pattern    │  │                         │ │
│  └────────────────┘  │                 │  │ yamux session           │ │
│         │            │                 │  │   ├─ stream 0 (control) │ │
│         ↓            │                 │  │   ├─ stream 1 (req A)   │ │
│  ┌────────────────┐  │                 │  │   ├─ stream 2 (req B)   │ │
│  │ FastAPI proxy   │  │  ←─ HTTP req ──│  │   └─ stream N (req N)  │ │
│  │ localhost:PORT  │  │  ── HTTP res →  │  └─────────────────────────┘ │
│  │ (Ollama/vLLM)  │  │                 │            ↑                 │
│  └────────────────┘  │                 │  ┌─────────┴───────────────┐ │
│                      │                 │  │ HTTPS ingress           │ │
└──────────────────────┘                 │  │ device-id.llmhosts.com  │ │
                                         │  │ /v1/chat/completions    │ │
         Outbound only.                  │  └─────────────────────────┘ │
         NAT-friendly.                   │                               │
         Works behind any firewall.      └───────────────────────────────┘
```

### Two Rust Crates

#### 1. `relay_core/` — Client library (ships in pip wheel)

Compiled Rust library exposed to Python via pyo3 + maturin. Follows the exact pattern established by `router_core/`.

```
relay_core/
├── Cargo.toml          # pyo3, tokio, tokio-tungstenite, yamux, snow, rustls
├── pyproject.toml      # maturin build backend
└── src/
    ├── lib.rs          # #[pymodule] — exports RelayClient class to Python
    ├── client.rs       # WebSocket connection, reconnection, lifecycle
    ├── noise.rs        # Noise NK handshake + transport encryption
    ├── mux.rs          # yamux session management + stream handling
    ├── forward.rs      # HTTP request forwarding to local proxy
    └── metrics.rs      # Latency, throughput, connection state
```

**Python interface** (exposed via pyo3):

```python
from _relay_core import RelayClient

client = RelayClient(
    relay_url="wss://llmhosts.com/api/relay/connect",
    device_token="lh_dev_...",
    local_port=11400,
    server_pubkey=b"...",  # Noise NK: client knows server's static key
)

# Async lifecycle
await client.connect()       # WSS + Noise handshake + yamux session
await client.wait_closed()   # Block until disconnected
await client.disconnect()    # Graceful shutdown

# State queries
client.is_connected          # bool
client.latency_ms            # float (last heartbeat RTT)
client.streams_active        # int (concurrent yamux streams)
client.bytes_relayed         # int (total bytes forwarded)
client.uptime_seconds        # float
```

#### 2. `llmhosts-relay` — Server binary (runs on VPS)

Standalone Rust binary deployed on the LLMHosts VPS. Not distributed to customers.

```
relay_server/
├── Cargo.toml          # tokio, hyper, tokio-tungstenite, yamux, snow, rustls, tracing
└── src/
    ├── main.rs         # Binary entrypoint, config loading, graceful shutdown
    ├── acceptor.rs     # WebSocket upgrade + Noise NK handshake
    ├── registry.rs     # Device ID → active yamux session map
    ├── proxy.rs        # HTTPS request → yamux stream → device
    ├── health.rs       # /health, /metrics, /ready endpoints
    └── config.rs       # TOML config: bind addr, TLS, max connections, timeouts
```

**VPS-side API endpoints**:

| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/api/relay/connect` | GET (WebSocket upgrade) | Device connects here |
| `/api/relay/health` | GET | Relay server health |
| `/api/relay/metrics` | GET | Prometheus metrics |
| `{device-id}.llmhosts.com/v1/*` | ANY | Proxied to device via relay |

### Protocol Layers

```
┌─────────────────────────────────────────┐
│  Layer 4: HTTP Request/Response         │  Application data
├─────────────────────────────────────────┤
│  Layer 3: yamux Stream Multiplexing     │  Concurrent streams, no HOL blocking
├─────────────────────────────────────────┤
│  Layer 2: Noise NK Transport            │  End-to-end encryption (relay can't read)
├─────────────────────────────────────────┤
│  Layer 1: WebSocket over TLS            │  NAT traversal, firewall traversal
└─────────────────────────────────────────┘
```

### Noise Protocol: NK Pattern

The **NK** (No key for initiator, Known key for responder) pattern is chosen because:

1. **Client doesn't need a static keypair** — authenticates via device token instead
2. **Server's public key is known** — distributed with the pip package or fetched at first auth
3. **Zero-knowledge relay** — the relay server routes encrypted yamux frames it cannot decrypt between the outer WSS layer and the Noise transport. Only the device and the end client can read payloads.

```
NK:
  <- s          # Server's static key is pre-shared
  ...
  -> e, es      # Client sends ephemeral, DH with server's static
  <- e, ee      # Server sends ephemeral, DH ephemeral-ephemeral
```

After handshake: both sides have a shared symmetric key. All yamux frames are encrypted.

### yamux Multiplexing

**Why yamux over HTTP/2 multiplexing or simple WebSocket framing:**

- **No head-of-line blocking**: Each LLM streaming response gets its own yamux stream. A slow 70B model response doesn't block a fast 3B model response.
- **Proven at scale**: Used by libp2p (IPFS), Consul Connect (HashiCorp), and Terraform.
- **Flow control**: Per-stream and connection-level flow control prevents any single request from starving others.
- **Lightweight**: ~200 lines of protocol logic. 4-byte header per frame.

**Stream allocation**:
- Stream 0: Control channel (heartbeat, auth refresh, capability negotiation)
- Stream 1+: One per HTTP request/response pair (opened by server, closed after response)

### Authentication Flow

Extends the existing Device Code Auth (RFC 8628) flow in `llmhosts/auth/device_flow.py`:

```
1. User runs `llmhosts up`
2. DeviceCodeFlow authenticates → gets Credentials (api_key, user_id)
3. Python requests relay token: POST /api/devices/relay-token
   Body: { device_id, api_key }
   Response: { relay_token: "lh_relay_...", server_pubkey: "base64..." }
4. relay_core.RelayClient connects with relay_token
5. Server validates token, registers device in relay registry
6. Relay token refreshed every 23 hours via control stream (stream 0)
```

### Reconnection Strategy

```
Disconnect detected
  → Wait 1s (jitter ±500ms)
  → Reconnect attempt 1
  → Wait 2s (jitter ±1s)
  → Reconnect attempt 2
  → Wait 4s (jitter ±2s)
  → ...
  → Cap at 60s between attempts
  → Never stop trying (device should always be reachable)

During reconnection:
  - Local proxy continues serving localhost traffic
  - Dashboard shows "Reconnecting..." status
  - Heartbeat to llmhosts.com switches to HTTP fallback
  - On reconnect: resume relay, re-register capabilities
```

### Graceful Degradation (Journey 6 Alignment)

The relay is progressive enhancement. LLMHosts works without it:

| Condition | Behavior |
|-----------|----------|
| No internet | Proxy runs locally, relay skipped |
| Relay server down | Fallback to Tailscale/Cloudflare if available |
| WebSocket blocked (rare) | Fallback to Tailscale/Cloudflare if available |
| Relay token expired | Auto-refresh via control stream, transparent to user |
| Device behind strict proxy | WSS looks like HTTPS, passes through |

### Provider Cascade (Updated Priority)

```
TunnelDetector.detect() priority order:
  1. LLMHosts Relay  -- built-in, zero config, no external software
  2. Tailscale       -- if user has it installed and connected
  3. Cloudflare      -- if cloudflared is installed

TunnelManager._start_provider() cascade:
  Try #1 (Relay) → if fails → Try #2 (Tailscale) → if fails → Try #3 (Cloudflare) → if all fail → local-only mode
```

## INTEGRATION POINTS

### Files to Modify

| File | Change |
|------|--------|
| `llmhosts/tunnel/models.py` | Add `RELAY = "relay"` to `TunnelProvider` enum |
| `llmhosts/tunnel/detector.py` | Add relay detection (check `relay_core` import + internet connectivity) |
| `llmhosts/tunnel/manager.py` | Add relay provider to `_start_provider()` cascade |
| `llmhosts/tunnel/__init__.py` | Export new relay module |
| `llmhosts/up.py` | Wire relay into Phase 4 (Connect), pass relay token from auth |
| `llmhosts/auth/device_flow.py` | Add `relay_token` to Credentials, add relay token endpoint |
| `pyproject.toml` | Add `relay_core` as build dependency (maturin) |
| `frontend/src/app/api/devices/route.ts` | Add relay-token endpoint |

### New Files

| File | Purpose |
|------|---------|
| `relay_core/` (entire crate) | Rust client library, pyo3 + maturin |
| `relay_server/` (entire crate) | Rust server binary for VPS |
| `llmhosts/tunnel/relay.py` | Python wrapper around `_relay_core.RelayClient` |
| `tests/test_tunnel_relay.py` | Unit tests for relay Python wrapper |
| `tests/test_relay_integration.py` | Integration tests (mock WebSocket server) |

### Existing Pattern: `router_core/`

The relay client follows the exact build pattern established by `router_core/`:

```toml
# relay_core/Cargo.toml
[package]
name = "relay_core"
version = "0.1.0"
edition = "2021"

[lib]
name = "_relay_core"
crate-type = ["cdylib"]

[dependencies]
pyo3 = { version = "0.25", features = ["extension-module"] }
tokio = { version = "1", features = ["full"] }
tokio-tungstenite = { version = "0.24", features = ["rustls-tls-native-roots"] }
yamux = "0.13"
snow = "0.9"
hyper = { version = "1", features = ["client", "http1"] }
tracing = "0.1"

[profile.release]
strip = "symbols"
lto = true
codegen-units = 1
```

```toml
# relay_core/pyproject.toml
[build-system]
requires = ["maturin>=1.0"]
build-backend = "maturin"

[tool.maturin]
features = ["pyo3/extension-module"]
```

### Build Targets

| Target | Platform | Size (est.) |
|--------|----------|-------------|
| `x86_64-unknown-linux-musl` | Linux x86_64 (static) | ~3MB |
| `aarch64-unknown-linux-musl` | Linux ARM64 (RPi, AWS Graviton) | ~3MB |
| `x86_64-apple-darwin` | Intel Mac | ~3MB |
| `aarch64-apple-darwin` | Apple Silicon Mac | ~3MB |
| `x86_64-pc-windows-msvc` | Windows x86_64 | ~3.5MB |

Cross-compiled via `cross` or GitHub Actions matrix. Wheels published to PyPI alongside the pure-Python `llmhosts` package.

## IMPLEMENTATION PLAN

### Milestone 1: Foundation (relay_core skeleton + integration stubs)

**Goal**: Relay client connects to a mock server, Python can import and call it.

1. Scaffold `relay_core/` crate with pyo3 + maturin (copy `router_core/` pattern)
2. Implement basic `RelayClient` struct with `connect()` / `disconnect()` / `is_connected`
3. Plain WebSocket (no Noise, no yamux yet) — prove the pyo3 async bridge works
4. Add `TunnelProvider.RELAY` to `models.py`
5. Create `llmhosts/tunnel/relay.py` wrapper class
6. Wire into `detector.py` and `manager.py` (relay as priority 1)
7. Unit tests for Python wrapper with mock WebSocket server
8. CI: add `maturin develop` to test pipeline

### Milestone 2: Noise Protocol Encryption

**Goal**: All relay traffic is end-to-end encrypted. Server cannot read payloads.

1. Implement Noise NK handshake in `relay_core/src/noise.rs`
2. Generate server keypair, distribute public key via auth endpoint
3. Encrypt all WebSocket frames after handshake
4. Verify encryption with packet capture (tcpdump shows only ciphertext)
5. Add `server_pubkey` to relay token response
6. Key rotation support (server can publish new keys, clients re-handshake)

### Milestone 3: yamux Multiplexing

**Goal**: Concurrent LLM streaming requests don't block each other.

1. Implement yamux session management in `relay_core/src/mux.rs`
2. Control stream (stream 0): heartbeat, auth refresh, capability ads
3. Request streams (stream 1+): one per HTTP request, server-initiated
4. Per-stream flow control (prevent one slow model from starving others)
5. Stress test: 10 concurrent streaming requests through relay
6. Measure: yamux overhead <2ms per frame

### Milestone 4: Relay Server (`llmhosts-relay` binary)

**Goal**: Production relay server running on VPS, accepting device connections.

1. Scaffold `relay_server/` crate with tokio + hyper + tungstenite
2. WebSocket acceptor + Noise NK responder
3. Device registry: token → active yamux session map
4. HTTPS proxy: route `{device}.llmhosts.com/v1/*` → correct device stream
5. Health + metrics endpoints (Prometheus-compatible)
6. Deploy to LLMHosts VPS behind existing Traefik
7. Wildcard DNS: `*.llmhosts.com` → VPS
8. Load test: 100 concurrent devices, 10 req/s each

### Milestone 5: Production Hardening

**Goal**: Relay is battle-tested, monitored, and handles all edge cases.

1. Reconnection with exponential backoff + jitter
2. Token refresh via control stream (23h cycle)
3. Connection draining for server restarts (SIGTERM → drain → close)
4. Rate limiting per device (prevent abuse)
5. Bandwidth monitoring + alerting
6. Update `llmhosts up` Phase 4 to use relay as primary tunnel
7. Update frontend docs: replace Tailscale/Cloudflare examples with relay URLs
8. E2E test: `llmhosts up` → relay connect → `curl https://device.llmhosts.com/v1/models`
9. Archive Headscale infrastructure (move to `_legacy/`)

## DOCUMENTATION & LEGACY CLEANUP

This ADR triggers a cascade of documentation updates:

| Document | Update |
|----------|--------|
| ADR-004 | Add revision note pointing to ADR-009 |
| ADR index | Add ADR-009 row |
| CLAUDE.md | Update ADR quick reference, tunnel references |
| AGENTS.md | Remove `HEADSCALE_API_KEY`, update test counts, tunnel strategy |
| CONTEXT.md | Update tunnel strategy section |
| README.md | Replace Tailscale/Cloudflare assumptions with relay |
| `.cursor/rules/project-context.mdc` | Remove "Phase 3: Headscale" reference |
| `docs/roadmap/*` | Replace Headscale references with relay milestones |
| Frontend docs (3 TSX files) | Replace `.ts.net` URLs with `device.llmhosts.com` |
| Frontend test fixtures (8 files) | Replace `*.ts.net` mock URLs |
| `.env.example` | Remove `HEADSCALE_*` vars, add `RELAY_*` vars |
| `deferred/P0-p5-external-connectivity.md` | Move to resolved |

### Legacy Code to Archive

| Path | Reason |
|------|--------|
| `llmhosts/hive/headscale.py` (230 lines) | Headscale client, replaced by relay |
| `tests/test_hive_headscale.py` (437 lines) | Tests for deprecated Headscale code |
| `infrastructure/headscale/` (4 files) | Headscale deployment infra |

## PERFORMANCE TARGETS

| Metric | Target | Rationale |
|--------|--------|-----------|
| Relay overhead (added latency) | <50ms | Negligible for LLM streaming (tokens arrive every 20-100ms) |
| WebSocket handshake + Noise NK | <200ms | One-time on connect |
| yamux stream open | <1ms | Per-request overhead |
| Reconnection time | <5s (p95) | Includes backoff jitter |
| Max concurrent streams per device | 256 | yamux default, configurable |
| Max concurrent devices per server | 10,000 | Limited by fd limits + memory |
| Binary size (relay_core .so) | <5MB | Fits comfortably in pip wheel |
| Memory per device connection | <2MB | Registry entry + yamux session state |

## SUCCESS CRITERIA

From GitHub issue #438:

- [ ] `llmhosts up` establishes relay connection with zero user config
- [ ] `curl https://device.llmhosts.com/v1/models` returns models from the device
- [ ] Works behind NAT, corporate firewall, hotel WiFi
- [ ] Reconnects automatically on network interruption
- [ ] Binary ships inside pip wheel (no separate install)
- [ ] <50ms latency overhead for relay (excluding network RTT)
- [ ] Works on Windows, Linux, macOS (all architectures)

Additional criteria from user success journeys:

- [ ] Journey 4: Multi-machine developer can set `OPENAI_BASE_URL=https://device.llmhosts.com/v1` and use any OpenAI-compatible tool
- [ ] Journey 5: All team members' GPUs accessible via relay for Hive pooling
- [ ] Journey 6: Local proxy continues working when relay is disconnected
- [ ] Journey 7: Marketplace operators maintain persistent relay connections for buyer traffic

## RATIONALE

1. **WebSocket over raw TCP**: Universal firewall traversal. WSS looks like HTTPS to any middlebox. Corporate proxies, hotel WiFi, strict NATs — all pass WebSocket traffic.

2. **Noise NK over TLS client certs**: Simpler key management (no PKI), proven cryptographic primitive (used by WireGuard, Lightning Network), and achieves zero-knowledge relay — the server routes encrypted traffic it cannot decrypt.

3. **yamux over HTTP/2 mux**: yamux is a stream multiplexer designed for exactly this use case. HTTP/2 mux carries unnecessary HTTP semantics overhead. yamux is used by libp2p, Consul Connect, and Terraform — battle-tested at scale.

4. **Rust over Go**: Consistent with `router_core/` (already Rust + pyo3). Single compiled binary with no runtime. Smaller binary size (~3MB vs ~10MB for Go). pyo3 + maturin pattern is proven in this codebase.

5. **maturin over standalone binary**: Ships inside the pip wheel. Users type `pip install llmhosts` and get the relay client. No separate download, no PATH manipulation, no install scripts. Same pattern as `uv` (Astral).

6. **Self-hosted over any third party**: Eliminates ToS risk (Tailscale), infrastructure dependency (Cloudflare), and operational complexity (Headscale). Full control over SLA, pricing, and feature roadmap.

## CONSEQUENCES

### Positive
- Zero third-party tunnel dependencies for paying customers
- Branded relay URL (`device.llmhosts.com`) reinforces product identity
- Full control over SLA, latency, and feature roadmap
- Zero-knowledge encryption provides genuine privacy story
- Progressive enhancement — local-first, relay optional, fallback to Tailscale/Cloudflare

### Negative
- Must operate relay server infrastructure (VPS, monitoring, on-call)
- Rust development expertise required for relay maintenance
- Cross-platform wheel building adds CI complexity (5 targets × 2 crates)
- yamux + Noise add protocol complexity vs. plain WebSocket forwarding
- Server keypair management (rotation, distribution, backup)

### Risks
- **WebSocket blocked by proxy**: Extremely rare (<1% of corporate environments). Fallback to Tailscale/Cloudflare handles this.
- **Relay server becomes SPOF**: Mitigated by deploying relay in multiple regions (Phase 2 of relay itself). For MVP, single VPS with health monitoring.
- **Binary size bloat**: Estimated ~3-5MB per target. Well within pip wheel norms (numpy ships 15MB+).

## REFERENCES

- GitHub issue: [#438](https://github.com/LookNoHandsMom/LLMHosts.com/issues/438)
- bore (MIT): https://github.com/ekzhang/bore — minimal Rust tunnel reference
- rathole (Apache-2.0): https://github.com/rapiz1/rathole — Noise Protocol + Rust tunnel
- yamux spec: https://github.com/hashicorp/yamux/blob/master/spec.md
- Noise Protocol: https://noiseprotocol.org/noise.html#interactive-handshake-patterns-fundamental
- uv (maturin distribution model): https://github.com/astral-sh/uv
- Existing `router_core/` pattern: `router_core/Cargo.toml`, `router_core/src/lib.rs`
