# Relay Architecture (LEGACY)

> **SUPERSEDED**: This document describes the original Next.js relay approach. The current relay architecture is a self-hosted Rust binary using WebSocket + yamux + Noise Protocol. See [ADR-009](../architecture/adr-009-relay-tunnel.md) for the accepted design. The request routing logic, streaming, and health checking concepts described here remain valid at a high level, but the implementation is changing from Next.js API routes to a standalone Rust binary.

> How requests flow from external clients through LLMHosts.com to the home GPU -- and how responses stream back.

This document details the server-side relay layer: the component that makes an external `POST /v1/chat/completions` request arrive at a user's home GPU and return the inference result. It covers request routing logic, streaming support, health checking, failover, and the complete connection lifecycle.

---

## Table of Contents

1. [Overview](#overview)
2. [Request Routing Logic](#request-routing-logic)
3. [Relay Data Flow](#relay-data-flow)
4. [Streaming Support (SSE)](#streaming-support-sse)
5. [Health Checking and Failover](#health-checking-and-failover)
6. [Connection Lifecycle](#connection-lifecycle)
7. [Error Handling and Retries](#error-handling-and-retries)
8. [Performance Characteristics](#performance-characteristics)
9. [Observability](#observability)
10. [Security in the Relay Layer](#security-in-the-relay-layer)
11. [Future Architecture](#future-architecture)

---

## Overview

The relay is the bridge between external API clients and the user's home GPU. It runs as a set of Next.js API routes on the LLMHosts.com server and handles:

1. **API key validation** -- Authenticate the client and resolve the owning user
2. **Device selection** -- Find the user's best available online device
3. **Request forwarding** -- Proxy the request to the device's tunnel URL
4. **Response streaming** -- Stream the response back to the client in real time
5. **Error handling** -- Graceful degradation when devices are offline or unhealthy

```
                            LLMHosts.com Server
                  +---------------------------------------+
                  |                                       |
Client ──HTTPS──> |  API Gateway                          |
                  |    |                                  |
                  |    ├─ Validate lh_sk_* key             |
                  |    ├─ Resolve userId                   |
                  |    ├─ Query online devices              |
                  |    ├─ Select best device                |
                  |    |                                  |
                  |    v                                  |
                  |  Relay Layer                          |
                  |    |                                  |
                  |    ├─ Forward request to tunnel_url    |
                  |    ├─ Stream response back              |
                  |    └─ Track metrics                    |
                  |                                       |
                  +------|--------------------------------+
                         |
                    HTTPS (tunnel)
                         |
                         v
                  Device (home GPU)
                    llmhosts proxy :4000
                         |
                    Ollama :11434
```

---

## Request Routing Logic

When an external client sends a request, the relay executes the following decision chain:

### Step 1: API Key Validation

```sql
-- Pseudocode for key validation
SELECT ak.userId, ak.revokedAt, ak.expiresAt, u.plan
FROM ApiKey ak
JOIN User u ON ak.userId = u.id
WHERE ak.key = $api_key
  AND ak.revokedAt IS NULL
  AND (ak.expiresAt IS NULL OR ak.expiresAt > NOW())
LIMIT 1;
```

**Decision tree:**

```
API key present in Authorization header?
  No  --> 401 Unauthorized { "error": "missing_api_key" }
  Yes --> Key exists in database?
            No  --> 401 Unauthorized { "error": "invalid_api_key" }
            Yes --> Key revoked?
                      Yes --> 403 Forbidden { "error": "key_revoked" }
                      No  --> Key expired?
                                Yes --> 403 Forbidden { "error": "key_expired" }
                                No  --> VALID (proceed with userId)
```

**Side effect:** Update `lastUsed` timestamp on the API key.

### Step 2: Device Selection

```sql
SELECT id, name, tunnelUrl, models, hardware, status, lastSeen, latencyMs
FROM Device
WHERE userId = $userId
  AND status IN ('online', 'degraded')
ORDER BY
  -- Prefer online over degraded
  CASE WHEN status = 'online' THEN 0 ELSE 1 END,
  -- Prefer lower latency
  latencyMs ASC NULLS LAST,
  -- Prefer more recently seen
  lastSeen DESC;
```

**Decision tree:**

```
User has online devices?
  No  --> 503 Service Unavailable { "error": "no_online_devices" }
  Yes --> Requested model available on any device?
            Yes --> Select device with requested model
                    (break ties by latency, then last_seen)
            No  --> Return error: 404 { "error": "model_not_available",
                    "available_models": [...] }
```

**Model matching:** The relay parses the JSON `models` column of each device and checks if the requested model name is present. Model names are case-sensitive and include the tag (e.g., `llama3.1:70b`, not just `llama3.1`).

**Multi-device selection (Phase 3+):** When a user has multiple devices, the relay selects the best one based on:

| Factor | Weight | Rationale |
|---|---|---|
| Model availability | Required | Device must have the requested model |
| Device status | High | `online` strongly preferred over `degraded` |
| Measured latency | Medium | Lower latency = faster response |
| Last seen recency | Low | More recent = more likely to be responsive |
| Hardware capability | Low | Larger VRAM = can handle larger models |

### Step 3: Request Forwarding

Once a device is selected, the relay constructs the forwarded request:

```
Original request:
  POST https://llmhosts.com/v1/chat/completions
  Authorization: Bearer lh_sk_AbCd...
  Content-Type: application/json
  X-Request-ID: req_abc123 (auto-generated if not present)
  Body: { "model": "llama3.1:70b", "messages": [...], "stream": true }

Forwarded request:
  POST https://abc123.trycloudflare.com/v1/chat/completions
  Content-Type: application/json
  X-Request-ID: req_abc123
  X-Forwarded-For: <client_ip>
  X-LLMHosts-Relay: true
  Body: { "model": "llama3.1:70b", "messages": [...], "stream": true }
```

**Key changes in forwarding:**
- The `Authorization` header is stripped (the device proxy has its own auth via tunnel key)
- `X-Request-ID` is passed through for end-to-end tracing
- `X-Forwarded-For` is set to the original client IP
- `X-LLMHosts-Relay: true` signals to the device proxy that this is a relayed request
- The body is passed through unmodified

---

## Relay Data Flow

### Non-Streaming Request

```
Client                     Relay                      Device
  |                          |                           |
  |  POST /v1/chat/completions                           |
  |  Body: { stream: false } |                           |
  |------------------------->|                           |
  |                          |  POST /v1/chat/completions|
  |                          |  Body: { stream: false }  |
  |                          |-------------------------->|
  |                          |                           |
  |                          |     (device processes     |
  |                          |      inference via        |
  |                          |      Ollama, checks       |
  |                          |      cache, runs router)  |
  |                          |                           |
  |                          |  200 OK                   |
  |                          |  { choices: [...] }       |
  |                          |<--------------------------|
  |  200 OK                  |                           |
  |  { choices: [...] }      |                           |
  |  X-LLMHosts-Model: ...   |                           |
  |  X-LLMHosts-Cost: 0.0    |                           |
  |  X-Request-ID: req_abc123|                           |
  |<-------------------------|                           |
```

**Timeout:** 120 seconds (matches `BACKEND_TIMEOUT` in dispatcher). If the device doesn't respond within this window, the relay returns `504 Gateway Timeout`.

### Streaming Request (SSE)

```
Client                     Relay                      Device
  |                          |                           |
  |  POST /v1/chat/completions                           |
  |  Body: { stream: true }  |                           |
  |------------------------->|                           |
  |                          |  POST /v1/chat/completions|
  |                          |  Body: { stream: true }   |
  |                          |-------------------------->|
  |                          |                           |
  |  HTTP 200 OK             |                           |
  |  Content-Type: text/event-stream                     |
  |  Transfer-Encoding: chunked                          |
  |<- - - - - - - - - - - - -|                           |
  |                          |                           |
  |                          |  data: {"choices":[...]}  |
  |                          |<--------------------------|
  |  data: {"choices":[...]} |                           |
  |<-------------------------|                           |
  |                          |                           |
  |                          |  data: {"choices":[...]}  |
  |                          |<--------------------------|
  |  data: {"choices":[...]} |                           |
  |<-------------------------|                           |
  |                          |                           |
  |            ...           |           ...             |
  |                          |                           |
  |                          |  data: [DONE]             |
  |                          |<--------------------------|
  |  data: [DONE]            |                           |
  |<-------------------------|                           |
```

---

## Streaming Support (SSE)

The relay must handle Server-Sent Events (SSE) streaming with zero buffering to achieve real-time token delivery.

### How SSE Streaming Through the Relay Works

```typescript
// Simplified relay implementation (Next.js API route)

export async function POST(request: Request) {
  // ... validate API key, select device ...

  const deviceUrl = `${device.tunnelUrl}/v1/chat/completions`;
  const body = await request.json();

  if (body.stream) {
    // Streaming mode: forward as SSE
    const deviceResponse = await fetch(deviceUrl, {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify(body),
    });

    // Create a TransformStream to pipe device SSE to client
    const { readable, writable } = new TransformStream();
    const writer = writable.getWriter();
    const reader = deviceResponse.body!.getReader();
    const decoder = new TextDecoder();

    // Pipe in background
    (async () => {
      try {
        while (true) {
          const { done, value } = await reader.read();
          if (done) break;
          await writer.write(value);  // Pass through with zero transformation
        }
      } finally {
        await writer.close();
      }
    })();

    return new Response(readable, {
      status: 200,
      headers: {
        "Content-Type": "text/event-stream",
        "Cache-Control": "no-cache",
        "Connection": "keep-alive",
        "X-Request-ID": requestId,
        "X-LLMHosts-Model": body.model,
      },
    });
  }

  // Non-streaming mode: buffer and forward
  // ...
}
```

### SSE Wire Format Preservation

The relay passes through SSE data verbatim. Each event frame:

```
data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1740268800,"model":"llama3.1:70b","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}\n\n
```

**Critical constraints:**
- Each `data:` line must end with `\n\n` (double newline)
- The relay must NOT buffer multiple chunks before forwarding
- The relay must NOT modify the JSON content
- The relay must NOT add extra SSE fields (like `event:` or `id:`)
- The terminal `data: [DONE]\n\n` must be forwarded as-is

### Streaming Through Cloudflare Tunnel

Cloudflare Quick Tunnel supports SSE natively. The tunnel acts as an HTTP/2 proxy:

```
Client <--HTTP/2--> Cloudflare Edge <--HTTP/2--> cloudflared <--HTTP/1.1--> localhost:4000
```

**Key behaviors:**
- Cloudflare does not buffer SSE events (they are forwarded immediately)
- HTTP/2 multiplexing allows multiple concurrent streams
- Connection timeout: Cloudflare's proxy timeout is 100 seconds for idle connections
- SSE connections are exempt from Cloudflare's request timeout (they are long-lived)

### Streaming Through Tailscale Tunnel

Tailscale Funnel exposes a local port over HTTPS via WireGuard:

```
Client <--HTTPS--> Tailscale DERP/relay <--WireGuard--> tailscaled <--HTTP/1.1--> localhost:4000
```

**Key behaviors:**
- WireGuard is a packet-level VPN; SSE events are forwarded at TCP level
- Lower overhead than Cloudflare (no HTTP/2 re-encoding)
- No proxy buffering

---

## Health Checking and Failover

### Heartbeat-Based Health

The relay relies on the heartbeat system to determine device health:

```
Device status states:

  ONLINE ──(1 missed beat)──> ONLINE (still trusted)
     ^                            |
     |                     (2 missed beats)
     |                            |
     |                            v
     |                       DEGRADED
     |                            |
     |                     (3+ missed beats)
     |                            |
     |                            v
     +────(heartbeat ok)──── OFFLINE
```

| Status | Heartbeat State | Relay Behavior |
|---|---|---|
| `online` | Beating normally | Full routing, preferred |
| `degraded` | 2 missed beats | Routable but deprioritized |
| `offline` | 3+ missed beats | Not routable, excluded from selection |

### Failover Between Devices

When a user has multiple devices registered:

```
User has 3 devices:
  Device A: online, latency=50ms, has llama3.1:70b
  Device B: online, latency=80ms, has llama3.1:70b, codestral:latest
  Device C: offline

Request for llama3.1:70b:
  1. Select Device A (online, lowest latency, has model)
  2. Forward request to Device A's tunnel
  3. If Device A returns 502/503/504:
     a. Mark Device A as degraded
     b. Retry with Device B (next best candidate)
  4. If Device B also fails:
     a. Return 503 to client with explanation
```

### Tunnel Health Probing

The relay does not actively probe tunnels (that would add latency). Instead, it relies on:

1. **Heartbeat data:** The device reports its own health every 30 seconds
2. **Passive failure detection:** If a forwarded request fails, the relay marks the device as degraded
3. **Recovery via heartbeat:** When the next heartbeat succeeds, the device returns to `online`

**Why not active probing?**
- Adding a health probe before every relay request would add 50-200ms latency
- The heartbeat system already provides 30-second health resolution
- Passive detection catches failures within a single request cycle

---

## Connection Lifecycle

### Tunnel Establishment

```
Phase 1: Device starts tunnel
  t=0s    llmhosts up started
  t=2s    Ollama confirmed running
  t=3s    FastAPI proxy started on :4000
  t=5s    cloudflared tunnel --url http://localhost:4000
  t=8s    Tunnel URL acquired (https://abc123.trycloudflare.com)
  t=10s   Device code flow initiated
  t=15s   User approves in browser (depends on user)
  t=16s   Device registered with server
  t=16s   First heartbeat sent
  t=16s   Device status: ONLINE

Phase 2: Steady state
  t=46s   Heartbeat #2 sent (30s interval)
  t=76s   Heartbeat #3 sent
  ...     (continues every 30s)

Phase 3: Request handling
  t=Ns    External client sends request
  t=Ns    Relay forwards to tunnel URL
  t=Ns    Device proxy receives, routes to Ollama
  t=Ns    Response streams back through tunnel -> relay -> client
```

### Tunnel URL Changes

Cloudflare Quick Tunnels generate a new URL on each restart. The system handles this:

```
Device restarts llmhosts up:
  1. Old tunnel process terminated
  2. New cloudflared process started
  3. New URL generated (https://xyz789.trycloudflare.com)
  4. Device re-registers:
     POST /api/devices/register
     { "name": "home-desktop", "tunnel_url": "https://xyz789.trycloudflare.com" }
  5. Server updates Device record with new tunnelUrl
  6. Next heartbeat confirms new URL works
  7. Any in-flight requests to old URL will fail (handled by retry logic)
```

### Graceful Shutdown

```
User presses Ctrl+C on llmhosts up:
  1. Signal handler caught
  2. Send final heartbeat with status: "shutting_down"
  3. Stop heartbeat loop
  4. Unregister device:
     DELETE /api/devices/<device_id>
  5. Stop tunnel (SIGTERM -> cloudflared)
  6. Stop FastAPI proxy
  7. Print "Device unregistered. GPU offline."
```

If the process is killed ungracefully (SIGKILL, power loss), the device will be marked offline after 3 missed heartbeats (90 seconds).

### Reconnection Logic

```
Network interruption detected (heartbeat fails):
  1. First failure: log warning, retry in 10s instead of 30s
  2. Second failure: attempt tunnel restart
     a. Stop current tunnel
     b. Start new tunnel (may get new URL)
     c. Re-register device with new URL
  3. Third failure: enter backoff mode
     Retry every 60s, 120s, 240s (exponential, max 5 min)
  4. Recovery: heartbeat succeeds
     a. Log recovery event
     b. Resume normal 30s heartbeat interval
     c. Server updates device to online
```

---

## Error Handling and Retries

### Relay Error Response Format

All relay errors follow this format:

```json
{
  "error": {
    "code": "device_unreachable",
    "message": "Could not reach your device. It may be offline.",
    "type": "relay_error",
    "request_id": "req_abc123",
    "device_id": "clxyz123...",
    "suggestion": "Run 'llmhosts up' on your device to bring it online."
  }
}
```

### Retry Policy

| Error Type | Retry? | Max Retries | Backoff | Failover? |
|---|---|---|---|---|
| Device timeout (504) | Yes | 1 | Immediate | Yes (next device) |
| Device connection refused | Yes | 1 | Immediate | Yes (next device) |
| Device 5xx error | Yes | 1 | 1 second | Yes (next device) |
| Device 4xx error | No | 0 | N/A | No (client error) |
| API key invalid | No | 0 | N/A | No |
| No online devices | No | 0 | N/A | No |

### Idempotency

Chat completions are NOT idempotent (each call may produce different output). The relay does NOT retry on success -- only on transport failures where the request definitely did not reach the inference engine.

**Streaming safety:** If a streaming response has already started sending tokens to the client and then the connection breaks mid-stream, the relay does NOT retry. The client receives a truncated stream and must handle reconnection.

---

## Performance Characteristics

### Latency Breakdown (Relay Overhead)

| Component | P50 | P95 | P99 | Notes |
|---|---|---|---|---|
| API key lookup | 1ms | 3ms | 5ms | SQLite indexed query |
| Device selection | 1ms | 2ms | 5ms | SQLite query + sorting |
| Request serialization | <1ms | 1ms | 2ms | JSON pass-through |
| DNS resolution (tunnel) | 5ms | 20ms | 50ms | Cloudflare DNS |
| TLS handshake (tunnel) | 10ms | 30ms | 80ms | Connection pooling reduces this |
| Tunnel transit | 20ms | 50ms | 100ms | Depends on geographic distance |
| **Total relay overhead** | **~40ms** | **~100ms** | **~240ms** | Excluding inference time |

### Connection Pooling

The relay maintains an HTTP connection pool to frequently-accessed tunnel URLs:

```
Connection pool settings:
  max_connections: 100 (per relay instance)
  max_connections_per_host: 10 (per tunnel URL)
  keepalive_timeout: 60s
  connection_timeout: 10s
  read_timeout: 120s (matches BACKEND_TIMEOUT)
```

Connection pooling eliminates the DNS + TLS handshake overhead for subsequent requests to the same device, reducing relay overhead from ~100ms to ~30ms at P50.

### Throughput

| Metric | Value | Bottleneck |
|---|---|---|
| Max concurrent relayed requests | 100 per relay instance | Node.js event loop |
| Max SSE streams | 100 concurrent | OS file descriptors |
| Bandwidth per stream | Limited by device upload | Home network upload speed |

**Home network bandwidth consideration:** A typical home network has 10-50 Mbps upload speed. A single LLM streaming response uses ~0.1 Mbps (tokens arrive slowly). This means a single home connection can serve 100+ concurrent streaming requests without bandwidth issues.

---

## Observability

### Request Tracing

Every relayed request gets an `X-Request-ID` header. This ID is:
1. Generated by the relay if not present in the client request
2. Passed to the device in the forwarded request
3. Returned to the client in the response
4. Logged at every hop for end-to-end tracing

```
Client -> Relay (log: request_received, request_id=req_abc123)
Relay -> Device (log: request_forwarded, request_id=req_abc123, device_id=clxyz)
Device (log: inference_started, request_id=req_abc123, model=llama3.1:70b)
Device -> Relay (log: response_received, request_id=req_abc123, ttft=450ms)
Relay -> Client (log: response_sent, request_id=req_abc123, total_ms=520ms)
```

### Metrics (Prometheus)

| Metric | Type | Labels | Description |
|---|---|---|---|
| `llmhosts_relay_requests_total` | Counter | `status`, `model` | Total relayed requests |
| `llmhosts_relay_latency_seconds` | Histogram | `model` | End-to-end relay latency |
| `llmhosts_relay_device_selection_ms` | Histogram | | Device selection time |
| `llmhosts_relay_errors_total` | Counter | `error_type` | Relay errors by type |
| `llmhosts_relay_active_streams` | Gauge | | Currently active SSE streams |
| `llmhosts_device_status` | Gauge | `device_id`, `status` | Device health status |
| `llmhosts_heartbeat_age_seconds` | Gauge | `device_id` | Time since last heartbeat |

### Structured Logging

```json
{
  "level": "info",
  "timestamp": "2026-02-22T10:00:00.123Z",
  "message": "relay_request_complete",
  "request_id": "req_abc123",
  "user_id": "cluser123",
  "device_id": "clxyz123",
  "device_name": "home-desktop",
  "model": "llama3.1:70b",
  "stream": true,
  "relay_latency_ms": 42,
  "total_latency_ms": 520,
  "tokens": 150,
  "cache_hit": false,
  "status": 200
}
```

---

## Security in the Relay Layer

### Request Sanitization

Before forwarding, the relay:
1. Strips the `Authorization` header (prevents key leakage to the device)
2. Validates the request body against the OpenAI/Anthropic schema
3. Rejects excessively large request bodies (>10MB)
4. Sets `X-Forwarded-For` to the client's real IP
5. Sets `X-LLMHosts-Relay: true` so the device knows this is a relayed request

### Response Sanitization

Before returning to the client, the relay:
1. Strips any `Set-Cookie` headers from the device response
2. Strips any internal headers (`X-Powered-By`, server version headers)
3. Adds standard LLMHosts headers (`X-Request-ID`, `X-LLMHosts-Model`, `X-LLMHosts-Cost`)

### Tunnel URL Validation

The relay validates tunnel URLs before forwarding:
- Must be HTTPS
- Must match known tunnel patterns (`*.trycloudflare.com`, `*.ts.net`)
- Cannot be localhost, private IP ranges, or link-local addresses
- URL is stored at device registration and cannot be changed via heartbeat (requires re-registration)

### Rate Limiting

| Layer | Limit | Scope |
|---|---|---|
| Per API key | 60 RPM (default), configurable per plan | Per key |
| Per user | 200 RPM | Across all of user's keys |
| Per device | 100 RPM | Per tunnel URL |
| Global | 10,000 RPM | Entire relay system |

---

## Future Architecture

### Phase 3: WebSocket Relay

The current architecture uses HTTP forwarding through tunnel URLs. Phase 3 introduces an optional WebSocket-based relay for lower latency:

```
Current (HTTP):
  Client --> HTTPS POST --> Relay --> HTTPS POST --> Tunnel --> Device

Future (WebSocket):
  Client --> HTTPS POST --> Relay --> WS message --> Device
  (Device maintains persistent WebSocket to relay)
```

**Benefits:**
- Eliminates DNS + TLS overhead for each request (persistent connection)
- Enables server-push notifications to devices
- Reduces latency by ~50ms per request

**Trade-off:**
- Requires the device to maintain a persistent outbound WebSocket
- More complex connection management
- Tunnels still work as fallback

### Phase 3: Multi-Device Load Balancing

Smart load balancing across multiple devices:

```
User has 3 devices:
  Device A: RTX 4090 (24GB), latency=50ms
  Device B: RTX 3080 (10GB), latency=30ms
  Device C: M2 Ultra (64GB), latency=100ms

Request for llama3.1:70b (requires 40GB VRAM):
  -> Only Device C has enough VRAM
  -> Route to Device C

Request for mistral:7b (requires 5GB VRAM):
  -> All devices qualify
  -> Route to Device B (lowest latency)

Request for codestral:latest (requires 8GB VRAM):
  -> Devices A and B qualify
  -> Device A preferred (more headroom for concurrent requests)
```

### Phase 4: Global Edge Relay

Deploy relay instances at multiple edge locations:

```
US West Client --> US West Relay --> Tunnel --> US device
EU Client --> EU Relay --> Tunnel --> EU device (or US device if closer)
Asia Client --> Asia Relay --> Tunnel --> nearest available device
```

This reduces the client-to-relay hop latency by placing relay instances geographically closer to clients.

---

## Related Documentation

- [README.md](README.md) -- GPU-as-API technical reference
- [Device Code Authentication](device-code-auth.md) -- How devices authenticate
- [Quickstart Guide](quickstart.md) -- User-facing setup guide
