# Quickstart: Your GPU as a Cloud API in 60 Seconds

> Go from a bare laptop with a GPU to a production API endpoint accessible from anywhere in the world.

---

## Prerequisites

- A machine with a GPU (NVIDIA recommended, Apple Silicon works too)
- [Ollama](https://ollama.com) installed (the inference engine)
- Python 3.12+ installed
- An internet connection (any kind -- home WiFi, cellphone hotspot, hotel WiFi, corporate network)

That's it. No cloud account, no infrastructure knowledge, no port forwarding, no DNS configuration.

---

## Step 1: Install LLMHosts

```bash
pip install llmhosts
```

This installs the core package (~50MB). For advanced features like intelligent routing and semantic caching, install the smart tier:

```bash
pip install llmhosts[smart]    # +kNN router, namespace cache, vCache semantic (~150MB)
pip install llmhosts[full]     # +everything including Qwen-0.5B reasoning router (~2GB)
```

Verify the installation:

```bash
llmhosts --version
# llmhosts v0.5.0
```

## Step 2: Pull Some Models

If you haven't already, pull at least one model with Ollama:

```bash
ollama pull llama3.1:8b       # Fast, good for general tasks (~5GB)
ollama pull codestral:latest  # Great for code (~14GB)
ollama pull llama3.1:70b      # Powerful, needs 40GB+ VRAM (~40GB)
```

## Step 3: Start Your GPU API

```bash
llmhosts up
```

That's the whole command. Here's what happens:

```
$ llmhosts up

  LLMHosts v0.5.0 -- Starting your GPU API...

  Phase 1: Discovering inference engines...
    Ollama at localhost:11434 -- 3 models
  Phase 2: Analyzing hardware...
    NVIDIA GeForce RTX 4090 (24.0GB), 64GB RAM
  Phase 3: Starting proxy...
    API ready at http://localhost:11400
  Phase 4: Connecting to LLMHosts.com...

  ┌──────────────────────────────────────────────────┐
  │                                                  │
  │   To connect this device, open your browser:     │
  │                                                  │
  │     https://llmhosts.com/activate                │
  │                                                  │
  │   And enter this code:                           │
  │                                                  │
  │              ABCD-1234                            │
  │                                                  │
  └──────────────────────────────────────────────────┘

  Waiting for approval... (expires in 15 minutes)
```

Open your browser, go to https://llmhosts.com/activate, log in (or create a free account), and enter the code. Click "Approve".

Back in your terminal:

```
  Approved! Device registered as "home-desktop"
  Phase 5: Running...
    Heartbeat active, savings tracking started

  ════════════════════════════════════════════════════════
   Your GPU is live!
  ════════════════════════════════════════════════════════

   Local:     http://localhost:11400/v1
   Remote:    https://home-desktop.llmhosts.com/v1
   API Key:   lh_sk_AbCdEfGh... (saved to ~/.llmhosts/credentials.json)
   Models:    llama3.1:8b, codestral:latest, llama3.1:70b
   Tunnel:    LLMHosts Relay (built-in, zero config)
   GPU:       NVIDIA RTX 4090 (24.0GB), 64GB RAM

   Try it now:

   curl https://llmhosts.com/v1/chat/completions \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer lh_sk_AbCdEfGh..." \
     -d '{
       "model": "llama3.1:8b",
       "messages": [{"role": "user", "content": "Hello!"}]
     }'

  ════════════════════════════════════════════════════════
   Press Ctrl+C to stop and unregister this device.
  ════════════════════════════════════════════════════════
```

## Step 4: Test It Works

From **any other machine** (your phone, a server, a coworker's laptop), run:

```bash
curl https://llmhosts.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer lh_sk_AbCdEfGh..." \
  -d '{
    "model": "llama3.1:8b",
    "messages": [{"role": "user", "content": "What is the meaning of life?"}]
  }'
```

You'll get back a standard OpenAI-format response, but the inference ran on YOUR GPU, in your house, for $0:

```json
{
  "id": "chatcmpl-abc123...",
  "object": "chat.completion",
  "model": "llama3.1:8b",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The meaning of life is a philosophical question that has been debated for centuries..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 14,
    "completion_tokens": 87,
    "total_tokens": 101
  }
}
```

### Test with streaming:

```bash
curl https://llmhosts.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer lh_sk_AbCdEfGh..." \
  -d '{
    "model": "llama3.1:8b",
    "messages": [{"role": "user", "content": "Write a haiku about GPUs"}],
    "stream": true
  }'
```

Tokens stream in real time, just like the OpenAI API.

## Step 5: Use It Everywhere

Because LLMHosts is a drop-in OpenAI API replacement, you can use it with any tool that supports the OpenAI API. Just set two environment variables:

```bash
export OPENAI_API_KEY=lh_sk_AbCdEfGh...
export OPENAI_BASE_URL=https://llmhosts.com/v1
```

Now every tool that uses the OpenAI SDK will automatically route through your GPU:

### Python (OpenAI SDK)

```python
from openai import OpenAI

client = OpenAI(
    api_key="lh_sk_AbCdEfGh...",
    base_url="https://llmhosts.com/v1",
)

response = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)
```

### LangChain

```python
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="llama3.1:8b",
    api_key="lh_sk_AbCdEfGh...",
    base_url="https://llmhosts.com/v1",
)

response = llm.invoke("Explain quantum computing in simple terms.")
print(response.content)
```

### Continue (VS Code AI Extension)

Edit `~/.continue/config.yaml`:

```yaml
models:
  - title: "My Home GPU"
    provider: openai
    model: llama3.1:70b
    apiKey: lh_sk_AbCdEfGh...
    apiBase: https://llmhosts.com/v1
```

### Cursor

In Cursor settings, add a custom model:

```
Provider: OpenAI Compatible
Base URL: https://llmhosts.com/v1
API Key: lh_sk_AbCdEfGh...
Model: llama3.1:70b
```

### Aider

```bash
aider --openai-api-key lh_sk_AbCdEfGh... \
      --openai-api-base https://llmhosts.com/v1 \
      --model llama3.1:70b
```

### Shell Alias (convenient one-liner)

Add to your `~/.bashrc` or `~/.zshrc`:

```bash
alias ask='curl -s https://llmhosts.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer lh_sk_AbCdEfGh..." \
  -d "{\"model\":\"llama3.1:8b\",\"messages\":[{\"role\":\"user\",\"content\":\"$1\"}]}" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)[\"choices\"][0][\"message\"][\"content\"])"'

# Usage:
ask "What is the capital of France?"
# Paris is the capital of France.
```

---

## Dashboard

Visit https://llmhosts.com/dashboard to see:

- **Devices:** All your registered devices, their status (online/offline), models, and hardware
- **API Keys:** Manage your keys (create, revoke, set expiration)
- **Usage:** Request counts, token usage, cost savings vs cloud APIs
- **Cache Stats:** Cache hit rates across all three tiers
- **Billing:** Current plan, usage against limits

---

## Configuration (Optional)

LLMHosts works great with zero configuration, but you can customize everything:

### Config file

Create `~/.llmhosts/config.toml`:

```toml
[server]
port = 4000              # Change local proxy port

[router]
prefer_local = true      # Always prefer local GPU over cloud
cost_optimize = true     # Optimize for lowest cost

[cache]
enabled = true           # Enable response caching
ttl_seconds = 3600       # Cache entries live for 1 hour

[security]
require_auth = true      # Require API key even on local proxy
rate_limit_rpm = 120     # Double the default rate limit
```

### Environment variables

Every config option can be set via environment variable:

```bash
export LLMHOST_SERVER_PORT=8080
export LLMHOST_CACHE_TTL_SECONDS=7200
export LLMHOST_ROUTER_PREFER_LOCAL=true
```

### CLI flags

```bash
llmhosts up --port 8080 --name "my-gpu-box" --tunnel cloudflare
```

---

## Stopping

Press `Ctrl+C` in the terminal where `llmhosts up` is running:

```
^C

  Shutting down...
  [1/3] Sending final heartbeat... done
  [2/3] Unregistering device "home-desktop"... done
  [3/3] Stopping tunnel... done

  Your GPU is offline. Run 'llmhosts up' again anytime.
```

Your device will show as "offline" in the dashboard. Any API requests will return a clear error message telling clients the device is unavailable.

---

## Troubleshooting

### "Tunnel not connecting"

The built-in LLMHosts Relay handles tunneling automatically. If the relay can't connect (rare), LLMHosts falls back to Tailscale or Cloudflare if installed:

```bash
# Install Tailscale (optional fallback): https://tailscale.com/download
# Install Cloudflare Tunnel (optional fallback):
# macOS: brew install cloudflared
# Linux: curl -L https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 -o /usr/local/bin/cloudflared
```

### "Ollama not reachable"

Make sure Ollama is running:

```bash
ollama serve
# Then in another terminal:
llmhosts up
```

Or if Ollama is on a different host:

```bash
export LLMHOST_OLLAMA_HOST=http://192.168.1.100:11434
llmhosts up
```

### "Device code expired"

The activation code is valid for 15 minutes. If it expires, just run `llmhosts up` again to get a new code.

### "Model not available"

The client is requesting a model that isn't installed on your device. Check what's available:

```bash
ollama list
```

Pull the requested model:

```bash
ollama pull <model-name>
```

### Slow responses

Check your internet upload speed. Streaming responses require upload bandwidth from your device to the client. Each streaming response uses about 0.1 Mbps, so even a slow connection can handle it, but multiple concurrent requests on a very slow connection may cause delays.

---

## FAQ

### Is this free?

The **Free tier** gives you 1 device and 1,000 requests/day. The **Pro tier** ($19/month) gives you 5 devices and unlimited requests. See https://llmhosts.com/pricing for all plans.

### Is my data private?

Yes. The relay uses outbound-only WebSocket tunnels — your device initiates the connection, so no incoming ports are required. End-to-end payload encryption (Noise NK protocol) is in active development and will be available in an upcoming release. For maximum privacy today, the built-in PII detector (IslandRun) automatically detects sensitive information (SSNs, credit cards, medical data) and routes those requests to your local GPU only, never through the relay.

You can also set `privacy_mode = "strict"` to force ALL requests to stay local, regardless of PII detection.

### Does this work behind my company's firewall?

Yes. LLMHosts uses outbound-only connections. Your device opens a WebSocket tunnel to the LLMHosts Relay (or falls back to Tailscale/Cloudflare if installed), which means no incoming ports need to be open. This works behind:
- Corporate firewalls
- Double NAT
- Cellphone hotspots
- Hotel WiFi
- University networks
- Any network that allows HTTPS outbound traffic

### What GPUs are supported?

Any GPU that Ollama supports:
- **NVIDIA:** GTX 1060+, RTX 2000/3000/4000/5000 series, Tesla, A100, H100
- **Apple Silicon:** M1, M2, M3, M4 (all variants)
- **AMD:** ROCm-supported GPUs (Radeon RX 7000 series, Instinct MI300)
- **CPU-only:** Works without a GPU, but much slower

### Can I add multiple GPUs/machines?

Yes (on Pro tier and above). Run `llmhosts up` on each machine. They all register with the same account and the relay automatically routes requests to the best available device based on model availability, latency, and load.

### What happens when my laptop sleeps?

The device goes offline after 90 seconds of missed heartbeats. API requests will return a clear error. When you wake it up and `llmhosts up` is still running, it automatically reconnects and re-registers.

### Is there a latency overhead?

The relay adds approximately 40-100ms of overhead per request (for API key validation, device selection, and tunnel transit). For comparison, the OpenAI API has similar latency just for their API gateway. The actual inference time (which dominates total latency) is the same as running locally.

### Can I use this with the Anthropic API format?

Yes. LLMHosts supports both the OpenAI format (`/v1/chat/completions`) and the Anthropic format (`/v1/messages`). Both work through the relay.

---

## Next Steps

- **Explore the dashboard:** https://llmhosts.com/dashboard
- **Add more models:** `ollama pull <model>` -- they'll appear automatically on next heartbeat
- **Add more devices:** Run `llmhosts up` on another machine with the same account
- **Enable smart caching:** `pip install llmhosts[smart]` for semantic caching that serves similar prompts from cache
- **Read the docs:** Full technical reference at [docs/gpu-as-api/README.md](README.md)
