# Why LLMHosts: The GPU-as-API Value Proposition

> You already own the hardware. LLMHosts turns it into real infrastructure.

---

## The Problem

You have a powerful GPU sitting under your desk. Maybe an RTX 4090, maybe an M2 Ultra Mac, maybe a multi-GPU workstation. You can run Ollama on it and get local inference. But the moment you close your laptop, leave the house, or want to use it from a different machine, your GPU becomes unreachable.

Meanwhile, you're paying OpenAI $20-200/month for API access. Or you're renting cloud GPUs at $2-5/hour. Or you're manually SSH tunneling and duct-taping configurations together every time you want remote access.

This is the gap LLMHosts closes.

---

## What LLMHosts Does

LLMHosts transforms your home GPU into a production cloud API endpoint that is:

- **Accessible from anywhere** -- behind any NAT, firewall, or VPN
- **Drop-in compatible** -- set two env vars and every OpenAI-compatible tool just works
- **Intelligent** -- routes, caches, and optimizes every request automatically
- **Private** -- your data stays on your hardware unless you explicitly allow otherwise
- **Zero-maintenance** -- one command, no infrastructure knowledge required

```bash
pip install llmhosts
llmhosts up
# Done. Your GPU is now a cloud API.
```

---

## The 7 Simultaneous Capabilities

No other product combines all seven of these capabilities in a single tool:

### 1. Home GPU Becomes a Cloud API

Your GPU serves the exact same API format as OpenAI. Any tool, library, or application that speaks the OpenAI API can use your GPU transparently. Change two environment variables and you're done:

```bash
export OPENAI_API_KEY=lh_sk_...
export OPENAI_BASE_URL=https://llmhosts.com/v1
```

LangChain, Continue, Cursor, Aider, custom Python scripts, shell scripts -- they all work without modification.

### 2. Works Behind Any NAT

LLMHosts uses outbound-only tunnel connections. Your GPU doesn't need a public IP address, open ports, or any network configuration. It works on:

- Home WiFi behind a consumer router
- Cellphone hotspot (double NAT)
- Corporate firewalls that only allow HTTPS outbound
- Hotel and airport WiFi
- University networks with restrictive policies
- Any connection that can reach the internet

The tunnel auto-detects Cloudflare Quick Tunnel (zero-config, no account needed) or Tailscale Funnel (stable URL, WireGuard encrypted).

### 3. Intelligent Three-Tier Routing

LLMHosts doesn't just forward requests. It makes intelligent routing decisions:

**Tier 0: Rule-based** (always available, <1ms)
- If you have the model locally, run it locally ($0 cost)
- If you request `gpt-4o` but have `llama3.1:70b` locally, suggest the local equivalent
- If a model is unavailable locally, fall back to cloud APIs

**Tier 1: kNN embedding router** (with `[smart]` install, <15ms)
- Learns which models work best for which types of queries
- Routes "write me a Python function" to Codestral, "explain this concept" to Llama

**Tier 2: ModernBERT classifier** (with `[smart]` install, <50ms)
- Transformer-based classifier that understands request intent
- Makes routing decisions that beat manual model selection

All three tiers have a confidence threshold. If Tier 0 is confident, it short-circuits immediately. Otherwise, it cascades to Tier 1, then Tier 2.

### 4. Verified Semantic Cache

The three-tier cache eliminates redundant inference:

**Tier 0: Exact hash** -- Identical prompts return cached results instantly (<1ms). Zero false positives.

**Tier 1: Namespace filter** -- Groups cached results by entity context. "What is the capital of France?" and "Tell me France's capital" share a namespace, so the second query can match a cached response.

**Tier 2: vCache semantic** -- Based on a paper from ICLR 2026 with formal correctness guarantees. Semantically similar prompts match cached responses with a provably bounded false positive rate (configurable, default 1%). This means "Explain quantum computing" can return a cached response from "What is quantum computing?" with mathematical guarantees about correctness.

In steady-state workloads, the combined cache achieves 60-85% hit rates, which means most of your requests are served in under 10ms with zero GPU utilization.

### 5. Drop-In OpenAI Replacement

LLMHosts implements the full OpenAI API spec:
- `POST /v1/chat/completions` (with streaming SSE)
- `POST /v1/messages` (Anthropic format)
- `GET /v1/models`
- Proper `usage` token counting
- Tool/function calling support
- All response headers (`X-Request-ID`, etc.)

This isn't a simplified subset. It's a complete API implementation that passes the OpenAI SDK's test suite.

### 6. Multi-Device Mesh

Run `llmhosts up` on multiple machines. They all register with the same account and form a mesh:

```
Home Desktop (RTX 4090, 24GB)  --> registered
Work Laptop (RTX 4070, 12GB)   --> registered
Mac Studio (M2 Ultra, 64GB)    --> registered
```

The relay automatically routes requests to the best device:
- Model availability: which device has the requested model?
- Latency: which device responds fastest?
- Load: which device has the most headroom?
- VRAM: which device can handle the model's memory requirements?

You get load balancing, failover, and multi-GPU inference across your entire fleet of hardware.

### 7. Privacy-First PII Routing

IslandRun is the built-in PII detection system that scans every request before routing:

- **Social Security Numbers** -- regex detection with checksum validation
- **Credit Card Numbers** -- Luhn algorithm validation
- **Email addresses, phone numbers, names** -- pattern and heuristic detection
- **Medical terminology** -- dictionary matching

When PII is detected, the request is forced to a local backend. It never leaves your machine. Never touches the cloud. Never traverses the relay.

You can also set `privacy_mode = "strict"` to force ALL requests local, or `privacy_mode = "paranoid"` for encrypted local-only routing.

---

## Comparison Table

| Feature | LLMHosts | Ollama Alone | ngrok + Ollama | RunPod / Vast.ai | OpenAI API |
|---|---|---|---|---|---|
| **Local inference** | Yes | Yes | Yes | No (cloud GPU) | No |
| **Remote access from anywhere** | Yes (auto-tunnel) | No | Yes (manual) | Yes | Yes |
| **Works behind NAT/firewall** | Yes (outbound tunnel) | N/A | Partial | N/A | N/A |
| **OpenAI-compatible API** | Yes (full spec) | No (Ollama format) | No | Vendor-specific | Yes |
| **Intelligent routing** | 3-tier hybrid | None | None | None | Internal |
| **Semantic caching** | 3-tier with formal guarantees | None | None | None | None |
| **PII detection & local routing** | Built-in | None | None | None | None |
| **Multi-device mesh** | Yes | No | No | No | N/A |
| **Setup time** | ~60 seconds | ~5 minutes | ~30 minutes | ~10 minutes | ~5 minutes |
| **Monthly cost (inference)** | $0 (your hardware) | $0 | $0 + ngrok plan | $0.50-5/hour | $20-200+/mo |
| **Data privacy** | Full control | Full control | You control | Cloud provider | OpenAI TOS |
| **Drop-in for existing tools** | Yes (env vars) | No (different API) | No (different URL format) | No | N/A (is the API) |
| **Automatic failover** | Yes | No | No | Manual | Yes |

### Head-to-Head Comparisons

#### vs. Ollama Alone

Ollama is fantastic for local inference. LLMHosts builds on top of it:
- Ollama exposes `localhost:11434` with its own API format. LLMHosts wraps it in the OpenAI API format and makes it accessible from anywhere.
- Ollama has no caching. LLMHosts adds three tiers of caching that can serve 60-85% of requests from cache.
- Ollama has no routing intelligence. LLMHosts picks the best model for each request automatically.
- Ollama has no privacy detection. LLMHosts scans for PII and enforces local routing.

LLMHosts uses Ollama as its inference engine. They are complementary, not competitive.

#### vs. ngrok + Manual Setup

You can technically expose Ollama via ngrok and build your own API wrapper. But:
- You need to manually set up the OpenAI API translation layer
- You need to manually configure authentication
- You need to handle reconnection when the ngrok URL changes
- You get no caching, no routing, no PII detection
- You need to build your own dashboard and monitoring
- Setup time: 30+ minutes of configuration vs. 60 seconds

LLMHosts automates all of this into a single command.

#### vs. RunPod / Vast.ai

Cloud GPU providers rent you remote GPUs:
- You pay $0.50-5/hour for GPU time (adds up fast)
- Your data runs on someone else's hardware
- You need to manage instances, deployments, and scaling
- No automatic routing or caching

LLMHosts is the opposite: use hardware you already own, pay nothing for inference, keep full control of your data.

#### vs. OpenAI API

OpenAI provides the best frontier models. LLMHosts provides infrastructure for open-source models on your own hardware:
- OpenAI costs $0.002-0.06 per 1K tokens. LLMHosts costs $0 (your hardware pays the electricity).
- OpenAI sees your data. LLMHosts keeps it on your machine.
- OpenAI has rate limits you can't control. LLMHosts has limits you configure.
- For tasks where Llama 3.1 70B or Codestral are sufficient (a lot of tasks), LLMHosts saves 100% of API costs.

LLMHosts supports BYOK (Bring Your Own Key) for cloud APIs too. You can route frontier-model requests to OpenAI through LLMHosts and get caching, routing, and PII detection on top.

#### vs. Clarifai Local Runners

Clarifai Local Runners let you run models on local hardware through their platform:
- Requires Clarifai account and their model ecosystem
- Tied to Clarifai's API format and tooling
- Commercial licensing
- No semantic caching or PII routing

LLMHosts is open-source, uses the universal OpenAI API format, works with any Ollama model, and includes caching and privacy features built in.

---

## Use Cases

### Individual Developers

**The daily driver.** You code with AI assistants (Continue, Cursor, Aider) and want to use your home GPU instead of paying for API credits. Set `OPENAI_BASE_URL=https://llmhosts.com/v1` and your development tools use your GPU from anywhere -- the coffee shop, a coworking space, on the train.

**Cost savings:** A developer making 500 API calls/day at $0.01 average per call pays $150/month to OpenAI. With LLMHosts Pro ($19/month) and their own GPU, they save $131/month.

### Small Teams (2-10 people)

**Shared GPU pool.** The team has a workstation with 2x RTX 4090s. Instead of each developer paying for their own OpenAI access, they share the GPUs via LLMHosts. The router automatically load-balances across both GPUs.

**Cost savings:** A team of 5 developers spending $100/month each on OpenAI = $500/month. LLMHosts Team plan ($49/seat for 5 seats = $245/month) + existing hardware = $255/month savings plus full data privacy.

### AI Startups

**Development and testing.** You're building an AI product and need to test against multiple models, run evaluations, and iterate quickly. LLMHosts lets you route between local models (free, fast iteration) and cloud models (frontier quality, for final testing) through the same API.

**Privacy compliance.** Your product handles user PII. LLMHosts' IslandRun ensures PII-containing requests never leave your infrastructure, while non-sensitive requests can optionally route to cloud models for better quality.

### Enterprise / On-Premises

**Air-gapped inference.** Regulations require that data never leaves the network. LLMHosts in `privacy_mode = "paranoid"` ensures all inference stays local. The semantic cache reduces GPU load by 60-85%, extending the capacity of existing hardware.

**GPU utilization.** Enterprise GPU clusters are often underutilized. LLMHosts can front-end these clusters with intelligent routing and caching, dramatically improving utilization rates.

---

## Cost Analysis

### Individual Developer Scenario

| | OpenAI API | LLMHosts (Free) | LLMHosts (Pro) |
|---|---|---|---|
| Hardware cost | $0 | $0 (already owned) | $0 (already owned) |
| Monthly subscription | $0 (pay-per-use) | $0 | $19 |
| Cost per request (~500 tokens) | ~$0.005 | $0 | $0 |
| 500 requests/day cost | ~$75/month | $0 | $0 |
| 2,000 requests/day cost | ~$300/month | $0 | $0 |
| Device limit | N/A | 1 | 5 |
| Privacy | OpenAI TOS | Full control | Full control |

**Break-even:** LLMHosts Pro pays for itself if you make more than ~4 API requests per day that would otherwise go to OpenAI.

### Team Scenario (5 developers)

| | OpenAI Team | LLMHosts Team |
|---|---|---|
| Monthly cost | $25/seat = $125 + usage (~$500) | $49/seat = $245 |
| Total per month | ~$625 | $245 |
| Annual savings | | **$4,560** |
| Data privacy | OpenAI TOS | Full control |
| Latency | 100-500ms (API) | 40-100ms (local network) |

### Electricity Cost

Running an RTX 4090 at full inference load consumes ~350W. At US average electricity rates ($0.16/kWh):
- 1 hour of continuous inference = $0.056
- 8 hours/day = $0.45/day = $13.50/month

Even accounting for electricity, local inference is 10-50x cheaper than cloud API pricing for equivalent model quality.

---

## What LLMHosts is NOT

To be honest about what we are and aren't:

- **Not a model provider.** We don't host models. You run them on your hardware via Ollama. We provide the infrastructure layer.
- **Not a replacement for frontier models.** GPT-4o and Claude Opus are still the best models for the hardest tasks. But for 80% of daily developer tasks, Llama 3.1 70B or Codestral running on your GPU is more than good enough -- and free.
- **Not a managed cloud service.** Your GPU, your hardware, your responsibility. If your laptop sleeps, your API goes offline (until you wake it up).
- **Not a VPN.** The tunnel is for API traffic only, not general networking.

---

## The Vision

Today, AI inference is centralized. A few companies control the APIs, the pricing, and the data. But the hardware is becoming commoditized. RTX 4090s cost $1,600. Mac Studios with 64GB unified memory cost $3,000. These machines can run models that were state-of-the-art 12 months ago.

LLMHosts is the infrastructure layer that turns this distributed hardware into a real platform. One command, and your GPU is a cloud API. Add more machines, and they form a mesh. The router optimizes every request. The cache eliminates redundant work. The privacy system keeps your data yours.

Your hardware. Real AI infrastructure. From anywhere.

---

## Getting Started

```bash
pip install llmhosts
llmhosts up
```

Read the [Quickstart Guide](quickstart.md) for the full walkthrough. See the [Technical Reference](README.md) for detailed architecture documentation.
