# LLMHosts Vision 2026 — Master Plan

**Date:** 2026-03-10
**Status:** Active — Engineering + Product Reference
**Author:** Claude (synthesis of 8 parallel research agents, 500+ searches)
**Epic:** [#471 — Vision 2026 Master Epic](https://github.com/LookNoHandsMom/LLMHosts.com/issues/471)
**Audience:** Engineering team, product team, investors

---

## Table of Contents

1. [Executive Summary](#1-executive-summary)
2. [Market Analysis](#2-market-analysis)
3. [The 10 Vision Pillars](#3-the-10-vision-pillars)
4. [Technology Architecture](#4-technology-architecture)
5. [Collective Intelligence Design](#5-collective-intelligence-design)
6. [Fine-Tuning Strategy](#6-fine-tuning-strategy)
7. [Benchmark Methodology](#7-benchmark-methodology)
8. [IP Protection Guide](#8-ip-protection-guide)
9. [User Journey — Micro + Macro](#9-user-journey--micro--macro)
10. [Implementation Roadmap](#10-implementation-roadmap)
11. [Risk Matrix](#11-risk-matrix)
12. [Success Metrics](#12-success-metrics)

---

## 1. Executive Summary

### What We're Building

LLMHosts turns consumer GPUs — the RTX 4090 on a developer's desk, the M4 Max MacBook Pro in a bag, the eight-GPU workstation in a home office — into production AI infrastructure. One command installs the software. One more command makes that GPU available from anywhere in the world, over a zero-trust encrypted tunnel, serving OpenAI-compatible and Anthropic-compatible APIs at latencies cloud providers cannot match.

By end of 2026, LLMHosts will be the definitive platform for the "bring your own GPU" economy. Not a wrapper around Ollama. Not another cloud console. A vertically-integrated stack from silicon to SaaS: our Rust inference engine, our relay protocol, our three-tier router, our semantic cache, our GPU marketplace, our fine-tuning pipeline — all orchestrated by a conversational fleet assistant that makes every user feel like they have an expert SRE running their infrastructure.

### Why Now

The market has arrived at a critical inflection:

- **Inference economics** have inverted: a $3,500 RTX 5090 pays for itself in cloud compute savings within 18 months at moderate usage
- **Model capability** has plateaued for most use cases: a 7B–13B quantized model beats GPT-4-class models on developer tasks run locally
- **Cloud trust is eroding**: enterprise data governance, GDPR Article 9, and CCPA opt-out rights are making cloud-first AI architecturally expensive beyond dollar cost
- **The local AI ecosystem is fragmented**: Ollama gave everyone inference but no distribution, no marketplace, no multi-device, no batching, no cross-platform

The window to define the category is 12–18 months. If we don't own "personal AI infrastructure," someone else will.

### The Bet

LLMHosts bets that:
1. The people who own GPUs are not hobbyists — they are the next generation of AI compute providers
2. Privacy-first local inference will grow from early-adopter curiosity to enterprise standard
3. Collective intelligence — anonymized learning across thousands of nodes — creates a compounding moat no single-node tool can match
4. Fine-tuning, once reserved for ML teams, will become a commodity accessed through a CLI

We are building the platform that makes all three bets pay off simultaneously.

---

## 2. Market Analysis

### 2.1 Market Size and Growth

**Inference spend is the new battleground.** By early 2026, inference accounts for 55% of AI cloud spend — approximately $37.5B globally — and is growing faster than training. This is the market we are disrupting.

| Market Segment | 2026 Estimate | CAGR | Our Angle |
|----------------|---------------|------|-----------|
| Self-hosted AI inference | $3–5B | 30% | Core market |
| GPU compute rental (consumer) | $1.5B | 45% | Marketplace (Phase 5) |
| Fine-tuning-as-a-service | $800M | 60% | Pillar 6 (#477) |
| Edge AI infrastructure | $12B | 35% | Expansion vector |
| AI developer tools | $5B | 40% | CLI + router |

Source: Gartner, IDC, Goldman Sachs AI infrastructure reports (2025–2026).

### 2.2 Competitive Landscape

#### Direct Competitors

**Ollama** (Go + llama.cpp, ~120K stars)
- Strength: Best UX benchmark, widest model support, active community
- Weakness: No batching, no PagedAttention, no multi-device, no marketplace, no datacenter features, NVIDIA + Apple only
- Our counter: We are 2–5× faster on the same hardware, cross-platform, and distributed-native

**SaladCloud** (distributed cloud, ~60K GPUs)
- Strength: 60,000+ consumer GPUs, RTX 4090 at $0.18/hr, massive scale
- Weakness: They are a marketplace for GPU time, not a product for GPU owners. Their UX is a procurement portal, not a developer tool
- Our counter: We monetize the same asset class (consumer GPUs) but the operator experience is an installable product, not a dashboard. We meet operators at `pip install llmhosts`

**vLLM** (Python + C++, ~45K stars)
- Strength: Best throughput for datacenter, continuous batching, PagedAttention, OpenAI-compatible
- Weakness: Complex install, Linux-only, NVIDIA-only (ROCm experimental), 2GB+ footprint, no model curation, no tunnel
- Our counter: We deliver equivalent features in a 15–20MB binary that works on Mac, Windows, AMD, and Intel Arc

**GPUStack** (Python, ~8K stars)
- Strength: Multi-GPU management dashboard
- Weakness: Wraps other backends (Ollama, llama.cpp, vLLM). Adds a UI layer but owns none of the stack. Feature velocity limited by upstream
- Our counter: We own the inference engine. We can ship features GPUStack can never have

#### Adjacent Competitors

| Platform | What They Do | Why We Win |
|----------|-------------|------------|
| CoreWeave | H100 cloud at $4.76/hr | We are 10–25× cheaper for equivalent capability |
| Together.ai | Cloud inference API | 200–500ms cloud TTFT vs our <20ms local TTFT |
| Replicate / Modal | Cloud GPU serverless | No cold starts, no egress fees, no per-call pricing |
| exo | LAN clustering (P2P) | No tunnel, no marketplace, no cloud spillover, single location only |

#### The Key Gap No Competitor Has Filled

No existing tool provides all of:
- One-command install
- Automatic GPU detection + model selection
- Continuous batching + PagedAttention
- Cross-platform (NVIDIA + AMD + Apple + Intel)
- Built-in encrypted tunnel with zero relay server dependency
- Cloud spillover when local is at capacity
- GPU marketplace for monetization
- Fine-tuning pipeline
- Collective intelligence from the fleet

LLMHosts Vision 2026 is the plan to build this complete stack.

### 2.3 Pricing Dynamics

**Cloud inference is declining** but not fast enough to close the gap with owned hardware:

| Service | 2025 Price | 2026 Trend | Local equivalent |
|---------|-----------|-----------|-----------------|
| GPT-4o (OpenAI) | $10/M tokens | Declining | $0.02/M tokens (owned RTX 4090) |
| Claude 3.5 Sonnet | $15/M tokens | Declining | $0.02/M tokens (owned RTX 4090) |
| Llama 3.1 70B (Together) | $0.40/M tokens | Declining fast | $0.005/M tokens (owned RTX 4090) |
| H100 (CoreWeave) | $4.76/hr | $3.50/hr by Q4 | $0.0 (already owned) |

The owned-hardware economics are not at risk from cloud price declines. The math works at $0.50/M tokens and still works if cloud drops to $0.10/M tokens.

**Egress** is the hidden multiplier: 70% of total cloud AI spend goes to egress for high-bandwidth applications. Local inference has zero egress. This is a $26B argument that accelerates over time.

### 2.4 The BYOG (Bring Your Own GPU) Economy

Consumer GPU VRAM has crossed the critical threshold:
- RTX 4090: 24GB — runs Llama 3.1 70B Q4 (the GPT-4-class open model) at ~25 tok/s
- RTX 5090: 32GB — runs Llama 3.1 70B Q5 at 35+ tok/s
- M4 Max: 128GB unified — runs the full 70B in half precision at 20+ tok/s
- A100 40GB (used): Available on eBay for $5,000 — better economics than cloud at >8 hours/day use

The people who own these GPUs are developers, researchers, and small studios. They are not running their hardware at full utilization 24/7. The idle capacity is our supply. The opportunity is aggregating it into a marketplace where demand — from developers who don't own hardware — can meet supply at prices 10× below hyperscalers.

LLMHosts captures both sides: the supply (GPU operators) and the demand (API consumers).

---

## 3. The 10 Vision Pillars

### Pillar 1 — Smart Install: Universal Platform Detection (#472)

**The Problem:** `pip install llmhosts` should be the last command a user ever needs to read documentation for. Today, we detect GPUs in Python code after install. This is too late — users with AMD ROCm or Intel Arc get worse experiences because we didn't know what we were installing for.

**The Vision:** Pre-install detection that selects the optimal binary variant, downloads the right compute backend, and configures everything before the user types their first real command.

**Architecture:**

```
pip install llmhosts
    ↓
setup.cfg / PEP 517 build hook (hatchling)
    ↓
detect_platform() → { gpu_vendor, compute_api, vram_gb, os }
    ↓
┌─────────────────────────────────────────────────────────────┐
│  Platform Matrix                                             │
│                                                             │
│  NVIDIA + CUDA 12.x  → llmhosts_core (cuda)     ★ first-class  │
│  AMD + ROCm 6.x      → llmhosts_core (wgpu→rocm) ★ first-class  │
│  Apple M-series      → llmhosts_core (wgpu→metal) ★ second-class │
│  Intel Arc + Level 0 → llmhosts_core (wgpu→vulkan)  experimental │
│  CPU only            → llmhosts_core (cpu+avx2)     fallback     │
│                                                                   │
│  Note: Apple Metal via wgpu is ~50-65% of native MPS performance │
│  (MPS not yet supported). NVIDIA CUDA is the performance leader.  │
└─────────────────────────────────────────────────────────────┘
    ↓
maturin wheel: correct binary for platform + GPU
    ↓
post-install: `llmhosts doctor` runs automatically
              prints: GPU detected, VRAM available, recommended model
```

**Implementation Path:**

1. `scripts/detect_platform.py` — runs at pip install time via hatchling hook
2. Platform enum → feature flag mapping (NVIDIA/CUDA, AMD/ROCm, Apple/Metal, Intel/Vulkan, CPU)
3. PyPI publish: one sdist + platform-specific wheels (manylinux2014, macosx_arm64, win_amd64)
4. `llmhosts/gpu/detect.py` — Python wrapper calling `llmhosts_gpu` Rust crate
5. `llmhosts doctor` — enhanced with GPU-specific diagnostic output

**Success Metrics:**
- Zero post-install configuration required for 95% of users
- `llmhosts doctor` passes first-time on NVIDIA, AMD, Apple, Intel Arc
- Install-to-first-inference under 5 minutes on hardware with pre-downloaded model
- Support matrix: CUDA 11.8+, ROCm 6.0+, Metal 3+, Vulkan 1.3+, AVX2 CPU

**GitHub Issue:** [#472](https://github.com/LookNoHandsMom/LLMHosts.com/issues/472)

---

### Pillar 2 — Self-Healing & Zero-Ops (#473)

**The Problem:** A GPU server running in a closet is not managed by an SRE. When the CUDA driver crashes, the model OOMs, the relay drops, or disk fills up — the user gets error messages they don't understand. We claim "it just works." Self-healing is what makes that claim true at 3am.

**The Vision:** LLMHosts monitors itself, diagnoses failures, and recovers automatically. When auto-recovery is not safe, it sends a plain-English notification explaining what happened and what the user needs to do.

**Failure Taxonomy and Auto-Recovery:**

| Failure Mode | Detection | Auto-Recovery | User Notification |
|-------------|-----------|--------------|-------------------|
| Model OOM | VRAM monitor exceeds 95% | Evict cached model, reduce batch size, retry | Only if recovery fails |
| Relay disconnect | Heartbeat timeout 30s | Reconnect with exponential backoff (1s, 2s, 4s, max 60s) | After 5 minutes of failed reconnects |
| CUDA driver error | Non-zero cudarc return code | Unload model, reset GPU context, reload | If reset fails after 3 attempts |
| Inference stall | Token generation > 60s for batch | Kill hung worker, restart inference process | Log only |
| Disk full | <1GB free on model cache dir | Evict LRU cached models | Before eviction (user may want to keep) |
| Backend crash | Process exit, port no longer bound | Systemd/launchd restart (auto-configured at install) | After 3 crashes in 5 minutes |
| Temperature throttle | GPU temp > 83°C (NVML threshold) | Reduce batch size 50%, alert | Immediately |
| Token budget exceeded | User-configured limit reached | Queue requests, reject new until reset | Immediately |

**Circuit Breaker Implementation:**

```rust
// llmhosts_core/src/compute/circuit_breaker.rs
pub struct CircuitBreaker {
    state: Arc<Mutex<CircuitState>>,
    failure_threshold: u32,  // trips after N failures
    recovery_timeout: Duration,  // half-open probe window
    success_threshold: u32,  // closes after N successes in half-open
}

pub enum CircuitState {
    Closed { failure_count: u32 },
    Open { opened_at: Instant },
    HalfOpen { success_count: u32 },
}
```

**Systemd / Launchd Auto-Configuration:**

At install, `llmhosts setup --service` configures:
- Linux: `/etc/systemd/system/llmhosts.service` with `Restart=always`, `RestartSec=5`, `WatchdogSec=30`
- macOS: `~/Library/LaunchAgents/com.llmhosts.plist` with `KeepAlive=true`, `ThrottleInterval=5`
- Windows: `New-Service llmhosts` with `StartupType=Automatic`, `RestartAfterFailure`

**Chaos Validation Protocol:**

Before shipping any self-healing feature, we run a chaos suite:
- Kill inference process mid-generation → verify restart + in-flight request handling
- Saturate VRAM → verify graceful degradation, not crash
- Disconnect relay → verify reconnect without user action
- Fill disk to 99% → verify LRU eviction without data loss
- Corrupt model file → verify detection + re-download prompt

**Success Metrics:**
- MTTR (mean time to recovery) < 60 seconds for all auto-recoverable failures
- Zero "unexplained errors" in user-visible surfaces — every error has a plain-English description + action
- Self-healing coverage: 80% of failure modes handled without user intervention
- 99.5% uptime SLA for relay connectivity (auto-reconnect handles transient failures)

**GitHub Issue:** [#473](https://github.com/LookNoHandsMom/LLMHosts.com/issues/473)

---

### Pillar 3 — Goal-Based Onboarding (#474)

**The Problem:** Current onboarding is tool-centric: "here are commands, here is a proxy, here is a dashboard." This creates friction because users don't think in tools — they think in goals: "I want to stop paying OpenAI $300/month," "I want to run code on my GPU while traveling," "I want my team to use my hardware without me being in the loop."

**The Vision:** LLMHosts onboards users by asking one question: "What do you want to do?" and then configures itself to do exactly that. The technical setup happens invisibly.

**The Three Goals:**

```
┌────────────────────────────────────────────────────────────────────┐
│                   llmhosts setup                                    │
│                                                                     │
│  What do you want to do?                                           │
│                                                                     │
│  [1] Stop paying cloud bills — use my GPU locally                  │
│      → Installs inference engine, configures proxy,                 │
│        detects best model for hardware, starts serving              │
│                                                                     │
│  [2] Use my GPU from anywhere — work remotely                      │
│      → All of the above + configures relay tunnel,                 │
│        generates shareable API endpoint with auth key               │
│                                                                     │
│  [3] Share my GPU + earn compute credits                           │
│      → All of the above + registers on marketplace,                 │
│        sets rate and availability schedule                          │
│                                                                     │
└────────────────────────────────────────────────────────────────────┘
```

**Onboarding Flow (Goal 1 — Local Savings):**

```
1. Hardware Detection (automatic, 3 seconds)
   GPU: NVIDIA RTX 4090 (24 GB VRAM)
   Recommended model: Llama 3.3 70B Q4_K_M (fits in VRAM with 2GB headroom)
   Expected performance: ~45 tokens/second

2. Baseline Capture (one question)
   "How much did you spend on AI APIs last month?"
   → Stores as baseline for savings dashboard

3. Model Download (background, with progress)
   Downloading llama3.3-70b-q4_k_m.gguf (26 GB) ... [=====>    ] 52%
   Estimated 8 minutes remaining

4. Proxy Config (automatic)
   OpenAI-compatible API: http://localhost:4000/v1
   Anthropic-compatible API: http://localhost:4000/v1/messages

5. First Request (automated verification)
   Testing inference... ✓ 47 tokens/second
   Testing cache... ✓ 0.2ms exact hit
   Testing routing... ✓

6. Your API is ready.
   Set OPENAI_BASE_URL=http://localhost:4000/v1
   Your current tooling (Cursor, VS Code, Continue) will now use your GPU.
```

**Adaptive Onboarding Intelligence:**

The onboarding flow learns from fleet behavior (Pillar 5):
- If 70% of users with RTX 4090 switch to a different model within 48 hours, we update the HIM recommendation
- If a hardware configuration fails setup more than 5% of the time, we flag it and insert a specialized diagnostic
- Common failure patterns surface as proactive warnings before they happen

**Delight Moments:**
- Real-time savings ticker starts the moment first inference request is served locally
- "Your first request saved $0.003 vs cloud" — granular and immediate
- Comparison benchmark auto-runs: "Your RTX 4090 is 12× faster than GPT-4o TTFT for this workload"

**Success Metrics:**
- Time from `pip install llmhosts` to first successful local inference: < 10 minutes (excluding model download)
- Onboarding completion rate: > 75% (users who run setup reach first inference)
- 7-day retention: > 60% (users still active after one week)
- NPS from onboarding cohort: > 60

**GitHub Issue:** [#474](https://github.com/LookNoHandsMom/LLMHosts.com/issues/474)

---

### Pillar 4 — Collective Intelligence & Privacy-Preserving Learning (#475)

**The Vision:** Every LLMHosts node contributes anonymized signals to a shared intelligence layer. The fleet, collectively, learns what models perform best on what hardware, which router decisions produce the best outcomes, where cache hit rates are low, and which failure modes are most common. This learning is available to every node in the fleet — making each individual installation better because of every other installation.

This is the compounding moat. It cannot be replicated by Ollama (single-node), vLLM (no fleet), or SaladCloud (they are the cloud, not the network).

See [Section 5](#5-collective-intelligence-design) for full technical design.

**GitHub Issue:** [#475](https://github.com/LookNoHandsMom/LLMHosts.com/issues/475)

---

### Pillar 5 — One Unified API + Intelligent Router (#476)

**The Problem:** Users juggle multiple AI providers — OpenAI for some tasks, Anthropic for others, their local GPU for price-sensitive workloads. The switching is manual, the context is lost, and the cost visibility is zero.

**The Vision:** One API endpoint that makes the right choice automatically — local GPU, cloud fallback, or marketplace node — based on privacy requirements, cost sensitivity, latency targets, and model capability requirements.

**The Unified API Contract:**

LLMHosts exposes:
- `POST /v1/chat/completions` — OpenAI-compatible
- `POST /v1/messages` — Anthropic-compatible
- `GET /v1/models` — lists all available backends (local + cloud + marketplace)
- `GET /v1/routing/explain` — returns last routing decision with full rationale

Every response includes headers:
```
X-LLMHosts-Backend: local|cloud|marketplace
X-LLMHosts-Model: llama3.3-70b-q4_k_m
X-LLMHosts-Latency-Ms: 23
X-LLMHosts-Cost-Per-1M-Tokens: 0.018
X-LLMHosts-Cache-Hit: false
X-LLMHosts-Privacy-Tier: personal|private|external
X-LLMHosts-Request-ID: req_01J8X...
```

**Router Architecture — Four Tiers:**

```
Request
  ↓
[Privacy Filter] — IslandRun: PII detected → personal (local only)
  ↓
[Tier 0: Rule-Based] — explicit model request, emergency fallback
  ↓
[Tier 1: kNN (FAISS)] — vector similarity, <15ms CPU, confidence ≥ 0.85
  ↓
[Tier 2: ModernBERT] — 6-signal classification, <50ms, confidence ≥ 0.75
  ↓
[Tier 3: Router LLM] — Qwen2.5-0.5B Q4, <500ms, edge case resolution
  ↓
[Backend Selection] — local GPU | cloud (BYOK) | marketplace node
  ↓
Response
```

**Privacy Tiers:**
- `personal` (trust=1.0): medical, financial, HR data → local GPU only, never routed externally
- `private` (trust=0.7): internal business context → local or enterprise-approved marketplace nodes
- `external` (trust=0.4): public context, code, general knowledge → any backend

**Cost-Latency Optimization:**

The router respects user-configured optimization targets:
```python
# User preference (stored in config)
optimization_target: "cost" | "latency" | "quality" | "balanced"

# Routing matrix
if target == "cost":
    prefer_local → prefer_marketplace → cloud_only_if_local_unavailable
if target == "latency":
    prefer_local → prefer_cloud (low TTFT) → marketplace
if target == "quality":
    model_capability_score determines backend regardless of cost
```

**Proprietary Router Model — The Moat:**

The Qwen-0.5B Tier 3 router is fine-tuned on our collective intelligence dataset (Pillar 4). Every routing decision in the fleet generates a training example. The model learns the mapping from `{prompt_embedding, user_prefs, hardware_profile, time_of_day}` → `{optimal_backend, confidence}`. This fine-tuned model is not open-source. It is the distillation of routing intelligence from millions of real requests.

See [Section 6](#6-fine-tuning-strategy) for the training pipeline.

**Success Metrics:**
- Router accuracy: > 90% agreement with human-labeled "optimal" backend choice
- Tier 1 latency: < 15ms (kNN lookup)
- Tier 3 latency: < 500ms (Qwen inference on CPU)
- Cloud spillover rate: < 5% when local GPU is available and appropriate

**GitHub Issue:** [#476](https://github.com/LookNoHandsMom/LLMHosts.com/issues/476)

---

### Pillar 6 — Fine-Tuning as a Service + Proprietary Model Moat (#477)

**The Vision:** LoRA fine-tuning on local hardware, initiated from the CLI, results in a private adapter that the user owns. The adapter can be deployed locally or, for Pro+ users, shared to marketplace nodes. The LLMHosts routing model is continuously fine-tuned on fleet data — creating a compounding moat.

See [Section 6](#6-fine-tuning-strategy) for full implementation detail.

**GitHub Issue:** [#477](https://github.com/LookNoHandsMom/LLMHosts.com/issues/477)

---

### Pillar 7 — World-Class Benchmark Suite (#478)

**The Vision:** LLMHosts publishes an honest, reproducible benchmark that compares our inference engine to Ollama, vLLM, and cloud APIs on real hardware. We do not cherry-pick benchmarks. We publish every result, including where we lose.

See [Section 7](#7-benchmark-methodology) for full methodology.

**GitHub Issue:** [#478](https://github.com/LookNoHandsMom/LLMHosts.com/issues/478)

---

### Pillar 8 — IP Protection & Marketing Language (#479)

**The Vision:** Our technical architecture (CubeCL + cudarc, PagedAttention implementation, router model weights) is our moat. We protect it through trade secret posture, careful marketing language, and strategic open-sourcing of non-critical components.

See [Section 8](#8-ip-protection-guide) for approved vocabulary and forbidden disclosures.

**GitHub Issue:** [#479](https://github.com/LookNoHandsMom/LLMHosts.com/issues/479)

---

### Pillar 9 — AI Advisory Council Session (#480)

**The Vision:** An internal AI-facilitated red team session that debates every architectural decision in the masterplan. Multi-perspective analysis: security, performance, privacy, business model, competitive response. Output is a structured debate document that becomes the grounding for Q2 2026 architecture reviews.

**Format:** Eight AI "advisors" each representing a perspective:
1. Performance Engineer: latency, throughput, VRAM efficiency
2. Security Researcher: attack surface, supply chain, data exfiltration
3. Privacy Advocate: GDPR compliance, federated learning guarantees
4. Competitive Analyst: how does each decision land vs. Ollama, vLLM, SaladCloud
5. Business Model Expert: monetization, unit economics, pricing
6. Developer Advocate: UX, documentation, adoption friction
7. Distributed Systems Engineer: fault tolerance, consensus, partitioning
8. Legal/Compliance: trade secrets, open-source licenses, CCPA

**Output format:** `docs/plans/2026-Q2-advisory-council-debate.md` — structured as pro/con/recommendation for each of the 10 pillars.

**GitHub Issue:** [#480](https://github.com/LookNoHandsMom/LLMHosts.com/issues/480)

---

### Pillar 10 — GitHub Harmony v2.0.0 Audit (#481)

**The Vision:** As the codebase crosses 10 Rust crates + Python monolith + Next.js frontend, GitHub becomes harder to navigate. GitHub Harmony v2 audits and restructures: issue taxonomy, milestone alignment, CI/CD efficiency, branch protection rules, and documentation cross-references.

**Scope:**
- Audit all 480+ issues: label consistency, milestone assignment, epic hierarchy
- CI optimization: parallel job matrix for Rust cross-compilation (NVIDIA/AMD/Apple wheels)
- Branch protection: add required status checks for new Rust crate tests
- Release automation: maturin build matrix → PyPI + GitHub Releases
- Documentation: all ADRs linked from relevant issues; AGENTS.md kept as primary cross-tool grounding

**GitHub Issue:** [#481](https://github.com/LookNoHandsMom/LLMHosts.com/issues/481)

---

## 4. Technology Architecture

### 4.1 System Overview

```
┌──────────────────────────────────────────────────────────────────────────┐
│                         LLMHosts Platform                                │
│                                                                          │
│  ┌─────────────────────┐    ┌─────────────────────┐                     │
│  │   Rust Layer        │    │   Python Layer       │                     │
│  │                     │    │                      │                     │
│  │  llmhosts_core      │    │  FastAPI Proxy       │                     │
│  │  ├── transformer    │◄───┤  ├── dispatcher.py   │                     │
│  │  ├── kv_cache       │    │  ├── router/         │                     │
│  │  ├── scheduler      │    │  ├── cache/          │                     │
│  │  └── generate       │    │  ├── metrics/        │                     │
│  │                     │    │  └── marketplace/    │                     │
│  │  llmhosts_gpu       │    │                      │                     │
│  │  ├── nvidia.rs      │    │  CLI (Click)         │                     │
│  │  ├── amd.rs         │    │  ├── up.py           │                     │
│  │  └── apple.rs       │    │  ├── serve           │                     │
│  │                     │    │  ├── doctor          │                     │
│  │  relay_core         │    │  └── finetune        │                     │
│  │  ├── tunnel/        │    │                      │                     │
│  │  └── noise_nk/      │    │  Hive.CEO Agents     │                     │
│  │                     │    │  └── monitoring      │                     │
│  │  router_core        │    │                      │                     │
│  │  └── knn/           │    │                      │                     │
│  └─────────────────────┘    └─────────────────────┘                     │
│                                        │                                 │
│  ┌─────────────────────────────────────▼──────────────────────────────┐  │
│  │                     Next.js Frontend                                │  │
│  │  Landing | Pricing | Auth | Dashboard | Marketplace | Docs | Token │  │
│  └────────────────────────────────────────────────────────────────────┘  │
│                                                                          │
│  ┌───────────────┐  ┌───────────────┐  ┌───────────────────────────┐   │
│  │ SQLite/Prisma │  │ Redis Cache   │  │ Prometheus + Grafana      │   │
│  └───────────────┘  └───────────────┘  └───────────────────────────┘   │
└──────────────────────────────────────────────────────────────────────────┘
          │                                        │
          │ pyo3/maturin bridge                    │ HTTPS / WSS
          ▼                                        ▼
   GPU Hardware                            Remote Client
   (CUDA/ROCm/Metal/Vulkan)               (curl, Python, Cursor, VS Code)
```

### 4.2 Rust Crate Architecture

#### `llmhosts_core` — Inference Engine

```
llmhosts_core/src/
├── lib.rs              — pyo3 module registration + public API surface
├── model/
│   ├── mod.rs          — LLMHostsModel trait + registry
│   ├── llama.rs        — LLaMA 2/3 architecture implementation
│   ├── qwen.rs         — Qwen 2.5 architecture
│   └── mistral.rs      — Mistral / Mixtral (planned F3)
├── gguf.rs             — GGUF format parser (zero-copy mmap)
├── tensor.rs           — LLMHostsTensor — shape, dtype, data pointer
├── quant.rs            — Q4_0, Q4_K_M, Q5_0, Q5_K_M, Q6_K, Q8_0
├── attention.rs        — Grouped Query Attention (GQA) implementation
├── ffn.rs              — SwiGLU feed-forward network
├── rope.rs             — Rotary Position Embedding (RoPE)
├── kv_cache.rs         — PagedAttention KV cache manager
├── generate.rs         — Autoregressive generation loop + sampling
├── compute/
│   ├── mod.rs          — ComputeBackend trait
│   ├── cpu.rs          — Reference CPU impl (AVX2 SIMD)
│   ├── cuda.rs         — cudarc NVIDIA backend
│   └── wgpu.rs         — CubeCL/WGPU cross-platform backend
├── memory/
│   ├── mod.rs          — MemoryManager
│   └── paged.rs        — PagedAttention block allocator
└── kernels/
    ├── matmul.cu       — CUDA matmul (cudarc)
    └── attention.cu    — Flash Attention v2 (cudarc, Ampere+)
```

**Key Types:**

```rust
// Public Python-facing API (pyo3)
#[pyclass]
pub struct LLMHostsEngine {
    model: Option<Arc<dyn LLMHostsModel + Send + Sync>>,
    scheduler: ContinuousBatchScheduler,
    kv_cache: PagedKVCache,
    compute: Box<dyn ComputeBackend>,
}

#[pymethods]
impl LLMHostsEngine {
    pub fn load_model(&mut self, path: &str) -> PyResult<()>;
    pub fn generate(&self, prompt: &str, params: GenerateParams) -> PyResult<GenerateResult>;
    pub fn generate_tokens(&self, token_ids: Vec<u32>, params: GenerateParams) -> PyResult<Vec<u32>>;
    pub fn unload_model(&mut self) -> PyResult<()>;
    pub fn stats(&self) -> PyResult<EngineStats>;
}
```

**Performance Targets:**

| Hardware | Model | Target tok/s | Baseline (Ollama) | Ratio |
|----------|-------|-------------|------------------|-------|
| RTX 4090 (24GB) | Llama 3.3 70B Q4_K_M | ≥ 50 | ~25 | 2× |
| RTX 4090 (24GB) | Qwen 2.5 7B Q5_K_M | ≥ 180 | ~90 | 2× |
| M4 Max (128GB) | Llama 3.3 70B Q4_K_M | ≥ 25 | ~20 | 1.25× |
| A100 40GB | Llama 3.3 70B Q4_K_M | ≥ 150 | N/A (vLLM ref: 80) | 1.9× |
| RTX 4060 (8GB) | Qwen 2.5 3B Q6_K | ≥ 300 | ~150 | 2× |

#### `llmhosts_gpu` — GPU Detection

```
llmhosts_gpu/src/
├── lib.rs              — pyo3 module + GPUInfo Python class
├── detect.rs           — platform dispatcher
├── nvidia.rs           — NVML runtime loading, per-device info
├── amd.rs              — ROCm sysfs + rocm-smi runtime loading
├── apple.rs            — IOKit Metal device enumeration
├── intel.rs            — Level Zero API (planned F3)
└── types.rs            — GPUVendor, GPUInfo, VRAMInfo structs
```

#### `relay_core` — Tunnel Connectivity

```
relay_core/src/
├── lib.rs              — pyo3 module + RelayClient Python class
├── tunnel/
│   ├── client.rs       — WSS client with yamux multiplexing
│   └── server.rs       — relay server (cloud-hosted, optional self-host)
├── noise/
│   ├── nk.rs           — Noise NK handshake implementation
│   └── session.rs      — encrypted session management
└── forward/
    ├── http.rs         — HTTP request forwarding over tunnel
    └── stream.rs       — SSE/WebSocket forwarding
```

#### `router_core` — kNN Inference

```
router_core/src/
├── lib.rs              — pyo3 module + RouterCore Python class
├── knn.rs              — FAISS-backed kNN router (< 15ms)
├── embed.rs            — embedding cache + ONNX runtime bridge
└── index.rs            — FAISS index management (load, update, save)
```

### 4.3 Data Flow — Request Lifecycle

```
1. Client sends POST /v1/chat/completions
   {"model": "auto", "messages": [...], "stream": true}

2. FastAPI dispatcher.py receives request
   a. Assign X-LLMHosts-Request-ID
   b. IslandRun PII classification → privacy_tier

3. Three-tier router selects backend
   a. Tier 0: Rule-based (explicit model name, emergency)
   b. Tier 1: kNN (FAISS) — prompt embedding → nearest routing decision
   c. Tier 2: ModernBERT — if kNN confidence < 0.85
   d. Tier 3: Qwen-0.5B Router LLM — if ModernBERT confidence < 0.75
   → Selected: local GPU (privacy_tier=personal) | cloud | marketplace

4. vCache semantic cache check
   a. Tier 0: SHA-256 exact hash lookup (0.1ms)
   b. Tier 1: Entity namespace filter
   c. Tier 2: Semantic similarity (ONNX, threshold per-prompt)
   → HIT: return cached response immediately, log savings

5. Backend dispatch
   a. Local: dispatcher calls LLMHostsEngine.generate() via pyo3
   b. Cloud: LiteLLM adapter → OpenAI/Anthropic API
   c. Marketplace: relay_core HTTP forward → remote node

6. Streaming response
   a. LLMHostsEngine yields tokens via async generator
   b. FastAPI streams SSE to client
   c. Concurrent: audit log written, metrics updated, collective intelligence signal queued

7. Post-request (async, non-blocking)
   a. Cost tracking updated
   b. Collective intelligence signal batch-flushed to aggregator (≥ 1000 signals or 60s interval)
   c. Cache population (if cacheable, below TTL threshold)
   d. Router training example recorded (prompt_embedding, chosen_backend, latency, quality_signal)
```

### 4.4 API Contracts

**Collective Intelligence Upload (client → aggregator):**
```json
POST /api/v1/intelligence/batch
{
  "node_id": "hmac_sha256(device_fingerprint, secret)",
  "signals": [
    {
      "type": "routing_decision",
      "prompt_embedding_hash": "sha256(embedding_bytes)",
      "chosen_backend": "local",
      "latency_ms": 23,
      "tokens_per_second": 47,
      "privacy_tier": "personal",
      "hardware_profile_hash": "sha256(gpu_vendor+vram_tier+os)"
    }
  ],
  "dp_noise_added": true,
  "epsilon": 0.5
}
```

**Hardware Intelligence Matrix Query (client ← aggregator):**
```json
GET /api/v1/him?gpu=rtx4090&vram_gb=24&os=linux
Response:
{
  "recommended_model": "llama3.3-70b-q4_k_m",
  "expected_toks_per_sec": 47,
  "vram_usage_gb": 22.1,
  "confidence": 0.94,
  "sample_count": 1247,
  "alternatives": [
    {"model": "qwen2.5-72b-q4_k_m", "expected_tps": 45, "notes": "better at code tasks"}
  ]
}
```

---

## 5. Collective Intelligence Design

### 5.1 What Gets Learned

The fleet generates five categories of anonymized signals:

| Signal Type | What It Tells Us | Example Application |
|-------------|-----------------|---------------------|
| Routing decisions | Which backend produces best outcome for prompt type | Improve Tier 1 kNN index |
| Hardware × model performance | tok/s, VRAM usage, temperature for each GPU × model pair | Update HIM recommendations |
| Cache hit patterns | Which prompt categories benefit most from semantic cache | Tune per-category thresholds |
| Failure modes | Which hardware configurations fail most, which errors repeat | Proactive warnings in `llmhosts doctor` |
| Fine-tune adapter quality | Which LoRA adapters generalize vs. overfit | Fine-tuning guidance improvements |

### 5.2 Privacy Architecture

**Absolute guarantees (written into code, not just policy):**
1. No raw prompt text ever leaves the device
2. No user identifiers in any signal (node_id is a rotated HMAC, never a user ID)
3. All embeddings are hashed before transmission — the server receives `sha256(embedding)`, not the embedding
4. Differential privacy noise (Gaussian mechanism, ε ≤ 1.0) added locally before transmission
5. k-anonymity: signals only included in aggregates if k ≥ 100 matching signals exist
6. Opt-out is a config flag: `telemetry: false` in `~/.config/llmhosts/config.toml` — verified with network trace in CI

**Data that does leave the device (collective intelligence signals):**
- Hardware profile hash (GPU vendor + VRAM tier bucket + OS family — not exact model or spec)
- Prompt embedding hash (not the embedding, not the prompt)
- Routing decision (backend chosen, latency, tokens/second)
- Cache tier hit/miss (no prompt content)
- Error codes (not stack traces, not user data)

**Federated Learning Architecture (Phase F4):**

For model weight updates (router LLM fine-tuning), we adopt the Flower framework pattern:
```
Server (aggregator)
  ├── Global model weights (router LLM)
  ├── FedAvg coordination
  └── Differential privacy budget tracking

Client (each LLMHosts node, opt-in only)
  ├── Local training on local routing decisions (100% local data)
  ├── Compute gradient updates only
  ├── Add DP noise to gradients (ε_local = 0.1)
  └── Send gradient updates (not data) to aggregator
```

This follows the Flower FlowerLLM pattern and is production-validated in healthcare (HIPAA) and satellite (Starcloud) deployments.

### 5.3 Hardware Intelligence Matrix (HIM)

The HIM is a continuously updated database mapping hardware profiles to optimal model configurations:

```
hardware_profile = {
    "gpu_vendor": "nvidia" | "amd" | "apple" | "intel" | "cpu",
    "vram_tier": "4gb" | "8gb" | "12gb" | "16gb" | "24gb" | "32gb" | "48gb" | "80gb" | "128gb+",
    "cpu_arch": "x86_64" | "arm64",
    "os_family": "linux" | "macos" | "windows"
}

model_recommendation = {
    "model_id": "llama3.3-70b-q4_k_m",
    "quantization": "Q4_K_M",
    "expected_tps": 47,
    "vram_usage_gb": 22.1,
    "ctx_length": 32768,
    "batch_size": 4,
    "confidence": 0.94,
    "sample_count": 1247,
    "last_updated": "2026-03-10"
}
```

**Initial HIM seeding (our own benchmarks, before fleet data):**

| GPU | VRAM | Model | Quant | tok/s (target) |
|-----|------|-------|-------|---------------|
| RTX 4060 Ti | 8 GB | Qwen 2.5 7B | Q4_K_M | 80 |
| RTX 4070 | 12 GB | Mistral 7B | Q5_K_M | 100 |
| RTX 4070 Ti | 12 GB | Llama 3.1 8B | Q5_K_M | 120 |
| RTX 4080 | 16 GB | Llama 3.1 13B | Q4_K_M | 90 |
| RTX 4090 | 24 GB | Llama 3.3 70B | Q4_K_M | 50 |
| RTX 5090 | 32 GB | Llama 3.3 70B | Q5_K_M | 70 |
| M2 Pro | 32 GB | Llama 3.3 70B | Q4_K_M | 20 |
| M4 Max | 128 GB | Llama 3.3 70B | Q8_0 | 18 |
| A100 40GB | 40 GB | Llama 3.1 70B | Q8_0 | 150 |

### 5.4 Compliance

**GDPR Article 9:** We process no sensitive categories of data. Prompt content never leaves the device. Embedding hashes are one-way transformations with no practical inversion risk.

**CCPA:** Right to opt-out is honoured by the `telemetry: false` config flag. Right to know: our privacy policy enumerates exactly which signal types are collected. Right to delete: node-level signals are keyed by rotated HMAC, not user ID — deletion is impossible because we have no linkage.

**HIPAA (for healthcare enterprise users):** Data Processing Agreement required. Collective intelligence disabled by default in enterprise tier. Federated learning (FL mode) available on Enterprise plan as an alternative.

---

## 6. Fine-Tuning Strategy

### 6.1 User-Facing Fine-Tuning (LoRA Pipeline)

**Target user:** A developer who wants a model fine-tuned on their company's internal documentation, codebase, or domain-specific task. They do not know what LoRA is. They have a folder of files and a GPU.

**CLI interface:**
```bash
# Step 1: prepare training data
llmhosts finetune prepare \
    --input ./docs/ \
    --format jsonl \
    --output ./finetune-data/

# Step 2: run fine-tuning
llmhosts finetune run \
    --base-model llama3.3-70b-q4_k_m \
    --data ./finetune-data/ \
    --adapter-name my-docs-adapter \
    --epochs 3 \
    --target-gpu rtx4090

# Step 3: test the adapter
llmhosts finetune test \
    --adapter my-docs-adapter \
    --prompt "What is our API rate limit policy?"

# Step 4: deploy locally
llmhosts serve --adapter my-docs-adapter

# Step 5 (Pro+): deploy to marketplace
llmhosts marketplace publish --adapter my-docs-adapter --private
```

**Technical stack:**

```python
# llmhosts/finetune/pipeline.py

class LoRAPipeline:
    """Manages end-to-end LoRA fine-tuning on local GPU."""

    def prepare(self, input_dir: Path, output_path: Path) -> DatasetStats:
        """Convert raw files to Alpaca/ShareGPT JSONL format."""
        # Uses LlamaIndex document loaders for PDF, DOCX, MD, code
        # Chunks, deduplicates, quality-filters
        # Returns stats: n_examples, token_count, quality_score

    def run(self, config: LoRAConfig) -> AdapterResult:
        """Execute fine-tuning via Unsloth (2× faster, 70% less VRAM)."""
        # Unsloth LoRA: only adapter weights trained (~1% of params)
        # PEFT with QLoRA: 4-bit base + 16-bit adapter
        # TRL Trainer: RLHF/DPO-ready pipeline
        # Checkpointing: resumes from last checkpoint on crash (self-healing)
        # Progress: SSE stream to dashboard and TUI

    def export(self, adapter_path: Path, format: str = "gguf") -> Path:
        """Export adapter as GGUF for deployment without full model copy."""
        # llama.cpp convert_lora_to_gguf.py integration
        # Result: small adapter file (10-100MB) + reference to base model
```

**Hardware requirements and time estimates:**

| GPU | VRAM | Base Model | Adapter Rank | Training Time |
|-----|------|-----------|-------------|--------------|
| RTX 4060 Ti | 8 GB | Llama 3.1 8B Q4 | r=16 | ~2 hours |
| RTX 4090 | 24 GB | Llama 3.3 70B Q4 | r=32 | ~12–20 hours |
| M4 Max | 128 GB | Llama 3.3 70B BF16 | r=64 | ~12 hours |
| A100 40GB | 40 GB | Llama 3.3 70B Q8 | r=64 | ~4 hours |

### 6.2 Proprietary Router Model Training — The Moat

The most strategically important fine-tuning work is not user-facing. It is the continuous fine-tuning of our Tier 3 router model (Qwen2.5-0.5B) on collective intelligence data.

**Training data generation:**

Every routing decision in the fleet becomes a training example:
```python
# Training example structure
{
    "input": {
        "prompt_embedding": [...],  # 384-dim MiniLM embedding
        "user_prefs": {
            "optimization_target": "cost",
            "privacy_tier": "external",
            "max_latency_ms": 2000
        },
        "available_backends": {
            "local": {"tps": 47, "model": "llama3.3-70b-q4"},
            "cloud": {"tps": None, "cost_per_1m": 10.0},
            "marketplace": [{"tps": 55, "cost_per_1m": 0.5}]
        },
        "time_context": {"hour": 14, "day_of_week": "tuesday"}
    },
    "label": {
        "optimal_backend": "marketplace",
        "reasoning": "local busy, marketplace 20% faster, 95% cost saving"
    },
    "quality_signal": 0.92  # user satisfaction proxy (session retention, no retry)
}
```

**Training pipeline:**
```
Fleet signals (anonymized) → Aggregator
    ↓
Weekly dedup + quality filter
    ↓
DP-noised training dataset
    ↓
DPO fine-tuning (Qwen2.5-0.5B)
    ↓
Eval on held-out routing decisions (accuracy ≥ 90% gate)
    ↓
Quantized to Q4_K_M (ships in pip wheel, ~300MB)
    ↓
A/B test: 10% of fleet routes via new model for 48h
    ↓
If quality ≥ current model: promote to 100%
```

**Why this is a moat:**
- The router model improves with every request across the entire fleet
- A competitor installing LLMHosts on 100 machines gets a router trained on our entire fleet's data
- We have 1-2 years of training data advantage before any competitor can replicate
- The model weights ship in the closed-source part of the wheel (not the BSD-3-licensed core)

### 6.3 Model Agent — Continuous Model Landscape Monitoring

The Model Agent is an AI background service that monitors the model landscape and pushes recommendations when better options emerge for a hardware profile.

```python
# llmhosts/agents/model_agent.py

class ModelAgent:
    """Monitors model landscape, updates HIM, notifies users of upgrades."""

    async def run_weekly_scan(self) -> ScanReport:
        """Scan HuggingFace, LM Studio community, and llm-benchmarks for new models."""
        new_models = await self._discover_new_models()
        for model in new_models:
            if self._is_candidate_for_him(model):
                benchmark = await self._run_benchmark(model)
                if benchmark.beats_current_recommendation(self.hardware_profile):
                    await self._notify_user(
                        f"New model available: {model.name} — {benchmark.improvement}% faster on your {self.gpu_name}"
                    )
                    await self._update_him(model, benchmark)

    def _is_candidate_for_him(self, model: ModelCard) -> bool:
        """Check if model is worth benchmarking for our hardware profiles."""
        return (
            model.license in APPROVED_LICENSES  # Apache 2.0, MIT, Llama 3 community
            and model.format == "gguf"
            and model.parameter_count in SUPPORTED_SIZES
            and model.architecture in SUPPORTED_ARCHITECTURES
        )
```

---

## 7. Benchmark Methodology

### 7.1 Philosophy

We publish every result. We compare to the competition's latest version. We document exactly what hardware was used. We publish the benchmark code.

**What we do not do:**
- Cherry-pick workloads where we win
- Compare against outdated versions of competitors
- Use non-standard hardware configurations
- Omit latency distributions in favour of only reporting mean

### 7.2 Benchmark Suite Architecture

**Repository:** `benchmarks/` subdirectory in LLMHosts.com repo (public, Apache 2.0 licensed even if the rest is not)

```
benchmarks/
├── README.md              — how to reproduce every result
├── harness/
│   ├── runner.py          — benchmark orchestrator
│   ├── metrics.py         — TTFT, ITL, throughput, memory
│   └── compare.py         — multi-system comparison + report generation
├── workloads/
│   ├── coding.jsonl       — 500 HumanEval-style tasks
│   ├── chat.jsonl         — 500 multi-turn conversations
│   ├── rag.jsonl          — 500 retrieval-augmented generation tasks
│   ├── long-context.jsonl — 50 128K-token tasks
│   └── batch.jsonl        — 1000 concurrent request simulation
└── results/
    ├── 2026-03-10/
    │   ├── rtx4090-llmhosts-v0.5.1.json
    │   ├── rtx4090-ollama-v0.5.0.json
    │   └── summary.md
    └── ...
```

### 7.3 Metrics

| Metric | Definition | Target (RTX 4090, 70B Q4) |
|--------|-----------|--------------------------|
| TTFT | Time to first token (P50, P95, P99) | P50: 180ms, P95: 350ms |
| ITL | Inter-token latency (P50) | 21ms (= ~47 tok/s) |
| Throughput | Tokens/second sustained over 60s continuous | ≥ 45 tok/s |
| Batch throughput | Total tokens/second with 16 concurrent requests | ≥ 200 tok/s |
| VRAM efficiency | GB VRAM used / model parameter count | < 0.93 GB/B parameters for Q4 |
| Cold start | Time from process start to first token | < 8 seconds (model pre-loaded) |
| Memory leak | VRAM delta after 1000 requests | < 50MB |

### 7.4 Hardware Tiers

We benchmark and publish results for all supported hardware tiers:

| Tier | Representative Hardware | Price Point |
|------|------------------------|-------------|
| Entry | RTX 4060 Ti 16GB | $400 |
| Mid | RTX 4080 16GB | $700 |
| High | RTX 4090 24GB | $1,600 |
| Enthusiast | RTX 5090 32GB | $3,500 |
| Professional | RTX 4000 Ada 20GB (workstation) | $1,250 |
| Apple | M4 Max 64GB / 128GB | $2,500 |
| AMD | RX 7900 XTX 24GB | $900 |
| Datacenter | A100 40GB SXM | $10,000+ (used) |

### 7.5 Comparison Targets

| System | Version | Why We Compare |
|--------|---------|---------------|
| Ollama | Latest at time of report | UX benchmark, most popular |
| vLLM | Latest stable | Performance benchmark, datacenter standard |
| llama.cpp | Latest | Raw inference baseline |
| Cloud (GPT-4o) | API, measured | Cost/latency reference |
| Cloud (Claude 3.5 Sonnet) | API, measured | Cost/latency reference |

### 7.6 Publication Strategy

**Frequency:** After every major LLMHosts Core release (F1, F2, F3, F4, F5)

**Channels:**
- `benchmarks/results/` in the repo (permanent, versioned)
- Blog post with narrative interpretation (llmhosts.com/blog)
- HuggingFace Open LLM Leaderboard submission (where eligible)
- Hacker News "Show HN" when results are notable
- Twitter/X thread with visual charts generated by the harness

**If we lose a benchmark:** We publish it anyway, with analysis of why. This builds trust. The narrative is: "Here's where Ollama beats us (cold start on models < 7B), here's why, here's what we're fixing in F3."

---

## 8. IP Protection Guide

### 8.1 Trade Secret Posture

**What is proprietary (do not publish, discuss in marketing, or include in open-source):**
1. Fine-tuned router model weights (Qwen2.5-0.5B + LoRA adapters trained on fleet data)
2. Collective intelligence aggregation pipeline code (`llmhosts/intelligence/aggregator.py`)
3. PagedAttention block allocator implementation (`llmhosts_core/src/memory/paged.rs`)
4. cudarc Flash Attention fused kernels (`llmhosts_core/src/kernels/attention.cu`)
5. Hardware Intelligence Matrix dataset (the actual HIM data, not the schema)
6. Relay server implementation (clients are open-source; the server we operate is not)

**What is open-source (BSD-3-Clause, builds trust + community):**
1. Benchmark harness (`benchmarks/`)
2. Python control plane (`llmhosts/`) — proxy, router wrapper, CLI
3. GGUF parser (`llmhosts_core/src/gguf.rs`) — community standard, no moat
4. GPU detection (`llmhosts_gpu/`) — helps us get bug reports from exotic hardware
5. Tunnel client (`relay_core/`) — client-side only

### 8.2 Approved Marketing Vocabulary

Use these terms when describing the inference engine:

| Instead of... | Use... |
|--------------|--------|
| "We use CubeCL" | "cross-platform GPU compute" |
| "We use cudarc" | "NVIDIA-native optimization layer" |
| "PagedAttention implementation" | "memory-efficient KV cache management" |
| "Qwen2.5-0.5B router" | "proprietary routing model" or "intelligent routing layer" |
| "Flash Attention v2 kernels" | "fused attention computation" |
| "Our maturin/pyo3 bridge" | "native Python integration" |
| "CubeCL kernels compile to PTX" | "hardware-compiled compute kernels" |

### 8.3 Forbidden Disclosures

**Never disclose publicly:**
- The specific Rust crates used in the inference engine (only "Rust-native" is sufficient)
- Exact router model architecture or training procedure
- Fleet size, signal volume, or collective intelligence data volume
- Relay server architecture, hosting provider, or geographic distribution
- HIM sample counts or confidence scores (aggregate HIM data is proprietary)
- Fine-tuned adapter architectures or hyperparameters used in router training

### 8.4 Competitor Research Protection

When competitors reverse-engineer our wheel:
- Cargo.toml is visible in the published wheel — do not name strategically sensitive crates in public Cargo.toml
- Consider separate `llmhosts_inference` crate name that obscures the relationship to CubeCL
- Strip debug symbols from release builds: `[profile.release] strip = "symbols"`
- WASM-compile non-critical path components to obscure logic
- Core kernels: compiled PTX embedded as binary blobs, not inspectable

### 8.5 Open-Source License Strategy

- **Apache 2.0** for benchmark harness (maximum adoption)
- **BSD-3-Clause** for Python control plane (permissive, keeps attribution)
- **Proprietary / Business Source License (BSL)** for inference engine core (converts to Apache 2.0 after 4 years)
- **Proprietary** for router model weights, collective intelligence pipeline, HIM dataset

The BSL model (used by HashiCorp Vault, MariaDB) allows free use for non-competing purposes while protecting our competitive moat. Self-hosters who do not run a competing SaaS can use the inference engine freely.

---

## 9. User Journey — Micro + Macro

### 9.1 Journey Map Overview

LLMHosts serves three primary user archetypes:

| Archetype | Goal | Success State |
|-----------|------|--------------|
| **The Developer** | Stop paying $200–500/mo for cloud AI | Full local inference stack, $0 cloud bills |
| **The Traveler** | Use home GPU while working remotely | Encrypted tunnel, <30ms latency to home |
| **The Operator** | Earn from idle GPU capacity | Marketplace listing, steady compute income |

All three archetypes share a single install path and diverge at `llmhosts setup`.

### 9.2 Stage-by-Stage Journey

#### Stage 1: Discovery
**Entry points:** Hacker News (most), GitHub search "ollama alternative", Twitter/X, colleague recommendation

**What the user feels:** Curious but skeptical. They've tried Ollama, it was too slow, or they want remote access, or they're frustrated with cloud costs.

**Delight moment:** The README's first line: "Your RTX 4090 is worth $47,000/year in cloud compute credits. Keep the money." With a live counter.

**Failure modes:**
- README is too long → one-screen pitch version at top, full docs below
- No benchmark comparison → Section 7 addresses this directly
- Unclear pricing → pricing page accessible from README without login

#### Stage 2: Install
```bash
pip install llmhosts          # < 30 seconds on fast connection (core wheel)
```

**What must happen automatically:**
- GPU detection (pre-install hook)
- Correct wheel variant selected
- `llmhosts doctor` runs automatically and prints a green summary

**Failure modes:**
- AMD user gets wrong compute backend → platform matrix in hatchling hook (#472)
- Windows user needs CUDA toolkit → `llmhosts doctor` detects and links to CUDA installer
- Apple Silicon: wrong arch wheel → universal2 wheel + CI matrix for macos_arm64

#### Stage 3: Setup
```bash
llmhosts setup
```

Goal-based wizard (see Pillar 3, #474). Completes with:
- Hardware profile saved
- Recommended model displayed
- Model download initiated (background)
- Baseline AI spend captured

**Delight moment:** During model download, the TUI shows a live "time until your first $0 inference" countdown.

#### Stage 4: First Inference
The moment trust is established. Must happen within 10 minutes of `llmhosts setup` completing (excluding download time).

```
$ curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "auto", "messages": [{"role": "user", "content": "hello"}]}'

# Response arrives in < 200ms TTFT
# Header: X-LLMHosts-Backend: local
# Header: X-LLMHosts-Latency-Ms: 187
```

**Delight moment:** Dashboard shows "Saved $0.003 vs GPT-4o on this request." Tiny, immediate, concrete.

#### Stage 5: Daily Use
The user sets `OPENAI_BASE_URL=http://localhost:4000/v1` in Cursor, VS Code + Continue, or their codebase. Everything just works. No code changes.

**What we do invisibly:**
- Route to cloud if local GPU is OOM (model too large)
- Cache repeated prompts (exact match + semantic)
- Track savings cumulative daily/weekly/monthly
- Self-heal relay on disconnect
- Update model recommendations when better models ship

**Failure modes to eliminate:**
- "Why is my request slow?" → TUI shows routing decision in real time
- "Why did it use cloud instead of local?" → `X-LLMHosts-Routing-Reason` header explains
- "It stopped working" → self-healing covers 80% of cases before user notices

#### Stage 6: Growth / Expansion
The user wants more: remote access, team sharing, or to earn from idle GPU.

**Conversion triggers:**
- Dashboard shows "you've saved $XXX — upgrade to Pro to use your GPU while traveling"
- Friend asks "can I use your GPU?" → "Share this API key" (requires Pro)
- GPU sits idle at night → marketplace notification "your GPU earned $12 last week doing nothing"

#### Stage 7: Remote Access (The Traveler)
```bash
llmhosts tunnel start
# Output:
# Your GPU is now accessible at: https://abc123.llmhosts.com/v1
# Encrypted with Noise NK protocol
# Rate limit: 100 req/min (Pro plan)
```

**Delight moment:** First request from laptop to home GPU completes in 23ms. "That's faster than cloud."

#### Stage 8: Marketplace (The Operator)
```bash
llmhosts marketplace enable \
    --rate 0.35 \         # $/hr
    --schedule "22:00-08:00"  \  # overnight only
    --model llama3.3-70b-q4_k_m
```

**Delight moment:** First payout email: "Your GPU earned $4.20 overnight. Running total: $4.20."

#### Stage 9: Fine-Tuning (Power User)
```bash
llmhosts finetune run \
    --base-model llama3.3-70b-q4_k_m \
    --data ./training-data/ \
    --adapter-name company-docs
```

**Delight moment:** "Your adapter achieves 94% accuracy on your validation set. Deploy with `llmhosts serve --adapter company-docs`."

### 9.3 Macro Journey — 12-Month Timeline

| Month | User State | LLMHosts Milestone |
|-------|-----------|-------------------|
| 0 | pip install | Smart install (#472), doctor, HIM v1 |
| 0.5 | First inference | F1 Foundation complete (CPU inference) |
| 1 | Daily use | F2 Performance (GPU compute, 2× Ollama) |
| 2 | Tunnel set up | Relay M2+ (stable tunnel, relay server) |
| 3 | Team sharing | Pro tier, team API keys |
| 4 | Marketplace | GPU marketplace listing, first payout |
| 6 | Fine-tuning | LoRA pipeline (#477), first adapter deployed |
| 8 | Collective intel | Fleet learning active, HIM powered by real data |
| 10 | Router model | Proprietary router model v1, beating manual routing |
| 12 | Platform | LLMHosts = category leader in self-hosted AI infra |

---

## 10. Implementation Roadmap

### 10.1 Phasing Principle

Vision 2026 is delivered in six waves, each building on the previous. No wave is "done" in isolation — they all contribute to the flywheel: more users → more collective intelligence → better recommendations → better user outcomes → more users.

**Wave 0 — Foundation Complete (Current):** F1 (CPU inference), F2 (model validation, Q5/Q6 quant), relay M1, PyPI v0.5.1

**Wave 1 — GPU Compute (Q1 2026, Weeks 1-6):**
- F2: WGPU (Metal, Vulkan, ROCm) + cudarc (CUDA hot paths)
- Achieve: ≥ 2× Ollama throughput on RTX 4090
- Issue: #452

**Wave 2 — Smart Install + Zero-Ops (Q1–Q2 2026, Weeks 6-12):**
- Smart install with pre-install platform detection (#472)
- Self-healing systemd/launchd integration (#473)
- Enhanced `llmhosts doctor` with GPU-specific diagnostics
- Issue: #472, #473

**Wave 3 — Goal-Based Onboarding + Unified API (Q2 2026, Weeks 12-18):**
- Goal-based setup wizard (#474)
- One unified API with routing transparency (#476)
- `llmhosts finetune` CLI skeleton (#477)
- Issue: #474, #476, #477

**Wave 4 — Collective Intelligence (Q2–Q3 2026, Weeks 18-26):**
- Privacy-preserving signal collection (#475)
- HIM v2 (fleet-powered recommendations)
- Router model v1 (Qwen fine-tuned on fleet data)
- Issue: #475

**Wave 5 — Fine-Tuning + Benchmarks (Q3 2026, Weeks 26-36):**
- Full LoRA pipeline with Unsloth backend (#477)
- Benchmark suite publication (#478)
- GGUF adapter export + marketplace sharing
- Issue: #477, #478

**Wave 6 — IP Protection + GitHub Harmony (Q3–Q4 2026, Weeks 36-48):**
- BSL licensing for inference engine core (#479)
- GitHub Harmony v2 audit (#481)
- Full user journey instrumentation (#482)
- Advisory council session (#480)
- Issue: #479, #480, #481, #482

### 10.2 Issue × Wave Mapping

| Issue | Title | Wave | Priority |
|-------|-------|------|----------|
| #471 | Master Epic | — | P0 |
| #452 | F2 Performance (GPU compute) | 1 | P0 |
| #472 | Smart Install | 2 | P0 |
| #473 | Self-Healing & Zero-Ops | 2 | P1 |
| #474 | Goal-Based Onboarding | 3 | P1 |
| #476 | One Unified API + Intelligent Router | 3 | P1 |
| #475 | Collective Intelligence | 4 | P1 |
| #477 | Fine-Tuning as a Service | 5 | P2 |
| #478 | World-Class Benchmark Suite | 5 | P2 |
| #479 | IP Protection & Marketing Language | 6 | P2 |
| #480 | AI Advisory Council Session | 6 | P3 |
| #481 | GitHub Harmony v2.0.0 | 6 | P3 |
| #482 | Full User Journey Map | 6 | P3 |

### 10.3 Dependency Graph

```
#452 (GPU compute)
    ↓
#472 (Smart install) → #473 (Self-healing)
    ↓
#474 (Goal-based onboarding) → #476 (Unified API)
    ↓
#475 (Collective intelligence)
    ↓
#477 (Fine-tuning)  →  #478 (Benchmarks)
    ↓
#479 (IP protection)  →  #480 (Advisory)  →  #481 (GitHub Harmony)
                                              ↓
                                          #482 (Journey map)
```

### 10.4 Resource Model

**Engineering allocation (single engineer + AI pair programming):**

| Wave | Weeks | Primary Focus | AI Assistance Level |
|------|-------|--------------|-------------------|
| 1 | 1-6 | Rust GPU compute (cudarc + WGPU) | High (boilerplate, test gen) |
| 2 | 6-12 | Python/Rust integration, systemd config | Medium |
| 3 | 12-18 | Python UX, API design | High (docs, tests) |
| 4 | 18-26 | Privacy engineering, aggregator | High (crypto, DP math) |
| 5 | 26-36 | LoRA training, benchmark tooling | Medium |
| 6 | 36-48 | Legal (BSL), process, audits | Low (judgment-heavy) |

---

## 11. Risk Matrix

### Top 10 Risks

| # | Risk | Likelihood | Impact | Mitigation |
|---|------|-----------|--------|-----------|
| R1 | CubeCL API instability breaks cross-platform build | Medium (30%) | High | Pin to tested versions; cudarc fallback for NVIDIA; track upstream changelog |
| R2 | Ollama ships batching + PagedAttention, closes performance gap | Medium (25%) | High | Our moat is cross-platform (AMD/Intel) + collective intelligence + marketplace, not just performance |
| R3 | AMD ROCm still too buggy on consumer cards by Q3 | High (60%) | Medium | Defer ROCm to F4; focus on CUDA + Metal first; publish AMD support timeline honestly |
| R4 | Privacy-preserving signals are re-identifiable despite DP | Low (10%) | Critical | Third-party privacy audit before launch; ε ≤ 0.5 initial budget; k-anonymity k ≥ 500 |
| R5 | Fine-tuned router model underperforms rule-based router | Medium (35%) | Medium | A/B test with automatic fallback; keep rule-based router as default until new model consistently beats it for 2 weeks |
| R6 | SaladCloud launches consumer CLI tool | Medium (20%) | High | We win on UX and vertical integration; they win on scale; our marketplace is a network of devices, not a cloud |
| R7 | Benchmark publication sparks hostile comparison from Ollama community | Low (15%) | Low | We publish where we lose; pre-notify Ollama maintainers; frame as "ecosystem comparison" not "we win" |
| R8 | BSL licensing alienates open-source community | Medium (30%) | Medium | Open-source Python control plane + benchmark harness; BSL only for inference engine core; 4-year Apache 2.0 conversion |
| R9 | Relay server costs exceed revenue at scale | Low (10%) | High | Relay architecture supports self-hosted relay; users pay for relay bandwidth above free tier (1GB/mo) |
| R10 | LLM-based router (Qwen-0.5B) adds latency that users notice | Low (15%) | Medium | Tier 3 is only invoked when Tiers 1-2 are below confidence threshold (< 25% of requests); result is cached |
| R11 | llama.cpp ships relay + marketplace in 12–18 months, commoditizing inference layer | Medium (25%) | High | Moat must be above the inference layer: HIM collective intelligence data, operator network effects, community trust, and seamless UX — not the engine itself. Accelerate HIM data flywheel before commoditization window closes. |

### Risk Response Protocol

For each P0 risk (R1, R2, R4, R6, R9):
1. Owner assigned at risk identification
2. Monthly risk review in engineering retro
3. Pre-mortems for R4 before collective intelligence launch
4. Competitive monitoring: weekly Ollama changelog + SaladCloud blog watch

---

## 12. Success Metrics

### 12.1 North Star Metrics

| Metric | Q2 2026 Target | Q4 2026 Target | Definition of Vision Achieved |
|--------|---------------|---------------|------------------------------|
| Weekly Active Nodes | 1,000 | 10,000 | Devices running llmhosts serve at least once/week |
| Token throughput (fleet) | 100M tok/week | 1B tok/week | Total tokens generated across all nodes |
| Inference performance | 2× Ollama on RTX 4090 | 2.5× Ollama | Sustained throughput, 70B Q4 model |
| Marketplace GMV | $5,000/mo | $50,000/mo | Gross marketplace compute revenue |
| Self-healing coverage | 70% | 85% | % of failures resolved without user action |
| Onboarding completion | 65% | 75% | Users who reach first inference after setup |
| Collective intelligence nodes | 500 | 5,000 | Nodes with telemetry enabled |
| HIM accuracy | — | 90% | % of HIM recommendations accepted without override |

### 12.2 Pillar-Level Success Metrics

**Pillar 1 — Smart Install (#472):**
- Install-to-inference: < 10 minutes (excl. download)
- Doctor pass rate (first run): > 92% across NVIDIA/AMD/Apple/Intel
- Platform detection accuracy: 100% (correct wheel selected)

**Pillar 2 — Self-Healing (#473):**
- MTTR: < 60 seconds for auto-recoverable failures
- False recovery rate: < 2% (self-healing makes things worse)
- User escalation rate: < 20% of failure events

**Pillar 3 — Goal-Based Onboarding (#474):**
- Wizard completion: > 75%
- 7-day retention: > 60%
- NPS (onboarding cohort): > 60

**Pillar 4 — Collective Intelligence (#475):**
- Privacy audit: pass (no PII leakage confirmed by third-party)
- Opt-in rate: > 60% (users keep telemetry enabled)
- HIM cold start accuracy: > 80% on first recommendation
- **HIM fleet threshold**: HIM fleet-powered recommendations require ≥ 500 opt-in nodes.
  Below this threshold, HIM provides no advantage over manual benchmarking. Marketing must
  not imply fleet-powered recommendations until threshold is reached — show current node
  count and estimated time to threshold in the dashboard.

**Pillar 5 — Unified API (#476):**
- Router accuracy: > 90% vs. human-labeled optimal backend
- Tier 1 latency: < 15ms P95
- Cloud spillover rate: < 5% when local is appropriate

**Pillar 6 — Fine-Tuning (#477):**
- Adapter training success rate: > 90% on supported hardware
- Adapter quality gate pass rate: > 70% (adapter beats base model on validation set)
- Time to first adapter: < 8 hours on RTX 4090

**Pillar 7 — Benchmarks (#478):**
- Benchmark reproducibility: 100% (any reader can reproduce)
- Published on major release: 100%
- Third-party verification: at least 2 independent reproductions by community

**Pillar 8 — IP Protection (#479):**
- Zero unintended disclosures of proprietary architecture in marketing/docs
- BSL compliance: legal review complete before enforcement
- Trade secret register: maintained and current

### 12.3 Revenue Metrics

| Quarter | MRR Target | Driver |
|---------|-----------|--------|
| Q1 2026 | $15,000 | Pro plan conversions from existing users |
| Q2 2026 | $40,000 | GPU marketplace GMV + Pro growth |
| Q3 2026 | $100,000 | Enterprise tier + fine-tuning |
| Q4 2026 | $250,000 | Scale: 10K WANs, marketplace GMV, enterprise |

**Unit economics target:**
- CAC: < $20 (open-source organic; zero paid acquisition)
- LTV: > $500 (Pro $19/mo × 26 months average retention)
- LTV/CAC: > 25×
- Gross margin: > 80% (infrastructure is the user's GPU, not ours)

### 12.4 How We Know Vision 2026 Is Achieved

Vision 2026 is achieved when:

1. **A developer with a consumer GPU** can install LLMHosts, run setup, and serve their first token in under 10 minutes — on NVIDIA, AMD, Apple, or Intel hardware — with no manual configuration.

2. **The inference engine** delivers ≥ 2× Ollama throughput on the RTX 4090 and is independently verified by at least two community members.

3. **The collective intelligence layer** is live, privacy-audited, and actively improving HIM recommendations with data from ≥ 1,000 nodes.

4. **The router model** outperforms rule-based routing on the held-out evaluation set by ≥ 10%, as measured by the routing accuracy benchmark.

5. **The GPU marketplace** has processed ≥ $50,000 in cumulative GMV, with operators reporting first-payout within 48 hours of listing.

6. **The benchmark suite** is published, reproducible, and honest — including where we lose.

7. **Revenue** has crossed $100,000 MRR with gross margin ≥ 80%.

8. **The category** "self-hosted AI infrastructure platform" exists in developer vocabulary, and LLMHosts is the answer when someone Googles it.

---

## Appendix A — Current State Snapshot (2026-03-10)

**Version:** v0.5.1 on PyPI
**Branch:** `feature/cuda-dogfooding-ready`

**Rust crates (present in repo):**
- `llmhosts_core` — CPU inference working, GGUF parsing, Q5_0/Q6K validated, basic GPU compute scaffolding
- `llmhosts_gpu` — GPU detection (NVML/sysfs/IOKit)
- `relay_core` — WSS tunnel (M1 complete)
- `router_core` — kNN skeleton

**Python layer (working):**
- FastAPI proxy (OpenAI + Anthropic dialects)
- Three-tier router (rule-based + kNN + ModernBERT)
- vCache semantic cache
- GPU marketplace with reputation scoring
- Token AI fleet assistant
- Stripe billing (Free/Pro/Team/Enterprise)
- Next.js dashboard

**Tests:** 2,373 Python + 808 frontend = 3,181+ total

**Open epics:** #450–455 (LLMHosts Core F1–F5), #438–443 (relay M1–M5), #472–#482 (Vision 2026)

---

## Appendix B — Glossary

| Term | Definition |
|------|-----------|
| HIM | Hardware Intelligence Matrix — maps GPU profiles to optimal model configurations |
| TTFT | Time to First Token — latency from request to first streamed token |
| ITL | Inter-Token Latency — average time between consecutive generated tokens |
| KV Cache | Key-Value cache — stores intermediate attention computations for fast generation |
| PagedAttention | Memory management technique for KV cache (OS virtual memory analogy) |
| LoRA | Low-Rank Adaptation — efficient fine-tuning method that trains < 1% of parameters |
| QLoRA | LoRA applied to 4-bit quantized base model — enables 70B fine-tuning on RTX 4090 |
| PEFT | Parameter-Efficient Fine-Tuning — umbrella term covering LoRA and adapters |
| CubeCL | Write-once GPU kernel language by Tracel.ai (CUDA + ROCm + Metal from single code) |
| cudarc | Rust bindings for CUDA driver + cuBLAS + cuDNN |
| pyo3 | Rust ↔ Python interop library |
| maturin | Build tool for Rust Python extensions (ships Rust code inside pip wheel) |
| Noise NK | Cryptographic handshake protocol used in WireGuard; used for relay tunnel auth |
| yamux | Multiplexing protocol over a single TCP/WebSocket connection (relay sessions) |
| BSL | Business Source License — proprietary with automatic Apache 2.0 conversion after N years |
| DP | Differential Privacy — mathematical privacy guarantee with formal ε budget |
| GQA | Grouped Query Attention — efficient multi-head attention variant in Llama 3+ |
| GGUF | GGML Unified Format — binary format for LLM model weights (llama.cpp standard) |
| IslandRun | LLMHosts privacy routing — classifies PII before routing to external backends |
| WANs | Weekly Active Nodes — our primary engagement metric |
| GMV | Gross Marketplace Value — total transaction volume through GPU marketplace |
| BYOK | Bring Your Own Key — user provides their own OpenAI/Anthropic API key for cloud fallback |
| BYOG | Bring Your Own GPU — the model operator who provides compute to the marketplace |

---

*This document is the canonical product + engineering reference for LLMHosts Vision 2026. All GitHub issues in the #472–#482 range derive their acceptance criteria from this document. When in conflict, this document governs. When this document is stale, update it.*

*Last updated: 2026-03-10 | Next review: 2026-04-10*
