# LLMHosts — Single Source of Truth

> **AI instruction**: Read this file first, every session. It is the complete project state.
> Do not start any work until you have read this file and run the commands in "Commands" below.
>
> **GitHub Harmony System v2.0.0** | **Last updated**: 2026-03-10 | **Commit**: aa24cc9 | **Updated by**: claude-sonnet-4-6 (harmony bootstrap)

---

## Identity

**What it is**: LLMHosts turns a home GPU into production AI infrastructure accessible from anywhere — a CLI-first tool + SaaS platform with three-tier intelligent router, verified semantic cache, and OpenAI/Anthropic compatible proxy.
**Tagline**: "Your hardware. Real AI infrastructure. From anywhere."
**Install**: `pip install llmhosts` | CLI: `llmhosts serve`
**Repo**: https://github.com/LookNoHandsMom/LLMHosts.com (private)
**Stack**: Python 3.12 / FastAPI / Pydantic v2 / Rust (Cargo/maturin) / Next.js 16 / Prisma / SQLite / Docker / Traefik
**Domain**: llmhosts.com
**License**: See LICENSE file
**Project board**: https://github.com/users/LookNoHandsMom/projects/5
**Macro issue**: #471 — LLMHosts Vision 2026 — World's Best Inference Platform

---

## Quick State

| Field | Value |
|-------|-------|
| Open issues | 30 |
| Open PRs | 6 (1 feature + 5 dependabot) |
| CI status | ✅ green (labeler, CodeQL) / CI queued on PR #483 |
| Current version | v0.5.1 (shipped 2026-03-10) |
| Current phase | Vision 2026 planning (post Phase 5 completion) |
| Active milestone | None set — Vision 2026 planning just started |
| Last shipped | v0.5.1 CUDA/WGPU dogfooding-ready (PR #470, 2026-03-10) |

Run to verify: `gh issue list --state open` · `gh pr list --state open` · `gh run list --limit 3`

---

## What Is Built (Phase 1–5 Complete — v0.5.1)

### Core CLI + Server
- **CLI entrypoint**: Click-based `llmhosts` command with 9 subcommands. `llmhosts/cli.py`. Multiple PRs.
- **FastAPI server**: Orchestrator mounting proxy, dashboard, hive admin. `llmhosts/server.py`.
- **Config system**: TOML + env var priority chain. `llmhosts/config.py`.
- **`llmhosts up`**: 5-phase pipeline from zero to running endpoint. `llmhosts/up.py`.

### Proxy + Router
- **OpenAI/Anthropic proxy**: `/v1/chat/completions`, `/v1/messages`, `/v1/models`. `llmhosts/proxy/`.
- **Three-tier hybrid router**: Tier 1 rule-based+kNN (FAISS/ONNX, <15ms CPU), Tier 2 ModernBERT (<50ms), Tier 3 Qwen2.5-0.5B (<500ms). `llmhosts/router/`. ADR-001.
- **IslandRun privacy routing**: PII detection before external routing, tiered trust levels. `llmhosts/router/engine.py`.
- **BackendDispatcher**: Routes to local/remote/cloud. `llmhosts/proxy/dispatcher.py`.

### LLMHosts Core (Rust Inference Engine)
- **Custom Rust engine**: CubeCL + cudarc, replaces Ollama entirely. ADR-010. `llmhosts_core/`. PR #465.
- **GPU compute backend**: CUDA/WGPU dogfooding-ready. `llmhosts_core/src/compute/`. PR #466.
- **maturin wheel**: Native Rust extension built via PyO3/maturin. `Cargo.toml`, `pyproject.toml`.
- **GGUF + chat templates**: Real model validation Q5_0/Q6K. PR #458.

### Caching
- **Three-tier cache**: Exact hash → entity namespace → vCache semantic (ICLR 2026). `llmhosts/cache/manager.py`.

### Tunnel / Relay
- **Self-hosted Rust relay**: WSS + yamux + Noise NK. ADR-009. `relay_core/`. PR #449 (merged).
- **Tailscale/Cloudflare**: Fully removed. Only relay + none providers remain.

### Discovery + Marketplace
- **Auto-discovery**: Scans for Ollama instances + GPU hardware zero-config. `llmhosts/discovery/`.
- **GPU Marketplace**: Device registration, reputation scoring, Stripe Connect billing, operator UI. Phase 5.
- **Reputation engine**: Tier system (Unverified/Standard/Verified/Elite), uptime aggregation. PR #404.
- **Remote routing**: `_dispatch_remote()` + MarketplaceRefresher 5min sync. PR #405.
- **At-capacity cloud fallback**: Auto-routes when local GPU busy. PR #396.

### Frontend + Auth
- **Next.js 16 dashboard**: App Router, Tailwind 4, shadcn/ui, Recharts. `frontend/`.
- **NextAuth v5**: Auth.js authentication. `frontend/src/app/api/auth/`.
- **Prisma + SQLite**: 21+ models including TokenConversation, MarketplaceBillingEvent. `frontend/prisma/`.
- **Stripe Connect billing**: 80/20 fee split, payout endpoint, earnings aggregation. PR #406.

### Token — Conversational Fleet Orchestrator
- **Token AI**: Streaming chat, multi-agent routing (Sonnet/Flash/DeepSeek-R1), fleet actions. `frontend/src/app/dashboard/token/`.
- **Intent classifier**: Gemini Flash Lite routes to correct model at confidence 0.6+.
- **Cost analytics + alerts**: Email notifications for device offline, budget alerts. Phase 3 complete.

### Agent System (Hive.CEO)
- **Embedded agent system**: CEO, Analyst, Watcher, Builder, Advisor agents. `hive/`.
- **Admin panel**: Mounts at `/admin/hive/` on FastAPI app.

### Docs + Infrastructure
- **Hybrid documentation**: 9 pages (5 public, 4 auth-gated). PR #407.
- **Monitoring stack**: Prometheus + Grafana + Loki. `monitoring/docker-compose.monitoring.yml`.
- **Docker + Traefik**: External Pangolin/Traefik deployment. `docker-compose.prod.yml`.
- **CI pipeline**: 17-check verification gate. Self-healing lint/format. `.github/workflows/ci.yml`.
- **CodeQL**: Security analysis. `.github/workflows/codeql.yml`.
- **Tests**: 2,373 Python + 808 frontend = 3,181+ total, all passing.

---

## What Is Open

### Vision 2026 Epics (Planning Phase — PR #483)
- **#471** — Epic: LLMHosts Vision 2026 (Master) — P0 — 11 sub-epics, 10 pillars
- **#472** — Epic: Smart Install — Universal Platform Detection — P1 — <60s from pip to endpoint
- **#473** — Epic: Self-Healing & Zero-Ops — P1 — autonomous operation, no maintenance
- **#474** — Epic: Goal-Based Intelligent Onboarding — P1 — user profile, router tuning
- **#475** — Epic: Collective Intelligence & Privacy-Preserving Learning — P2 — federated fleet learning
- **#476** — Epic: One Unified API + Intelligent Multi-Backend Router — P1 — single endpoint for all backends
- **#477** — Epic: Fine-Tuning as a Service + Proprietary Model Strategy — P2 — local adapters, moat
- **#478** — Epic: World-Class Benchmark Suite — P2 — 2× Ollama proof, public dashboard
- **#479** — Epic: IP Protection & Marketing Language Guide — P1 — Rust-agnostic messaging
- **#480** — Epic: AI Advisory Council Vision Review — P1 — multi-agent research synthesis
- **#481** — Epic: GitHub Harmony v2.0.0 Audit — P1 — THIS ISSUE (being executed now)
- **#482** — Epic: Full User Journey Map — P1 — micro + macro verbosity

### LLMHosts Core Engine (F1 done, F2-F5 pending)
- **#450** — Epic: LLMHosts Core Custom Rust Inference Engine — P0
- **#452** — LLMHosts Core F2: Performance (PagedAttention, continuous batching) — P0
- **#453** — LLMHosts Core F3: Intelligence (HIM, hot-swap) — P1
- **#454** — LLMHosts Core F4: Scale (phase-split, multi-node) — P2
- **#455** — LLMHosts Core F5: Polish — P3

### Relay (M1 merged, M2-M5 pending)
- **#438** — Epic: Self-hosted LLMHosts Relay tunnel — P0
- **#440** — M2: Noise Protocol NK encryption — P0
- **#441** — M3: yamux stream multiplexing — P0
- **#442** — M4: relay_server binary (VPS-side) + deployment — P0
- **#443** — M5: Production hardening, docs cascade — P1
- **#447** — Relay server VPS deployment + DNS + TLS + monitoring — P1

### Launch Gates
- **#134** — Soft launch decision and announcement — P0-blocker
- **#445** — Production launch checklist — P0-blocker
- **#446** — Post-launch: monitoring, support, feedback — P2
- **#448** — Install UX: pipx/uvx recommendations — P1
- **#435** — fix: recommend pipx/uvx install — P1

### E2E Hardware Testing
- **#319** — E2E Master Plan (Alienware + Mac Mini + Multi-GPU + Large GPU) — P0
- **#359** — [E2E] Run Round 1 Alienware — P0-blocker — requires human + hardware
- **#360** — [E2E] Run Round 2 Mac Mini — P1 — requires human + hardware
- **#361** — [E2E] Run Round 2 Multi-GPU — P1 — requires human + hardware
- **#362** — [E2E] Run Round 2 Large GPU — P2 — requires human + hardware
- **#363** — [E2E] Gap analysis after Round 2 — P1

---

## What Is Next (Top 5)

> Verify with: `gh issue list --state open --limit 20`

1. **#452 LLMHosts Core F2: Performance** [P0]
   Why first: F1 foundation shipped (PR #465), F2 unblocked — PagedAttention + continuous batching is the performance moat
   Start here: `llmhosts_core/src/compute/mod.rs`, `docs/plans/2026-03-09-llmhosts-core-f1-foundation.md`
   Design: `docs/architecture/adr-010-llmhosts-core-inference-engine.md`

2. **#440 M2: Noise Protocol NK encryption** [P0]
   Why: relay_core M1 merged (PR #449), M2 unblocked — encryption needed before production relay
   Start here: `llmhosts_core/src/lib.rs`, `relay_core/`

3. **#445 Production launch checklist** [P0-blocker]
   Why: launch is blocked until all gates pass — this is the coordination issue
   Start here: `.github/workflows/release.yml`, `docs/LAUNCH_GO_NOGO.md`

4. **#359 [E2E] Run Round 1 Alienware** [P0-blocker — human]
   Why: hardware E2E required for launch confidence; requires physical Alienware hardware
   Note: This is a human-action issue — the AI cannot run this

5. **#481 GitHub Harmony v2.0.0 Audit** [P1]
   Why: structural improvements to GitHub-as-SSOT; currently in progress
   Deliverables: SSOT.md (this file), BUILD-READINESS-INDEX.md, invoke.md, milestone, epic AC

---

## File Map (Critical Paths)

```
.github/
├── SSOT.md                         ← This file. AI reads first, always.
├── CODEOWNERS                      ← Auto-assigns @LookNoHandsMom to all paths
├── SECURITY.md                     ← Vulnerability reporting (private: security@llmhosts.com)
├── CONTRIBUTING.md                 ← Dev setup + PR workflow
├── FUNDING.yml                     ← GitHub sponsors config
├── dependabot.yml                  ← pip + npm + docker + actions, weekly
├── labeler.yml                     ← Path-based PR auto-labeling rules
├── pull_request_template.md        ← PR harmony checklist
├── copilot-setup-steps.yml         ← GitHub Copilot agent env setup
├── ISSUE_TEMPLATE/
│   ├── feature.yml                 ← Feature request form
│   ├── bug.yml                     ← Bug report form
│   ├── chore.yml                   ← Maintenance/chore form
│   └── config.yml                  ← Disables blank issues
└── workflows/
    ├── ci.yml                      ← 17-check verification gate (self-healing)
    ├── codeql.yml                  ← Security analysis
    ├── release.yml                 ← Wheel + PyPI release
    ├── labeler.yml                 ← PR auto-labeling
    ├── stale.yml                   ← Stale issue management (60d → 14d close)
    └── post-deploy-verify.yml      ← Post-deployment verification

.github-harmony/
├── invoke.md                       ← Project identity + session entry point
└── journal/                        ← Audit logs

llmhosts/                           ← Python package (PyPI: llmhosts)
├── cli.py                          ← Click CLI entrypoint
├── server.py                       ← FastAPI app orchestrator
├── config.py                       ← TOML + env config
├── proxy/                          ← OpenAI + Anthropic proxy routes
│   └── dispatcher.py               ← Backend selection + routing
├── router/                         ← Three-tier hybrid router
│   └── engine.py                   ← IslandRun privacy routing
├── cache/                          ← Three-tier semantic cache
│   └── manager.py                  ← Cache coordination
├── discovery/                      ← Auto-discovery + provisioner
├── tunnel/                         ← Relay integration
├── keys/                           ← BYOK encrypted key storage
├── up.py                           ← `llmhosts up` 5-phase pipeline
└── uninstall.py                    ← Clean uninstall

llmhosts_core/                      ← Rust inference engine (ADR-010)
├── Cargo.toml
└── src/
    ├── lib.rs                      ← PyO3 bridge
    └── compute/mod.rs              ← GPU compute kernels (CubeCL + cudarc)

relay_core/                         ← Self-hosted relay (Rust, ADR-009)

hive/                               ← Embedded agent system (Hive.CEO)
frontend/                           ← Next.js 16 dashboard
├── src/app/
│   ├── dashboard/                  ← Dashboard routes
│   │   └── token/                  ← Token AI conversational interface
│   └── api/                        ← Next.js API routes
├── src/lib/token/                  ← Token agents, classifier, analytics
└── prisma/schema.prisma            ← Database schema (21+ models)

docs/
├── architecture/                   ← 10 ADRs (ADR-001 to ADR-010)
├── plans/                          ← Implementation design docs
├── knowledge-base/                 ← Technology deep-dives
└── roadmap/                        ← 3-layer synchronized roadmap
```

---

## Architecture Decisions (ADR Summary)

| ADR | Decision | Status |
|-----|----------|--------|
| 001 | Three-tier hybrid router (kNN + ModernBERT + Qwen-0.5B) | Active |
| 002 | SUPERSEDED by ADR-010 | Superseded |
| 003 | LiteLLM + Anthropic dialect API proxy | Active |
| 004 | SUPERSEDED by ADR-009 | Superseded |
| 005 | Next.js + shadcn/ui frontend | Active |
| 006 | Model Context Protocol (MCP) for tools | Active |
| 007 | Docker Compose (dev) + K3s (scale) | Active |
| 008 | Tiered pip install (core/smart/full) | Active |
| 009 | Self-hosted Rust relay (WSS + yamux + Noise NK) | Active |
| 010 | LLMHosts Core custom Rust inference engine (CubeCL + cudarc) | Active |

---

## CI/CD Status

| Check | Status |
|-------|--------|
| Python type check (mypy) | ✅ Required |
| Python lint (ruff, self-healing) | ✅ Required |
| Python tests (pytest, 2373 tests) | ✅ Required |
| Frontend lint + type check | ✅ Required |
| Frontend tests (Vitest, 808 tests) | ✅ Required |
| Security scan (Bandit + npm audit) | ✅ Required |
| Docker build | ✅ Required |
| Frontend E2E (Playwright) | ⚠️ Needs `SEED_USER_EMAIL` + `SEED_USER_PASSWORD` secrets |
| Verification gate | ✅ Required (blocks merge) |
| CodeQL | ✅ Required |
| Release workflow | ✅ Active |
| Stale management | ✅ Active (60d stale, 14d close) |

**Branch protection**: `main-protection` ruleset active — PR required + CI gate + no force push.

---

## Label Taxonomy

| Namespace | Labels | Purpose |
|-----------|--------|---------|
| `type:*` | bug, feature, chore, docs, test, security (6) | What kind of work |
| `priority:*` | critical, high, medium, low (4) | When to do it |
| `size:*` | xs, s, m, l, xl (5) | Effort estimate |
| `status:*` | blocked, in-review, needs-spec, deferred, done (5) | Current state |
| `epic:*` | relay, compute, conductor, cache, security, observability, hive, frontend, onboarding, email, self-evolution (11) | Which epic |
| `phase:*` | 1, 2, 3, 4 (4) | Which phase |
| `agent:*` | human, copilot, cursor (3) | Who/what implements |

Missing from harmony canonical taxonomy: `vision-2026` (project-specific, OK to keep).

---

## Dependabot Configuration

| Ecosystem | Directory | Schedule | PR Limit |
|-----------|-----------|----------|----------|
| pip | `/` | Weekly (Monday) | 5 |
| npm | `/frontend` | Weekly (Monday) | 5 |
| docker | `/` | Monthly | (default) |
| github-actions | `/` | Weekly | (default) |

5 open Dependabot PRs (#460–#464) — review and merge when CI passes.

---

## Known Blockers / Human Actions Required

| Issue | Blocker | Unblock action |
|-------|---------|----------------|
| #359 | Hardware E2E — Alienware required | Human must run physical hardware test |
| #360–362 | Hardware E2E — Mac Mini + multi-GPU | Human must run physical hardware tests |
| #134 | Soft launch decision | Human decision required |
| #445 | Launch checklist gates | Multiple gates — see issue |
| E2E CI | `SEED_USER_EMAIL` + `SEED_USER_PASSWORD` not set | Add to GitHub Actions secrets |

---

## 41-Check Audit Results (2026-03-10)

**Score: 34/40 passed (85%)** — 1 skipped (secret scanning — requires Team/Enterprise plan)

| # | Check | Result | Notes |
|---|-------|--------|-------|
| P1.1 | Squash merge enabled | ✅ Pass | |
| P1.2 | Auto-delete branches on merge | ✅ Pass | |
| P1.3 | Repository topics set | ✅ Pass | 10 topics |
| P1.4 | CODEOWNERS exists | ✅ Pass | `.github/CODEOWNERS` |
| P1.5 | Pull request template | ✅ Pass | `.github/pull_request_template.md` |
| P1.6 | SECURITY.md exists | ✅ Pass | Root `SECURITY.md` |
| P2.1 | Bug report issue form | ✅ Pass | `.github/ISSUE_TEMPLATE/bug.yml` |
| P2.2 | Feature/capability issue form | ✅ Pass | `.github/ISSUE_TEMPLATE/feature.yml` |
| P2.3 | Chore/maintenance issue form | ✅ Pass | `.github/ISSUE_TEMPLATE/chore.yml` |
| P2.4 | Blank issues disabled | ✅ Pass | `config.yml: blank_issues_enabled: false` |
| P3.1 | type:* labels | ✅ Pass | 6 labels |
| P3.2 | priority:* labels | ✅ Pass | 4 labels |
| P3.3 | size:* labels | ✅ Pass | 5 labels |
| P3.4 | status:* labels | ✅ Pass | 5 labels |
| P4.1 | Project board exists | ✅ Pass | 9 boards |
| P4.2 | Custom fields configured | ✅ Pass | 16 fields |
| P4.3 | Issues tracked on board | ✅ Pass | |
| P4.4 | Open issues exist | ✅ Pass | 30 open |
| P5.1 | CI workflow exists | ✅ Pass | 6 workflows |
| P5.2 | Dependabot configured | ✅ Pass | 4 ecosystems |
| P5.3 | Auto-labeler configured | ✅ Pass | |
| P5.4 | Stale management | ✅ Pass | `stale.yml` |
| P5.5 | Release workflow | ✅ Pass | `release.yml` |
| P5.6 | CI concurrency | ✅ Pass | cancel-in-progress |
| P5.7 | CI caching | ✅ Pass | |
| P6.1 | SECURITY.md | ✅ Pass | |
| P6.2 | CODEOWNERS | ✅ Pass | |
| P6.3 | Dependabot vulnerability alerts | ❌ Fail | Enable in Settings → Security |
| P6.4 | Secret scanning | ⊘ Skip | Private repo — requires Team/Enterprise |
| P6.5 | CodeQL analysis | ✅ Pass | `codeql.yml` |
| P7.1 | Discussions enabled | ✅ Pass | |
| P7.2 | CONTRIBUTING.md | ✅ Pass | Root `CONTRIBUTING.md` |
| P7.3 | Wiki enabled | ❌ Fail | Intentional — docs live in `/docs` |
| P7.4 | Draft PRs used | ❌ Fail | Advisory — use draft PRs for WIP |
| P8.1 | Ruleset exists | ✅ Pass | `main-protection` (active) |
| P8.2 | Require PR reviews | ✅ Pass* | Ruleset has `pull_request` rule |
| P8.3 | Require status checks | ✅ Pass* | Ruleset has `required_status_checks` |
| P8.4 | Block force push | ✅ Pass* | Ruleset has `non_fast_forward` |
| P9.1 | SSOT.md exists | ✅ Pass | Created this session |
| P9.2 | BUILD-READINESS-INDEX.md | ❌ Fail | To be created |
| P9.3 | Harmony instance directory | ✅ Pass | Created this session |

*pro-audit.sh reports P8.2-8.4 as fail because it checks the list endpoint (which omits rule details). The ruleset at ID 12906150 contains all three rules — verified via `gh api repos/.../rulesets/12906150`.

**Actual failures (requiring action)**:
1. **P6.3** — Dependabot vulnerability alerts: `gh api -X PUT repos/LookNoHandsMom/LLMHosts.com/vulnerability-alerts`
2. **P7.3** — Wiki: intentional. Docs live in `/docs`. Mark as accepted.
3. **P7.4** — Draft PRs: advisory only. Use draft PRs for future WIP.
4. **P9.2** — BUILD-READINESS-INDEX.md: create at `docs/BUILD-READINESS-INDEX.md`

---

## Vision 2026 — Epic Structure

All sub-epics (#472–#482) lack:
- [ ] Milestones (none exist — create "Vision 2026 — Planning" milestone)
- [ ] Acceptance criteria sections
- [ ] Parent issue cross-links back from sub-epic bodies

These are tracked in #481 (this audit). Fixes being applied this session.

---

## Commands (Run Every Session)

```bash
# Orientation (run all 5)
gh issue list --state open --limit 20
gh pr list --state open
gh run list --limit 5
git log --oneline -5
git status

# Start work
gh issue view [N]                           # Read before implementing
git checkout -b feature/[N]-[short-desc]    # New branch per issue

# Before committing
pytest tests/ -v                            # Python tests
npm run test --prefix frontend              # Frontend tests
ruff check .                                # Lint
mypy llmhosts/ --ignore-missing-imports    # Type check

# PR
gh pr create --title "[type(scope): description]" --body "..."
```
