# Day 1: Full Codebase & State Audit

**Date:** 2026-03-10
**Facilitator:** AI Advisory Council Facilitator
**Session Type:** Systematic Audit — All Subsystems
**Output:** 30+ GitHub Issues Created

---

## Opening Statement from the Facilitator

The Advisory Council convenes for its first day with a single mandate: see the project as it is, not as we wish it to be. We have 40 open GitHub issues, a Rust inference engine in active development, a Python proxy that runs in production, and a Next.js frontend serving paying customers. Our job today is to audit every subsystem with brutal honesty — rating completion, identifying gaps, and producing a precise backlog that reflects reality.

Council members were asked to prepare independently. Their assessments follow a structured rubric: completion percentage, what works, what is missing, gap severity (P0/P1/P2), and effort estimate. The facilitator has final authority on all severity ratings.

---

## Competitive Calibration (Research Summary)

Before auditing ourselves, we calibrated against the market as it stands in March 2026:

**Ollama** (v0.17.5, March 2026): Supports cloud model offloading, web search APIs, multimodal, streaming tool calls, thinking models. **Does NOT implement PagedAttention or continuous batching** — serializes requests by design. With 10 concurrent users, response times degrade from 2s to 45s+. Ollama's architecture is a developer convenience tool, not a production serving engine. This is our opening.

**vLLM**: Production serving engine with PagedAttention + continuous batching. 23x throughput improvement over naive serving. Requires complex Python/CUDA environment. ~2GB install. No consumer-friendly packaging, no tunnel, no marketplace, no auto-configuration. This is our ceiling if we do not build.

**llama.cpp**: Added router mode (multi-model dynamic load/unload). No built-in tunnel relay — all networking is external (Cloudflare, Tailscale). No continuous batching. No auto-configuration. No marketplace. Focused on raw inference speed and format support.

**mistral.rs**: Rust inference engine with PagedAttention on CUDA + Metal, FlashAttention V2/V3, continuous batching. Closest technical competitor to LLMHosts Core — but no marketplace, no tunnel, no package distribution, no HIM auto-selection.

**SaladCloud**: RTX 4090 at $0.16/hr, RTX 5090 at $0.25/hr. Consumer GPU marketplace exists but operators are third parties (gamers, e-sports arenas), not developers who opted in for passive income.

**RunPod**: Best developer experience in cloud GPU rental, $1.74/hr for A100, $1.99/hr for H100 community instances. No self-hosted option, no local-first architecture.

**Key Insight**: No competitor combines (a) custom inference engine, (b) self-hosted relay tunnel, (c) operator-first marketplace, (d) single `pip install`. The combination is our moat. Today we audit how close we are to delivering it.

---

## Subsystem 1: LLMHosts Core Inference Engine (`llmhosts_core/`)

**Council Lead:** Marcus Chen (ML Systems Engineering)

### Current State Assessment

The F1 Foundation has been delivered per PR #465 "Core-first wheel — maturin, GPU compute, no Ollama." The codebase ships in `llmhosts_core/src/` with the following modules:

| File | Lines | Status |
|------|-------|--------|
| `lib.rs` | 183 | PyO3 bindings + exports |
| `gguf.rs` | 455 | GGUF model loading |
| `tensor.rs` | 327 | CPU tensor operations |
| `attention.rs` | 151 | Attention mechanisms |
| `generate.rs` | 242 | Token generation loop |
| `kv_cache.rs` | 84 | KV cache (basic) |
| `compute/` | ~400 | CPU, CUDA, WGPU backends |
| `model/` | ~600 | Config, transformer, weights |
| `memory/` | 476 | Block allocator |

**Total F1 code:** ~2,920 lines of Rust. This is a skeleton, not a finished engine.

### What Works

- GGUF file loading with mmap (zero-copy model loading confirmed)
- Q5_0 and Q6_K quantization parsing validated (PR #458)
- Basic CPU tensor operations (matmul, rmsnorm, rope, silu, softmax)
- KV cache data structure exists (simple, no paging)
- WGPU backend compiles (GPU code path exists)
- PyO3 bindings expose `LLMHostsEngine` to Python
- Maturin wheel builds and installs (`pip install llmhosts`)
- Token generation loop exists in `generate.rs`
- Chat template parsing (F1 delivered)
- CUDA backend stub exists (`compute/cuda.rs`)

### What Is Missing

**Critical (F2 — Wave 2):**
- PagedAttention: The `kv_cache.rs` (84 lines) is a flat array, not block-paged. No block table, no page fault mechanism, no eviction policy. The memory allocator (`allocator.rs`, 476 lines) does not implement virtual-to-physical block mapping.
- Continuous batching: `generate.rs` processes one request at a time. No scheduler exists. No request queue, no iteration-level batch management, no iteration-level completion detection.
- FlashAttention: `attention.rs` (151 lines) is standard attention, no tiling, no fused softmax, no memory-efficient backward pass.
- Speculative decoding: Not present.
- Multi-GPU: No tensor parallelism, no pipeline parallelism.
- ROCm/HIP: CubeCL's ROCm support is marked "work in progress" upstream. Our AMD path is not validated.

**High (F3 — Wave 3):**
- Hardware Intelligence Matrix (HIM): No model selection logic. Users must specify models manually.
- Hot-swap protocol: Not implemented. Changing models requires engine restart.
- Model Agent: Not implemented.
- Phase-split serving: Not implemented. Prefill and decode run on the same hardware sequentially.
- Adaptive quantization: Not implemented.

**Medium (F4 — Wave 4):**
- Cluster coordination: No multi-node tensor parallelism.
- KV cache transfer: No cross-node KV cache sharing.
- Fine-tuning interface: Not present.

### Completion Assessment

| Capability | Status | Completion |
|-----------|--------|------------|
| GGUF loading | Working | 85% |
| Basic CPU inference | Working | 70% |
| GPU inference (WGPU) | Compiles, untested | 30% |
| GPU inference (CUDA) | Stub only | 10% |
| KV cache (basic) | Working | 50% |
| KV cache (PagedAttention) | Not started | 0% |
| Continuous batching | Not started | 0% |
| PyO3 bridge | Working | 80% |
| Maturin wheel | Working | 75% |
| Chat templates | Working | 75% |
| HIM auto-selection | Not started | 0% |
| Hot-swap | Not started | 0% |
| FlashAttention | Not started | 0% |
| Speculative decoding | Not started | 0% |

**Overall F1 completion: 62%**
**F2 readiness: 15%**

### Gap Severity

- **P0**: PagedAttention + continuous batching (blocks Wave 2 entirely)
- **P0**: CUDA backend validation (most of our users are NVIDIA)
- **P1**: FlashAttention kernel (2x+ performance gap vs. competition)
- **P1**: HIM auto-selection (required for "just works" promise)
- **P2**: Speculative decoding, hot-swap, phase-split

### Council Discussion

Marcus Chen: "The kv_cache.rs file at 84 lines concerns me deeply. PagedAttention's block table alone is typically 200+ lines. The allocator exists but maps physical memory blocks without virtual addressing. We need to implement proper block tables before claiming PagedAttention support."

Dr. Kai Tanaka: "The WGPU backend is promising for cross-platform coverage but we have zero benchmark data on it. CubeCL's maturity is still rough — I've seen the 'rough edges' admission from the Burn team. We should have a CUDA validation harness before claiming Wave 2 readiness."

**Facilitator ruling:** F1 is ~62% complete. The engine generates tokens. It does not serve concurrent users. Wave 2 has a hard dependency on PagedAttention + continuous batching. Estimate: 3 months of dedicated Rust engineering for F2.

---

## Subsystem 2: relay_core Tunnel (`relay_core/`)

**Council Lead:** Dr. Amara Osei (Distributed Systems & Networking)

### Current State Assessment

`relay_core/src/lib.rs` is 212 lines. This is M1 (milestone 1) — the skeleton described in issue #438.

### What Works

- Rust crate compiles and is included in workspace
- PyO3 bridge exposes relay functionality to Python
- Basic TCP connection structure exists

### What Is Missing

- Noise Protocol NK encryption (M2, issue #440)
- yamux stream multiplexing (M3, issue #441) — critical for concurrent LLM request streams
- relay_server binary for VPS (M4, issue #442)
- DNS + TLS configuration for relay server
- Production hardening (M5, issue #443)

### Completion Assessment

**Overall relay_core completion: 20%**
M1 delivered. M2-M5 not started.

### Gap Severity

- **P0**: Noise NK encryption — users cannot use relay safely without this
- **P0**: yamux multiplexing — single stream cannot handle concurrent LLM requests
- **P0**: relay_server binary — no VPS-side component exists
- **P1**: DNS + TLS + monitoring for relay server

### Council Notes

Dr. Amara Osei: "The QUIC question deserves consideration. HTTP/3 + QUIC is becoming the standard for web traffic in 2026, with CDNs and browsers broadly supporting it. Our TCP-based relay with Noise NK is a solid choice for the LLM inference case — the connection persistence and backpressure handling of TCP with yamux is actually better for long-running inference streams than QUIC's connection migration features. I'd recommend we stay with TCP + Noise NK + yamux for M2-M3 and revisit QUIC in Wave 4 if needed."

---

## Subsystem 3: Python Proxy / FastAPI (`llmhosts/`)

**Council Lead:** Sofia Rodriguez (Developer Experience)

### Current State Assessment

The Python proxy is the most mature subsystem. It has been in production since Phase 2.

### What Works

- FastAPI app with OpenAI + Anthropic compatible endpoints
- BackendDispatcher with router integration
- LiteLLM adapter for cloud backends
- Authentication (API key enforcement)
- PII detection and IslandRun privacy routing
- Prometheus metrics export
- Audit logging (SQLite)
- Health check endpoints
- `llmhosts serve` command
- `llmhosts doctor` diagnostic tool
- `llmhosts tunnel start` command
- `llmhosts suggest-models` command
- `llmhosts setup` wizard

### What Is Missing

- **LLMHosts Core integration**: The proxy currently uses Ollama as backend. The inference module (`llmhosts/inference/core.py`) exists but LLMHosts Core is not fully wired as the default.
- **Streaming from LLMHosts Core**: Core generates tokens; streaming SSE from Rust to Python to client is partially wired.
- **Concurrent request handling**: Without PagedAttention, concurrent requests serialize.
- **Model management CLI**: `llmhosts models list/pull/remove` needs full implementation.

### Completion Assessment

**Overall Python proxy completion: 82%**

### Gap Severity

- **P1**: Core engine integration (streaming + concurrent) — blocking production use of Core
- **P1**: Model management CLI completeness
- **P2**: Dynamic model hot-swap from CLI

---

## Subsystem 4: Three-Tier Router (`llmhosts/router/`)

**Council Lead:** Marcus Chen

### Current State Assessment

The three-tier router (`knn.py`, `modernbert.py`, `qwen.py`, `engine.py`) routes requests based on content to appropriate backends.

### What Works

- Tier 1: kNN with FAISS/ONNX — rule-based + similarity matching
- Tier 2: ModernBERT classifier integration
- Tier 3: Qwen2.5-0.5B fallback (when configured)
- Privacy enforcement (PII detection → IslandRun routing)
- Cost optimization (route cheap requests to local, expensive to cloud)
- Latency measurement per tier

### What Is Missing

- **VRAM budget enforcement**: Router models themselves consume VRAM. On 8GB devices, loading router + inference model simultaneously is not validated.
- **Router performance under concurrent load**: No load test data.
- **Tier 3 (Qwen) production validation**: Not confirmed to run within 500ms target on consumer hardware.
- **Router metadata in GGUF**: Custom GGUF extensions for routing metadata not implemented.

### Completion Assessment

**Overall router completion: 72%**

---

## Subsystem 5: Semantic Cache (`llmhosts/cache/`)

**Council Lead:** Dr. Priya Mehta (Privacy) + Marcus Chen

### Current State Assessment

Three-tier cache: `exact.py` (hash), `namespace.py` (entity), `vcache.py` (semantic), coordinated by `manager.py`.

### What Works

- Exact hash cache (100% hit rate for identical requests)
- Namespace-based entity cache
- vCache semantic cache with FAISS vector similarity
- Cache hit headers (`X-LLMHosts-Cache-Hit`)
- Privacy-aware cache: PII-tagged requests bypass semantic cache

### What Is Missing

- **Verified cache correctness guarantees**: Our vCache is based on the February 2026 paper but the online learning algorithm for per-prompt threshold estimation is not fully implemented. We use a static threshold.
- **Cache federation**: Per-node cache only. No cross-node cache warming.
- **Cache hit rate instrumentation**: No dashboard showing cache efficiency.
- **Cache invalidation on model swap**: When model changes, semantic similarity changes. Cache does not invalidate.

### Completion Assessment

**Overall semantic cache completion: 68%**

---

## Subsystem 6: Frontend Dashboard (`frontend/`)

**Council Lead:** Sofia Rodriguez

### Current State Assessment

Next.js 16 App Router, React 19, TypeScript, TailwindCSS 4, Prisma + SQLite.

### What Works

- Landing page with pricing
- Authentication (Auth.js v5, email/password)
- Dashboard with device list
- Stripe billing (live keys configured)
- Marketplace operator UI
- Device registration via Device Code Auth (RFC 8628)
- Token AI assistant (conversational fleet orchestrator)
- Live pipeline stream (SSE)
- 9 documentation pages (5 public, 4 auth-gated)
- Fleet management (device list, status, VRAM)
- Buyer dashboard (marketplace usage)
- Email verification + reset

### What Is Missing

- **LLMHosts Core model management UI**: No "Models" page showing installed models, VRAM usage, HIM recommendations.
- **Real-time inference metrics**: TTFT, tokens/sec, queue depth not displayed.
- **Model pull progress UI**: No visual feedback for `llmhosts pull` command.
- **Operator earnings dashboard**: Stripe payouts are functional but no charts/history.
- **Hardware profile page**: No display of GPU detection results.
- **WCAG 2.2 AA full compliance**: Some areas flagged from prior audits (PR #429 addressed many).

### Completion Assessment

**Overall frontend completion: 79%**

---

## Subsystem 7: GPU Marketplace (Billing, Routing, Reputation)

**Council Lead:** Lena Björk (Business Development) + James Park (GTM)

### Current State Assessment

Phase 5 is marked complete across 12 PRs (#303, #311, #313, #314, #393-398, #404-407, #430).

### What Works

- Device registration and catalog
- Marketplace router with retry logic
- Stripe Connect KYC and payouts
- Reputation scoring engine (weighted scoring, tier system)
- Auto-removal of low-reputation nodes (<30% uptime)
- Buyer dashboard with usage tracking
- Operator agreement workflow
- Prepaid credits system
- At-capacity cloud fallback
- GPU remit system (read-only discovery guarantee)

### What Is Missing

- **Marketplace without Ollama**: All marketplace nodes currently serve via Ollama. When LLMHosts Core replaces Ollama, marketplace nodes need to upgrade.
- **Dynamic pricing**: Static pricing per operator. No demand-based pricing.
- **Fleet management at scale**: Tested for single-user scenarios, not validated at 100+ nodes.
- **Operator income projections**: No forecasting UI.
- **SLA enforcement**: No automated SLA breach handling beyond reputation scoring.

### Completion Assessment

**Overall marketplace completion: 76%**

---

## Subsystem 8: Authentication & Authorization

**Council Lead:** Nina Petrova (Security)

### What Works

- Auth.js v5 (JWT-based)
- API key management (`llmhosts keys add`)
- Device Code Auth (RFC 8628) for CLI-to-SaaS auth
- Role-based access (user / admin)
- OWASP security headers
- Rate limiting (Redis-backed)
- CrowdStrike WAF (via Pangolin/Traefik)

### What Is Missing

- **SOC 2 Type II controls**: No formal audit trail for SOC 2 compliance. Enterprise prospects will require this.
- **Zero-trust relay authentication**: When relay M2 ships with Noise NK, the identity model needs to bind device identity to relay session.
- **API key rotation**: No automated rotation policy.
- **Enterprise SSO (SAML/OIDC)**: Not implemented. Enterprise procurement requires this.
- **Audit log export**: Audit logs stored in SQLite but no export UI or SIEM integration.

### Completion Assessment

**Overall auth/security completion: 71%**

---

## Subsystem 9: Test Coverage and CI/CD

**Council Lead:** Sofia Rodriguez + Raj Patel

### Current State

- **Backend**: 2,469 pytest tests collected (121 test files)
- **Frontend**: 808+ Vitest tests
- **E2E**: 29 Playwright tests
- **CI**: 17-check verification gate
- **Coverage**: Not currently measured across all paths

### What Is Missing

- **LLMHosts Core Rust tests**: The Rust crates have minimal test coverage. No benchmarks.
- **Concurrent load testing**: No load test that validates behavior under 10+ concurrent requests.
- **GPU integration tests**: No CI runner with actual GPU. All GPU code paths are untested in CI.
- **relay_core tests**: Near zero (212 lines of code, no test file).
- **E2E with Core engine**: No E2E test that exercises the full path through LLMHosts Core.
- **Chaos testing**: No fault injection tests.

### Completion Assessment

**Overall test coverage completion: 61%**

---

## Subsystem 10: Documentation & Onboarding

**Council Lead:** Alex Thompson (Developer Marketing)

### What Works

- 9 documentation pages (quickstart, API, CLI, pricing, home, configuration, deployment, troubleshooting, fleet)
- `AGENTS.md` (universal grounding)
- `CLAUDE.md` (project conventions)
- 10 ADRs
- OpenAPI spec (`docs/architecture/openapi.yaml`)
- Multiple design documents

### What Is Missing

- **LLMHosts Core documentation**: No user-facing documentation for the Core engine. No "how inference works" page.
- **Video content**: No quickstart video, no demo.
- **Benchmark documentation**: No published performance numbers.
- **Model compatibility matrix**: No public list of supported models.
- **Community docs**: No contributing guide (CONTRIBUTING.md exists but is basic).
- **Troubleshooting for GPU-specific issues**: Common CUDA/VRAM errors not documented.

### Completion Assessment

**Overall documentation completion: 58%**

---

## Subsystem 11: Installation & Packaging (maturin wheel)

**Council Lead:** Dr. Kai Tanaka

### What Works

- `pip install llmhosts` installs Python package
- Maturin wheel builds Rust crates into the wheel
- Tiered install (`llmhosts`, `llmhosts[smart]`, `llmhosts[full]`)
- GitHub Actions CI builds wheels
- maturin 1.12.6 (March 2026 — latest)

### What Is Missing

- **CUDA wheel variants**: Currently the wheel includes CUDA code but ships a single binary. NVIDIA's approach (wheel variants) for targeting specific compute capabilities is not implemented. The wheel may include unnecessary code for non-CUDA users.
- **Cross-compilation**: Windows builds require WSL or cross-compilation setup. Not automated.
- **Homebrew formula**: No Homebrew tap for `brew install llmhosts`.
- **winget/apt/nixpkgs**: Not distributed via any system package manager.
- **pipx/uvx optimization**: Issue #448 open — PATH and venv edge cases not fully handled.
- **Binary size**: CUDA binary size optimization per NVIDIA's 2026 guidance not applied.

### Completion Assessment

**Overall packaging completion: 64%**

---

## Subsystem 12: OS Compatibility (Linux/Mac/Windows)

**Council Lead:** Dr. Kai Tanaka

### What Works

- Linux (Ubuntu 22.04, 24.04) — primary platform, well-tested
- macOS (Apple Silicon via WGPU backend) — compiles, limited testing
- Windows — CLI works via WSL, native path is challenging

### What Is Missing

- **macOS ANE/Metal validation**: The Apple Neural Engine situation is complex. As of March 2026, Apple M5 embeds Neural Accelerators in each GPU core, programmable via Metal 4 Tensor APIs. Our WGPU backend uses Metal but does not target the Neural Engine directly.
- **Windows native binary**: No native Windows `.exe`. Requires WSL2 or Python-only mode.
- **AMD ROCm validation**: CubeCL's ROCm support is "work in progress" — our AMD path is unvalidated.
- **Intel GPU**: No Intel Level Zero support implemented.
- **Apple Silicon VRAM budget**: 4GB offset for Apple Silicon (shared memory architecture) needs validation.

### Completion Assessment

**Overall OS compatibility completion: 52%**

---

## Day 1 Summary Dashboard

| Subsystem | Completion | P0 Gaps | P1 Gaps | P2 Gaps |
|-----------|------------|---------|---------|---------|
| LLMHosts Core F1 | 62% | PagedAttention, CUDA validation | FlashAttention, HIM | Speculative decoding |
| relay_core | 20% | Noise NK, yamux, relay_server | DNS/TLS | Monitoring |
| Python Proxy | 82% | — | Core streaming integration | Hot-swap CLI |
| Three-Tier Router | 72% | — | VRAM budget enforcement | GGUF metadata |
| Semantic Cache | 68% | — | Verified threshold, invalidation | Federation |
| Frontend | 79% | — | Models UI, real-time metrics | Earnings charts |
| GPU Marketplace | 76% | — | Core migration, fleet scale | Dynamic pricing |
| Auth/Security | 71% | — | Zero-trust relay, SOC 2 | SSO, key rotation |
| Test Coverage | 61% | Rust tests, GPU CI | Load testing, chaos | E2E with Core |
| Documentation | 58% | — | Core docs, benchmarks | Video, matrix |
| Packaging | 64% | — | CUDA variants, Windows | Homebrew, apt |
| OS Compatibility | 52% | AMD ROCm | macOS ANE, Windows native | Intel GPU |

**Weighted average completion: 65%**

---

## Critical Path (P0 Issues — Must Resolve Before Wave 2)

1. **PagedAttention in llmhosts_core** — blocks concurrent inference entirely
2. **Continuous batching scheduler** — blocks production serving
3. **CUDA backend validation** — 80% of target users are NVIDIA
4. **Noise NK encryption in relay_core** — relay is not production-safe without it
5. **yamux multiplexing** — relay cannot handle concurrent LLM streams
6. **relay_server VPS binary** — no server-side relay without this

---

## Issues Created Today (See GitHub)

Day 1 produces 30 GitHub issues covering all gap areas above. Each issue is linked to the appropriate Wave milestone and carries the `retreat:day1-audit` label.
