# Master E2E Test Plan — Full Capability, Harmony, Production-Ready

**META**
- **Created**: 2026-02-27
- **SSOT**: GitHub (issues, project board, labels). This doc is the narrative and index; execution and results live in GitHub and in `P5-full-test-guide.md` + runbook/checklist.
- **Goal**: Prove every aspect of the user experience and application capability end-to-end — app, install, CLI, proxy, router, cache, discovery, marketplace, billing, resiliency — with no hack job. All warnings, errors, or design gaps captured in GitHub. End state: production-grade, customer-ready, masterclass harmony.
- **Product stance**: We build **our own solution**. External projects (e.g. OpenClaw) are **optional leverage** — we may reuse specific code or patterns only where they fill gaps and match our objectives (GPU remit, user control, read-only access). We do not replicate them. See `docs/plans/2026-02-27-gpu-remit-and-user-control.md`.

---

## 1. Vision and bar

- **Full capability from the user’s perspective**: Every user journey, every dashboard route, every API and CLI command that the product exposes must work and be testable. Install flow, app flow (login → keys → devices → topology → marketplace → billing), local inference, tunnel, relay (when used), router, cache, discovery, and cost/savings must align.
- **Harmony**: What the app says and what the CLI/proxy do must match. Docs, UI, and behavior are one coherent experience. Any mismatch is a bug or doc gap → GitHub issue → fix → re-run.
- **Masterclass**: Smart routing (right model, best utilization), resiliency (local at capacity → seamless external/cloud where designed), learning/adaptation and value communication are in scope where already built; testing proves them and captures gaps for future work (e.g. explicit “at capacity” fallback, AI assistant suggestions, multi-GPU orchestration).
- **Production-ready**: By the end of the test campaign we know what works, what’s missing, and what’s needed to reach 100% customer-ready (see Gap analysis below).

---

## 2. Test hierarchy (docs and GitHub)

| Doc | Role |
|-----|------|
| **docs/testing/MASTER-E2E-PLAN.md** | **This file.** Strategy, rounds, machine matrix, user journeys/capabilities index, gap analysis. Single place for “what we’re proving” and “what’s next.” |
| **docs/testing/P5-full-test-guide.md** | **Execution master.** Part A (site), Part B (Alienware battery + Phase 0), Part C (recap + Unresolved issues + Suggestions). Real external user, app-led, no hack. Use for step-by-step runs and report-back. |
| docs/testing/P5-external-connectivity-runbook.md | Scenario/codebase reference; execution → P5 full test guide. |
| docs/testing/P5-external-connectivity-human-checklist.md | Abbreviated Alienware flow; full steps → P5 full test guide. |
| docs/testing/P5-external-connectivity-results-template.md | Optional per-step log; recap format → P5 full test guide Part C. |
| docs/testing/README.md | Index and “start here” for AI/human runners. |

**GitHub**: All failures, warnings, and design gaps → issues (e.g. `[P5]`, `[E2E]`, `[Alienware]`). Labels: `testing`, `P0-blocker` / `P1-critical`, `human` where manual. Project board: test phases, status, owner. Milestones: POC Testing Phase, Production readiness.

---

## 3. Full capability index (what we prove)

Alienware-first battery must touch every capability below. Anything not exercised is a gap; document and create issues as needed.

### 3.1 App (llmhosts.com) — user-facing

- **Auth**: Login (brendonurban@gmail.com), logout, session persistence, redirect to dashboard when authenticated.
- **Dashboard routes** (from `DashboardLayoutClient`): Overview, Spending, Savings, Add Entry, Projections, Achievements, API Keys (list/create/delete, plan limits), Cache, Feedback, Topology, Devices, Marketplace, LAN Nodes, Token, Team, Settings (plan, billing portal, profile).
- **API Keys**: Create key from app → use for proxy auth (Bearer). Plan limits (Free 2, Pro 10, etc.).
- **Devices**: Registered devices, tunnel URL, status, marketplace opt-in; consistent with proxy/relay when device reports.
- **Topology**: Backends/nodes from proxy; no 5xx; consistent with actual backends.
- **Marketplace**: Usage, credits (prepaid), catalog/uptime if wired; Stripe Connect for operators.
- **Billing**: Plan display, upgrade (Stripe Checkout), billing portal (Stripe), webhook handling.
- **Settings**: Plan, “Manage billing,” profile (email, name).

### 3.2 CLI (local machine)

- **Install**: `pip install llmhosts` (tiered: core/smart/full as applicable).
- **doctor**: Python, config, Ollama (and/or vLLM, etc.), BYOK, proxy port, tier, disk, **hardware (GPU detection — any GPU, not assumed)**, database, cache, tunnel tools.
- **serve**: Start proxy (and optional TUI); health, models, completions locally.
- **tunnel start**: Tailscale or Cloudflare; tunnel URL printed; reachable from external network.
- **keys**: Add/list (BYOK); used when proxy requires cloud.
- **setup**: First-run wizard where present.
- **suggest-models**: Hardware-aware suggestions (e.g. for Alienware GPU); no assumption on GPU model.

### 3.3 Proxy and API

- **Health**: `GET /health` → status, version, timestamp.
- **Models**: `GET /v1/models` → list from dispatcher (Ollama/vLLM/cloud/remote).
- **Completions**: `POST /v1/chat/completions` (stream and non-stream), `POST /v1/messages` (Anthropic) where supported.
- **Headers**: `X-Request-ID`, `X-LLMHosts-Model`, `X-LLMHosts-Cost` (or equivalent).
- **Auth**: 401 when required and missing/invalid API key; 200 with valid key from app.
- **Dashboard API** (proxy): `/api/backends`, `/api/topology`, `/api/config`, `/api/cache/*`, `/api/savings`, `/api/metrics`, `/api/requests` (if exposed to frontend).

### 3.4 Router, cache, discovery

- **Router**: Rule-based (and kNN/ModernBERT/Qwen where tier installed); correct backend selected; PII/privacy (IslandRun) where enabled.
- **Cache**: vCache tiers (exact, namespace, semantic); hit/miss behavior; no wrong responses.
- **Discovery**: Ollama (and vLLM, etc. where present) discovered; `GET /v1/models` reflects discovered backends; multi-engine on one machine (different ports) if applicable.

### 3.5 Backend selection and fallback (current behavior)

- **Local first**: Healthy local backend serving the model is used.
- **Cloud**: Model detected as cloud + healthy LiteLLM backend → cloud.
- **Fallback**: No local candidate → healthy litellm backend; else any healthy non-litellm. (No explicit “at capacity” queue yet — fallback when unhealthy/no candidate; document for future “at capacity” + seamless external.)
- **Remote (marketplace)**: Devices from catalog registered as remote backends; priority; requests forwarded to tunnel URL when selected.

### 3.6 Resiliency and robustness

- **Tunnel**: Disconnect/reconnect; tunnel URL change (e.g. Cloudflare Quick) or stable (Tailscale); device offline after missed heartbeats.
- **Backend down**: Ollama killed → proxy/relay behavior; restart Ollama → recovery without restarting proxy.
- **Concurrent/sequential**: Multiple requests; no spurious 5xx; queue or graceful degradation.
- **Large context**: 2000+ token prompt; completes or clear error.

### 3.7 Billing and marketplace (Phase 5)

- **Credits**: Prepaid balance; deduction on marketplace usage; low balance behavior.
- **Stripe Connect**: Operator onboarding, login-link; marketplace listing where implemented.
- **Reputation/probes**: Uptime/probes for marketplace nodes; dashboard/API consistency.

### 3.8 Observability and value communication

- **Cost tracking**: Request cost and savings visible (dashboard, headers).
- **Metrics**: Prometheus `/metrics` where mounted; no PII in logs.
- **Audit**: If enabled; export/endpoints as designed.

---

## 4. Round 1: Alienware-first (single GPU host)

**Machine**: Alienware laptop, one GPU onboard (type agnostic — solution must adapt).

**Objective**: Maximize coverage on one host. Every user journey and capability in §3 that can be exercised on a single machine: app (all routes), CLI (install, doctor, serve, tunnel, keys, suggest-models, setup), proxy (health, models, completions, auth), router/cache/discovery, tunnel from external network, Scenario A (client) + B (host), cross-check (Topology, Devices, Marketplace), resiliency (reconnect, backend down, concurrent, large context), billing flows (plan, keys, portal). Capture every failure, warning, or design shortfall as a GitHub issue; re-run until 100% pass or documented known gaps.

**Execution**: Follow `docs/testing/P5-full-test-guide.md` (Part A, Phase 0, Part B, Part C). Expand step coverage to explicitly hit every dashboard route and every CLI command above; add steps where the guide is summary-level (e.g. “A.3.6 optional” → explicit pass over Spending, Savings, Add Entry, Projections, Achievements, Cache, Feedback, LAN Nodes, Token, Team). Same guide, same recap format (Unresolved issues, Suggestions); report-back to project owner → dev agent.

**Exit condition**: All steps pass or every failure has an issue; recap + Unresolved + Suggestions in GitHub (#255 or successor); deferred/README and LAUNCH_STATUS updated. Then proceed to Round 2.

---

## 5. Round 2: Regression + platform and hardware diversity

**Objective**: Regression on the same battery plus scenarios that only appear on different hardware/OS.

### 5.1 Mac Mini (OS and stack diversity)

- **Profile**: Different OS (macOS), possibly Apple Silicon (M1/M2) or Intel; Ollama (and/or other backends) as available.
- **Prove**: Same E2E battery as Round 1; install, doctor, serve, tunnel, inference; dashboard consistency. OS-specific issues (e.g. tunnel tool, GPU detection) → GitHub.
- **Execution**: Same P5 full test guide; record Host profile (Mac Mini, OS, backend); Part C recap; Unresolved + Suggestions.

### 5.2 Multi-GPU server (2× 4090s)

- **Profile**: One machine, 2 GPUs; same or different models per GPU; possible patterns: 2× same model (e.g. 2× qwen2.5:14b), or 1× model A on GPU0 and 1× model B on GPU1, or adaptive (e.g. 4 instances of same model across 2 GPUs if supported by engine).
- **Prove**: Discovery sees backends (Ollama/vLLM/etc. on this host); dispatcher selects correct backend; utilization and routing behave as designed; no wrong-model or single-GPU-only assumption. Document current behavior (e.g. one Ollama instance vs multi-instance vLLM) and capture gaps (e.g. explicit multi-GPU load balancing, adaptive instance count) as issues/suggestions.
- **Execution**: Phase 0 (Host discovery) on this machine; run Part B (and Part A if needed); recap with Host profile “2× 4090, …”; Unresolved + Suggestions.

### 5.3 Large GPU (e.g. NVIDIA Spark 128K)

- **Profile**: Large memory / long context (e.g. 128K); may change model choice, context handling, or timeouts.
- **Prove**: doctor and discovery; model list and completions; large-context request handling; any platform-specific behavior or limits. Gaps (e.g. context-length handling, timeouts) → GitHub.
- **Execution**: Phase 0 + Part B (and A as needed); recap with Host profile; Unresolved + Suggestions.

**Exit condition**: All three (Mac Mini, multi-GPU, large GPU) run with recaps; regression pass; new issues from diversity captured. Then run Gap analysis (§7).

---

## 6. AI assistant and “masterclass” (current and future)

**Already in product (test as part of full capability)**:
- **Router**: Tier 0 (rule) + Tier 1 (kNN) + Tier 2 (ModernBERT) + Tier 3 (Qwen) → best-fit model selection; test via varied prompts and model list.
- **suggest-models**: Hardware-aware recommendations; test on each host type.
- **Cost/savings**: Dashboard and headers; value visible to user.
- **Privacy (IslandRun)**: PII detection and local-only routing where configured; test per design.

**Not yet built (capture as suggestions / roadmap)**:
- **At-capacity fallback**: Seamless use of external/cloud when local is at capacity (dispatcher today falls back when no healthy local candidate, not on queue depth).
- **AI assistant**: Onboarding grounding, usage-based profile suggestions, “optimized LLM remit” recommendations; learning/adaptation over time.
- **Multi-GPU orchestration**: Adaptive instance count (e.g. 4× same model on 2 GPUs), load balancing across GPUs, SLO-aware scheduling.
- **Explicit resiliency/autoscaling**: Platform-level “local full → fail over to cloud” with clear user communication.

Testing proves what exists; Unresolved issues and Suggestions in Part C feed the backlog for these items.

---

## 7. Gap analysis (after Round 2)

After Round 1 (Alienware) and Round 2 (Mac Mini, multi-GPU, large GPU):

1. **List all Unresolved issues** (from recaps) and ensure each has a GitHub issue; prioritize P0/P1.
2. **List all Suggestions** (UX, resiliency, multi-GPU, assistant, fallback); create backlog issues or append to epics (e.g. #290).
3. **Coverage gaps**: Capabilities in §3 that could not be tested (e.g. missing hardware, env, or feature). For each: “To prove X we need Y” (e.g. “To prove multi-GPU load balancing we need Z in dispatcher”).
4. **Production readiness**: What must be true for customer-ready (e.g. all P0/P1 resolved, billing live, email verified, relay/device flow stable). Publish a short “Production readiness checklist” (in LAUNCH_STATUS or new doc) and mark items done or blocked.
5. **Next steps**: Ordered list: fix P0 → re-run from failure step; then P1; then address suggestions in priority order; then re-run Round 2 on any machine that had failures.

If after Rounds 1 and 2 some capabilities still cannot be proven to 100%, the gap analysis explicitly states: “To reach 100% E2E masterclass and production-ready we need: [list]. Until then: [current confidence and remaining risks].”

---

## 8. GitHub as SSOT (Feb 2026)

- **Issues**: Every test failure, warning, or design gap → issue. Labels: `testing`, `e2e`, `P5`, `alienware`, `mac-mini`, `multi-gpu`, `human`, `P0-blocker`, `P1-critical`, etc. Link to recap or step ID.
- **Project board**: “LLMHosts.com Roadmap” (or dedicated test view): Test Phase (P0–P7, Round 1, Round 2), Owner (AI/Human), Status (Not Started / In Progress / Pass / Fail / Blocked). Use custom fields for machine (Alienware, Mac Mini, 2×4090, Spark) and round.
- **Milestones**: “POC Testing Phase”, “Production readiness”. Link issues to milestones.
- **Docs**: Runbooks and master guide live in repo; execution and results reference GitHub (issue numbers, links). No duplicate state — GitHub is the source of truth for open/closed and priorities.

---

## 9. Quick reference

- **Execution**: `docs/testing/P5-full-test-guide.md` (Part A → Phase 0 → Part B → Part C).
- **Test account**: brendonurban@gmail.com; password from project owner; API key from app (Dashboard → API Keys).
- **Recap**: `docs/testing/P5-test-run-YYYY-MM-DDTHHmmss.md`; Unresolved issues + Suggestions; paste or link in GitHub.
- **Failure loop**: Issue → fix → re-run from failed step until pass or documented.
- **Machines**: Round 1 = Alienware (1 GPU). Round 2 = Mac Mini, 2×4090 server, large GPU (e.g. Spark 128K).

---

## 10. External battery — runner checklist (handoff)

**For whoever runs the battery on real hardware (AI or human):**

1. **Read first:** `docs/testing/README.md` → then `docs/testing/P5-full-test-guide.md` (full steps). Strategy: this file (MASTER-E2E-PLAN).
2. **Round 1 — Alienware:** Run Part A + Phase 0 + Part B + Part C. Hit every dashboard route and CLI command in §3. Every failure → GitHub issue; re-run until 100% pass or documented.
3. **Round 2 — in order:** (a) Mac Mini (same battery; Host profile + recap); (b) multi-GPU server (2×4090s; Phase 0 + Part B; document behavior and gaps); (c) large GPU (e.g. Spark 128K; Phase 0 + Part B; document context/limits).
4. **Recap per run:** Create `docs/testing/P5-test-run-YYYY-MM-DDTHHmmss.md` with Part C template (Environment, Phase 0, Part A/B, Summary, **Unresolved issues**, **Suggestions**). Paste or link in GitHub (e.g. comment on #319 or the run’s issue).
5. **Report-back:** Project owner shares Unresolved + Suggestions with dev agent; fix → re-run from failed step until all pass or gap analysis is complete.
6. **Gap analysis (after Round 2):** Per §7 — list Unresolved, Suggestions, coverage gaps, production readiness checklist, and “To reach 100% we need: …”.

**What to send back:** Each recap file (or link) + list of new GitHub issues created. No separate “report” needed if recaps and issues are in GitHub.
