# Full-Stack E2E Test Battery — LLMHosts.com

> **Date**: 2026-03-02
> **Purpose**: Validate the entire LLMHosts.com product lifecycle using real hardware, from frontend login to GPU inference, ensuring every layer works in harmony.
> **First Target**: Alienware RTX 4090 Laptop (Windows 11, i9-14900HX, 32 cores, 64GB RAM, 16GB VRAM)
> **Access**: SSH via WireGuard mesh (100.64.0.2, user: brend)

## Philosophy

1. **The laptop stays immaculate** — zero legacy debt. Every test cleans up completely.
2. **Full stack, not just CLI** — frontend, backend API, CLI, proxy, database, billing — all tested together.
3. **Scientific** — known starting state, controlled conditions, measurable outcomes.
4. **Customer perspective** — every test simulates what a real user sees and does.
5. **Fix as we go** — when something breaks, fix it, retest, move on.

## State Management Protocol

### Before Every Test
```
1. SSH to Alienware: audit installed packages, running processes, ports, services, config dirs
2. Save snapshot as pre-test baseline
3. Configure laptop to required starting state for this test
4. Verify starting state matches expectations
```

### After Every Test
```
1. Stop any llmhosts/Ollama processes
2. Remove pip packages we installed
3. Delete ~/.llmhosts/, ~/.config/llmhosts/
4. If we installed Ollama: uninstall + remove ~/.ollama/
5. Clear LLMHOST_* env vars
6. Kill anything on port 4000
7. Snapshot again, diff against pre-test baseline
8. FAIL if any residue found
```

### Tracking
Every file, directory, service, process, and env var we create is logged. Cleanup verification compares against the pre-test snapshot.

---

## Test Battery: 6 Stages, 50+ Scenarios

### Stage 1: Frontend & Dashboard Baseline (Browser-based)

Before the laptop does anything, verify the web platform is healthy.

| # | Test | Method | Expected |
|---|------|--------|----------|
| 1.1 | Load https://llmhosts.com | Playwright | Landing page renders, no console errors |
| 1.2 | Login with seed user | Playwright | Redirect to /dashboard |
| 1.3 | Dashboard renders empty fleet | Playwright | "Connect your first device" CTA visible |
| 1.4 | API keys page accessible | Playwright | Can create/list/delete API keys |
| 1.5 | Cost tracking page accessible | Playwright | Empty state, no errors |
| 1.6 | Documentation pages load | Playwright | All 9 doc pages render |
| 1.7 | No console errors on any page | Playwright | Zero JS errors in browser console |
| 1.8 | API health check | curl | GET /api/health returns 200 |
| 1.9 | Device list API empty | curl | GET /api/devices returns { devices: [], count: 0 } |

### Stage 2: Clean Install & First Run on Alienware (SSH)

Starting state: no llmhosts, no Ollama on the laptop.

| # | Test | Command (via SSH) | Expected |
|---|------|-------------------|----------|
| 2.1 | Install core tier | `pip install llmhosts` | Exit 0, `llmhosts --version` works |
| 2.2 | Help output | `llmhosts --help` | All commands listed |
| 2.3 | Doctor — no engine | `llmhosts doctor` | 14 checks run, GPU detected, Ollama reported missing |
| 2.4 | Doctor — GPU accuracy | Parse doctor output | "NVIDIA GeForce RTX 4090 Laptop GPU" + "16376 MiB" |
| 2.5 | Suggest models | `llmhosts suggest-models` | Recommends models matching 16GB VRAM |
| 2.6 | Discover — no engine | `llmhosts discover` | Reports no engines found (not a crash) |

### Stage 3: The Autopilot — `llmhosts up` (SSH + Frontend)

Starting state: llmhosts installed, no Ollama. This is the flagship test.

| # | Test | What Happens | Expected |
|---|------|--------------|----------|
| 3.1 | Phase 1: Assess | Hardware scan + engine probe | RTX 4090 detected, no engine found |
| 3.2 | Phase 2: Provision | Auto-install Ollama on Windows | Ollama installed, correct model pulled (T3 tier for 16GB) |
| 3.3 | Phase 2: Model pull | Stream download | Progress reporting works, model available after pull |
| 3.4 | Phase 3: Start proxy | FastAPI on :4000 | /health returns 200, /v1/models lists pulled model |
| 3.5 | Phase 4: Connect | Device auth + registration | Device registered with llmhosts.com API |
| 3.6 | **Dashboard shows device** | Check frontend | Alienware appears in fleet with correct GPU, status=online |
| 3.7 | **Dashboard detail view** | Click device | Hardware profile matches: i9-14900HX, 32 cores, 64GB RAM, RTX 4090 16GB |
| 3.8 | Phase 5: Run | Heartbeat + watcher | Status remains "online" over 60s, heartbeat visible in API |

### Stage 4: Real Inference — Full Round Trip (SSH + curl + Frontend)

Starting state: llmhosts up and running on Alienware, device registered.

| # | Test | Method | Expected |
|---|------|--------|----------|
| 4.1 | Non-streaming inference | `POST /v1/chat/completions` via proxy | Valid response, model name in header |
| 4.2 | Streaming inference | `POST /v1/chat/completions` stream:true | SSE chunks arrive, final chunk has usage |
| 4.3 | OpenAI SDK compatibility | Python `openai.OpenAI(base_url="http://100.64.0.2:4000/v1")` | Standard SDK works unchanged |
| 4.4 | Response headers | Check headers | X-Request-ID, X-LLMHosts-Model, X-LLMHosts-Cost present |
| 4.5 | Multiple models | Pull 2nd model, request each | Both route correctly |
| 4.6 | Concurrent requests | 5 parallel requests | All complete, no crashes |
| 4.7 | Cost tracking in frontend | Check dashboard after requests | Request count + cost savings updated |
| 4.8 | Audit log | Check /audit endpoint or frontend | Requests appear in audit trail |
| 4.9 | Cache hit | Repeat exact same prompt | 2nd request faster, cache stats show hit |
| 4.10 | 20-request benchmark | Sequential timing | Capture p50/p95/p99 latency, tokens/sec |

### Stage 5: Error Conditions & Edge Cases (SSH)

Each test sets up a broken state, verifies graceful handling, then cleans up.

| # | Test | Setup | Expected |
|---|------|-------|----------|
| 5.1 | Port conflict | Bind port 4000 before serve | Clear error message, doesn't crash |
| 5.2 | Corrupt config | Write invalid TOML to ~/.llmhosts/config.toml | Falls back to defaults, warns user |
| 5.3 | No engine, no keys | Remove Ollama, remove keys | Starts in degraded mode with warning |
| 5.4 | Bad model name | `llmhosts pull nonexistent-xyz` | Clear error, no residue files |
| 5.5 | Kill Ollama mid-request | Send request, kill Ollama during | Proxy returns error, doesn't crash, recovers |
| 5.6 | Stop agent | Ctrl+C the agent | Dashboard shows offline within 60s |
| 5.7 | Restart agent | Stop then start again | Dashboard shows online again, no duplicate device |

### Stage 6: Uninstall & Residue Verification (SSH)

The MOST important stage. Validates we left no trace.

| # | Test | Command | Expected |
|---|------|---------|----------|
| 6.1 | Uninstall llmhosts | `pip uninstall llmhosts -y` | Package removed |
| 6.2 | Uninstall Ollama | Platform-specific removal | Ollama removed |
| 6.3 | Remove config | `rm -rf ~/.llmhosts/` | Directory gone |
| 6.4 | Remove Ollama data | `rm -rf ~/.ollama/` | Models + data gone |
| 6.5 | Kill processes | Kill anything on port 4000, stop Ollama service | No orphans |
| 6.6 | **Full residue scan** | Compare laptop state to pre-battery snapshot | ZERO differences |
| 6.7 | Dashboard reflects removal | Check frontend | Device shows offline/removed |

---

## Execution Model

### Test Harness
A Python script (`tests/e2e/full_stack_battery.py`) that:
- Connects to Alienware via SSH (asyncssh, mesh IP 100.64.0.2)
- Connects to LLMHosts.com frontend via Playwright (browser automation)
- Connects to LLMHosts.com API via httpx (direct API calls)
- Manages state snapshots and cleanup
- Runs tests in order, captures results
- Generates JSON + markdown report

### How It Runs
```bash
# From VPS
python tests/e2e/full_stack_battery.py --device alienware --stage all

# Run specific stage
python tests/e2e/full_stack_battery.py --device alienware --stage 4

# Continue after failure (for debugging)
python tests/e2e/full_stack_battery.py --device alienware --stage all --continue-on-failure
```

### Results
- JSON report with every test: pass/fail, duration, measurements
- Markdown summary postable to GitHub issue
- Screenshots from Playwright for visual tests
- Laptop state diff (should be empty after cleanup)

---

## Future Hardware Targets

Once Alienware passes all stages:

| Device | Customer Profile | What It Tests |
|--------|-----------------|---------------|
| **Mac Mini M2** | macOS + Apple Silicon | Metal GPU detection, brew install path, unified memory |
| **NVIDIA Jetson/Spark** | Small form factor | Low VRAM model selection, ARM architecture |
| **Multi-GPU Server** | Power user / operator | Multi-GPU VRAM aggregation, model sharding, operator marketplace |
| **Raspberry Pi 5** | Edge / hobby | CPU-only fallback, minimal resource footprint |
| **Cloud VM (no GPU)** | Cloud-only user | BYOK routing, no local engine, pure cloud proxy |

Each device gets the same battery. Differences are captured in the device profile and assertions adjust accordingly.

---

## Fix-as-You-Go Protocol

When a test fails:
1. Log the failure with full context (command, output, expected vs actual)
2. Diagnose the root cause
3. Fix it in the codebase
4. Re-run ONLY the failed test to confirm the fix
5. Re-run the full stage to confirm no regressions
6. Continue to next stage

This is iterative refinement, not a pass/fail gate.
