# POC Testing Phase Design

**Date**: 2026-02-22
**Status**: Approved
**Goal**: Full end-to-end validation of LLMHosts.com before soft launch. Claude runs automated tests; human tests real GPU inference, external connectivity, payments, and email.

---

## Signup Lockdown

**Approach**: Email allowlist via `ALLOWED_SIGNUP_EMAILS` env var.
- Only `brendonurban@gmail.com` can register
- All other signups get `403 Signups are currently closed`
- Allowlist is comma-separated, easy to add more testers later
- Implemented in `frontend/src/app/api/auth/register/route.ts`

---

## Test Ownership Split

### Claude (Automated — ~170 tests)
- All public pages load, zero console errors
- Auth flows (register, verify, login, password reset)
- Dashboard pages (all 10+), navigation, empty states
- API endpoints (/v1/models, /health, /docs, relay)
- SEO (sitemap, robots, JSON-LD, OG images, llms.txt, RSS)
- Security headers (CSP, HSTS, X-Frame-Options, etc.)
- Accessibility (WCAG 2.2 AA via axe-core)
- Performance (Lighthouse, Core Web Vitals)
- Visual regression (multi-viewport screenshots)
- Container health and log inspection
- Responsive design (desktop, tablet, mobile viewports)

### Human (Manual — ~50 tests)
- Real GPU inference from external network
- Tunnel connectivity (Tailscale/Cloudflare from outside)
- Stripe payment (test card checkout, plan upgrade, billing portal)
- Email delivery (verification email, password reset email)
- CLI installation (`pip install llmhosts` on Alienware)
- TUI dashboard (Textual in real terminal)
- First-run wizard (`llmhosts setup`)
- Mobile browser (real device touch interactions)
- `llmhosts doctor` diagnostics on real hardware
- Model suggestion accuracy (`llmhosts suggest-models` with 4090)

---

## Phase Breakdown (Critical-to-Quality Order)

### P0: Infrastructure Smoke — Claude (~15 tests)
Verify site is up, all services healthy, zero errors.

| Test | Method | Pass Criteria |
|------|--------|---------------|
| All 26 pages return 200 | Playwright against prod | HTTP 200, no network errors |
| Zero console errors on every page | `page.on('console')` | No error-type messages |
| `/health` endpoint | HTTP GET | 200 + JSON |
| `/v1/models` responds | HTTP GET | 200 + model list |
| `/sitemap.xml` valid | HTTP GET + XML parse | Valid, all URLs resolve |
| `/robots.txt` present | HTTP GET | 200, references sitemap |
| `/llms.txt` present | HTTP GET | 200, correct content |
| `/feeds/articles.xml` RSS | HTTP GET | Valid XML |
| `/og?title=Test` OG image | HTTP GET | 200, image/png |
| Docker containers healthy | `docker compose ps` | Both healthy |
| No container restart loops | `docker inspect` | RestartCount = 0 |
| Container logs clean | `docker logs` | No ERROR/EXCEPTION/Traceback |
| Traefik routing correct | curl /v1/models vs / | Correct backend per route |
| SSL certificate valid | openssl s_client | Valid, not expiring <30d |
| HTTPS redirect works | curl http://llmhosts.com | 301 → https |

### P1: Auth & Security — Claude + Human (~30 tests)

**Claude (automated):**

| Test | Method | Pass Criteria |
|------|--------|---------------|
| Register with allowlisted email | Playwright form fill | 201, account created |
| Register with non-allowlisted email blocked | Playwright | 403 response |
| Email verification link works | Playwright (if Resend configured) | Account verified |
| Login with valid credentials | Playwright | Redirect to dashboard |
| Login with wrong password | Playwright | Error message, no redirect |
| Login with unverified account | Playwright | Blocked with message |
| Forgot password flow | Playwright | Token generated |
| Password reset with valid token | Playwright | Password changed |
| Password reset with expired token | Playwright | Error message |
| Session persistence | Playwright | Dashboard accessible after reload |
| Logout | Playwright | Redirected to login |
| Unauthenticated dashboard access | Playwright | Redirected to login |
| Rate limiting on login | Playwright (11 attempts) | 429 after 10 |
| Rate limiting on register | Playwright (6 attempts) | 429 after 5 |
| Security headers on all responses | Playwright `response.headers()` | CSP, HSTS, X-Frame, X-Content-Type, Referrer, Permissions |
| No server fingerprinting | Response headers | No Server/X-Powered-By |
| CSRF protection | Attempt cross-origin POST | Blocked |
| PII not in page source | Regex scan | No email/name in HTML source |

**Human (manual):**

| Test | What to do |
|------|------------|
| Verification email arrives | Check brendonurban@gmail.com inbox |
| Email content correct | Verify branding, link works |
| Password reset email arrives | Trigger reset, check inbox |
| Reset link works from email | Click link, set new password |

### P2: Dashboard & Features — Claude (~40 tests)

| Test | Method | Pass Criteria |
|------|--------|---------------|
| Dashboard overview loads | Playwright (authenticated) | 200, no errors |
| Spending page loads | Playwright | Table/chart renders |
| API Keys page loads | Playwright | Key list renders |
| Create API key | Playwright form | Key created, shown once |
| API key masked in list | Playwright | Only last 4 chars visible |
| Delete API key | Playwright | Key removed |
| Free plan limit (2 keys) | Create 3rd key | Error: plan limit |
| Cache page loads | Playwright | Stats display |
| Feedback page loads | Playwright | Form renders |
| Submit feedback | Playwright form | 201, confirmation |
| Settings page loads | Playwright | User info displayed |
| Update profile name | Playwright form | Name changed |
| Topology page loads | Playwright | Map/list renders |
| Achievements page loads | Playwright | Badges display |
| Projections page loads | Playwright | Charts render |
| Team page loads | Playwright | Team info or create prompt |
| LAN Nodes page loads | Playwright | Node list or empty state |
| Add device page loads | Playwright | Form renders |
| Sidebar navigation works | Click each nav item | Correct page loads |
| Mobile sidebar toggle | Mobile viewport | Hamburger menu works |
| Empty states display correctly | Fresh account | Helpful messages, CTAs |
| All dashboard pages zero console errors | Console tracking | No errors |
| All dashboard pages screenshot captured | toHaveScreenshot() | Baseline established |

### P3: API Proxy — Claude + Human (~25 tests)

**Claude (automated):**

| Test | Method | Pass Criteria |
|------|--------|---------------|
| GET /v1/models returns model list | curl/fetch | 200 + models array |
| POST /v1/chat/completions with API key | curl | 200 + completion |
| POST /v1/chat/completions without key | curl | 401 |
| POST /v1/chat/completions with invalid key | curl | 401 |
| POST /v1/messages (Anthropic format) | curl | 200 or 404 if not wired |
| Response includes X-Request-ID | curl | Header present |
| Response includes X-LLMHosts-Model | curl | Header present |
| Response includes X-LLMHosts-Cost | curl | Header present |
| /health returns backend status | curl | 200 + status JSON |
| /docs renders OpenAPI UI | Playwright | Swagger/Redoc loads |
| Rate limiting on proxy | Rapid requests | 429 after limit |
| Streaming response works | curl with stream | SSE chunks received |
| Large prompt handled | curl 4k token prompt | Response or proper error |
| Invalid model name | curl | 400/404 with message |
| Malformed JSON body | curl | 422 validation error |

**Human (manual — from Alienware on external network):**

| Test | What to do |
|------|------------|
| Connect Alienware to external network | Phone hotspot or neighbor WiFi |
| Install llmhost CLI | `pip install llmhosts` |
| Run `llmhosts doctor` | Verify GPU detected (4090 18GB) |
| Run `llmhosts serve` | Server starts, TUI shows |
| Run `llmhosts suggest-models` | Reasonable suggestions for 18GB |
| Start tunnel | `llmhosts tunnel start` |
| Send inference request from server | curl from llmhosts.com server → Alienware |
| Verify model response | Correct completion returned |
| Check latency | Response time reasonable |
| Test with different models | Try 2-3 models Ollama serves |

### P4: SEO & Marketing — Claude (~30 tests)

| Test | Method | Pass Criteria |
|------|--------|---------------|
| Landing page meta title | Playwright `page.title()` | Contains "LLMHosts" |
| Landing page meta description | `meta[name=description]` | Present, 120-160 chars |
| Each page has unique title | Crawl all pages | No duplicates |
| Each page has meta description | Crawl all pages | All present |
| JSON-LD on landing page | Parse `script[type=application/ld+json]` | Valid WebSite schema |
| JSON-LD on pricing page | Parse | Valid Product/Offer schema |
| JSON-LD on docs page | Parse | Valid TechArticle schema |
| JSON-LD on contact page | Parse | Valid ContactPage schema |
| OG tags on all public pages | `meta[property^=og:]` | title, description, image, url |
| Twitter card tags | `meta[name^=twitter:]` | card, title, description |
| Canonical URLs present | `link[rel=canonical]` | Present on all pages |
| No broken internal links | Crawl all pages | All internal links resolve |
| Sitemap lists all public pages | Parse sitemap.xml | All 10+ public pages listed |
| Sitemap URLs match canonical | Compare | Consistent |
| Robots.txt allows crawling | Parse | No disallow on public pages |
| /llms.txt content accurate | Compare to site | Matches reality |
| RSS feed valid | XML validation | Well-formed, items present |
| OG image generates correctly | Fetch /og?title=X | Image renders, correct size |
| Lighthouse SEO score | Lighthouse CI | ≥90 |
| Lighthouse Performance score | Lighthouse CI | ≥80 |
| Lighthouse Accessibility score | Lighthouse CI | ≥90 |
| Lighthouse Best Practices | Lighthouse CI | ≥90 |
| Page load time <3s | Playwright timing | LCP <2.5s |
| No render-blocking resources | Lighthouse audit | Pass |
| Images have alt text | axe-core | All images |
| Heading hierarchy correct | Parse h1-h6 | One h1, logical order |
| Mobile-friendly | Google mobile test equivalent | Viewport, tap targets |
| Content API (/api/content/v1/pages) | curl | 200, JSON pages list |

### P5: External Connectivity — Human (~15 tests)

| Test | What to do | Pass Criteria |
|------|------------|---------------|
| Alienware connects to WiFi/hotspot | Join external network | Connected, different IP than home |
| Ollama running on Alienware | `ollama list` | Models available |
| llmhost discovers Ollama | `llmhosts doctor` | Ollama detected |
| Tailscale tunnel establishes | `llmhosts tunnel start` | Tunnel active |
| Server sees Alienware as backend | Check /v1/models from server | Alienware models listed |
| Inference routes to Alienware | POST /v1/chat/completions | Response from GPU model |
| Latency acceptable | Measure response time | <10s for short prompts |
| Streaming works through tunnel | curl --no-buffer | Tokens stream progressively |
| Disconnect/reconnect resilience | Kill tunnel, restart | Reconnects, health recovers |
| Multiple sequential requests | 5 requests in a row | All succeed |
| Concurrent requests | 3 simultaneous | All succeed or queue gracefully |
| Large context request | 2000+ token prompt | Completes without timeout |
| Different model selection | Specify model in request | Correct model used |
| Backend health monitoring | Dashboard shows GPU status | Heartbeat visible |
| Cost tracking | After requests | Cost entries appear in dashboard |

### P6: Billing — Human (~10 tests)

| Test | What to do | Pass Criteria |
|------|------------|---------------|
| Pricing page displays plans | Visit /pricing | All tiers shown, prices correct |
| Free plan limits enforced | Try >2 API keys | Blocked |
| Checkout button works | Click "Upgrade to Pro" | Redirects to Stripe |
| Stripe checkout loads | Complete with test card 4242... | Payment page works |
| Subscription activates | After payment | Plan upgrades to Pro |
| Pro limits applied | Create >2 API keys | Allowed (up to 10) |
| Billing portal accessible | Settings → Manage Billing | Stripe portal loads |
| Cancel subscription | Via Stripe portal | Plan downgrades at period end |
| Webhook processes correctly | After Stripe events | DB updated correctly |
| Invoice/receipt available | Stripe portal | Download works |

### P7: Accessibility & Performance — Claude (~20 tests)

| Test | Method | Pass Criteria |
|------|--------|---------------|
| All public pages WCAG 2.2 AA | axe-core/Playwright | Zero violations |
| Dashboard pages WCAG 2.2 AA | axe-core (authenticated) | Zero violations |
| Keyboard navigation works | Tab through pages | All interactive elements reachable |
| Focus indicators visible | Tab through | Clear focus rings |
| Color contrast passes | axe-core | All text passes 4.5:1 |
| Screen reader landmarks | axe-core | main, nav, header, footer |
| Form labels present | axe-core | All inputs labeled |
| Alt text on images | axe-core | All images have alt |
| Desktop viewport (1920x1080) | Playwright screenshot | Layout correct |
| Tablet viewport (768x1024) | Playwright screenshot | Responsive, readable |
| Mobile viewport (375x812) | Playwright screenshot | Mobile-optimized |
| FCP <1.5s | Lighthouse | Pass |
| LCP <2.5s | Lighthouse | Pass |
| CLS <0.1 | Lighthouse | Pass |
| TBT <300ms | Lighthouse | Pass |
| No memory leaks | Navigate 10 pages, check heap | Stable memory |
| API response time <500ms | Timed requests | /v1/models, /health |
| Static assets cached | Response headers | Cache-Control present |
| Gzip/Brotli compression | Accept-Encoding test | Compressed responses |
| No unused JS >50KB | Lighthouse audit | Pass |

---

## GitHub Sync Plan

### Milestone: "POC Testing Phase"
- Due: TBD (based on testing velocity)
- All phase issues linked to milestone

### Issues (8 total):
1. `test(poc): P0 — Infrastructure smoke` (label: testing, P0-blocker)
2. `test(poc): P1 — Auth & security` (label: testing, P0-blocker)
3. `test(poc): P2 — Dashboard & features` (label: testing, P1-critical)
4. `test(poc): P3 — API proxy` (label: testing, P1-critical)
5. `test(poc): P4 — SEO & marketing` (label: testing, P2-normal)
6. `test(poc): P5 — External connectivity` (label: testing, P0-blocker, human)
7. `test(poc): P6 — Billing` (label: testing, P1-critical, human)
8. `test(poc): P7 — Accessibility & performance` (label: testing, P2-normal)

Plus implementation issues:
9. `feat(auth): Add email allowlist for signup lockdown` (label: enhancement, P0-blocker)
10. `test(e2e): Production Playwright test suite for POC` (label: testing, P0-blocker)

### Projects V2 Board
- Add "POC Testing" view to existing board
- Custom fields: Test Phase (P0-P7), Owner (Claude/Human), Status (Not Started/In Progress/Pass/Fail/Blocked)

---

## Implementation Sequence

1. **Implement signup allowlist** — add `ALLOWED_SIGNUP_EMAILS` env var check to register route
2. **Deploy allowlist** — rebuild frontend container, set env var in production
3. **Seed test account** — register brendonurban@gmail.com through the UI (validates the flow)
4. **Create GitHub milestone + issues** — sync all test cases to GitHub
5. **Run P0** (Claude) — infrastructure smoke
6. **Run P1** (Claude automated + human email verification)
7. **Run P2** (Claude) — dashboard features
8. **Run P3** (Claude automated API tests + human Alienware GPU tests)
9. **Run P4** (Claude) — SEO validation
10. **Run P5** (Human) — external connectivity from Alienware
11. **Run P6** (Human) — Stripe billing
12. **Run P7** (Claude) — accessibility and performance
13. **Fix any failures** — create issues for bugs, fix, re-test
14. **Sign-off** — all phases green → ready for soft launch (#134)

---

## Success Criteria

- **All 220 tests pass** (or known-acceptable exceptions documented)
- **Zero console errors** across all pages
- **Zero container errors** in logs
- **Lighthouse scores**: Performance ≥80, Accessibility ≥90, SEO ≥90, Best Practices ≥90
- **WCAG 2.2 AA compliant** on all pages
- **Real GPU inference works** from external network through tunnel
- **Stripe billing complete cycle** works (upgrade, use, downgrade)
- **Email delivery confirmed** (verification + password reset)
- **Visual regression baselines** established for all pages

When all criteria met → close #134 (soft launch decision) and proceed to production launch.
