# Day 3: North Star — Success Criteria & KPIs

**Date:** 2026-03-10
**Retreat:** 7-Day Advisory Council Strategic Retreat
**Session Lead:** Marcus Okonkwo (GTM), Fatima Al-Rashid (BD), Amara Nwosu (DevRel)
**Purpose:** Define every measurable success criterion. The checkpoints we cannot ship without.

---

## Framework

For each KPI:
- **Target**: The number we're aiming for
- **Threshold**: The minimum acceptable — below this we have a problem
- **Baseline**: Where we are today (March 2026)
- **Measurement**: How we measure it

---

## Section 1: Technical KPIs — Inference Performance

### 1.1 Time to First Token (TTFT)

| Hardware | Model | Target | Threshold | Baseline | Measurement |
|----------|-------|--------|-----------|----------|-------------|
| RTX 4090 | 7B Q5_K_M | <80ms | <150ms | ~290ms | Automated benchmark every commit |
| RTX 4090 | 70B Q4_K_M | <200ms | <400ms | ~600ms | Automated benchmark every commit |
| M4 Max | 7B Q5_K_M | <100ms | <200ms | ~350ms | Automated benchmark |
| Via relay | 7B Q5_K_M | <230ms | <500ms | N/A (M2 not deployed) | Relay latency test |
| Via marketplace | 7B Q5_K_M | <350ms | <700ms | N/A | Marketplace latency test |

**Why these targets:** At >500ms TTFT, users notice the delay as "lag." At <100ms, inference feels instant. Our competitive claim is "faster than cloud" — GPT-4o TTFT averages 400-800ms.

### 1.2 Throughput (tokens/second)

| Hardware | Model | Target | Threshold | Baseline | Measurement |
|----------|-------|--------|-----------|----------|-------------|
| RTX 4090 | 7B Q5_K_M | >180 tok/s | >120 tok/s | ~100 tok/s (Ollama baseline) | llmhosts_bench suite |
| RTX 4090 | 70B Q4_K_M | >50 tok/s | >30 tok/s | ~25 tok/s (Ollama) | llmhosts_bench suite |
| RTX 4090 | 70B Q4_K_M concurrent (4 req) | >120 tok/s total | >80 tok/s | N/A (serial today) | Concurrent benchmark |
| M4 Max | 70B Q4_K_M | >25 tok/s | >18 tok/s | ~20 tok/s (Ollama Metal) | Benchmark |
| CPU only | 7B Q4_K_M | >8 tok/s | >4 tok/s | ~5 tok/s | CPU benchmark |

**Competitive context:** Ollama is the baseline. Our 2× claim requires: RTX 4090 + 70B: >50 vs Ollama's ~25. This requires PagedAttention + continuous batching — not yet implemented.

### 1.3 Cache Performance

| Metric | Target | Threshold | Baseline | Measurement |
|--------|--------|-----------|----------|-------------|
| Exact cache TTFT | <1ms | <5ms | ~0.2ms | Unit test, p99 |
| Semantic cache TTFT | <30ms | <80ms | ~50ms (ONNX lookup) | Integration test |
| Semantic cache hit rate (repeat queries) | >40% | >20% | ~25% (estimated) | Production telemetry |
| Cache false positive rate | <0.1% | <0.5% | Unknown | Manual audit |
| Cache size limit enforcement | 100% | 100% | Not implemented | Automated test |

### 1.4 Router Performance

| Metric | Target | Threshold | Baseline | Measurement |
|--------|--------|-----------|----------|-------------|
| Tier 1 kNN latency | <15ms CPU | <30ms | Not benchmarked | Automated test |
| Tier 2 ModernBERT latency | <50ms | <100ms | Not deployed | Integration test |
| Tier 3 Qwen latency | <500ms | <1000ms | Not deployed | Integration test |
| Router accuracy vs oracle | >90% | >80% | Not measured | Labeled test set |
| Privacy tier classification accuracy | >99% | >97% | Not measured | Red team test |
| Cloud spillover rate (when local available) | <5% | <10% | ~30% (no smart routing) | Production log |

### 1.5 Relay Performance

| Metric | Target | Threshold | Baseline | Measurement |
|--------|--------|-----------|----------|-------------|
| Relay connection establishment | <2s | <5s | N/A (M2 not deployed) | E2E test |
| Relay added latency (overhead vs direct) | <50ms | <150ms | N/A | Relay latency test |
| Relay reconnect after drop | <5s | <15s | N/A | Chaos test |
| Relay uptime | >99.5% | >99% | N/A | Synthetic monitoring |
| Relay encryption throughput | >500 MB/s per connection | >100 MB/s | N/A | Throughput test |

### 1.6 System Reliability

| Metric | Target | Threshold | Baseline | Measurement |
|--------|--------|-----------|----------|-------------|
| Request error rate | <0.5% | <1% | Unknown | Prometheus/Grafana |
| p99 latency (local, 7B) | <500ms | <1000ms | Unknown | Prometheus histogram |
| MTTR (mean time to recovery) | <60s | <180s | Unknown | Chaos testing |
| Model OOM recovery time | <30s | <120s | Unknown | Chaos test |
| Memory leak over 24h | Zero | <500MB drift | Unknown | Long-running test |

---

## Section 2: CI/CD KPIs

| Metric | Target | Threshold | Baseline | Measurement |
|--------|--------|-----------|----------|-------------|
| Python test coverage | >85% | >75% | ~82% (estimated) | pytest --cov |
| Rust test coverage | >80% | >70% | ~60% (estimated) | cargo tarpaulin |
| Frontend test coverage | >80% | >70% | ~72% | Vitest --coverage |
| CI build time (no cache) | <8 min | <15 min | ~12 min | GitHub Actions |
| CI build time (cached) | <3 min | <6 min | ~6 min | GitHub Actions |
| Clippy warnings | 0 | 0 | 0 (achieved) | CI gate |
| Type errors (mypy) | 0 | 0 | 0 (achieved) | CI gate |
| Benchmark regression gate | <5% degradation | <10% | Not implemented | Per-commit bench |

---

## Section 3: Product KPIs — User Experience

### 3.1 Install & Onboarding

| Metric | Target | Threshold | Baseline | Measurement |
|--------|--------|-----------|----------|-------------|
| Install-to-first-inference time (model pre-downloaded) | <5 min | <10 min | ~8 min (Anya journey) | E2E timer |
| Install-to-first-inference time (first-time download, 7B) | <12 min | <20 min | ~12 min | E2E timer |
| `llmhosts up` zero-config success rate (NVIDIA+Win) | >95% | >85% | ~80% (estimated) | Hardware E2E |
| `llmhosts up` zero-config success rate (Mac M-series) | >95% | >85% | ~75% | Hardware E2E |
| `llmhosts doctor` all-green rate on first run | >80% | >65% | ~60% | Hardware E2E |
| Onboarding completion (install → first request) | >75% | >55% | Unknown | Analytics funnel |

### 3.2 Retention

| Metric | Target | Threshold | Baseline | Measurement |
|--------|--------|-----------|----------|-------------|
| 7-day retention (any user type) | >60% | >45% | Unknown | Cohort analysis |
| 30-day retention (single-device users) | >45% | >30% | Unknown | Cohort analysis |
| 90-day retention (single-device users) | >35% | >20% | Unknown | Cohort analysis |
| 30-day retention (marketplace operators) | >65% | >50% | Unknown | Cohort analysis |
| 90-day retention (marketplace operators) | >55% | >40% | Unknown | Cohort analysis |
| Daily active usage (requests/active-user/day) | >50 requests | >20 requests | Unknown | Request logs |

### 3.3 Error Experience

| Metric | Target | Threshold | Baseline | Measurement |
|--------|--------|-----------|----------|-------------|
| API error rate (all users) | <1% | <2% | Unknown | Prometheus |
| "Unexplained errors" (no user-visible reason) | 0% | <0.5% | Unknown | Support tickets |
| Model OOM events per day | <1 per 1000 users | <5 per 1000 | Unknown | Error telemetry |
| Relay disconnect events per day | <1 per 100 users | <5 per 100 | Unknown | Relay telemetry |

### 3.4 Net Promoter Score

| Cohort | Target NPS | Threshold | Baseline | Measurement |
|--------|-----------|-----------|----------|-------------|
| Onboarding cohort (first 30 days) | >60 | >40 | Unknown | In-app survey |
| Active users (30-90 days) | >55 | >35 | Unknown | Monthly survey |
| Marketplace operators | >65 | >45 | Unknown | Monthly survey |
| Enterprise accounts | >70 | >50 | Unknown | Quarterly survey |

---

## Section 4: Business KPIs

### 4.1 User Growth

| Metric | Wave 1 (Q2) | Wave 2 (Q3) | Wave 4 (Q4) | Measurement |
|--------|-------------|-------------|-------------|-------------|
| Total registered users | 1,000 | 5,000 | 25,000 | Auth DB |
| Monthly Active Users (MAU) | 400 | 2,000 | 12,000 | Request logs |
| New user registrations/week | 50 | 150 | 500 | Auth DB |
| Organic (non-paid) acquisition rate | >70% | >60% | >55% | UTM tracking |

### 4.2 Marketplace Economics

| Metric | Q3 2026 Target | Q4 2026 Target | Q2 2027 Target | Measurement |
|--------|----------------|----------------|----------------|-------------|
| Active marketplace operators | 100 | 500 | 2,000 | Marketplace DB |
| Marketplace GMV/month | $5,000 | $50,000 | $400,000 | Stripe |
| Average operator revenue/month | $30 | $80 | $150 | Stripe / operator count |
| Operator churn rate/month | <20% | <12% | <8% | Cohort analysis |
| Buyer satisfaction (SLA compliance) | >90% | >95% | >99% | Request success logs |
| Path to $1M GMV/month: | 10K operators × $80 avg = $800K. Requires Q2 2027 operator count. Achievable. | | | |

### 4.3 Revenue

| Metric | Q2 2026 | Q3 2026 | Q4 2026 | Q2 2027 | Measurement |
|--------|---------|---------|---------|---------|-------------|
| MRR (all tiers) | $2,000 | $15,000 | $75,000 | $300,000 | Stripe |
| Pro tier subscribers ($19/mo) | 100 | 700 | 3,500 | 12,000 | Stripe |
| Enterprise accounts | 0 | 2 | 10 | 30 | CRM |
| Platform fee from marketplace | $1,000 | $10,000 | $80,000 | $250,000 | Stripe (20% of GMV) |
| ARPU (Pro) | $19 | $19 | $22 | $28 | Stripe / active users |

### 4.4 Unit Economics

| Metric | Target | Threshold | Measurement |
|--------|--------|-----------|-------------|
| CAC (Customer Acquisition Cost) | <$5 | <$15 | Ad spend / new paying users |
| LTV (Pro, 12-month) | >$150 | >$100 | Cohort revenue |
| LTV/CAC ratio | >30 | >10 | Calculated |
| Gross margin (platform fee) | >85% | >70% | Revenue - relay COGS |
| Relay COGS per active user | <$0.50/mo | <$1.50/mo | Infrastructure costs |

### 4.5 Enterprise Sales

| Metric | Q3 2026 | Q4 2026 | Q2 2027 | Measurement |
|--------|---------|---------|---------|-------------|
| Enterprise pipeline value | $50K | $200K | $800K | CRM |
| Average contract value (ACV) | $12K | $18K | $24K | Stripe |
| Sales cycle length | 60 days | 45 days | 30 days | CRM |
| Enterprise features complete | 60% | 85% | 100% | GitHub milestones |

---

## Section 5: Community KPIs

### 5.1 GitHub

| Metric | Q2 2026 | Q3 2026 | Q4 2026 | Q2 2027 | Measurement |
|--------|---------|---------|---------|---------|-------------|
| GitHub stars (public repo) | 500 | 2,000 | 8,000 | 25,000 | GitHub API |
| Forks | 50 | 200 | 800 | 2,500 | GitHub API |
| External contributors | 5 | 20 | 75 | 200 | GitHub contributors |
| Issues from community (bug reports, features) | 10/mo | 40/mo | 120/mo | 300/mo | GitHub Issues |
| PRs from community | 2/mo | 8/mo | 25/mo | 60/mo | GitHub PRs |

**Stars milestones and triggers:**
- 100 stars: tweet thread from founder, post on r/LocalLLaMA
- 500 stars: Show HN post
- 1,000 stars: Product Hunt launch
- 5,000 stars: Benchmark blog post (honest comparison vs Ollama, vLLM)
- 10,000 stars: "State of LLMHosts" annual report
- 50,000 stars: Series A announcement, opens enterprise GTM

### 5.2 Community

| Metric | Q2 2026 | Q3 2026 | Q4 2026 | Measurement |
|--------|---------|---------|---------|-------------|
| Discord members | 200 | 1,000 | 5,000 | Discord |
| Discord daily active | 30 | 150 | 800 | Discord |
| Reddit r/LLMHosts members | 100 | 500 | 2,000 | Reddit |
| Twitter/X followers | 500 | 2,000 | 8,000 | Twitter |
| YouTube subscribers | 50 | 500 | 3,000 | YouTube |
| Newsletter subscribers | 200 | 1,000 | 5,000 | Email provider |

### 5.3 Contributor Ladder

| Level | Criteria | Benefit |
|-------|----------|---------|
| User | Uses LLMHosts | — |
| Community Contributor | 1+ comment/issue in GitHub Discussions | Discord role |
| Code Contributor | 1+ merged PR | Listed in CONTRIBUTORS.md |
| Recurring Contributor | 5+ merged PRs | Invited to #contributors Discord channel |
| Maintainer | Core team nomination | Write access, paid if revenue allows |
| Advisory Council | Strategic input | Listed in AGENTS.md and advisory docs |

---

## Section 6: Quality Gate Criteria (ship blockers)

These are binary gates that block release. If any is not green, we do not ship.

### Wave 1 Quality Gates
- [ ] `llmhosts up` completes without error on Windows 11 + RTX 4090 (E2E test)
- [ ] `llmhosts up` completes without error on macOS M-series (E2E test)
- [ ] All API endpoints return correct responses (contract tests)
- [ ] Relay M1 connects and forwards requests end-to-end (E2E test)
- [ ] No regression in any existing pytest suite (CI gate)
- [ ] No clippy warnings, no mypy errors (CI gate)
- [ ] `llmhosts doctor` returns actionable output on first run

### Wave 2 Quality Gates (add to above)
- [ ] Concurrent inference: 4 simultaneous requests all complete without serialization timeout
- [ ] Relay M2: all relay traffic encrypted, verified with Wireshark trace
- [ ] PagedAttention: 2× throughput improvement vs baseline on RTX 4090 (benchmark regression)
- [ ] TTFT <150ms for 7B model on RTX 4090 (regression test)

### Wave 3 Quality Gates (add to above)
- [ ] Collective intelligence opt-in flow tested by 5 external beta users
- [ ] GDPR Data Protection Impact Assessment completed
- [ ] LDP epsilon budget documented and tested
- [ ] HIM returns accurate recommendations for 10 hardware profiles (seeded data)

---

## Day 3 Summary

**The 10 Most Critical KPIs (the council's consensus):**

1. **Install-to-first-inference: <5 minutes** — This is the product promise. Failure here is existential.
2. **TTFT <150ms local (7B, RTX 4090)** — The performance claim must be real.
3. **7-day retention >60%** — If users try and leave, we have a product problem, not a marketing problem.
4. **Relay M2 uptime >99.5%** — The remote access value prop dies if relay is unreliable.
5. **Concurrent inference: 4 req simultaneous on RTX 4090** — Required before any multi-user claim.
6. **Marketplace operator churn <12%/month** — Above this, GMV growth stalls regardless of acquisition.
7. **Router accuracy >90%** — Wrong routing decisions waste money and erode trust.
8. **Cache hit rate >40% on repeat queries** — This is the savings amplifier.
9. **GitHub stars 1,000** — Community signal that the product has found its audience.
10. **NPS >60 from onboarding cohort** — Users who love it immediately become our marketing team.

---

*Advisory Council — Day 3 Session — 2026-03-10*
*Lead: Marcus Okonkwo, Fatima Al-Rashid, Amara Nwosu | All 12 advisors contributed*
