# Test Strategy - LLMHosts.com

## META
- **created**: 2026-02-14
- **references**: PRD, implementation-blueprint.md, openapi.yaml, system-architecture.md
- **status**: Active

## 1. PHILOSOPHY

### Test-Driven Development (TDD)
Write tests first for every feature. Tests define the behavior before implementation begins.

### Living Spec Tests
Tests ARE the specification - if a test passes, the feature works to spec. Tests serve as executable documentation that never goes stale.

### Reusable Battery
Tests double as ongoing health/audit system, not just dev-time gates. The same test suite validates correctness during development, CI/CD, and post-deployment monitoring.

### Three Test Audiences
1. **Developer** (during build): Fast feedback loop, fail-fast unit tests
2. **CI** (on every PR): Comprehensive suite excluding slow performance tests
3. **Operator** (post-deploy monitoring): Health checks, metrics, debug hooks

### Inspiration
- **TestForge iterative refinement**: Tests evolve with understanding
- **SWE-Gym executable specs**: Tests as formal specification
- **Nexus oracle synthesis**: Tests generate expected behavior

## 2. TEST PYRAMID

### Layer 1 - Unit Tests (70%)
- **Tool**: pytest + pytest-asyncio
- **Scope**: Every function, class, module in isolation
- **Speed**: <1s per test, <30s total suite
- **Examples**:
  - Router rule matching
  - Cache key generation
  - PII regex patterns
  - Cost calculator
  - Model discovery parser
- **Mock**: All external APIs (Ollama, OpenAI, Anthropic) mocked via pytest-httpx or respx
- **Coverage target**: 90%+ line coverage

### Layer 2 - Integration Tests (20%)
- **Tool**: pytest with Docker fixtures (testcontainers-python)
- **Scope**: Component interactions (router+cache, proxy+backend, CLI+server)
- **Speed**: <5min total
- **Examples**:
  - Full request through proxy to mocked backend
  - Cache hit/miss flow
  - BYOK key encryption/retrieval
  - Router tier escalation (rule->knn->bert)
- **Docker**: testcontainers for SQLite/PostgreSQL, mock API servers
- **Key tests**:
  - OpenAI format request -> response with LLMHosts extensions
  - Anthropic format request -> response
  - Cache invalidation on model change
  - Tunnel connection lifecycle

### Layer 3 - E2E Tests (10%)
- **Tool**: pytest + httpx for API, Playwright for dashboard
- **Scope**: Full system running in Docker
- **Speed**: <10min total
- **Examples**:
  - `pip install -> llmhosts serve -> first request -> dashboard shows stats`
  - `tunnel start -> remote request -> tunnel stop`
  - Upgrade flow (free -> pro feature gate)
- **Docker compose**: Full stack with Ollama simulator, LLMHosts proxy, dashboard

## 3. SPEC VALIDATION TESTS (Living Specification)

These tests validate the product works to PRD specification. They're derived directly from PRD acceptance criteria.

For each PRD feature (F-001 through F-025), create a test file: `tests/spec/test_f001_proxy.py` etc.

### Example Mappings

- **F-001 OpenAI API Proxy**: Test `POST /v1/chat/completions` returns valid OpenAI response + llmhost extension fields
- **F-003 Ollama Auto-Discovery**: Test that Ollama models are detected within 5s of server start
- **F-007 vCache**: Test same query returns cache hit on second call, FP rate < 1%
- **F-009 Dashboard**: Test `/dashboard` returns 200, WebSocket connects, stats update in real-time
- **F-013 Tunnel**: Test tunnel start creates accessible URL, remote request reaches local proxy
- **F-014 kNN Router**: Test embedding + FAISS lookup returns valid routing decision in <15ms
- **F-023 vLLM Integration**: Test vLLM backend registers and serves inference requests
- **F-024 RBAC**: Test team roles enforce correct permissions on all endpoints
- **F-025 npx Scaffolder**: Test npx create-llmhosts generates valid project structure

### Structure
Each spec test file should:
1. Import PRD acceptance criteria as docstrings
2. Test happy path
3. Test edge cases
4. Test error conditions
5. Assert performance requirements (if applicable)

## 4. CONTRACT TESTS

Validate API contracts match `openapi.yaml` spec.

- **Tool**: schemathesis (property-based testing from OpenAPI spec)
- **Run**: `schemathesis run openapi.yaml --base-url http://localhost:4000`
- **Validates**:
  - All endpoints return correct status codes
  - Response shapes match schema
  - Required fields present
  - Request validation (400 on invalid input)
- **Also**: Snapshot test for `openapi.yaml` - any breaking change detected

### Contract Test Structure
```python
# tests/contract/test_openapi_contract.py
import schemathesis

schema = schemathesis.from_file("openapi.yaml")

@schema.parametrize()
def test_api_contract(case):
    response = case.call(base_url="http://localhost:4000")
    case.validate_response(response)
```

## 5. PERFORMANCE TESTS

- **Tool**: locust or k6
- **Benchmarks** (from PRD non-functional requirements):
  - Cold start proxy: <2s
  - Router decision (rule tier): <5ms
  - Router decision (knn tier): <50ms
  - Cache lookup (exact): <10ms
  - Full request roundtrip (cached): <100ms
  - Dashboard WebSocket latency: <500ms
- **Run**: On every release candidate, not every PR

### Performance Test Structure
```python
# tests/performance/locustfile.py
from locust import HttpUser, task, between

class LLMHostsUser(HttpUser):
    wait_time = between(1, 3)
    
    @task
    def chat_completion(self):
        self.client.post("/v1/chat/completions", json={
            "model": "test-model",
            "messages": [{"role": "user", "content": "test"}]
        })
```

## 6. SECURITY TESTS

- **Tool**: bandit (Python SAST), safety (dependency vulnerabilities)
- **PII Detection tests**: Verify regex catches SSN, email, API keys, credit cards
- **BYOK encryption tests**: Verify keys encrypted at rest, never logged, never sent to analytics
- **Rate limiting tests**: Verify 429 responses under load

### Security Test Examples
```python
# tests/security/test_pii_detection.py
def test_ssn_detection():
    assert detect_pii("SSN: 123-45-6789") == True

def test_api_key_detection():
    assert detect_pii("sk-1234567890abcdef") == True

# tests/security/test_key_encryption.py
def test_keys_encrypted_at_rest():
    key = "test-key"
    encrypted = encrypt_key(key)
    assert encrypted != key
    assert decrypt_key(encrypted) == key

def test_keys_not_logged():
    # Verify no key appears in logs
    pass
```

## 7. TEST INFRASTRUCTURE

### Directory Structure
```
tests/
  conftest.py           # shared fixtures
  unit/
    test_router.py
    test_cache.py
    test_discovery.py
    test_cost.py
    test_pii.py
    test_config.py
  integration/
    test_proxy_flow.py
    test_cache_flow.py
    test_router_escalation.py
    test_key_management.py
    conftest.py          # Docker fixtures
  e2e/
    test_first_run.py
    test_dashboard.py
    test_tunnel.py
    conftest.py          # Docker compose fixtures
  spec/
    test_f001_discovery.py
    test_f003_openai_format.py
    test_f007_vcache.py
    test_f009_dashboard.py
    test_f014_tunnel.py
    ...per PRD feature
  contract/
    test_openapi_contract.py
  performance/
    locustfile.py
  security/
    test_pii_detection.py
    test_key_encryption.py
```

### Fixtures

#### Shared Fixtures (`tests/conftest.py`)
- `mock_ollama`: Fake Ollama server returning canned responses
- `mock_openai`: Fake OpenAI API
- `mock_anthropic`: Fake Anthropic API
- `llmhost_server`: Running LLMHosts proxy instance
- `fresh_db`: Clean SQLite database
- `seeded_db`: Database with sample data
- `sample_requests`: Collection of test prompts covering routing tiers

#### Integration Fixtures (`tests/integration/conftest.py`)
- `docker_postgres`: PostgreSQL container via testcontainers
- `docker_sqlite`: SQLite file in temp directory
- `mock_backend`: Mock LLM backend server

#### E2E Fixtures (`tests/e2e/conftest.py`)
- `docker_compose`: Full stack Docker Compose instance
- `ollama_simulator`: Fake Ollama for testing
- `dashboard_browser`: Playwright browser instance

### CI Configuration

```yaml
# pytest.ini or pyproject.toml
[pytest]
markers =
    unit: Unit tests (fast, no deps)
    integration: Integration tests (Docker required)
    e2e: End-to-end tests (full stack required)
    spec: Specification validation tests
    contract: API contract tests
    performance: Performance benchmarks
    security: Security tests
    slow: Tests that take >10s

testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*

# Coverage
[coverage:run]
source = llmhost
omit = 
    */tests/*
    */__pycache__/*

[coverage:report]
precision = 2
show_missing = True
skip_covered = False
```

### GitHub Actions Example
```yaml
# .github/workflows/test.yml
name: Tests

on: [push, pull_request]

jobs:
  unit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.12'
      - run: pip install -e ".[test]"
      - run: pytest tests/unit/ -v --cov=llmhost --cov-report=xml
  
  integration:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_PASSWORD: test
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
      - run: pip install -e ".[test]"
      - run: pytest tests/integration/ -v
  
  contract:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: pip install schemathesis
      - run: schemathesis run openapi.yaml --base-url http://localhost:4000
```

## 8. AUDIT AND MONITORING HOOKS

### Post-Deploy Health Check

**Command**: `llmhosts doctor --full`

Runs:
- Unit tests (fast subset)
- Health endpoint check
- Cache connectivity
- Backend discovery
- Tunnel reachability

**Exit codes**:
- `0`: Healthy
- `1`: General error
- `2`: Cache unavailable
- `3`: Backend discovery failed
- `4`: Tunnel unreachable

**Machine-readable output**: `llmhosts doctor --format json`
```json
{
  "status": "healthy",
  "checks": {
    "unit_tests": {"status": "pass", "duration_ms": 250},
    "health_endpoint": {"status": "pass", "response_time_ms": 12},
    "cache": {"status": "pass", "hit_rate": 0.85},
    "backend_discovery": {"status": "pass", "models_found": 3},
    "tunnel": {"status": "pass", "url": "https://tunnel.llmhosts.com/abc123"}
  },
  "timestamp": "2026-02-14T10:30:00Z"
}
```

### Continuous Monitoring

**Metrics endpoint**: `/metrics` (Prometheus format)

Exposes:
- `request_count` (counter)
- `error_rate` (gauge)
- `cache_hit_rate` (gauge)
- `router_distribution` (histogram by tier)
- `latency_p50`, `latency_p95`, `latency_p99` (histogram)

**Alert thresholds** (defined in config):
- `error_rate > 5%`
- `cache_fp_rate > 1%`
- `latency_p99 > 2s`

### Debug Hooks

**Request replay**: `llmhosts debug request <id>`
- Replay a request with verbose logging
- Output: Structured JSON with routing decisions, cache hits/misses, backend calls

**Router debug**: `llmhosts debug router <query>`
- Show routing decision tree
- Output: Which tier matched, confidence scores, fallback path

**Cache debug**: `llmhosts debug cache <query>`
- Show cache lookup path and result
- Output: Hash key, namespace, similarity scores, hit/miss reason

**All debug output**: Structured JSON for programmatic consumption

### Restore/Rollback

**Config versioning**: `llmhosts config rollback <version>`
- Configs versioned in SQLite
- List versions: `llmhosts config versions`
- View diff: `llmhosts config diff <v1> <v2>`

**Cache management**:
- `llmhosts cache clear --namespace <ns>`: Clear specific namespace
- `llmhosts cache clear --all`: Clear entire cache
- `llmhosts cache stats`: Show hit rates, sizes, namespaces

**Database migration**: `llmhosts db migrate --to <version>`
- Rollback: `llmhosts db migrate --to <previous_version>`
- Status: `llmhosts db status`

## 9. TEST-DRIVEN BUILD ORDER

When building features, the implementation blueprint step order becomes:

1. **Write spec test** (from PRD acceptance criteria) - **RED**
2. **Write unit tests** for the component - **RED**
3. **Implement minimal code** to pass unit tests - **GREEN**
4. **Write integration test** - **RED**
5. **Implement integration glue** - **GREEN**
6. **Refactor** - **REFACTOR**
7. **Verify spec test passes** - **GREEN**
8. **Commit**

### Example Flow

**Feature**: F-007 vCache verified semantic caching

1. Write `tests/spec/test_f007_vcache.py`:
   ```python
   def test_cache_hit_on_identical_query():
       # RED
       pass
   ```

2. Write `tests/unit/test_cache.py`:
   ```python
   def test_cache_key_generation():
       # RED
       pass
   ```

3. Implement `llmhosts/cache/vcache.py`:
   ```python
   def generate_cache_key(query):
       # GREEN
       return hash(query)
   ```

4. Write `tests/integration/test_cache_flow.py`:
   ```python
   def test_cache_hit_miss_flow():
       # RED
       pass
   ```

5. Implement cache integration in proxy:
   ```python
   # GREEN
   ```

6. Refactor for performance/clarity

7. Verify spec test passes

8. Commit with message: "feat: implement vCache (F-007)"

## 10. REUSABLE TEST ECOSYSTEM

Tests are designed to be run at any point in the product lifecycle:

### During Development
```bash
pytest tests/unit/ -x  # fail fast
```

### Before PR Merge
```bash
pytest tests/ -m "not performance and not slow"
```

### Before Release
```bash
pytest tests/  # full suite including perf
```

### Post-Deploy
```bash
llmhosts doctor --full
```

### Ongoing Monitoring
- `/metrics` endpoint + alerting
- Prometheus scraping + Grafana dashboards

### Regression After Pivot
```bash
pytest tests/spec/  # do all PRD features still work?
```

### After Dependency Update
```bash
pytest tests/contract/  # do APIs still match spec?
```

### Security Audit
```bash
bandit -r llmhost/
safety check
pytest tests/security/
```

## 11. TEST DATA MANAGEMENT

### Test Fixtures Location
- `tests/fixtures/`: Static test data (JSON responses, sample prompts)
- `tests/fixtures/models/`: Model response samples
- `tests/fixtures/requests/`: Sample API requests

### Test Data Principles
- **Deterministic**: Same input always produces same output
- **Minimal**: Smallest dataset that exercises the feature
- **Realistic**: Based on real-world usage patterns
- **Versioned**: Test data changes tracked in git

### Mock Data
- Ollama responses: `tests/fixtures/ollama_responses/`
- OpenAI responses: `tests/fixtures/openai_responses/`
- Anthropic responses: `tests/fixtures/anthropic_responses/`

## 12. TESTING BEST PRACTICES

### Naming Conventions
- Test files: `test_<module>.py`
- Test functions: `test_<feature>_<scenario>()`
- Test classes: `Test<Feature>`

### Test Organization
- One test per behavior
- Arrange-Act-Assert pattern
- Descriptive test names (can be long)
- Group related tests in classes

### Assertions
- Use specific assertions: `assert response.status_code == 200` not `assert response.ok`
- Include context in failure messages: `assert cache_hit, f"Cache miss for query: {query}"`
- Test both positive and negative cases

### Async Testing
- Use `pytest-asyncio` for async functions
- Mark async tests: `@pytest.mark.asyncio`
- Use `asyncio.create_task()` for concurrent operations

### Test Isolation
- Each test is independent
- No shared state between tests
- Use fixtures for setup/teardown
- Clean up resources (files, DB records, containers)

## 13. CONTINUOUS IMPROVEMENT

### Test Metrics
- Track test execution time
- Monitor flaky tests
- Measure coverage trends
- Track test-to-code ratio

### Regular Reviews
- Monthly test suite review
- Remove obsolete tests
- Consolidate duplicate tests
- Update tests when requirements change

### Test Documentation
- Each test file has a docstring explaining what it tests
- Complex test logic is commented
- Test fixtures are documented

---

**Last Updated**: 2026-02-14
**Next Review**: 2026-03-14
