# Production Gap Closure Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Close all 10 production gaps to make LLMHosts.com customer-ready for onboarding.

**Architecture:** Extend the existing FastAPI middleware stack with auth, rate limiting, and CORS enforcement, all driven by a new `[security]` config section. Add Alembic for schema migrations, optional Sentry for observability, and complete the Stripe webhook lifecycle. Wire everything together with E2E tests and deployment documentation.

**Tech Stack:** FastAPI middleware, Pydantic v2 config, Alembic (SQLite), Sentry SDK (optional), Stripe webhooks, pytest ASGI transport.

---

## Task 1: Extend Config with SecurityConfig and ObservabilityConfig

**Files:**
- Modify: `llmhosts/config.py`
- Test: `tests/test_config.py`

**Step 1: Add SecurityConfig and ObservabilityConfig models**

Add after `EngineConfig` in `llmhosts/config.py`:

```python
class SecurityConfig(BaseModel):
    """Proxy security settings."""
    require_auth: bool = False
    rate_limit_rpm: int = 60
    rate_limit_burst: int = 10
    cors_restrict: bool = False

class ObservabilityConfig(BaseModel):
    """Observability and error tracking settings."""
    sentry_dsn: str = ""
    log_format: str = "text"  # "text" | "json"
```

Add to `LLMHostsConfig`:
```python
security: SecurityConfig = Field(default_factory=SecurityConfig)
observability: ObservabilityConfig = Field(default_factory=ObservabilityConfig)
```

Add to `_apply_env_overrides` section_models:
```python
"security": SecurityConfig,
"observability": ObservabilityConfig,
```

Update `save_default_config` template with new sections.

**Step 2: Add tests for new config sections**

```python
def test_security_config_defaults():
    cfg = load_config(tmp_path / "nonexistent.toml")
    assert cfg.security.require_auth is False
    assert cfg.security.rate_limit_rpm == 60

def test_security_config_env_override(monkeypatch):
    monkeypatch.setenv("LLMHOST_SECURITY_REQUIREAUTH", "true")
    cfg = load_config(tmp_path / "nonexistent.toml")
    assert cfg.security.require_auth is True
```

**Step 3: Run tests**

```bash
pytest tests/test_config.py -v
```

**Step 4: Commit**

```bash
git add llmhosts/config.py tests/test_config.py
git commit -m "feat(security): add SecurityConfig and ObservabilityConfig sections"
```

---

## Task 2: API Key Authentication Middleware

**Files:**
- Create: `llmhosts/proxy/auth.py`
- Create: `tests/test_auth.py`

**Step 1: Write auth middleware**

`llmhosts/proxy/auth.py`:
- `ProxyKeyStore` class: manages proxy API keys in `~/.llmhosts/proxy_keys.json`
  - `create_key(label) -> (key_id, raw_key)` — generates a random key, stores SHA-256 hash
  - `validate_key(raw_key) -> key_id | None` — checks against stored hashes
  - `list_keys() -> list[KeyInfo]` — returns key metadata (id, label, created, last_used)
  - `revoke_key(key_id) -> bool`
- `AuthMiddleware(BaseHTTPMiddleware)`:
  - Extracts `Authorization: Bearer <key>` header
  - Validates against `ProxyKeyStore`
  - Sets `request.state.authenticated`, `request.state.api_key_id`
  - Exempt paths: `/health`, `/docs`, `/openapi.json`, `/redoc`
  - When `require_auth=False`: all requests pass, state set to anonymous
  - When `require_auth=True`: returns 401 `{"error": {"message": "...", "type": "authentication_error"}}` (OpenAI-compatible error format)

**Step 2: Write tests**

`tests/test_auth.py`:
- `test_auth_disabled_allows_all` — default config, no header, 200
- `test_auth_enabled_rejects_no_header` — require_auth=True, no header, 401
- `test_auth_enabled_rejects_invalid_key` — require_auth=True, bad key, 401
- `test_auth_enabled_accepts_valid_key` — create key, use it, 200
- `test_auth_skips_health` — require_auth=True, GET /health without header, 200
- `test_key_store_create_and_validate` — unit test key lifecycle
- `test_key_store_revoke` — revoke key, then validate fails
- `test_key_store_list` — list shows metadata, not raw keys

**Step 3: Run tests, verify pass**

```bash
pytest tests/test_auth.py -v
```

**Step 4: Commit**

```bash
git add llmhosts/proxy/auth.py tests/test_auth.py
git commit -m "feat(security): add API key authentication middleware and key store"
```

---

## Task 3: Rate Limiting Middleware

**Files:**
- Create: `llmhosts/proxy/ratelimit.py`
- Create: `tests/test_ratelimit.py`

**Step 1: Write rate limiter**

`llmhosts/proxy/ratelimit.py`:
- `TokenBucket` class: per-key in-memory rate limiter
  - `consume() -> (allowed, retry_after_seconds)`
  - Auto-refills at `rate_limit_rpm / 60` tokens per second
  - Burst capacity from config
- `RateLimitMiddleware(BaseHTTPMiddleware)`:
  - Keys by `request.state.api_key_id` (from auth) or client IP if anonymous
  - Adds headers: `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`
  - Returns 429 with `Retry-After` header when exhausted
  - Exempt: `/health`
  - Periodic cleanup of stale buckets (>1hr idle)

**Step 2: Write tests**

`tests/test_ratelimit.py`:
- `test_ratelimit_allows_under_limit` — 1 request, passes
- `test_ratelimit_blocks_over_limit` — burst+1 requests, last gets 429
- `test_ratelimit_includes_headers` — check X-RateLimit-* headers
- `test_ratelimit_retry_after` — 429 includes Retry-After header
- `test_ratelimit_per_key_isolation` — two keys, independent limits
- `test_ratelimit_exempt_health` — /health not rate limited
- `test_token_bucket_refill` — bucket refills over time

**Step 3: Run tests**

```bash
pytest tests/test_ratelimit.py -v
```

**Step 4: Commit**

```bash
git add llmhosts/proxy/ratelimit.py tests/test_ratelimit.py
git commit -m "feat(security): add token-bucket rate limiting middleware"
```

---

## Task 4: Wire Middleware + CORS Hardening into App

**Files:**
- Modify: `llmhosts/proxy/app.py`
- Modify: `llmhosts/server.py`

**Step 1: Update `create_app()` to accept config**

Modify `create_app(config: LLMHostsConfig | None = None)`:
- If config provided, use `config.server.cors_origins` when `config.security.cors_restrict` is True
- Add `AuthMiddleware` before `RequestLoggingMiddleware`
- Add `RateLimitMiddleware` after `AuthMiddleware`
- Store `ProxyKeyStore` on `app.state.key_store`

Middleware order (Starlette processes in reverse add order):
```python
app.add_middleware(RequestLoggingMiddleware)
app.add_middleware(RateLimitMiddleware, config=config.security if config else SecurityConfig())
app.add_middleware(AuthMiddleware, key_store=key_store, require_auth=config.security.require_auth if config else False)
app.add_middleware(RequestIdMiddleware)
```

**Step 2: Update `LLMHostsServer._create_app()`**

Pass config to `create_app(config=self.config)`.

**Step 3: Update CORS block**

```python
cors_origins = ["*"]
if config and config.security.cors_restrict:
    cors_origins = config.server.cors_origins
app.add_middleware(CORSMiddleware, allow_origins=cors_origins, ...)
```

**Step 4: Run all existing tests to verify no regressions**

```bash
pytest tests/ -v
```

**Step 5: Commit**

```bash
git add llmhosts/proxy/app.py llmhosts/server.py
git commit -m "feat(security): wire auth, rate limiting, and CORS into app middleware stack"
```

---

## Task 5: Observability Module

**Files:**
- Create: `llmhosts/observability.py`
- Modify: `llmhosts/server.py`
- Modify: `pyproject.toml`

**Step 1: Write observability module**

`llmhosts/observability.py`:
- `init_observability(config: ObservabilityConfig)`:
  - If `sentry_dsn` is non-empty, try import `sentry_sdk` and init with FastAPI integration
  - If `log_format == "json"`, configure `logging.Formatter` with JSON output
  - Guarded import: if sentry_sdk not installed, log warning and skip
- `setup_json_logging()` — configures root logger with JSON formatter

**Step 2: Add monitoring optional dependency**

In `pyproject.toml`:
```toml
monitoring = [
    "sentry-sdk[fastapi]>=2.0.0",
]
```

**Step 3: Wire into server startup**

In `LLMHostsServer.startup()`, after report generation:
```python
from llmhost.observability import init_observability
init_observability(self.config.observability)
```

**Step 4: Run tests**

```bash
pytest tests/ -v
```

**Step 5: Commit**

```bash
git add llmhost/observability.py llmhosts/server.py pyproject.toml
git commit -m "feat(observability): add optional Sentry integration and JSON logging"
```

---

## Task 6: Database Migrations with Alembic

**Files:**
- Create: `llmhosts/migrations/__init__.py`
- Create: `llmhosts/migrations/env.py`
- Create: `llmhosts/migrations/script.py.mako`
- Create: `llmhosts/migrations/versions/001_initial_audit_schema.py`
- Create: `alembic.ini`
- Modify: `llmhosts/cli.py` — add `llmhosts db` subgroup
- Modify: `pyproject.toml` — add alembic dependency

**Step 1: Add alembic to dependencies**

```toml
dependencies = [
    ...
    "alembic>=1.14.0",
]
```

**Step 2: Create Alembic config**

`alembic.ini` at project root pointing to `llmhosts/migrations/`.
`llmhosts/migrations/env.py` configured for SQLite at `~/.llmhosts/llmhost.db`.

**Step 3: Create initial migration**

`001_initial_audit_schema.py`:
- Creates `audit_log` table matching current schema in `llmhosts/audit/`
- Idempotent: checks if table exists before creating

**Step 4: Add CLI command**

```python
@cli.group()
def db():
    """Database migration commands."""

@db.command()
def upgrade():
    """Run pending migrations."""

@db.command()
def downgrade():
    """Revert last migration."""
```

**Step 5: Run tests**

```bash
pytest tests/ -v
```

**Step 6: Commit**

```bash
git add alembic.ini llmhost/migrations/ llmhosts/cli.py pyproject.toml
git commit -m "feat(db): add Alembic migrations with initial audit schema"
```

---

## Task 7: Deployment Documentation + Production Docker Compose

**Files:**
- Create: `docs/deployment/bare-metal.md`
- Create: `docs/deployment/docker-production.md`
- Create: `docs/deployment/vps-guide.md`
- Create: `docker-compose.prod.yml`
- Create: `contrib/llmhost.service` (systemd unit)

**Step 1: Write bare-metal guide**

Covers: pip install, config, systemd service, reverse proxy (nginx/caddy), TLS.

**Step 2: Write Docker production guide**

Covers: docker-compose.prod.yml, environment variables, volumes, health checks, restart policies.

**Step 3: Write VPS guide**

Covers: Hetzner/OVH setup, firewall, Docker install, deployment, monitoring.

**Step 4: Create production docker-compose**

`docker-compose.prod.yml`:
- LLMHosts service with resource limits, restart policy, health check
- Caddy reverse proxy with automatic TLS
- Volume for persistent data

**Step 5: Create systemd unit file**

`contrib/llmhost.service` for bare-metal deployments.

**Step 6: Commit**

```bash
git add docs/deployment/ docker-compose.prod.yml contrib/
git commit -m "docs(deployment): add bare-metal, Docker, and VPS deployment guides"
```

---

## Task 8: Stripe Webhook Completion

**Files:**
- Modify: `frontend/src/app/api/webhooks/stripe/route.ts`

**Step 1: Review current webhook handler**

Read existing implementation, identify missing event types.

**Step 2: Complete webhook handlers**

Add handlers for:
- `checkout.session.completed` — activate subscription
- `customer.subscription.updated` — update plan tier
- `customer.subscription.deleted` — deactivate subscription
- `invoice.payment_failed` — flag account, send notification

Add idempotency: track processed event IDs to prevent double-processing.
Add signature verification with `stripe.webhooks.constructEvent()`.

**Step 3: Commit**

```bash
git add frontend/src/app/api/webhooks/
git commit -m "feat(billing): complete Stripe webhook lifecycle handlers"
```

---

## Task 9: E2E Test Suite

**Files:**
- Create: `tests/test_e2e.py`

**Step 1: Write E2E tests**

Using ASGI transport (same as existing tests):

- `test_e2e_unauthenticated_rejected` — require_auth=True, no key, 401
- `test_e2e_full_request_flow` — create key, authenticate, send chat completion, verify response headers (X-Request-ID, X-RateLimit-*), verify 200
- `test_e2e_rate_limit_enforcement` — burst requests, verify 429 on overflow
- `test_e2e_cors_restricted` — cors_restrict=True, verify preflight blocked for unauthorized origin
- `test_e2e_cors_allowed` — cors_restrict=True, verify preflight allowed for configured origin
- `test_e2e_health_no_auth` — /health accessible without auth even when required
- `test_e2e_models_authenticated` — /v1/models works with valid key

**Step 2: Run full test suite**

```bash
pytest tests/ -v --tb=short
```

**Step 3: Commit**

```bash
git add tests/test_e2e.py
git commit -m "test(e2e): add end-to-end security and request flow tests"
```

---

## Task 10: Integration Verification & Final Polish

**Step 1: Run full test suite with coverage**

```bash
pytest tests/ -v --cov=llmhost --cov-report=term-missing
```

**Step 2: Run linter and type checker**

```bash
ruff check . && ruff format --check . && mypy llmhost/ --ignore-missing-imports
```

**Step 3: Verify wheel builds**

```bash
python -m build --wheel --outdir dist/
```

**Step 4: Update CLAUDE.md and MEMORY.md with new state**

**Step 5: Final commit**

```bash
git add -A
git commit -m "chore: production gap closure -- all 10 gaps resolved"
```

---

## Parallelization Notes

These tasks can be parallelized:
- **Group A (independent):** Task 1 (config) must run first
- **Group B (parallel after Task 1):** Tasks 2, 3, 5, 6, 7, 8 are independent
- **Group C (depends on 2+3):** Task 4 (wire middleware)
- **Group D (depends on 4):** Task 9 (E2E tests)
- **Group E (last):** Task 10 (verification)

## Success Criteria

- [ ] All existing 253 tests still pass
- [ ] New tests for auth, ratelimit, E2E all pass
- [ ] `llmhosts serve` starts with default config (no auth, no breaking change)
- [ ] `LLMHOST_SECURITY_REQUIREAUTH=true llmhosts serve` enforces auth
- [ ] Wheel builds successfully
- [ ] Ruff + mypy clean
- [ ] Deployment docs are complete and actionable
