# Token Operations Runbook

Production operational procedures for Token — the conversational fleet orchestrator.

## 1. Deployment

### Environment Variables

| Variable | Required | Description |
|----------|----------|-------------|
| `OPENROUTER_API_KEY` | Yes | OpenRouter API key for LLM access |
| `DATABASE_URL` | Yes | Prisma database URL (SQLite/PostgreSQL) |
| `AUTH_SECRET` | Yes | NextAuth session secret |
| `RESEND_API_KEY` | No | Email notifications (Resend). Omit to log-only. |

### Docker Build & Deploy

```bash
# Build frontend image (includes Prisma generate)
docker compose -f docker-compose.prod.yml build llmhosts-frontend

# Apply database migrations
docker compose exec llmhosts-frontend npx prisma migrate deploy

# Restart with new image
docker compose -f docker-compose.prod.yml up -d llmhosts-frontend
```

### Health Check

```bash
# Verify frontend is responding
curl -s http://localhost:3000/api/health | jq .

# Verify Token chat endpoint (auth required — use browser or session cookie)
curl -s http://localhost:3000/api/token/config | head -c 200

# Verify Prometheus metrics are emitting
curl -s http://localhost:3000/api/token/metrics | head -20
```

### Post-Deploy Verification

1. Load `/dashboard/token` in browser — confirm chat UI renders
2. Send a test message — confirm streaming response
3. Check `/api/token/metrics` — confirm `token_messages_total` increments
4. Check Grafana Token dashboard — confirm panels populate

## 2. Monitoring

### Key Metrics

| Metric | Source | Alert Threshold |
|--------|--------|-----------------|
| `token_messages_total` | Prometheus | N/A (observational) |
| `token_cost_usd_total` | Prometheus | Track rate for budget burn |
| `token_latency_ms{quantile="0.95"}` | Prometheus | > 10,000ms = investigate |
| `token_alerts_total{severity="critical"}` | Prometheus | Any increase = investigate |
| Budget usage % | `/api/token/analytics` | > 80% = warning, > 95% = critical |

### Grafana Dashboard

- **Location**: Grafana → Dashboards → LLMHosts Token
- **File**: `monitoring/grafana/dashboards/llmhosts-token.json`
- **Panels**: Messages/min, Cost/time, Latency percentiles, Active alerts, Budget gauge

### Prometheus Scrape Config

Token metrics are scraped from the Next.js frontend:
```yaml
# monitoring/prometheus/prometheus.yml
- job_name: "token-metrics"
  metrics_path: /api/token/metrics
  static_configs:
    - targets: ["llmhosts-frontend:3000"]
```

## 3. Troubleshooting

### OpenRouter API Down / Errors

**Symptoms**: Chat returns 500, error banner in UI, `token_latency_ms` spikes or drops to zero.

**Steps**:
1. Check OpenRouter status: https://status.openrouter.ai
2. Check logs: `docker compose logs llmhosts-frontend --tail 100 | grep token_chat`
3. If prolonged outage, Token will be unavailable (no local fallback for LLM brain)
4. No user action needed — Token automatically retries on next message

### Budget Exhausted (100%)

**Symptoms**: Chat returns error "Monthly budget exhausted", users see error banner.

**Steps**:
1. Check current spend: `GET /api/token/analytics` → `spending.budgetUsedPct`
2. Option A — increase budget: `PATCH /api/token/config` with `{ "monthlyBudgetUsd": <new_limit> }`
3. Option B — wait for month reset (TokenConfig.totalSpentUsd is not auto-reset; admin must reset)
4. To reset manually: update `TokenConfig.totalSpentUsd = 0` via Prisma Studio or API

### Device Shows Offline (False Positive)

**Symptoms**: Device-offline alert fires, but device is actually running.

**Steps**:
1. Check if device is in a maintenance window: `GET /api/token/schedule`
2. Check device heartbeat: look at `Device.lastSeen` timestamp
3. If the device is behind NAT/firewall, heartbeat may be delayed — increase threshold if needed
4. Run `llmhosts doctor` on the device machine to diagnose connectivity

### High Latency (> 5s responses)

**Symptoms**: Users report slow Token responses, `token_latency_ms` P95 > 5000.

**Steps**:
1. Check which model is slow: Grafana → Latency panel → filter by model
2. Check OpenRouter model status — some models have queue delays
3. Consider switching to a faster model in Token config: `PATCH /api/token/config` with `{ "chatModel": "google/gemini-2.5-flash" }`
4. Check if the system prompt is too large (many devices, many alerts) — reduce fleet size in prompt

### Conversation Load Fails

**Symptoms**: Clicking a conversation in sidebar shows error or empty state.

**Steps**:
1. Check browser console for network errors
2. Verify database is accessible: `docker compose exec llmhosts-frontend npx prisma studio`
3. Check if conversation exists: query `TokenConversation` by ID
4. If database is corrupted, restore from backup (see section 6)

## 4. Scaling

### Increasing Budget

```bash
# Via API (as admin user)
curl -X PATCH http://localhost:3000/api/token/config \
  -H "Content-Type: application/json" \
  -H "Cookie: <session_cookie>" \
  -d '{"monthlyBudgetUsd": 100}'
```

### Adding Models to Config

Token uses 4 model slots configured in `TokenConfig`:
- `chatModel` — general conversation (default: gemini-2.5-flash)
- `researchModel` — deep research tasks (default: claude-sonnet-4.6)
- `reasoningModel` — complex reasoning (default: deepseek-r1)
- `classificationModel` — intent classification (default: gemini-2.5-flash-lite)

Update via API or Prisma Studio. Changes take effect on the next message.

### Multi-Instance Considerations

Token uses **in-memory** stores for:
- Prometheus metrics (lib/token/metrics.ts) — per-instance, not aggregated
- Alert acknowledgments (lib/token/alerts.ts) — 1h TTL per instance
- Email cooldowns (lib/token/notifications.ts) — per-instance

For multi-instance deployments:
- Prometheus will scrape each instance separately — use `sum()` in PromQL for aggregation
- Alert acks may re-fire on different instances — acceptable for 1h TTL
- Email cooldowns may send duplicates across instances — acceptable with cooldown windows

For true HA, consider moving these to Redis (future enhancement).

## 5. Incident Response

### Disable Token Entirely

```bash
# Via API
curl -X PATCH http://localhost:3000/api/token/config \
  -H "Content-Type: application/json" \
  -H "Cookie: <session_cookie>" \
  -d '{"enabled": false}'
```

When disabled, the chat endpoint returns `{ "error": "Token is disabled" }` and the UI shows accordingly.

### Emergency Budget Override

If budget is exhausted but service is critical:
```sql
-- Via Prisma Studio or direct DB access
UPDATE TokenConfig SET totalSpentUsd = 0, monthlyBudgetUsd = 200 WHERE id = 'singleton';
```

### OpenRouter API Key Rotation

1. Generate new key at https://openrouter.ai/keys
2. Update `.env`: `OPENROUTER_API_KEY=sk-or-v1-...`
3. Restart frontend: `docker compose restart llmhosts-frontend`
4. Verify: send a test message in Token chat

### Conversation Data Corruption

If conversations are showing incorrect data:
1. Check recent migrations: `npx prisma migrate status`
2. Verify schema matches: `npx prisma db pull` and compare
3. If needed, re-run generate: `npx prisma generate`
4. Last resort: restore database from backup

## 6. Backup & Recovery

### What to Back Up

| Data | Location | Frequency |
|------|----------|-----------|
| SQLite database | Docker volume `/data/llmhosts.db` | Daily |
| TokenConfig | Part of database | With DB backup |
| Conversations + Messages | Part of database | With DB backup |
| Maintenance Windows | Part of database | With DB backup |
| Environment variables | `.env` file | On change |

### Database Backup

```bash
# Copy SQLite database from Docker volume
docker compose exec llmhosts-frontend cp /data/llmhosts.db /data/llmhosts-backup-$(date +%Y%m%d).db

# Or from host
docker cp llmhosts-frontend:/data/llmhosts.db ./backups/llmhosts-$(date +%Y%m%d).db
```

### Config Export / Import

```bash
# Export Token config
curl -s http://localhost:3000/api/token/config \
  -H "Cookie: <session_cookie>" | jq . > token-config-backup.json

# Export conversations for a user (use export endpoint)
curl -s "http://localhost:3000/api/token/conversations/<conv_id>/export?format=json" \
  -H "Cookie: <session_cookie>" > conversation-backup.json
```

### Recovery

1. Stop frontend: `docker compose stop llmhosts-frontend`
2. Restore database file to volume
3. Run migrations: `docker compose exec llmhosts-frontend npx prisma migrate deploy`
4. Start frontend: `docker compose start llmhosts-frontend`
5. Verify: load Token dashboard, check conversations are present
