# Daemon & Server Health Check

Diagnose daemon push failures, server auth issues, and connectivity problems. Uses `.prod.env` for credentials — never hardcode or expose API keys.

**Auto-trigger**: Use this command when the user asks about daemon health, push failures, 401 errors, device connectivity, or server status.

## Input

The user may provide a device ID, email, org name, or describe a symptom:
- `/daemon-health-check b51ede5e...`
- `/daemon-health-check vaibhav's daemon is failing`
- `/daemon-health-check all pratilipi devices`

If no argument: check ALL devices (not just ones with errors).

## Step 1: Server health

Verify the ingest server is up and DB is connected:

```bash
curl -s https://trace.quickcall.dev/health
```

Expected: `{"status": "ok", "db": "connected"}`

If this fails, the server is down — check Azure Container Apps status.

## Step 2: Auth check

Test auth using `X-API-Key` header (NOT `Authorization: Bearer` — the server uses `X-API-Key`):

```bash
source /path/to/.prod.env
curl -s -w "\nHTTP_CODE: %{http_code}\n" -X POST \
  -H "X-API-Key: $ORG_PRATILIPI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[]' \
  https://trace.quickcall.dev/ingest
```

Expected: `{"ingested": 0}` with HTTP 200.

If 401: the server's `QC_TRACE_PUSH_KEYS` env var doesn't include this key. Check Azure:

```bash
source .prod.env
az containerapp show --name $AZURE_CONTAINER_APP --resource-group $AZURE_RESOURCE_GROUP \
  --query "properties.template.containers[0].env[?name=='QC_TRACE_PUSH_KEYS' || name=='QC_TRACE_ADMIN_KEYS'].name" -o tsv
```

Do NOT print the actual key values — just confirm they exist.

## Step 3: Check daemon heartbeats

Query the `daemon_heartbeats` table for the device/org:

```sql
-- By device ID (prefix match)
SELECT device_name, org, daemon_version, status,
       last_push_at, last_seen_at, recent_errors
FROM daemon_heartbeats
WHERE device_id LIKE '<prefix>%';

-- By org
SELECT device_id, device_name, daemon_version, status,
       last_push_at, last_seen_at, recent_errors
FROM daemon_heartbeats
WHERE org = '<org>'
ORDER BY last_seen_at DESC;

-- All devices (default when no argument given)
SELECT device_id, device_name, org, daemon_version,
       status, last_push_at, last_seen_at, recent_errors
FROM daemon_heartbeats
ORDER BY last_seen_at DESC;
```

## Step 4: Determine if errors are transient

If a device shows errors (e.g., `HTTP Error 401: Unauthorized`), check if it has **pushed messages successfully after the error timestamp**:

```sql
SELECT device_name, last_push_at, recent_errors,
       (SELECT MAX(m.timestamp) FROM messages m
        JOIN sessions s ON s.id = m.session_id
        WHERE s.device_id = dh.device_id) as latest_msg
FROM daemon_heartbeats dh
WHERE device_id LIKE '<prefix>%';
```

**Transient error criteria**: If `last_push_at` or `latest_msg` is MORE RECENT than the error timestamp in `recent_errors`, the issue was transient (e.g., during a deploy window) and has self-resolved. Report it as transient.

Common transient causes:
- Server redeployment (container restart window, typically 1-2 minutes)
- Network blip
- Azure Container Apps scaling event

## Step 5: Cross-check with sessions table

If the daemon appears stuck (no recent pushes), check what data exists:

```sql
SELECT source, COUNT(*) as sessions,
       MAX(first_seen) as last_session
FROM sessions
WHERE device_id LIKE '<prefix>%'
GROUP BY source
ORDER BY last_session DESC;
```

## Step 6: Identify the device owner

If only a device ID is provided, find the associated user:

```sql
SELECT DISTINCT user_email, device_name, org
FROM sessions
WHERE device_id LIKE '<prefix>%';
```

## Output format

**CRITICAL**: Run Steps 1-3 in parallel (server health, auth check, heartbeats query). Then analyze the results and produce the COMPLETE output below in a SINGLE response. Do NOT ask follow-up questions or split into multiple responses.

Present ALL results as a single markdown table + summary. The output MUST look exactly like this:

```
## All Devices (N total)

| # | Device | Org | Version | Status | Last Push | Last Seen | Errors |
|---|--------|-----|---------|--------|-----------|-----------|--------|
| 1 | device-name | org | 0.4.48 | pushing | Mar 7 | Mar 7 12:31 | ✅ None |
| 2 | device-name | org | 0.4.48 | pushing | Mar 6 | Mar 7 12:30 | ⚡ Transient timeout (Mar 5) |
| 3 | device-name | org | 0.4.48 | idle | Feb 20 | Feb 25 12:33 | ⚠️ Stale — not seen in N days |

### Summary
- **Server**: ✅ Healthy / ❌ Down
- **Auth**: ✅ Working / ❌ Broken
- **Active devices** (seen today): X of Y
- <any notable issues: transient errors, stale devices, old versions>
```

Rules for the Errors column:
- `✅ None` — no errors, device is healthy
- `✅ None (heartbeating but no new data)` — device is seen recently but last_push is old
- `⚡ Transient <error type> (<date>)` — error occurred but device pushed successfully after
- `⚠️ Stale — not seen in N days` — device hasn't been seen in 3+ days
- `❌ Active error: <description>` — error is ongoing, device has NOT pushed since the error

A device is "stale" if `last_seen_at` is more than 3 days ago.
An error is "transient" if `last_push_at` is more recent than the error timestamp in `recent_errors`.

## Important

- NEVER print or expose API keys in output — only confirm presence/absence
- Use `source .prod.env` to load keys, never hardcode them
- The server uses `X-API-Key` header, NOT `Authorization: Bearer`
- `recent_errors` in `daemon_heartbeats` is a JSON array with `ts`, `count`, and `error` fields
- Errors with count=1-5 during a deploy window are almost always transient
- A device with `status: "pushing"` and recent `last_seen_at` is healthy even if it has old errors in `recent_errors`
