# Daemon & Server Health Check

Diagnose daemon push failures, server auth issues, and connectivity problems. Uses `.prod.env` for credentials — never hardcode or expose API keys.

**Auto-trigger**: Use this command when the user asks about daemon health, push failures, 401 errors, device connectivity, or server status.

## Input

The user may provide a device ID, email, org name, or describe a symptom:
- `/daemon-health-check b51ede5e...`
- `/daemon-health-check vaibhav's daemon is failing`
- `/daemon-health-check all pratilipi devices`

If no argument: check all devices with recent errors.

## Step 1: Server health

Verify the ingest server is up and DB is connected:

```bash
curl -s https://trace.quickcall.dev/health
```

Expected: `{"status": "ok", "db": "connected"}`

If this fails, the server is down — check Azure Container Apps status.

## Step 2: Auth check

Test auth using `X-API-Key` header (NOT `Authorization: Bearer` — the server uses `X-API-Key`):

```bash
source /path/to/.prod.env
curl -s -w "\nHTTP_CODE: %{http_code}\n" -X POST \
  -H "X-API-Key: $ORG_PRATILIPI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[]' \
  https://trace.quickcall.dev/ingest
```

Expected: `{"ingested": 0}` with HTTP 200.

If 401: the server's `QC_TRACE_PUSH_KEYS` env var doesn't include this key. Check Azure:

```bash
source .prod.env
az containerapp show --name $AZURE_CONTAINER_APP --resource-group $AZURE_RESOURCE_GROUP \
  --query "properties.template.containers[0].env[?name=='QC_TRACE_PUSH_KEYS' || name=='QC_TRACE_ADMIN_KEYS'].name" -o tsv
```

Do NOT print the actual key values — just confirm they exist.

## Step 3: Check daemon heartbeats

Query the `daemon_heartbeats` table for the device/org:

```sql
-- By device ID (prefix match)
SELECT device_name, org, daemon_version, status,
       last_push_at, last_seen_at, recent_errors
FROM daemon_heartbeats
WHERE device_id LIKE '<prefix>%';

-- By org
SELECT device_id, device_name, daemon_version, status,
       last_push_at, last_seen_at, recent_errors
FROM daemon_heartbeats
WHERE org = '<org>'
ORDER BY last_seen_at DESC;

-- All devices with errors
SELECT device_id, device_name, org, daemon_version,
       last_push_at, last_seen_at, recent_errors
FROM daemon_heartbeats
WHERE recent_errors != '[]' AND recent_errors IS NOT NULL
ORDER BY last_seen_at DESC;
```

## Step 4: Determine if errors are transient

If a device shows errors (e.g., `HTTP Error 401: Unauthorized`), check if it has **pushed messages successfully after the error timestamp**:

```sql
SELECT device_name, last_push_at, recent_errors,
       (SELECT MAX(m.timestamp) FROM messages m
        JOIN sessions s ON s.id = m.session_id
        WHERE s.device_id = dh.device_id) as latest_msg
FROM daemon_heartbeats dh
WHERE device_id LIKE '<prefix>%';
```

**Transient error criteria**: If `last_push_at` or `latest_msg` is MORE RECENT than the error timestamp in `recent_errors`, the issue was transient (e.g., during a deploy window) and has self-resolved. Report it as transient.

Common transient causes:
- Server redeployment (container restart window, typically 1-2 minutes)
- Network blip
- Azure Container Apps scaling event

## Step 5: Cross-check with sessions table

If the daemon appears stuck (no recent pushes), check what data exists:

```sql
SELECT source, COUNT(*) as sessions,
       MAX(first_seen) as last_session
FROM sessions
WHERE device_id LIKE '<prefix>%'
GROUP BY source
ORDER BY last_session DESC;
```

## Step 6: Identify the device owner

If only a device ID is provided, find the associated user:

```sql
SELECT DISTINCT user_email, device_name, org
FROM sessions
WHERE device_id LIKE '<prefix>%';
```

## Diagnosis summary

After gathering data, report:
1. **Server status**: healthy / degraded / down
2. **Auth status**: working / broken (and for which keys)
3. **Device status**: pushing / stale / erroring
4. **Error assessment**: transient (self-resolved) / active (needs intervention)
5. **Recommended action**: if any

## Important

- NEVER print or expose API keys in output — only confirm presence/absence
- Use `source .prod.env` to load keys, never hardcode them
- The server uses `X-API-Key` header, NOT `Authorization: Bearer`
- `recent_errors` in `daemon_heartbeats` is a JSON array with `ts`, `count`, and `error` fields
- Errors with count=1-5 during a deploy window are almost always transient
- A device with `status: "pushing"` and recent `last_seen_at` is healthy even if it has old errors in `recent_errors`
