# Reingest

Trigger a reingest command for cursor_vscdb (or other sources) across daemons. Uses `.prod.env` for credentials — never hardcode or expose API keys.

**Auto-trigger**: Use this command when the user asks to reingest, re-push, or re-process data on daemons after a transform fix or schema change.

## Input

The user may provide a source, version filter, device name, or org:
- `/reingest` — reingest cursor_vscdb on all daemons at latest version
- `/reingest cursor_vscdb on 0.4.42` — filter by daemon version
- `/reingest cursor_vscdb on sounak` — filter by device name (partial match)
- `/reingest all sources on pratilipi` — reingest all sources for an org's devices

If no argument: reingest `cursor_vscdb` on all daemons running the latest version.

## Step 1: Find target daemons

Query daemon_heartbeats to find devices matching the filter:

```bash
source /Users/sagar/work/all-things-quickcall/trace/.prod.env
psql "$QC_TRACE_DSN" -c "
  SELECT device_id, device_name, org, daemon_version, status, last_seen_at
  FROM daemon_heartbeats
  ORDER BY last_seen_at DESC"
```

Apply filters:
- **Version filter**: `WHERE daemon_version = '<version>'`
- **Device name filter**: `WHERE device_name ILIKE '%<name>%'`
- **Org filter**: `WHERE org = '<org>'`
- **Default (no filter)**: all daemons at the highest daemon_version

Show the user the list of target daemons and confirm before proceeding.

## Step 2: Check for existing pending commands

Before sending reingest, check if any target devices already have pending/acked reingest commands:

```bash
psql "$QC_TRACE_DSN" -c "
  SELECT dc.id, dh.device_name, dc.status, dc.params
  FROM daemon_commands dc
  JOIN daemon_heartbeats dh ON dh.device_id = dc.device_id
  WHERE dc.command = 'reingest' AND dc.status IN ('pending', 'acked')"
```

If any exist, warn the user — a device can only have one pending reingest at a time (the API returns 409 Conflict).

## Step 3: Send reingest commands

Use the `/api/reingest` endpoint with the admin key:

```bash
source /Users/sagar/work/all-things-quickcall/trace/.prod.env
ADMIN_KEY=$(echo "$QC_TRACE_ADMIN_KEYS" | cut -d',' -f1)

curl -s -X POST https://trace.quickcall.dev/api/reingest \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $ADMIN_KEY" \
  -d '{"device_id": "<device_id>", "source": "<source>"}'
```

The `source` parameter is optional — omit it to reingest all sources. Include it to limit reingest to a specific source (e.g., `cursor_vscdb`).

Loop through all target devices and fire the requests. Report command IDs.

## Step 4: Monitor progress

Poll the daemon_commands table every 60-90 seconds until all commands complete or the user tells you to stop:

```bash
psql "$QC_TRACE_DSN" -c "
  SELECT dc.id, dh.device_name, dc.status, dc.completed_at, dc.result
  FROM daemon_commands dc
  JOIN daemon_heartbeats dh ON dh.device_id = dc.device_id
  WHERE dc.id >= <first_command_id>
  ORDER BY dc.id"
```

Report a summary table with:
- Device name
- Status (pending → acked → completed)
- Messages pushed (from `result->>'messages_pushed'`)
- Any errors

Commands are picked up on the next heartbeat (up to 5 minutes). If a command stays `pending` for more than 10 minutes, the daemon may be offline.

## Step 5: Verify results

After all commands complete, run a quick count of the affected source:

```bash
psql "$QC_TRACE_DSN" -c "
  SELECT source, count(*) AS sessions, sum(msg_count) AS messages
  FROM (
    SELECT s.source, s.id, count(m.id) AS msg_count
    FROM sessions s
    LEFT JOIN messages m ON m.session_id = s.id
    WHERE s.source = '<source>'
    GROUP BY s.source, s.id
  ) sub
  GROUP BY source"
```

## Important

- NEVER print or expose API keys — use `source .prod.env` and reference env vars
- The reingest API is admin-only (`QC_TRACE_ADMIN_KEYS`)
- A device can only have ONE pending/acked reingest at a time (409 Conflict if duplicate)
- Reingest resets file_progress state for the source, causing the daemon to re-read and re-push all files
- Messages use `ON CONFLICT DO UPDATE` so re-pushing is idempotent — no duplicates
- Daemons pick up commands via heartbeat response (every ~5 minutes)
- Large vscdb files can take several minutes to process (19K+ messages observed)
