# Plan: Reingest Command (Issue #82)

## Context

When server-side data gets stale or a daemon's local state drifts out of sync, there's no way to tell a running daemon to re-push data without manual restarts or DB surgery. This adds a server-initiated command queue, starting with a `reingest` command delivered via the heartbeat response.

**Important caveat:** Messages currently use `ON CONFLICT DO NOTHING`, so re-pushing existing messages won't update them. Reingest is useful for: recovering from missed/failed pushes, re-pushing after local state corruption, and pushing files the daemon skipped. Schema evolution (updating existing message fields) would require changing the writer's conflict strategy — that's out of scope for this issue.

## Approach: Command queue delivered via heartbeat response

The daemon already heartbeats every 5 minutes. Extend the heartbeat response to include pending commands. No new polling endpoint needed.

## Changes

### 1. New DB migration (v9): `daemon_commands` table

**File:** `qc_trace/db/migrations.py`

```sql
CREATE TABLE IF NOT EXISTS daemon_commands (
    id SERIAL PRIMARY KEY,
    device_id TEXT NOT NULL,
    command TEXT NOT NULL,
    params JSONB DEFAULT '{}',
    status TEXT NOT NULL DEFAULT 'pending',
    created_at TIMESTAMPTZ DEFAULT NOW(),
    acked_at TIMESTAMPTZ,
    completed_at TIMESTAMPTZ,
    result JSONB
);
CREATE INDEX IF NOT EXISTS idx_daemon_commands_pending
    ON daemon_commands (device_id, status) WHERE status = 'pending';
```

Statuses: `pending` → `acked` → `completed` | `failed`

### 2. Server: `POST /api/reingest` endpoint

**File:** `qc_trace/server/handlers.py` + `qc_trace/server/app.py`

> Note: Using `/api/reingest` with `device_id` in the body (not path param) because the router uses exact string matching — no path parameter support.

- Admin auth required
- Body includes `device_id` + scope params:

| Scope | Body example | Behaviour |
|-------|-------------|-----------|
| All | `{"device_id": "x", "scope": "all"}` | Reset all file progress, re-push everything |
| By source/IDE | `{"device_id": "x", "source": "claude_code"}` | Re-push only files from that tool |
| Single file | `{"device_id": "x", "file": "/path/to/session.jsonl"}` | Re-push one specific file |
| By date | `{"device_id": "x", "since": "2026-02-01"}` | Re-push files modified after date |

- Validates `device_id` exists in `daemon_heartbeats`
- Rejects if device already has a `pending` or `acked` reingest command (prevents duplicates)
- **Deletes matching `file_progress` rows server-side** (critical — otherwise `_reconcile()` restores old state)
- Inserts row into `daemon_commands`
- Returns `{"ok": true, "command_id": <id>}`

### 3. Server: extend `handle_heartbeat()` response

**File:** `qc_trace/server/handlers.py`

After upserting heartbeat, query pending commands for the device. Return:
```json
{"ok": true, "commands": [{"id": 1, "command": "reingest", "params": {"scope": "all"}}]}
```

Mark returned commands as `status = 'acked'`, `acked_at = NOW()`.

Also: recover stale commands — any command in `acked` state for >15 minutes reverts to `pending` (handles daemon crash after ack but before execution).

### 4. Server: `POST /api/command-complete` endpoint

**File:** `qc_trace/server/handlers.py` + `qc_trace/server/app.py`

> Note: Using `/api/command-complete` with `command_id` in the body (not path param) for same routing reason.

- Push auth (daemon calls this)
- Body: `{"command_id": 1, "status": "completed", "result": {...}}` or `{"command_id": 1, "status": "failed", "result": {"error": "..."}}`
- Updates the `daemon_commands` row with status, result, and `completed_at`

### 5. Server: admin notification on command completion/failure

**File:** `qc_trace/server/handlers.py`

When a command is marked `failed`:
- Log at WARNING level with device_id, command details, and error
- Include command status in `GET /api/devices` response so admin dashboard shows it
- Include recent commands (last 10) in `GET /api/monitor` response

When a command is marked `completed`:
- Log at INFO level

(Full notification system like email/Slack is out of scope — admin visibility via existing endpoints is sufficient for now.)

### 6. Daemon: process commands from heartbeat response

**File:** `qc_trace/daemon/main.py`

- In `_send_heartbeat()`: `pusher.send_heartbeat()` now returns parsed response dict. Extract `commands` array from it. **Do NOT execute commands inside `_send_heartbeat()`** — instead store them in `self._pending_commands` and return.
- In main `run()` loop, after `_poll_cycle()` + `_send_heartbeat()`, check `self._pending_commands`. If non-empty, execute them before the next sleep.

**Reingest execution flow (avoids recursive `_poll_cycle` and watcher conflicts):**

```
run() loop:
  1. _poll_cycle()          ← normal watcher + push
  2. _send_heartbeat()      ← picks up commands, stores in self._pending_commands
  3. if self._pending_commands:
       self._paused = True  ← signal watcher to skip
       _handle_commands()   ← resets state, runs _reingest_poll_cycle()
       self._paused = False ← resume watcher
  4. _sleep()
```

- New method `_handle_commands(commands: list[dict])`:
  - Iterates commands, dispatches by `command` field
  - For `reingest`: calls `_handle_reingest(command_id, params)`
- `_handle_reingest(command_id, params)`:
  - Based on scope, reset matching entries via `state_mgr` methods
  - Call `_reingest_poll_cycle()` — a dedicated poll that only processes the reset files (not a recursive call to `_poll_cycle()`)
  - **Timeout guard**: `_reingest_poll_cycle()` has a `max_duration=600` (10 min) parameter. If reingest takes longer, it stops mid-cycle, reports partial completion, and resumes normal operation. This prevents the watcher from being paused indefinitely.
  - On success: `pusher.report_command_complete(command_id, "completed", {"files_reset": N, "messages_pushed": M})`
  - On failure: `pusher.report_command_complete(command_id, "failed", {"error": str(e)})`
  - Wrapped in try/finally so `self._paused = False` always runs — **watcher is never stuck paused**

- `_reingest_poll_cycle(max_duration=600)`:
  - Same logic as `_poll_cycle()` but:
    - Checks `time.time() - start > max_duration` after each file — breaks if exceeded
    - Returns `{"files_processed": N, "messages_pushed": M, "timed_out": bool}`
    - Does NOT call `_send_heartbeat()` at the end (avoids recursion)

### 7. Daemon StateManager: scope-aware reset methods

**File:** `qc_trace/daemon/state.py`

- `reset_all()` — clear entire state dict
- `reset_by_source(source: str)` — clear entries where source matches
- `reset_by_file(path: str)` — already exists as `reset_state()`
- `reset_since(cutoff: str)` — clear entries for files with mtime after cutoff date

### 8. Daemon Pusher: heartbeat response + command completion reporting

**File:** `qc_trace/daemon/pusher.py`

- **Change `send_heartbeat()`** — currently returns `bool`, must return `dict | None` with parsed response body (so daemon can read `commands` array). Change line 208 from `return resp.status < 300` to `return json.loads(resp.read()) if resp.status < 300 else None`
- **Add `report_command_complete(command_id, status, result)`** — POST to `/api/command-complete`

## Files to modify

| File | Change |
|------|--------|
| `qc_trace/db/migrations.py` | Add v9 migration for `daemon_commands` table |
| `qc_trace/server/handlers.py` | Add reingest + command-complete endpoints, extend heartbeat response |
| `qc_trace/server/app.py` | Register 2 new routes (`/api/reingest`, `/api/command-complete`) |
| `qc_trace/daemon/main.py` | Handle commands from heartbeat, execute reingest |
| `qc_trace/daemon/state.py` | Add `reset_all()`, `reset_by_source()`, `reset_since()` |
| `qc_trace/daemon/pusher.py` | Change `send_heartbeat()` to return response body, add `report_command_complete()` |

## Parallel Agent Execution Guide

This feature is split into sub-issues designed for parallel Claude Code agent execution.

### GitHub Issues

| Phase | Issue | Task |
|-------|-------|------|
| 1 | #84 | DB migration v9: `daemon_commands` table |
| 1 | #85 | Daemon StateManager scope-aware reset methods |
| 2 | #86 | Server endpoints (reingest + command-complete + heartbeat) |
| 2 | #87 | Daemon command processing + reingest execution |
| 3 | #88 | Integration tests |

### How to run with Claude Code agents

**Phase 1** — Launch 2 agents in parallel (no dependencies):
```
Agent A → "Implement #84: read qc_trace/db/migrations.py, add v9 migration for daemon_commands table following existing v4-v8 pattern. Create branch feat/reingest-db-migration."

Agent B → "Implement #85: read qc_trace/daemon/state.py, add reset_all(), reset_by_source(source), reset_since(cutoff) methods. reset_by_file already exists as reset_state(). Add unit tests. Create branch feat/reingest-state-reset."
```

**Phase 2** — After Phase 1 merges, launch 2 agents in parallel:
```
Agent C → "Implement #86: read qc_trace/server/handlers.py and app.py. Add POST /api/reingest (admin auth, deletes matching file_progress rows, inserts into daemon_commands), POST /api/command-complete (push auth), and extend handle_heartbeat() to return pending commands + recover stale acked commands. Create branch feat/reingest-server-endpoints."

Agent D → "Implement #87: read qc_trace/daemon/main.py and pusher.py. Change pusher.send_heartbeat() to return parsed response JSON instead of bool. In main.py _send_heartbeat(), read commands from response. Add _handle_commands() and _handle_reingest() methods. Add report_command_complete() to pusher.py. Create branch feat/reingest-daemon-commands."
```

**Phase 3** — After Phase 2 merges:
```
Agent E → "Implement #88: add tests/test_reingest.py covering server handler tests (reingest endpoint, command-complete endpoint, heartbeat with commands), StateManager reset tests, and daemon command processing tests. Create branch feat/reingest-tests."
```

### Key files each agent needs to read first

| Agent | Must read |
|-------|-----------|
| A | `qc_trace/db/migrations.py` |
| B | `qc_trace/daemon/state.py` |
| C | `qc_trace/server/handlers.py`, `qc_trace/server/app.py`, `qc_trace/server/auth.py` |
| D | `qc_trace/daemon/main.py`, `qc_trace/daemon/pusher.py`, `qc_trace/daemon/state.py` |
| E | All of the above + `tests/` for patterns |

## Assumptions

| # | Assumption | Verified? | Impact if wrong |
|---|-----------|-----------|-----------------|
| 1 | **Daemon is single-threaded** — `_handle_reingest` can safely call `_poll_cycle()` synchronously | Yes — `run()` is a `while not self._shutdown` loop, no threads | Race conditions in state/pusher |
| 2 | **`file_progress` has no `device_id` column** — PK is `(raw_file_path, source)`. Deleting rows during reingest affects ALL devices sharing those paths | Yes — confirmed in schema v4 | Multi-device setups break. Mitigated: in practice file paths contain user-specific home dirs (`/Users/sagar/...`) so collisions are unlikely. For v1 this is acceptable; multi-device isolation needs `device_id` added to `file_progress` PK (separate issue) |
| 3 | **FileWatcher re-detects files after state reset** — `_check_file()` returns `None` only if `existing.last_mtime == mtime AND existing.last_size == size`. After `reset_state()` deletes the entry, `existing` is `None` → file is returned as changed | Yes — `watcher.py:70-74` | Files silently skipped after reingest |
| 4 | **`send_heartbeat()` discards response body** — currently reads `resp.status` only, never calls `resp.read()`. Must be changed to return parsed JSON | Yes — `pusher.py:207-208` | Commands never delivered to daemon |
| 5 | **`reset_since` uses `FileState.last_mtime`** — the mtime stored at last processing, not current disk mtime. If a file was processed long ago but not modified since, `last_mtime` is old and won't match a recent `since` cutoff | Assumption — matches intent (re-push files processed after a date) | Could confuse admins who expect "files modified after date" vs "files last processed after date". Document clearly in API |
| 6 | **Heartbeat interval is 5 minutes** — max latency between reingest request and daemon pickup is ~5min | Yes — `_HEARTBEAT_INTERVAL = 300` | Not a bug, just UX. Acceptable for v1 |
| 7 | **`ON CONFLICT DO NOTHING` on messages** — re-pushed messages that already exist in DB are silently dropped. Reingest does NOT update existing rows | Yes — `writer.py:119-124` | Admins expecting data refresh for existing messages will be disappointed. Documented as caveat |
| 8 | **Single reingest per device at a time** — plan rejects new reingest if one is already `pending`/`acked` | Design decision | Could be relaxed later if needed |
| 9 | **`_poll_cycle()` cannot be called recursively** — `_send_heartbeat()` is called from within `_poll_cycle()`. If reingest calls `_poll_cycle()` again from inside heartbeat handling, we get recursion + watcher conflicts | Yes — confirmed call chain: `run()` → `_poll_cycle()` → `_send_heartbeat()` | Stack overflow or state corruption. Mitigated: commands stored in `_pending_commands`, executed after heartbeat returns, in dedicated `_reingest_poll_cycle()` |
| 10 | **Reingest won't block forever** — 10-minute timeout on `_reingest_poll_cycle()`, try/finally ensures `self._paused = False` always runs | Design decision | Without timeout, a large `scope: all` reingest on a device with thousands of files could pause normal operation for hours |
| 11 | **Watcher pause is implicit (single-threaded)** — `self._paused` flag is technically redundant since reingest runs in the same thread as the main loop. The flag exists as a safety guard and for logging clarity ("reingest in progress, watcher paused") | Yes — single-threaded | Flag is belt-and-suspenders; remove if it adds complexity |
| 12 | **`since` scope — server can't filter by file mtime** — `file_progress` has `updated_at` (last daemon report time) but no file mtime. Server-side deletion for `since` scope must use `updated_at` as a proxy, or skip server-side deletion and rely on daemon-side reset only | Yes — schema v4 has only `updated_at` | If daemon restarts before reingest completes, `_reconcile()` may restore state for `since`-scoped files. Acceptable for v1 — `since` is a convenience scope, not the primary use case |
| 13 | **`send_heartbeat()` return type change breaks existing test** — `test_heartbeat.py:59` asserts `result is True`. Must update test to expect `dict` with `{"ok": True}` | Yes — `tests/test_heartbeat.py:59` | Test failure blocks CI |
| 14 | **Backward compatibility: server/daemon version skew** — New server + old daemon: response includes `commands: []`, old daemon discards body (safe). Old server + new daemon: response is `{"ok": true}` with no `commands` key, new daemon must handle missing key gracefully with `.get("commands", [])` | Design decision | Daemon crash on old server if not handled |

## Gaps addressed in this plan

| Gap | Risk | Mitigation |
|-----|------|------------|
| Server `file_progress` restores old state via `_reconcile()` | Reingest has no effect after daemon restart | `POST /api/reingest` deletes matching `file_progress` rows server-side |
| Router has no path params | 404 on `/api/devices/{id}/reingest` | Use flat endpoints: `/api/reingest`, `/api/command-complete` with IDs in body |
| Acked commands stuck after daemon crash | Command never executes | Heartbeat handler reverts `acked` commands older than 15min back to `pending` |
| Duplicate reingest spam | Wasted work (harmless due to `DO NOTHING` but noisy) | Reject if device already has `pending`/`acked` reingest |
| Messages use `ON CONFLICT DO NOTHING` | Re-pushed messages with new fields won't update | Out of scope — documented as caveat. Schema evolution needs separate writer change |
| `send_heartbeat()` discards response body | Commands never reach daemon | Change `pusher.send_heartbeat()` to return parsed response JSON |
| `file_progress` has no `device_id` | Reingest for one device could clear another's progress | Acceptable for v1 (paths contain user home dirs). Future: add `device_id` to `file_progress` PK |
| Recursive `_poll_cycle()` call | `_handle_reingest` inside `_send_heartbeat` inside `_poll_cycle` creates recursion | Commands stored in `_pending_commands`, executed after heartbeat returns in dedicated `_reingest_poll_cycle()` |
| Reingest could block watcher indefinitely | Large reingest pauses normal operation for hours | 10-min timeout on `_reingest_poll_cycle()`, try/finally ensures `self._paused = False` always runs |
| `since` scope can't delete server `file_progress` by mtime | No mtime column in `file_progress` | Use `updated_at` as proxy for server-side deletion; daemon-side uses `FileState.last_mtime`. Document the difference |
| `send_heartbeat()` return type change | Existing test `test_heartbeat.py:59` expects `True` | Update test to expect `dict`. Test mock server already returns `{"ok":true}` (line 30) |
| Server/daemon version skew | New daemon + old server could crash on missing `commands` key | Daemon uses `.get("commands", [])` — safe fallback. New server + old daemon is safe (old daemon ignores response body) |
| Timed-out reingest — partial completion | Some files processed, some not. What status? | Report `completed` with `{"timed_out": true, "files_processed": N, "files_remaining": M}`. Admin can trigger another reingest for remaining files |

## Verification

### Automated (unit tests — Phase 3, issue #88)
1. Server handler tests: reingest endpoint creates command + deletes file_progress
2. Server handler tests: command-complete endpoint updates status
3. Server handler tests: heartbeat returns pending commands, recovers stale acked
4. Server handler tests: duplicate reingest rejected
5. StateManager tests: each reset method clears correct entries
6. Daemon tests: `_handle_commands()` dispatches correctly, failures don't crash
7. Daemon tests: `_reingest_poll_cycle()` respects timeout, returns partial result
8. Daemon tests: watcher is never stuck paused (try/finally)
9. Pusher tests: **update existing `test_heartbeat.py`** — `send_heartbeat()` now returns `dict` not `bool`
10. Daemon tests: handles missing `commands` key in heartbeat response (backward compat with old server)

### Manual end-to-end smoke test
```bash
# 1. Start infra
docker compose -f analysis-feb2026/docker-compose.yml up -d

# 2. Run migration
uv run python -c "
import asyncio
from qc_trace.db.migrations import ensure_schema
asyncio.run(ensure_schema('postgresql://localdev@localhost:15432/trace'))
"

# 3. Start server + daemon (separate terminals)
uv run python -m qc_trace.server
uv run python -m qc_trace.daemon

# 4. Verify heartbeat returns empty commands
curl -s localhost:8080/api/heartbeat  # (or check daemon logs)

# 5. Trigger reingest
curl -X POST localhost:8080/api/reingest \
  -H "Authorization: Bearer <admin-key>" \
  -H "Content-Type: application/json" \
  -d '{"device_id": "<id>", "scope": "all"}'

# 6. Verify command was created
psql -h localhost -p 15432 -U localdev trace \
  -c "SELECT * FROM daemon_commands;"

# 7. Wait for next heartbeat (~5min) or check daemon logs for command pickup

# 8. Verify file_progress was cleared
psql -h localhost -p 15432 -U localdev trace \
  -c "SELECT * FROM file_progress;"

# 9. Verify command completed
psql -h localhost -p 15432 -U localdev trace \
  -c "SELECT id, status, result, completed_at FROM daemon_commands;"
```
