# Sync prod data to local Docker Postgres

Dump data from production Postgres (`.prod.env`) and restore into the local Docker Postgres (`qctrace-local` container on port 15432, user `localdev`).

**Auto-trigger**: Use this command when the user asks to sync prod data locally, export prod to docker, or refresh local DB.

## Input

The user may optionally specify filters:
- `/sync-prod-to-local` — full sync, all data
- `/sync-prod-to-local --since 2026-02-01` — only sessions with `first_seen >= date`
- `/sync-prod-to-local --until 2026-02-23` — only sessions with `first_seen <= date`
- `/sync-prod-to-local --org pratilipi` — only data for a specific org
- `/sync-prod-to-local --org pratilipi --since 2026-02-01` — combine filters

If no argument: full sync of all data.

## Prerequisites

1. Docker must be running
2. Local postgres container must be up:
   ```bash
   docker compose -f /Users/sagar/work/all-things-quickcall/trace/analysis/docker-compose.yml up -d
   ```
3. Verify container is running:
   ```bash
   docker ps --filter name=qctrace-local
   ```

## Method A: Full sync (no filters)

Uses `pg_dump -Fc` (custom format, compressed) with `pg_restore -j 4` (4 parallel jobs). This is the fastest approach.

### Step 1: Drop and recreate local schema

```bash
psql "postgresql://localdev:localdev@localhost:15432/qc_trace" \
  -c "DROP SCHEMA public CASCADE; CREATE SCHEMA public;"
```

### Step 2: Dump from prod using pg16 docker image

```bash
source /Users/sagar/work/all-things-quickcall/trace/.prod.env
docker run --rm --network host -v /tmp:/tmp postgres:16 \
  pg_dump "$QC_TRACE_DSN" -Fc --no-owner --no-acl \
  -f /tmp/prod_dump.custom
```

### Step 3: Copy dump into container and restore with parallel jobs

```bash
docker cp /tmp/prod_dump.custom qctrace-local:/tmp/dump.custom
docker exec qctrace-local pg_restore -U localdev -d qc_trace \
  --no-owner --no-acl --disable-triggers -j 4 /tmp/dump.custom
```

### Step 4: Verify

```bash
docker exec qctrace-local psql -U localdev -d qc_trace -c "
  SELECT 'sessions' as tbl, count(*) FROM sessions
  UNION ALL SELECT 'messages', count(*) FROM messages
  UNION ALL SELECT 'token_usage', count(*) FROM token_usage
  UNION ALL SELECT 'tool_calls', count(*) FROM tool_calls
  UNION ALL SELECT 'tool_results', count(*) FROM tool_results
  ORDER BY 1;"
```

## Method B: Filtered sync (date range and/or org)

When the user specifies `--since`, `--until`, or `--org`, we can't use pg_dump directly. Instead, use SQL COPY with filters.

### Step 1: Ensure local schema exists

```bash
psql "postgresql://localdev:localdev@localhost:15432/qc_trace" \
  -f /Users/sagar/work/all-things-quickcall/trace/qc_trace/db/schema.sql
```

### Step 2: Clear existing data in local

```bash
psql "postgresql://localdev:localdev@localhost:15432/qc_trace" \
  -c "TRUNCATE daemon_commands, tool_results, tool_calls, token_usage, messages, sessions, file_progress, daemon_heartbeats, version_config, schema_version CASCADE;"
```

### Step 3: Build the WHERE clause

Construct the filter based on user input. Examples:

- `--since 2026-02-01`: `WHERE s.first_seen >= '2026-02-01'`
- `--until 2026-02-23`: `WHERE s.first_seen <= '2026-02-23'`
- `--org pratilipi`: `WHERE s.org = 'pratilipi'`
- Combined: `WHERE s.org = 'pratilipi' AND s.first_seen >= '2026-02-01' AND s.first_seen <= '2026-02-23'`

### Step 4: Export filtered data from prod as CSV via COPY

Use `\COPY` to export filtered data. Run these from the prod connection:

```bash
source /Users/sagar/work/all-things-quickcall/trace/.prod.env

# Sessions (filtered)
psql "$QC_TRACE_DSN" -c "\COPY (SELECT * FROM sessions s <WHERE_CLAUSE>) TO '/tmp/sync_sessions.csv' WITH CSV HEADER"

# Messages for those sessions
psql "$QC_TRACE_DSN" -c "\COPY (SELECT m.* FROM messages m JOIN sessions s ON m.session_id = s.id <WHERE_CLAUSE>) TO '/tmp/sync_messages.csv' WITH CSV HEADER"

# Token usage for those messages
psql "$QC_TRACE_DSN" -c "\COPY (SELECT tu.* FROM token_usage tu JOIN messages m ON tu.message_id = m.id JOIN sessions s ON m.session_id = s.id <WHERE_CLAUSE>) TO '/tmp/sync_token_usage.csv' WITH CSV HEADER"

# Tool calls
psql "$QC_TRACE_DSN" -c "\COPY (SELECT tc.* FROM tool_calls tc JOIN messages m ON tc.message_id = m.id JOIN sessions s ON m.session_id = s.id <WHERE_CLAUSE>) TO '/tmp/sync_tool_calls.csv' WITH CSV HEADER"

# Tool results
psql "$QC_TRACE_DSN" -c "\COPY (SELECT tr.* FROM tool_results tr JOIN messages m ON tr.message_id = m.id JOIN sessions s ON m.session_id = s.id <WHERE_CLAUSE>) TO '/tmp/sync_tool_results.csv' WITH CSV HEADER"

# Standalone tables (always full copy — small)
psql "$QC_TRACE_DSN" -c "\COPY daemon_heartbeats TO '/tmp/sync_daemon_heartbeats.csv' WITH CSV HEADER"
psql "$QC_TRACE_DSN" -c "\COPY daemon_commands TO '/tmp/sync_daemon_commands.csv' WITH CSV HEADER"
psql "$QC_TRACE_DSN" -c "\COPY version_config TO '/tmp/sync_version_config.csv' WITH CSV HEADER"
psql "$QC_TRACE_DSN" -c "\COPY schema_version TO '/tmp/sync_schema_version.csv' WITH CSV HEADER"
psql "$QC_TRACE_DSN" -c "\COPY file_progress TO '/tmp/sync_file_progress.csv' WITH CSV HEADER"
```

### Step 5: Load CSVs into local (order matters for FK constraints)

```bash
LOCAL_DSN="postgresql://localdev:localdev@localhost:15432/qc_trace"

# Disable triggers for speed
psql "$LOCAL_DSN" -c "SET session_replication_role = 'replica';"

# Load in FK order: parent tables first
for t in schema_version version_config daemon_heartbeats daemon_commands file_progress sessions messages token_usage tool_calls tool_results; do
  echo "Loading $t..."
  psql "$LOCAL_DSN" -c "\COPY $t FROM '/tmp/sync_${t}.csv' WITH CSV HEADER"
done

# Re-enable triggers
psql "$LOCAL_DSN" -c "SET session_replication_role = 'origin';"
```

### Step 6: Verify counts

```bash
docker exec qctrace-local psql -U localdev -d qc_trace -c "
  SELECT 'sessions' as tbl, count(*) FROM sessions
  UNION ALL SELECT 'messages', count(*) FROM messages
  UNION ALL SELECT 'token_usage', count(*) FROM token_usage
  UNION ALL SELECT 'tool_calls', count(*) FROM tool_calls
  UNION ALL SELECT 'tool_results', count(*) FROM tool_results
  ORDER BY 1;"
```

## Cleanup

After successful sync, clean up temp files:

```bash
rm -f /tmp/prod_dump.custom /tmp/sync_*.csv
docker exec qctrace-local rm -f /tmp/dump.custom
```

Ask the user before cleaning up — they may want to keep the dump.

## Important

- NEVER print or expose credentials — always `source .prod.env` and reference env vars
- Local Docker Postgres: port `15432`, user `localdev`, password `localdev`, db `qc_trace`
- The `pg_dump` must use a postgres:16 docker image (local pg_dump is v14, prod is v16)
- Method A (full sync) is ~10x faster than Method B — prefer it when no filters needed
- For very large filtered exports, the CSV approach with `\COPY` handles embedded newlines and special characters correctly
