# Sync prod data to local Docker Postgres

Dump data from production Postgres (`.prod.env`) and restore into the local Docker Postgres (`qctrace-local` container on port 25432, user `localdev`).

**Auto-trigger**: Use this command when the user asks to sync prod data locally, export prod to docker, or refresh local DB.

## Prerequisites

1. Docker must be running
2. Local postgres container must be up:
   ```bash
   docker compose -f analysis/docker-compose.yml up -d
   ```
3. Verify container is running:
   ```bash
   docker ps --filter name=qctrace-local
   ```

## Steps

### 1. Drop and recreate local schema

```bash
psql "postgresql://localdev:localdev@localhost:25432/qc_trace" \
  -c "DROP SCHEMA public CASCADE; CREATE SCHEMA public;"
```

### 2. Dump from prod using pg16 docker image

```bash
source .prod.env
docker run --rm --network host -v /tmp:/tmp postgres:16 \
  pg_dump "$QC_TRACE_DSN" -Fc --no-owner --no-acl \
  -f /tmp/prod_dump.custom
```

### 3. Copy dump into container and restore with parallel jobs

```bash
docker cp /tmp/prod_dump.custom qctrace-local:/tmp/dump.custom
docker exec qctrace-local pg_restore -U localdev -d qc_trace \
  --no-owner --no-acl --disable-triggers -j 4 /tmp/dump.custom
```

### 4. Verify

```bash
docker exec qctrace-local psql -U localdev -d qc_trace -c "
  SELECT 'sessions' as tbl, count(*) FROM sessions
  UNION ALL SELECT 'messages', count(*) FROM messages
  UNION ALL SELECT 'token_usage', count(*) FROM token_usage
  UNION ALL SELECT 'tool_calls', count(*) FROM tool_calls
  UNION ALL SELECT 'tool_results', count(*) FROM tool_results
  ORDER BY 1;"
```

## Selective / partial sync (e.g. specific orgs or date ranges)

When the user asks to import only specific orgs or date-filtered data **without wiping existing data**:

1. **Do NOT drop the schema** — skip step 1 entirely.
2. **Still use `pg_dump -Fc`** to dump the full prod DB (step 2). Never use CSV exports or `COPY TO STDOUT` — always use the pg_dump/pg_restore workflow.
3. **Restore into a temporary database** inside the same container, then cherry-pick rows:
   ```bash
   # Create temp DB and restore full dump into it
   docker exec qctrace-local psql -U localdev -d postgres \
     -c "DROP DATABASE IF EXISTS qc_trace_tmp; CREATE DATABASE qc_trace_tmp;"
   docker exec qctrace-local pg_restore -U localdev -d qc_trace_tmp \
     --no-owner --no-acl --disable-triggers -j 4 /tmp/dump.custom
   ```
4. **Use dblink** to copy filtered rows from temp DB into the main DB:
   ```bash
   docker exec qctrace-local psql -U localdev -d qc_trace -c "CREATE EXTENSION IF NOT EXISTS dblink;"
   # Then INSERT ... SELECT * FROM dblink('dbname=qc_trace_tmp user=localdev', '<query>') AS t(<col defs>)
   # ON CONFLICT DO NOTHING;
   ```
   - Insert in FK order: `sessions` → `messages` → `token_usage`, `tool_calls`, `tool_results`
   - Filter on `sessions.org` and/or `sessions.first_seen` in the dblink query
   - Use `ON CONFLICT (pkey) DO NOTHING` to avoid duplicates
   - Get column definitions from `information_schema.columns`
5. **Delete stale data first** if refreshing existing orgs:
   ```bash
   # CASCADE on sessions FK deletes related messages, token_usage, tool_calls, tool_results
   DELETE FROM sessions WHERE org LIKE 'sagar%';
   ```
6. **Clean up** the temp DB when done:
   ```bash
   docker exec qctrace-local psql -U localdev -d postgres -c "DROP DATABASE qc_trace_tmp;"
   ```

## Important

- NEVER print or expose credentials — always `source .prod.env` and reference env vars
- NEVER spin up new postgres containers just to run `psql` — use host `psql` (available at `/usr/bin/psql`) or `docker exec qctrace-local psql`
- NEVER use CSV-based workflows (`COPY TO STDOUT`, `\copy`, writing CSV files) — always use `pg_dump -Fc` + `pg_restore`
- Local Docker Postgres: port `25432`, user `localdev`, password `localdev`, db `qc_trace`
- The `pg_dump` must use a postgres:16 docker image (local pg_dump may be older, prod is v16)
- All paths are relative to project root
