# Orchestration Runbook: Running Parallel Claude Code Agents

Companion to `docs/plan-traced-postgres-06022026.md`. This is the step-by-step playbook for setting up, launching, monitoring, and merging work from 4 parallel Claude Code agents using git worktrees.

---

## Prerequisites

```bash
# Verify Claude Code is installed
claude --version

# Verify git worktree support
git worktree list

# Create shared directories
mkdir -p /home/sagar/trace/worktrees
mkdir -p /home/sagar/trace/project_logs
```

---

## Step 1: Create Git Worktrees + Branches

Run from the main repo (`/home/sagar/trace`):

```bash
# Create feature branches
git branch feat/traced-infra
git branch feat/traced-ingest
git branch feat/traced-daemon-v2
git branch feat/traced-ops

# Create worktrees (each gets its own directory + branch)
git worktree add worktrees/infra-db feat/traced-infra
git worktree add worktrees/ingest-server feat/traced-ingest
git worktree add worktrees/daemon-core feat/traced-daemon-v2
git worktree add worktrees/ops-cli feat/traced-ops

# Verify
git worktree list
```

Each worktree is a full working directory sharing the same `.git` history. Agents can commit independently on their branches without conflicts.

---

## Step 2: Create CLAUDE.md in Each Worktree

Each agent needs a worktree-specific `CLAUDE.md` with its task, constraints, and coordination instructions. These are gitignored so they stay local.

### worktrees/infra-db/CLAUDE.md

```markdown
# Agent: infra-db

You are building the infrastructure layer for qc-traced.

## Your Task
Build Docker Postgres setup, database schema, connection pool, and batch writer.
Read the full plan: ../docs/plan-traced-postgres-06022026.md (Worktree 1 section)

## Deliverables
- docker-compose.yml (Postgres 16, port 5432, volume: qc_trace_pgdata)
- qc_trace/db/__init__.py
- qc_trace/db/schema.sql (sessions, messages, token_usage, tool_calls, tool_results)
- qc_trace/db/connection.py (psycopg3 AsyncConnectionPool wrapper)
- qc_trace/db/writer.py (batch COPY writer for NormalizedMessage)
- qc_trace/db/migrations.py (schema version tracking)
- scripts/dev-db.sh (start/stop dev database)
- tests/test_db_writer.py
- Update pyproject.toml with optional [daemon] deps

## Constraints
- Use psycopg[binary]>=3.1.0 and psycopg_pool>=3.1.0
- Use async COPY for batch inserts (not row-by-row)
- Keep qc_trace core zero-dependency; DB deps are optional
- Reuse NormalizedMessage from qc_trace/schemas/unified.py
- All tests must pass: pytest tests/

## Coordination
- You have NO blockers — start immediately
- Write progress to: /home/sagar/trace/project_logs/infra-db_progress.json
- Other agents depend on you. When done, update provides_context with file paths and API details.
- Use conventional commits: feat:, fix:, test:, etc.
- No Claude/Anthropic attribution in commits.
```

### worktrees/ingest-server/CLAUDE.md

```markdown
# Agent: ingest-server

You are building the HTTP ingest server that sits between the daemon and Postgres.

## Your Task
Build a lightweight HTTP API server that accepts NormalizedMessage batches and writes to Postgres.
Read the full plan: ../docs/plan-traced-postgres-06022026.md (Worktree 2 section)

## Deliverables
- qc_trace/server/__init__.py
- qc_trace/server/app.py (HTTP server using stdlib http.server)
- qc_trace/server/handlers.py (POST /ingest, POST /sessions, GET /health)
- qc_trace/server/batch.py (accumulate + flush: 100 rows or 5s timeout)
- Dockerfile for the ingest server
- tests/test_ingest_server.py

## Constraints
- Use stdlib http.server (no Flask/FastAPI)
- Import DB writer from qc_trace.db.writer (built by infra-db agent)
- POST /ingest accepts JSON array of NormalizedMessage dicts
- POST /sessions upserts session records
- GET /health returns {"status": "ok", "db": "connected|disconnected"}
- Batch accumulator: flush at 100 messages or 5 second timeout

## Coordination
- BLOCKED BY: infra-db agent (you need qc_trace/db/ module)
- Check blocker status: cat /home/sagar/trace/project_logs/infra-db_progress.json
- When infra-db status is "completed", read its provides_context for file paths
- Write progress to: /home/sagar/trace/project_logs/ingest-server_progress.json
- Use conventional commits: feat:, fix:, test:, etc.
```

### worktrees/daemon-core/CLAUDE.md

```markdown
# Agent: daemon-core

You are building the daemon that watches coding agent session files and pushes data to the ingest server.

## Your Task
Build the file watcher, source collectors, state tracker, and HTTP pusher.
Read the full plan: ../docs/plan-traced-postgres-06022026.md (Worktree 3 section)

## Deliverables
- qc_trace/daemon/__init__.py
- qc_trace/daemon/config.py (daemon configuration dataclass)
- qc_trace/daemon/watcher.py (file discovery for claude_code, codex_cli, gemini_cli, cursor)
- qc_trace/daemon/collector.py (source-specific collectors using existing transforms)
- qc_trace/daemon/state.py (file processing state, JSON on disk, atomic writes)
- qc_trace/daemon/pusher.py (HTTP POST to ingest server, retry queue, exponential backoff)
- qc_trace/daemon/main.py (main poll-collect-push loop)
- qc_trace/daemon/__main__.py (entry point: python -m qc_trace.daemon)
- tests/test_daemon_*.py

## Key Source Locations to Watch
- Claude Code: ~/.claude/projects/*/*.jsonl
- Codex CLI: ~/.codex/sessions/*/*/*/*/rollout-*.jsonl
- Gemini CLI: ~/.gemini/tmp/*/chats/session-*.json
- Cursor: ~/.cursor/projects/*/agent-transcripts/*.txt

## Existing Transforms to Reuse
- qc_trace/schemas/claude_code/transform.py → transform_claude_v1()
- qc_trace/schemas/codex_cli/transform.py → transform_codex_v1() + CodexTransformContext
- qc_trace/schemas/gemini_cli/transform.py → transform_gemini_v1()
- qc_trace/schemas/cursor/transform.py → transform_cursor_v1()
- qc_trace/utils/cursor_parser.py → parse_agent_transcript()

## Constraints
- Poll interval: 5 seconds (configurable)
- HTTP POST batches to localhost:7777/ingest
- Retry on failure: exponential backoff 1s, 2s, 4s, 8s, 16s, max 60s
- State file: ~/.qc-trace/state.json (atomic writes via tempfile+rename)
- JSONL files: resume from last_line_processed (don't re-read entire file)
- JSON/txt files: hash-based change detection (re-process if hash changes)

## Coordination
- BLOCKED BY: ingest-server agent (you need the API contract)
- Check blocker: cat /home/sagar/trace/project_logs/ingest-server_progress.json
- Write progress to: /home/sagar/trace/project_logs/daemon-core_progress.json
- Use conventional commits: feat:, fix:, test:, etc.
```

### worktrees/ops-cli/CLAUDE.md

```markdown
# Agent: ops-cli

You are building the CLI, logging, monitoring, and service files for qc-traced.

## Your Task
Build the user-facing CLI and operational tooling.
Read the full plan: ../docs/plan-traced-postgres-06022026.md (Worktree 4 section)

## Deliverables
- qc_trace/cli/__init__.py
- qc_trace/cli/traced.py (argparse CLI: start, stop, status, logs, db init)
- qc_trace/daemon/progress.py (progress JSON writer/reader)
- scripts/qc-traced.service (systemd unit)
- scripts/com.quickcall.traced.plist (launchd plist)
- scripts/install.sh (install daemon as OS service)
- scripts/uninstall.sh (remove daemon service)
- tests/test_cli.py

## CLI Commands
- qc-traced start: start daemon in background (write PID file)
- qc-traced stop: send SIGTERM to daemon PID
- qc-traced status: show PID, uptime, server health, files watched, messages count
- qc-traced logs: tail the log file
- qc-traced db init: POST to server to initialize schema

## Constraints
- Daemon log: ~/.qc-trace/qc-traced.log (structured, timestamped)
- PID file: ~/.qc-trace/qc-traced.pid
- systemd: Type=simple, Restart=on-failure, RestartSec=5
- launchd: RunAtLoad=true, KeepAlive=true
- install.sh should detect OS (Linux vs macOS) and install the appropriate service

## Coordination
- NO BLOCKERS — start immediately, work in parallel with infra-db
- Write progress to: /home/sagar/trace/project_logs/ops-cli_progress.json
- Use conventional commits: feat:, fix:, test:, etc.
```

---

## Step 3: Launch Agents

Open 4 terminal sessions (or tmux panes). Launch Claude Code in each worktree.

### Phase 1 — Launch independent agents first

```bash
# Terminal 1: infra-db (no blockers)
cd /home/sagar/trace/worktrees/infra-db
claude --dangerously-skip-permissions

# Terminal 2: ops-cli (no blockers)
cd /home/sagar/trace/worktrees/ops-cli
claude --dangerously-skip-permissions
```

### Phase 2 — Launch after infra-db completes

Monitor infra-db progress:
```bash
# From any terminal, poll until completed
watch -n 10 cat /home/sagar/trace/project_logs/infra-db_progress.json
```

Once `"status": "completed"`:
```bash
# Terminal 3: ingest-server
cd /home/sagar/trace/worktrees/ingest-server

# First pull in infra-db's work so this agent can import qc_trace.db
git merge feat/traced-infra

claude --dangerously-skip-permissions
```

### Phase 3 — Launch after ingest-server completes

Monitor:
```bash
watch -n 10 cat /home/sagar/trace/project_logs/ingest-server_progress.json
```

Once `"status": "completed"`:
```bash
# Terminal 4: daemon-core
cd /home/sagar/trace/worktrees/daemon-core

# Pull in dependencies
git merge feat/traced-infra
git merge feat/traced-ingest

claude --dangerously-skip-permissions
```

---

## Step 4: Monitor All Agents

### Dashboard (single terminal)

```bash
# Quick status of all agents
for f in /home/sagar/trace/project_logs/*_progress.json; do
  echo "=== $(basename $f) ==="
  python3 -c "
import json, sys
d = json.load(open('$f'))
print(f\"  Status: {d.get('status', '?')}\")
print(f\"  Task:   {d.get('current_task', '?')}\")
print(f\"  Done:   {len(d.get('completed_tasks', []))} tasks\")
print(f\"  Errors: {len(d.get('errors', []))}\")
" 2>/dev/null || echo "  (not started)"
  echo
done
```

Save this as `scripts/agent-status.sh` and run with `watch -n 15`:
```bash
watch -n 15 bash /home/sagar/trace/scripts/agent-status.sh
```

### Git activity across worktrees

```bash
# See latest commit in each worktree
git worktree list | while read dir branch rest; do
  echo "=== $dir ($branch) ==="
  git -C "$dir" log --oneline -3 2>/dev/null || echo "  (no commits)"
  echo
done
```

---

## Step 5: Handle Agent Failures / Restarts

If a Claude Code session dies (context exhaustion, crash, network):

```bash
# 1. Check what the agent accomplished
cd /home/sagar/trace/worktrees/<agent-name>
git log --oneline -10

# 2. Check progress file
cat /home/sagar/trace/project_logs/<agent-name>_progress.json

# 3. Restart Claude Code in the same worktree
# The CLAUDE.md and git history give the new session full context
claude --dangerously-skip-permissions
```

The agent picks up where it left off because:
- Git commits preserve all completed work
- The progress JSON shows what's done vs remaining
- The CLAUDE.md contains the full task specification

### If an agent is stuck / looping

```bash
# Kill the session (Ctrl+C or from another terminal)
# Review its recent commits
cd /home/sagar/trace/worktrees/<agent-name>
git log --oneline -5
git diff HEAD~1  # see last change

# Optionally revert a bad commit
git revert HEAD

# Add a hint to CLAUDE.md then restart
echo "\n## Hint\nThe previous approach of X didn't work. Try Y instead." >> CLAUDE.md
claude --dangerously-skip-permissions
```

---

## Step 6: Merge All Worktrees

Once all 4 agents report `"status": "completed"`:

```bash
cd /home/sagar/trace

# 1. Merge in dependency order
git checkout main

git merge --no-ff feat/traced-infra -m "feat: merge infra-db (Docker + Postgres schema + writer)"
# Run tests
pytest tests/

git merge --no-ff feat/traced-ingest -m "feat: merge ingest-server (HTTP API)"
# Run tests
pytest tests/

git merge --no-ff feat/traced-daemon-v2 -m "feat: merge daemon-core (watcher + collectors + pusher)"
# Run tests
pytest tests/

git merge --no-ff feat/traced-ops -m "feat: merge ops-cli (CLI + logging + service files)"
# Run tests
pytest tests/

# 2. If there are conflicts, resolve them manually or launch a 5th Claude session:
# claude --dangerously-skip-permissions
# "Resolve the merge conflicts. Run pytest tests/ to verify."

# 3. Verify full stack
docker compose up -d
python -m qc_trace.cli.traced db init
python -m qc_trace.cli.traced start
python -m qc_trace.cli.traced status
```

### Cleanup worktrees after merge

```bash
git worktree remove worktrees/infra-db
git worktree remove worktrees/ingest-server
git worktree remove worktrees/daemon-core
git worktree remove worktrees/ops-cli

# Delete merged branches
git branch -d feat/traced-infra feat/traced-ingest feat/traced-daemon-v2 feat/traced-ops
```

---

## Step 7: Create PR

```bash
gh pr create \
  --title "feat: qc-traced v2 — Postgres backend with ingest server" \
  --body "$(cat <<'EOF'
## Summary
- Silent daemon watches Claude Code, Codex CLI, Gemini CLI, and Cursor session files
- Pushes normalized data via HTTP to an ingest server
- Ingest server batch-writes to Postgres via async COPY
- CLI for start/stop/status/logs
- systemd + launchd service files with install scripts

## Built by 4 parallel Claude Code agents
- infra-db: Docker + Postgres schema + DB writer
- ingest-server: HTTP ingest API
- daemon-core: File watcher + collectors + HTTP pusher
- ops-cli: CLI + logging + service files

## Test plan
- [ ] docker compose up -d starts Postgres
- [ ] qc-traced db init creates schema
- [ ] qc-traced start begins watching files
- [ ] New session files appear in Postgres within 10s
- [ ] qc-traced status shows healthy state
- [ ] pytest tests/ passes
EOF
)"
```

---

## Quick Reference: tmux Layout

If you prefer tmux for managing all 4 agents:

```bash
# Create session with 4 panes
tmux new-session -d -s agents -c /home/sagar/trace/worktrees/infra-db
tmux split-window -h -c /home/sagar/trace/worktrees/ops-cli
tmux split-window -v -c /home/sagar/trace/worktrees/ingest-server
tmux select-pane -t 0
tmux split-window -v -c /home/sagar/trace/worktrees/daemon-core

# Attach
tmux attach -t agents

# In each pane, run: claude --dangerously-skip-permissions
```

### Pane layout:
```
┌─────────────────┬─────────────────┐
│   infra-db      │    ops-cli      │
│   (Agent 1)     │    (Agent 4)    │
├─────────────────┼─────────────────┤
│   daemon-core   │  ingest-server  │
│   (Agent 3)     │    (Agent 2)    │
└─────────────────┴─────────────────┘
```

---

## Timeline Estimate

```
Phase 1: infra-db + ops-cli in parallel     (~1-2 hours)
Phase 2: ingest-server                       (~1 hour)
Phase 3: daemon-core                         (~1-2 hours)
Phase 4: merge + integration testing         (~30 min)
─────────────────────────────────────────────
Total:                                       ~3-5 hours active
```

The agents can run unattended. Check in periodically via the progress dashboard.
