# Dobby: Meta-Agent Orchestrator for Multi-Project Development

**Date:** 2026-03-21
**Status:** Draft
**Author:** Simon Yu + Claude

## Problem

Simon manages 40+ project directories across multiple active research projects (SPARE, STLM, MemAgent, EGAO, etc.). He is the serial bottleneck — holding ideas, context, and execution capacity across all projects. Context-switching destroys throughput. Ideas get lost between sessions. GPU jobs require manual SSH and monitoring.

## Solution

Dobby is a persistent daemon powered by Claude that acts as a meta-agent orchestrator. It manages a task queue across all projects, dispatches Claude Code agents to execute work autonomously, monitors GPU jobs on remote servers, and reports results back via CLI and Slack.

## Design Principles

- **Queue-first**: Ideas never get lost. Every idea becomes a task in a persistent queue.
- **Fire-and-forget by default**: Most tasks run autonomously. Opt-in checkpoints for high-risk work.
- **Project-aware**: Each project has a registered profile with repo location, context, and infra config. Dispatched agents inherit the project's CLAUDE.md automatically.
- **Local-first, remote-ready**: Runs on a laptop day 1. Migrates to a VPS with near-zero changes.
- **Interface-flexible**: CLI as primary, Slack for quick idea capture, web dashboard optional.

## Core Data Model

### Project Profile

```yaml
project:
  name: "spare"
  repo: "/Users/simonyu/local/local_orby/spiral_refactor"
  git_remote: "git@github.com:..."
  context: "Dual-role RL framework for SPARE training"
  claude_md: true
  infra:
    - name: "gpu-box-1"
      type: ssh
      host: "simon@gpu-box-1"
      gpus: "4xA100"
      conda_env: "/home/simon/miniconda3/envs/spare"
    - name: "modal"
      type: modal
      gpu: "H100"
```

Projects are registered once via `dobby register`. Dobby reads existing CLAUDE.md files in each project automatically.

### Task

```yaml
task:
  id: "t-20260321-001"
  project: "spare"
  description: "Add learning rate warmup scheduler to training loop"
  mode: "quick"                   # "quick" (skip planning) or "planned"
  plan: []                        # populated after drafting phase for planned tasks
  exit_criteria: "PR ready"       # "PR ready", "report results", "run and collect logs", custom
  checkpoint: false               # true = ask before big decisions
  priority: "normal"              # low / normal / high / urgent
  status: "pending"               # drafting → planned → pending → running → completed / failed / cancelled
                                   # recurring tasks: active / paused / cancelled
  source: "slack"                 # "slack" or "cli"
  planning_thread: null           # slack thread ID or claude-code session ID
  planning_messages: []            # conversation history for planned tasks (stored in DB)
  schedule:
    type: "once"                  # "once", "cron", "interval", "after"
    expr: null                    # cron expression (e.g., "0 3 * * *")
    every: null                   # interval (e.g., "30m")
    until: null                   # optional expiry for recurring tasks
    depends_on: null              # task ID for "after" type
  created_at: "2026-03-21T14:00:00Z"
```

#### Task Lifecycle

```
drafting → planned → pending → running → completed / failed
```

- **drafting**: Interactive planning session between user and agent. Happens in a Slack thread or Claude Code session. The agent and user align on scope, break work into steps, agree on the plan.
- **planned**: Plan is locked in. Task has concrete steps. Waiting in queue.
- **pending**: Simple/quick tasks skip straight here.
- **running**: Agent is actively working on the task.
- **completed / failed**: Done. Results are stored.

Quick tasks (`mode: "quick"`) skip `drafting` and `planned`, going directly to `pending`.

#### Scheduling

Tasks can be scheduled in four ways:

- **once** (default): Run once when dequeued.
- **cron**: Run on a cron schedule (e.g., nightly evals at 3am).
- **interval**: Run every N minutes/hours (e.g., check training loss every 30m).
- **after**: Run after another task completes (single dependency only — e.g., "run eval after training finishes"). Complex multi-dependency DAGs are deferred to v2.

Recurring tasks accumulate results as a list for historical tracking.

### Result

```yaml
result:
  task_id: "t-20260321-001"
  status: "completed"
  summary: "Added cosine warmup scheduler. PR #42 ready."
  artifacts:
    - type: "pr"
      url: "https://github.com/.../pull/42"
    - type: "diff"
      path: "results/t-20260321-001/changes.diff"
    - type: "log"
      path: "results/t-20260321-001/agent.log"
  metrics: {}
  gpu_info: null
  completed_at: "2026-03-21T15:30:00Z"
```

For GPU jobs, results include structured metrics and GPU info:

```yaml
metrics:
  final_loss: 0.342
  eval_accuracy: 0.871
  epochs: 3
  wall_time: "2h14m"
gpu_info:
  server: "gpu-box-1"
  gpu_type: "4xA100"
  duration: "2h14m"
```

## Architecture

### Components

```
┌─────────────────────────────────────────────────┐
│                  Dobby Daemon                    │
│  ┌───────────┐ ┌────────────┐ ┌──────────────┐ │
│  │  Queue     │ │ Dispatcher │ │  Collector   │ │
│  │  Manager   │ │            │ │              │ │
│  └─────┬─────┘ └─────┬──────┘ └──────┬───────┘ │
│        │              │               │          │
│        └──────────────┼───────────────┘          │
│                       │                          │
│                ┌──────┴──────┐                   │
│                │ DobbyStore  │ (interface)        │
│                └──────┬──────┘                   │
│                       │                          │
└───────────────────────┼──────────────────────────┘
                        │
              ┌─────────┴─────────┐
              │  SQLite (v1)      │
              │  MongoDB (later)  │
              └───────────────────┘

External interfaces:
  ├── Dobby CLI (local terminal)
  ├── Slack Bot (Slack Bolt)
  └── Web Dashboard (optional, later)
```

### 1. Dobby Core (daemon)

A Python process that runs persistently. Main loop: dequeue → dispatch → monitor → collect.

```
dobby/
  core/
    daemon.py          # main loop
    queue.py           # task queue with priority + scheduling
    dispatcher.py      # spawns claude processes, manages concurrency
    collector.py       # parses results, stores artifacts
    projects.py        # project registry, context loading
  store/
    interface.py       # DobbyStore protocol
    sqlite.py          # SQLite implementation (v1)
    mongodb.py         # MongoDB implementation (later)
  slack/
    bot.py             # Slack Bolt app
    parser.py          # NL → task parsing via Claude API
  cli/
    main.py            # CLI entry point
    commands.py        # add, plan, status, results, etc.
  config/
    projects.yaml      # registered project profiles
    dobby.yaml         # global config
  db/
    dobby.db           # SQLite database
  results/             # per-task artifact storage
```

### 2. Dobby CLI

Thin wrapper that talks to the daemon via API (localhost when local, SSH tunnel or HTTP when remote).

```bash
dobby add "implement X" --project spare                    # quick task
dobby add "run eval" --project spare --cron "0 3 * * *"    # scheduled task
dobby add "run training" --project spare --infra gpu-box-1 # GPU task
dobby plan "refactor Y" --project memagent                 # planned task
dobby status                                                # all tasks
dobby status spare                                          # project tasks
dobby results t-20260321-001                                # specific result
dobby results spare --last 5                                # recent results
dobby projects                                              # list projects
dobby register ./path/to/repo --name myproject              # add project
dobby logs t-20260321-001                                   # tail agent logs
dobby cancel t-20260321-001                                 # cancel task
dobby retry t-20260321-001                                   # re-queue failed task
dobby stats                                                  # token usage, cost, queue stats
dobby pause / dobby resume                                   # pause/resume queue
```

### 3. Slack Bot

Lightweight Python app using Slack Bolt.

**Capabilities:**
- Natural language task creation: `@dobby add to SPARE: implement warmup scheduler, make a PR`
- Planning conversations in threads: `@dobby plan: refactor orchestrator in memagent`
- Status queries: `@dobby status`
- Completion/failure notifications pushed to user's DM or a channel

**NL parsing:** A single Claude API call interprets the user's message into a structured task. Not a full agent session — just extraction of project, description, exit criteria, priority, and schedule from natural language.

**Planning flow:**
1. User sends `@dobby plan: ...`
2. Dobby creates a thread, spawns a Claude session for planning
3. User and agent discuss in the thread. Dobby maintains conversation history in the DB (stored per-task as `planning_messages`), re-sending full history with each Claude API call to simulate a persistent session.
4. User reacts with checkmark to approve the plan
5. Task moves from `drafting` → `planned` → `pending`

### 4. Web Dashboard (v2, optional)

Minimal single-page app:
- Queue view: pending / running / completed / failed
- Per-project view: recent tasks and results
- Task detail: logs, diffs, metrics
- Not a v1 priority — CLI + Slack cover 90% of needs

## Daemon API Protocol

The daemon exposes a **REST API over HTTP** on a configurable port (default: `8420`).

- **Local mode**: Binds to `127.0.0.1:8420`. CLI talks to `localhost:8420`.
- **Remote mode**: Binds to `0.0.0.0:8420` behind SSH tunnel or API key auth.
- All endpoints accept/return JSON.

### Endpoints

```
POST   /tasks              # create task
GET    /tasks               # list tasks (query params: project, status, last_n)
GET    /tasks/:id           # get task detail
PATCH  /tasks/:id           # update task (cancel, update priority, etc.)
DELETE /tasks/:id           # delete task

GET    /results             # list results (query params: project, last_n)
GET    /results/:task_id    # get result for task

POST   /projects            # register project
GET    /projects             # list projects
GET    /projects/:name      # get project detail

GET    /health              # daemon health check
GET    /stats               # queue stats, running agents, token usage
```

### Authentication

- **v1 (local only)**: No auth. Daemon binds to localhost only.
- **v2 (remote)**: API key in `Authorization: Bearer <key>` header. Key stored in `dobby.yaml`. SSH tunnel is an alternative (no auth needed if tunneled).

### Implementation

Lightweight HTTP server using `aiohttp` (the daemon is asyncio-based — see Process Model below).

## Daemon Process Model

The daemon uses **Python asyncio** as its concurrency model:

- **Main event loop**: runs the queue polling, scheduling, and API server concurrently
- **Agent dispatch**: each running agent is a subprocess managed via `asyncio.create_subprocess_exec`, with stdout/stderr captured asynchronously
- **Slack bot**: runs in the same event loop (Slack Bolt has async support via `AsyncApp`)
- **SQLite access**: all DB access goes through the daemon process (single writer). SQLite is opened in **WAL mode** for safe concurrent reads.

This avoids threading complexity. The daemon is I/O-bound (waiting on subprocesses and API calls), which is asyncio's sweet spot.

## Dispatch Mechanism

### How Claude Code is invoked

Dobby dispatches agents using `claude` CLI in **headless mode**:

```bash
claude -p "task prompt" \
  --project-dir /path/to/project \
  --output-format json \
  --max-turns 200 \
  --allowedTools "Bash,Read,Write,Edit,Glob,Grep" \
  2>&1 | tee results/t-xxx/agent.log
```

Key flags:
- `-p "prompt"`: Non-interactive print mode. The full task prompt (description, exit criteria, infra config) is passed here.
- `--project-dir`: Points to the project repo. Claude Code loads the project's CLAUDE.md automatically.
- `--output-format json`: Structured JSON output for reliable parsing by the Collector.
- `--max-turns`: Safety limit to prevent runaway agents.
- `--allowedTools`: Explicitly grant tool permissions so the agent doesn't block on permission prompts.

**Wall-time timeout**: Each agent subprocess has a default wall-clock timeout of 30 minutes (configurable per-task via `--timeout`). GPU Phase 1 (launch only) uses 10 minutes. The timeout is enforced by the daemon, not the agent.

The `--dangerously-skip-permissions` flag is always used — agents run headless with no terminal to approve permission prompts. The `checkpoint` field controls whether the agent should pause and ask the user (via Slack/CLI) before making high-level decisions, not tool-level permissions.

**Fallback**: If Claude Code CLI is unavailable or unreliable, the dispatcher can fall back to the **Anthropic API directly** using the `anthropic` Python SDK with tool use. This is more work to set up (must replicate tool definitions) but eliminates CLI dependency.

### Agent output parsing contract

The Collector parses agent output using a **structured + heuristic hybrid approach**:

1. **Primary (structured)**: When `--output-format json` is used, Claude Code returns a JSON object with `result`, `cost_usd`, `total_tokens`, etc. The task prompt instructs the agent to include a structured summary block at the end:

```
When you are done, output a summary in this exact format:
---DOBBY_RESULT---
{
  "status": "completed" or "failed",
  "summary": "one-line summary of what was done",
  "pr_url": "url or null",
  "files_changed": ["list", "of", "files"],
  "metrics": { ... any numeric results ... },
  "error": "error message if failed, null otherwise"
}
---END_DOBBY_RESULT---
```

2. **Fallback (heuristic)**: If the structured block is missing (agent crashed, forgot, etc.), the Collector:
   - Checks exit code (0 = likely success, non-zero = failure)
   - Scans stdout for PR URLs (`github.com/.*/pull/\d+`)
   - Scans for common error patterns
   - Stores raw stdout as the result summary

3. **Always captured**: Full agent log (stdout + stderr) is saved to `results/t-xxx/agent.log` regardless.

### Token/cost tracking

Every task result includes:
```yaml
cost:
  input_tokens: 45000
  output_tokens: 12000
  total_tokens: 57000
  cost_usd: 0.42
```

Dobby tracks cumulative token usage and cost per project and globally. `dobby stats` shows spending breakdown.

## Dispatch Flow

### Standard task dispatch

```
User ──→ CLI/Slack ──→ Dobby API ──→ Queue (SQLite)
                                        │
                                  daemon loop picks up task
                                        │
                                  Dispatcher reads project profile
                                  Loads context, CLAUDE.md path
                                        │
                                  Spawns: claude -p "task prompt" \
                                          --project-dir /path/to/project \
                                          --output-format json
                                        │
                                  Agent works autonomously
                                  (writes code, runs tests, creates PR)
                                        │
                                  Collector parses JSON output + DOBBY_RESULT block
                                  Extracts summary, PR URL, metrics, cost
                                        │
                                  Result stored in DB + artifacts/
                                        │
                                  Slack notification sent
```

### GPU job dispatch

GPU jobs are split into two phases to handle long-running training without keeping a Claude session alive for hours:

**Phase 1 — Setup & Launch (Claude agent):**
The dispatched agent SSHs into the GPU server, pulls code, configures the run, and launches training as a background process with output piped to a log file. The agent exits after confirming the job is running.

**Phase 2 — Monitor & Collect (Dobby daemon):**
Dobby itself polls the GPU server via SSH to check if the training process is still running. When it completes, Dobby spawns a short Claude agent session to parse the logs, extract metrics, and produce a structured result.

```
Dispatcher ──→ Claude agent (Phase 1)
                  │ SSH into GPU server
                  │ git pull, configure run
                  │ nohup python train.py ... > train.log 2>&1 &
                  │ Confirm PID is running
                  │ Exit with: { "gpu_pid": 12345, "log_path": "/home/.../train.log" }
                  ▼
Dobby daemon ──→ SSH poll loop (Phase 2)
                  │ Every 5 min: check if PID 12345 is still running
                  │ When done: scp train.log, metrics files
                  ▼
Dispatcher ──→ Claude agent (short session)
                  │ Parse logs, extract final metrics
                  │ Produce DOBBY_RESULT
                  ▼
Collector ──→ Store result + notify
```

This avoids keeping a Claude API session alive for hours of training.

**Agent receives (Phase 1):**
```
You are working on project "spare".
Infra config:
  - GPU server: ssh://simon@gpu-box-1
  - Conda env: /home/simon/miniconda3/envs/spare
  - Working dir: /home/simon/projects/spare

Task: "Run SPARE training with lr=1e-4, warmup=500, 3 epochs."
Exit criteria: "Launch training as background process and confirm it's running."

IMPORTANT: Launch the training with nohup and redirect output to a log file.
Report back the PID and log file path in your DOBBY_RESULT block.
Do NOT wait for training to complete.
```

### Concurrency management

- Configurable max concurrent agents in `dobby.yaml` (default: 5)
- **Per-project lock**: Only one agent runs per project at a time by default. This prevents git conflicts from concurrent modifications. Can be overridden with `--allow-concurrent` for read-only tasks (evals, monitoring).
- Priority queue: urgent > high > normal > low
- Excess tasks wait in queue
- Rate limiting: respects Claude API limits, backs off on 429s with exponential delay

## Error Handling & Retry

### Agent failures

| Failure mode | Detection | Response |
|---|---|---|
| Agent exits with non-zero code | Exit code check | Mark task as `failed`, store error in result, notify via Slack |
| Agent times out (exceeds `max_turns` or wall-time limit) | Subprocess timeout | SIGTERM the process, mark `failed`, store partial output |
| Agent crashes (OOM, segfault) | Subprocess exit signal | Mark `failed`, store whatever output was captured |
| API rate limit (429) | Agent stdout / exit code | Re-queue task with exponential backoff (max 3 retries) |
| Network failure during SSH | Agent reports SSH error | Mark `failed`, notify. User can `dobby retry t-xxx` to re-queue. |

### Retry policy

- **Default**: No automatic retry. Failed tasks stay `failed`. User can manually retry with `dobby retry <task_id>`.
- **Opt-in auto-retry**: `dobby add ... --retries 3` allows up to 3 automatic retries with exponential backoff (1min, 5min, 15min).
- Each retry creates a new result entry linked to the same task, so the failure history is preserved.

### Daemon resilience

- **Daemon crash**: On restart, the daemon reads the DB. Tasks that were `running` when the daemon died are marked `failed` with reason `"daemon_restart"`. The user is notified to decide whether to retry.
- **Queue durability**: All task state is in SQLite (WAL mode, fsync on commit). No in-memory-only state. A kill -9 loses at most the current transaction.
- **Graceful shutdown** (`SIGTERM`): Daemon stops picking up new tasks, waits for running agents to complete (up to 60s timeout), then exits. Running agents beyond the timeout are SIGTERM'd and marked `failed`.

### Task cancellation

`dobby cancel <task_id>`:
- **If pending**: Removed from queue, marked `cancelled`.
- **If running**: Sends SIGTERM to the Claude subprocess, waits 10s, then SIGKILL if still alive. Marks `cancelled`. Any uncommitted code changes are left in the working directory (not cleaned up automatically — the user may want to inspect them).
- **If running a GPU job (Phase 2 monitoring)**: SSH into the server, `kill <gpu_pid>`, mark `cancelled`.

### Recurring task lifecycle

- Recurring tasks (cron/interval) have a persistent parent status of `active`. Each run creates a child result.
- If a single run fails, the parent stays `active` — next scheduled run still fires.
- `dobby cancel` on a recurring task stops all future runs (parent → `cancelled`).
- `dobby pause <task_id>` suspends a recurring task without cancelling it.
- The `until` field means "do not start new runs after this datetime."

## Storage Layer

### DobbyStore Protocol

```python
class DobbyStore(Protocol):
    # Tasks
    async def create_task(self, task: Task) -> str: ...
    async def get_task(self, task_id: str) -> Task: ...
    async def update_task(self, task_id: str, **fields) -> None: ...
    async def list_tasks(self, project: str = None, status: str = None) -> list[Task]: ...
    async def get_pending_tasks(self, limit: int = 10) -> list[Task]: ...
    async def get_scheduled_tasks(self) -> list[Task]: ...

    # Results
    async def store_result(self, result: Result) -> None: ...
    async def get_result(self, task_id: str) -> Result: ...
    async def list_results(self, project: str = None, last_n: int = None) -> list[Result]: ...

    # Projects
    async def register_project(self, project: Project) -> None: ...
    async def get_project(self, name: str) -> Project: ...
    async def list_projects(self) -> list[Project]: ...
```

All methods are async to match the asyncio daemon. The SQLite implementation uses `aiosqlite` (or `asyncio.to_thread()` wrapping standard `sqlite3`) to avoid blocking the event loop.

**v1:** `SqliteStore` — zero-setup, single-file database.
**Later:** `MongoStore` — drop-in replacement for multi-machine access.

## Local → Remote Transition

### Day 1: Local mode

```
Your laptop
├── dobby daemon (background process)
├── dobby.db (SQLite)
├── ~/local/local_orby/* (all project repos)
└── claude CLI (dispatched agents run here)
```

### Day N: Remote mode

```
VPS (small Vultr/Hetzner box)
├── dobby daemon (systemd-managed, always on)
├── dobby.db (SQLite) or MongoDB
├── ~/projects/* (shallow git clones)
├── claude CLI
├── Slack bot (always listening)
└── SSH keys (to GPU servers)

Your laptop
├── dobby CLI → talks to VPS
└── Slack (works from anywhere)
```

### Migration steps

1. Push Dobby code to git
2. Clone on VPS, copy config + DB
3. `dobby start --daemon` (systemd manages it)
4. `dobby config set remote vps:8420` on laptop
5. Done — same CLI, same Slack, different host

### Key design decisions enabling this

- Projects have `git_remote` field — VPS clones repos instead of reading local paths
- Daemon exposes API — CLI doesn't care if it's localhost or remote
- Results stored in DB + local artifacts directory — accessible via API from anywhere
- Slack bot is host-agnostic

## v1 Scope

Split into two phases to reduce risk:

### v1a — Core loop (build first)

1. **DobbyStore (SQLite, WAL mode)** — task/result/project CRUD
2. **Data models** — Task, Result, Project as Python dataclasses
3. **Project registry** — `dobby register`, loading CLAUDE.md
4. **Daemon** — asyncio main loop: dequeue → dispatch → monitor → collect
5. **REST API** — aiohttp server on localhost:8420
6. **Dispatcher** — spawn `claude -p` headless processes with `--output-format json`
7. **Collector** — parse DOBBY_RESULT block + heuristic fallback, token/cost tracking
8. **GPU job support** — two-phase dispatch (agent launches, daemon monitors via SSH poll)
9. **Error handling** — failure detection, manual retry, graceful shutdown
10. **CLI** — `add`, `plan`, `status`, `results`, `projects`, `register`, `cancel`, `logs`, `retry`, `stats`

### v1b — Slack integration (layer on top)

1. **Slack bot** — Slack Bolt AsyncApp running in daemon event loop
2. **NL parser** — Claude API call to extract task fields from natural language
3. **Notifications** — DM or channel notifications on task completion/failure
4. **Status queries** — `@dobby status` in Slack
5. **Planning threads** — threaded conversation with conversation history persistence

### Deferred to v2

- Web dashboard
- MongoDB backend
- Remote/VPS deployment
- Hybrid mode (local + remote)
- Multi-server GPU scheduling (agent picks infra)
- Task dependency graphs (complex DAGs beyond simple "after")
- Budget controls / spend alerts
- API key auth for remote access
