# Loom Phase 1 — Implementation Plan

**Project:** Loom — Multi-Agent Project Orchestration System
**Scope:** Phase 1 (Foundation)
**PRD:** `PRD-loom-v1.1.md` in this directory (read it first for full architecture context)
**Status:** Ready for implementation

---

## What Is Loom

Loom is a production-grade project orchestration system for large, parallel software projects executed by multiple AI agents. It provides:

- A **persistent, dependency-aware task graph** backed by PostgreSQL (source of truth) and Redis (live cache + event bus)
- A **reliable inter-agent message bus** using Redis pub/sub, streams, and queues
- An **MCP server** exposing all of the above as Claude Code-native tools
- A **CLI** for human interaction (`loom init`, `loom up`, `loom status`, etc.)

Installed globally via `pipx install loom-agents`. Runs locally via Docker Compose.

## Phase 1 Goal

> A Claude Code agent connects to the Loom MCP server, calls `loom_ready`, claims a manually-created task, executes it, and marks it done. State persists across MCP server restarts.

---

## Resolved Architecture Decisions

| Decision | Choice | Notes |
|----------|--------|-------|
| Python version | **3.12** | Modern type hints (`str \| None`), best perf |
| Dev tool | **uv** | For env + dependency management during development |
| Distribution | **pipx + PyPI** as `loom-agents` | Global install, no venv friction |
| Build backend | **hatchling** | Standard, works well with uv |
| Data models | **Pydantic BaseModel** | Not dataclass — better serialization for MCP tool I/O |
| Dev setup | **Native MCP server + Docker DBs** | MCP runs as Python process; Postgres + Redis in Docker |
| MCP transport | **stdio** (local) / **SSE** (cloud) | Claude Code spawns MCP server as subprocess via stdio |
| MCP registration | **`.mcp.json`** at project root | Auto-written by `loom up`; Claude Code reads it automatically |
| Testing | **pytest + testcontainers** | Integration tests against real Postgres/Redis in Docker |
| Build order | **Bottom-up** | db → graph → bus → mcp → cli |

## Prerequisites

- **Docker Desktop** installed and running
- **uv** installed (`curl -LsSf https://astral.sh/uv/install.sh | sh`)
- **Python 3.12+** (uv can install this: `uv python install 3.12`)

---

## Python Dependencies

```toml
# pyproject.toml [project] dependencies
dependencies = [
    "asyncpg>=0.30.0",
    "redis[hiredis]>=5.0.0",
    "fastmcp>=2.0.0",
    "click>=8.1.0",
    "pydantic>=2.5.0",
    "pyyaml>=6.0",
]

[dependency-groups]
dev = [
    "pytest>=8.0",
    "pytest-asyncio>=0.24.0",
    "testcontainers[postgres,redis]>=4.0",
]
```

---

## Package Structure

```
loom/
├── __init__.py
├── cli.py                  # Click CLI: loom init, up, down, status
├── config.py               # 3-layer config: global → project → env vars
├── ids.py                  # Task ID generation: loom-{8hex}
├── exceptions.py           # LoomError hierarchy
│
├── graph/
│   ├── __init__.py
│   ├── task.py             # Task/TaskStatus/Priority Pydantic models
│   ├── store.py            # ALL Postgres CRUD for tasks (only writer)
│   ├── cache.py            # Redis sync layer (only reader, falls back to store)
│   ├── deps.py             # Dependency resolution + ready queue management
│   └── project.py          # Project CRUD
│
├── bus/
│   ├── __init__.py
│   ├── channels.py         # ALL Redis key/channel name functions (no literals elsewhere)
│   ├── events.py           # EventType enum
│   ├── publisher.py        # Redis publish helpers (stream + pub/sub)
│   ├── subscriber.py       # Redis subscribe helpers (minimal Phase 1)
│   └── queue.py            # Escalation queue (LPUSH/BRPOP)
│
├── db/
│   ├── __init__.py
│   ├── connection.py       # asyncpg pool setup/teardown
│   └── migrations/
│       ├── __init__.py
│       └── 001_initial.sql # Full schema (see PRD Section 4)
│
└── mcp/
    ├── __init__.py
    ├── __main__.py          # Entry point: mcp.run(transport="stdio")
    ├── server.py            # FastMCP server with lifespan (pool + redis)
    └── tools.py             # 11 MCP tool implementations (thin, ≤15 lines each)
```

### Module Contracts (Critical)

- **`graph/store.py`** — The ONLY module that writes to Postgres for task data. All modules needing persistent data go through store.py.
- **`graph/cache.py`** — The ONLY module that reads from Redis for task data. Falls back to store.py on cache miss, then syncs Redis. Callers never choose Redis vs Postgres.
- **`bus/channels.py`** — ALL Redis key and channel name patterns as typed functions. No string literals like `"loom:tasks:ready"` appear anywhere else in the codebase.
- **`mcp/tools.py`** — Tool functions ≤15 lines each. Business logic belongs in graph/ or bus/. Tools are thin coordinators only.

---

## Build Steps (Ordered)

### Step 0: Project Scaffolding

**Files:** `pyproject.toml`, `.gitignore`, all `__init__.py` files, `loom/ids.py`, `loom/exceptions.py`, `tests/conftest.py`

**`pyproject.toml`:**
- Build backend: hatchling
- Package name: `loom-agents`
- Entry point: `[project.scripts] loom = "loom.cli:cli"`
- `requires-python = ">=3.12"`
- All dependencies listed above
- `[tool.pytest.ini_options] asyncio_mode = "auto"`

**`loom/ids.py`:**
```python
import secrets

def task_id() -> str:
    """Generate conflict-free task ID: loom-{8hex}."""
    return f"loom-{secrets.token_hex(4)}"
```

**`loom/exceptions.py`:**
```python
class LoomError(Exception):
    """Base for all Loom errors."""

class TaskNotFoundError(LoomError): ...
class TaskStateError(LoomError): ...       # invalid status transition
class ProjectNotFoundError(LoomError): ...
class DependencyCycleError(LoomError): ...
class ClaimConflictError(LoomError): ...   # task already claimed
```

---

### Step 1: Configuration System

**File:** `loom/config.py`
**Depends on:** Step 0

Pydantic models for config sections:
```python
class DatabaseConfig(BaseModel):
    url: str = "postgresql://loom:loom_local@localhost:5432/loom"

class RedisConfig(BaseModel):
    url: str = "redis://localhost:6379"

class McpConfig(BaseModel):
    port: int = 8765
    host: str = "0.0.0.0"

class LoomConfig(BaseModel):
    project_name: str = ""
    project_id: str = ""
    database: DatabaseConfig = DatabaseConfig()
    redis: RedisConfig = RedisConfig()
    mcp: McpConfig = McpConfig()
    log_level: str = "INFO"
```

`load_config(project_dir: Path | None = None) -> LoomConfig`:
1. Start with defaults
2. Deep-merge global config (`~/.loom/config.yaml`)
3. Deep-merge project config (`.loom/config.yaml` relative to `project_dir` or cwd)
4. Apply env var overrides: `LOOM_DATABASE_URL`, `LOOM_REDIS_URL`, `LOOM_MCP_PORT`, `LOOM_LOG_LEVEL`
5. Return `LoomConfig.model_validate(config_dict)`

---

### Step 2: Database Connection + Migrations

**Files:** `loom/db/connection.py`, `loom/db/migrations.py`, `loom/db/migrations/001_initial.sql`
**Depends on:** Steps 0, 1

**`connection.py`** — asyncpg pool singleton:
- `init_pool(config: LoomConfig) -> asyncpg.Pool` — creates pool (min_size=2, max_size=10)
- `get_pool() -> asyncpg.Pool` — returns initialized pool or raises RuntimeError
- `close_pool()` — closes pool, resets to None

**`migrations.py`** — simple migration runner:
- Creates `_loom_migrations` tracking table if not exists
- Reads `.sql` files from `loom/db/migrations/` sorted by filename
- Applies unapplied migrations in a transaction
- Idempotent — safe to run on every startup

**`001_initial.sql`** — full schema from PRD Section 4:
- Tables: `projects`, `tasks`, `task_deps`, `events`, `escalations`, `skill_runs`
- All indexes from PRD Section 4.2
- Task statuses: `pending|claimed|done|failed|blocked|epic`
- Priorities: `p0|p1|p2`
- Task IDs: TEXT (format `loom-{8hex}`)
- Project IDs: UUID
- Context/output fields: JSONB

---

### Step 3: Domain Models

**File:** `loom/graph/task.py`
**Depends on:** Step 0

```python
class TaskStatus(StrEnum):
    PENDING = "pending"
    CLAIMED = "claimed"
    DONE = "done"
    FAILED = "failed"
    BLOCKED = "blocked"
    EPIC = "epic"

class Priority(StrEnum):
    P0 = "p0"
    P1 = "p1"
    P2 = "p2"

PRIORITY_SCORES = {Priority.P0: 300, Priority.P1: 200, Priority.P2: 100}
```

**`Task(BaseModel)`** fields:
- `id: str` (default_factory=generate_task_id)
- `project_id: str`
- `title: str`
- `status: TaskStatus` (default pending)
- `priority: Priority` (default p1)
- `assignee: str | None`
- `parent_id: str | None`
- `context: dict` (JSONB — free-form, no required fields)
- `output: dict` (JSONB)
- `done_when: str | None`
- `created_at, updated_at, claimed_at, done_at: datetime | None`
- `depends_on: list[str]` (populated from task_deps on read, not stored in tasks table)

Class method: `Task.from_record(dict)` → `cls.model_validate(dict(record))`

**`ProjectStatus(BaseModel)`**: project_id, project_name, counts (dict[str,int]), ready_count, blocked_tasks (list[str]), open_escalations (int)

---

### Step 4: Postgres CRUD (Store Layer)

**Files:** `loom/graph/store.py`, `loom/graph/project.py`
**Depends on:** Steps 2, 3

**`store.py`** — all functions take `pool: asyncpg.Pool` as first param:

| Function | Description |
|----------|-------------|
| `create_task(pool, task)` | Insert task + dep edges in single transaction. Returns Task. |
| `get_task(pool, task_id)` | Fetch task by ID, joins task_deps for depends_on list. Returns Task or None. |
| `claim_task(pool, task_id, agent_id)` | **SELECT FOR UPDATE SKIP LOCKED**. Atomically claims pending task. Returns Task or None if unavailable. |
| `complete_task(pool, task_id, output)` | Sets status=done, stores output JSONB, sets done_at=NOW(). |
| `fail_task(pool, task_id, reason)` | Sets status=failed. |
| `update_task(pool, task_id, **fields)` | Updates mutable fields only (title, context, priority, done_when). |
| `update_task_status(pool, task_id, status)` | Direct status change (used by deps.py for blocked→pending). |
| `get_ready_tasks(pool, project_id, priority?, limit?)` | Tasks that are pending AND have no unresolved deps. SQL LEFT JOIN on task_deps. |
| `get_dependents(pool, task_id)` | Task IDs that depend on this task (from task_deps). |
| `get_task_dependencies(pool, task_id)` | Task IDs this task depends on (from task_deps). |
| `replace_dependencies(pool, task_id, depends_on)` | Delete old deps, insert new ones. |
| `list_active_tasks(pool, project_id)` | All tasks NOT in done/failed status. For cache rebuild. |
| `list_all_tasks(pool, project_id)` | All tasks. For graph display. |
| `create_escalation(pool, project_id, task_id, message)` | Insert into escalations table. |
| `record_event(pool, project_id, event_type, task_id?, agent_id?, payload)` | Insert into events table (audit log). |

**Claim implementation** (critical — from PRD):
```python
async def claim_task(pool, task_id, agent_id):
    async with pool.acquire() as conn:
        async with conn.transaction():
            row = await conn.fetchrow(
                "SELECT * FROM tasks WHERE id = $1 AND status = 'pending' FOR UPDATE SKIP LOCKED",
                task_id,
            )
            if row is None:
                return None
            updated = await conn.fetchrow(
                """UPDATE tasks SET status='claimed', assignee=$1,
                   claimed_at=NOW(), updated_at=NOW() WHERE id=$2 RETURNING *""",
                agent_id, task_id,
            )
    return Task.from_record(dict(updated))
```

**`project.py`**: `create_project(pool, name, desc?)`, `get_project(pool, project_id)`, `get_project_status(pool, project_id)`

---

### Step 5: Redis Bus Layer

**Files:** `loom/bus/channels.py`, `loom/bus/events.py`, `loom/bus/publisher.py`, `loom/bus/subscriber.py`, `loom/bus/queue.py`
**Depends on:** Step 0 (can be built in parallel with Step 4)

**`channels.py`** — typed functions, no hardcoded strings:
```python
def task_key(project_id, task_id) -> str:     # "loom:{pid}:task:{tid}"
def status_set(project_id, status) -> str:    # "loom:{pid}:tasks:{status}"
def ready_queue(project_id) -> str:           # "loom:{pid}:tasks:ready"
def event_stream(project_id) -> str:          # "loom:{pid}:events"
def updates_channel(project_id) -> str:       # "loom:{pid}:tasks:updates"
def escalation_channel(project_id) -> str:    # "loom:{pid}:escalations"
def agent_channel(project_id, agent_id) -> str: # "loom:{pid}:messages:{aid}"
def broadcast_channel(project_id) -> str:     # "loom:{pid}:broadcast"
```

**`events.py`** — EventType enum:
```
task.created, task.claimed, task.done, task.failed, task.blocked,
task.updated, task.unblocked, message.sent, escalation.raised,
project.decomposed (Phase 2), project.complete, escalation.resolved
```

**`publisher.py`**:
- `publish_event(redis, project_id, event_type, payload)` — XADD to stream + PUBLISH to pub/sub
- `publish_escalation(redis, project_id, task_id, message)` — PUBLISH to escalation channel
- `send_agent_message(redis, project_id, to, message, thread_id?)` — PUBLISH to agent channel

**`subscriber.py`** — minimal Phase 1 skeleton (async generator for event stream)

**`queue.py`** — `push_escalation(redis, project_id, payload)` / `pop_escalation(redis, project_id, timeout?)`

---

### Step 6: Redis Cache Layer

**File:** `loom/graph/cache.py`
**Depends on:** Steps 4, 5

| Function | Description |
|----------|-------------|
| `sync_task(redis, task)` | Write task to Redis hash + update status sets (SREM from all, SADD to current) |
| `add_to_ready_queue(redis, project_id, task_id, priority)` | ZADD to sorted set with priority score |
| `remove_from_ready_queue(redis, project_id, task_id)` | ZREM from sorted set |
| `get_task(redis, pool, project_id, task_id)` | HGETALL from Redis → on miss: read from Postgres, sync, return |
| `get_ready_tasks(redis, pool, project_id, limit?, priority?)` | ZREVRANGE from sorted set → on empty: fallback to Postgres, rebuild |
| `rebuild_cache(redis, pool, project_id)` | Full rebuild: read all active tasks from Postgres, sync each to Redis, rebuild ready queue |
| `rebuild_ready_queue(redis, pool, project_id)` | Delete + rebuild sorted set from Postgres |

**Gotcha:** Redis hashes store everything as strings. `sync_task` must `json.dumps()` dict/list fields. The reverse deserialization function must `json.loads()` them back.

---

### Step 7: Dependency Resolution

**File:** `loom/graph/deps.py`
**Depends on:** Steps 4, 6

| Function | Description |
|----------|-------------|
| `check_and_unblock(pool, redis, completed_task_id, project_id)` | Find dependents of completed task. For each, check if ALL their deps are done. If so: update status blocked→pending, sync cache, add to ready queue. Returns list of unblocked task IDs. |
| `detect_cycle(pool, task_id, new_deps)` | Iterative DFS from each new_dep looking for task_id. Returns True if cycle found. Run BEFORE adding deps. |
| `compute_initial_status(pool, depends_on)` | No deps → pending. All deps done → pending. Any dep not done → blocked. |

---

### Step 8: MCP Server + Tools

**Files:** `loom/mcp/server.py`, `loom/mcp/tools.py`, `loom/mcp/__main__.py`
**Depends on:** All previous steps

**`server.py`** — FastMCP with lifespan:
```python
@asynccontextmanager
async def lifespan():
    config = load_config(Path(os.environ.get("LOOM_PROJECT_DIR", ".")))
    pool = await init_pool(config)
    await run_migrations(pool)
    redis = Redis.from_url(config.redis.url, decode_responses=True)
    if config.project_id:
        await rebuild_cache(redis, pool, config.project_id)
    yield {"pool": pool, "redis": redis, "config": config}
    await redis.aclose()
    await close_pool()

mcp = FastMCP(name="loom", instructions="...", lifespan=lifespan)
```

**`tools.py`** — 11 tools registered via `@mcp.tool`:

| Tool | Delegates To | Key Behavior |
|------|-------------|--------------|
| `loom_ready(priority?, limit?)` | cache.get_ready_tasks | Returns list of task dicts |
| `loom_claim(task_id, agent_id)` | store.claim_task + cache.sync + publisher | Atomic claim via SELECT FOR UPDATE SKIP LOCKED |
| `loom_done(task_id, output)` | store.complete + cache.sync + deps.check_and_unblock | Cascades: unblocks dependents |
| `loom_fail(task_id, reason)` | store.fail + cache.sync + escalation | Creates escalation record |
| `loom_escalate(task_id, message)` | store.update_status(blocked) + escalation | Flags task as blocked |
| `loom_create(title, context, depends_on?, priority?, parent_id?, done_when?)` | deps.compute_initial_status + store.create + cache.sync | Auto-determines pending vs blocked |
| `loom_status(task_id?)` | cache.get_task or project.get_project_status | Task detail or project overview |
| `loom_message(to, message, thread_id?)` | publisher.send_agent_message | Direct message to agent |
| `loom_decompose(goal, spec_path?, confirm?)` | — | **Phase 2 stub**: returns not_implemented |
| `loom_graph(format?)` | store.list_all_tasks | json, mermaid, or summary format |
| `loom_update(task_id, title?, context?, priority?, depends_on?, done_when?)` | store.update + cache.sync | Mutable fields only, cycle detection on deps |

Every tool: writes Postgres first → syncs Redis → publishes event → records audit event.

**`__main__.py`:**
```python
from loom.mcp.server import mcp
mcp.run(transport="stdio")
```

**How stdio works:** Claude Code spawns `python -m loom.mcp.server` as a subprocess. Communication is JSON-RPC over stdin/stdout. FastMCP handles the protocol. The lifespan connects to Postgres + Redis, and the process stays alive for the entire Claude Code session.

---

### Step 9: CLI

**File:** `loom/cli.py`
**Depends on:** Steps 1, 8

**`loom init`:**
1. Create `.loom/` directory structure (config.yaml, skills/, workflows/, logs/)
2. Generate project UUID
3. Write `.loom/config.yaml` with project_name, project_id, default DB/Redis URLs
4. Write `docker-compose.loom.yml` (Postgres 16 + Redis 7 with healthchecks, port mappings 5432/6379, persistent volumes)
5. Write `AGENTS.md` (basic agent instructions referencing Loom tools)
6. Write `.loom/.gitignore` for logs/

**`loom up`:**
1. Verify `docker-compose.loom.yml` exists
2. `docker compose -f docker-compose.loom.yml up -d postgres redis`
3. Wait for healthchecks (retry with backoff)
4. Run database migrations against Postgres
5. Write/update `.mcp.json` at project root:
```json
{
  "mcpServers": {
    "loom": {
      "command": "uv",
      "args": ["run", "--with", "loom-agents", "python", "-m", "loom.mcp.server"],
      "env": { "LOOM_PROJECT_DIR": "<absolute path to project>" }
    }
  }
}
```
6. Print instructions to start Claude Code

**`loom down`:** `docker compose -f docker-compose.loom.yml down`

**`loom status [task-id]`:** Connects directly to Postgres (not MCP), shows project overview or task detail.

**Docker Compose (generated, local only — no MCP service):**
```yaml
services:
  postgres:
    image: postgres:16-alpine
    environment: { POSTGRES_DB: loom, POSTGRES_USER: loom, POSTGRES_PASSWORD: loom_local }
    ports: ["5432:5432"]
    volumes: [postgres-data:/var/lib/postgresql/data]
    healthcheck: { test: ["CMD-SHELL", "pg_isready -U loom"], interval: 2s, timeout: 5s, retries: 10 }
  redis:
    image: redis:7-alpine
    ports: ["6379:6379"]
    volumes: [redis-data:/data]
    command: redis-server --appendonly yes
    healthcheck: { test: ["CMD", "redis-cli", "ping"], interval: 2s, timeout: 5s, retries: 10 }
volumes:
  postgres-data:
  redis-data:
```

---

### Step 10: End-to-End Integration Tests

**File:** `tests/integration/test_e2e.py`
**Depends on:** All steps

**Test 1 — Full Agent Workflow (proves Phase 1 deliverable):**
1. Create project and task in Postgres
2. Rebuild Redis cache (simulates server startup)
3. Call `get_ready_tasks` — verify task appears
4. Call `claim_task` — verify status=claimed
5. Call `complete_task` with output — verify status=done
6. Verify ready queue is empty
7. Flush Redis, rebuild cache (simulates restart)
8. Verify task is still done with correct output (persistence)

**Test 2 — Dependency Cascade:**
1. Create tasks A, B, C where C depends on A and B
2. Verify only A and B are in ready queue
3. Complete A — verify C still blocked
4. Complete B — verify C now unblocked and in ready queue

**Test fixtures (`tests/conftest.py`):**
- Session-scoped: `postgres_container`, `redis_container` (testcontainers)
- Function-scoped: `pool` (asyncpg, runs migrations, cleans tables between tests), `redis` (flushdb between tests)

---

## Verification Checklist

- [ ] `uv run pytest` — all tests pass
- [ ] `loom init` creates correct directory structure and files
- [ ] `loom up` starts Postgres + Redis, runs migrations, writes `.mcp.json`
- [ ] Open Claude Code in project directory — Loom MCP tools appear in tool list
- [ ] Agent calls `loom_create` → `loom_ready` → `loom_claim` → `loom_done` successfully
- [ ] `loom down && loom up` — task state persists, cache rebuilds correctly
- [ ] Concurrent claims: two agents claiming same task — exactly one succeeds

---

## Known Gotchas

1. **asyncpg Records are not dicts** — use `dict(record)` before passing to Pydantic
2. **Redis hashes store strings** — must `json.dumps()` dict/list fields in `sync_task`, `json.loads()` on read
3. **FastMCP lifespan API** — pin FastMCP version; verify `ctx.lifespan_context` returns the dict from `yield`
4. **stdio + long-lived connections** — MCP server process stays alive for entire Claude Code session; asyncpg pool handles reconnection; Redis should use `retry_on_timeout=True`
5. **Migration timing** — `loom up` must wait for Postgres healthcheck before running migrations; use retry with backoff
6. **LOOM_PROJECT_DIR** — MCP server reads config from this env var (set in `.mcp.json`); `load_config` must accept directory parameter

---

## What Phase 1 Does NOT Include

- Skills system (Phase 2)
- Decomposition logic (Phase 2, `loom_decompose` is a stub)
- Workflows (Phase 2+)
- Orchestrator agent loop (Phase 3)
- Claim TTL / auto-release (Phase 3)
- GCP deployment (Phase 4)
- Slack/Obsidian integrations (Phase 4)
