# Roadmap: vmcluster-mcp

**Milestone:** v1.0 — Core Infrastructure Orchestration
**Granularity:** Standard
**Total Phases:** 7
**Coverage:** 33/33 v1 requirements ✓

---

## Phases

- [ ] **Phase 1: Foundation** — Server skeleton, ToolResult error contract, config loading, topology parsing
- [ ] **Phase 2: Core VM Lifecycle** — cluster_define/start/stop/destroy/status/handle with libvirt integration
- [ ] **Phase 3: SSH Execution** — node_exec/exec_all with asyncssh connection pool and reconnect
- [x] **Phase 4: Snapshot Management** — Atomic cluster-wide snapshot create/revert/list/delete (completed 2026-03-23)
- [x] **Phase 5: Artifact Pipeline** — artifact_register/sync/install/list/diff with content-addressed dedup (completed 2026-03-23)
- [ ] **Phase 6: Kernel Observability** — dmesg_mark/collect with oops/panic/KASAN/UBSAN classification
- [x] **Phase 7: Network & Node Fault Injection** — net_partition/impair/heal + node_crash + persistent fault registry (completed 2026-03-24)
- [x] **Phase 8: Retroactive Phase Verification (02 & 03)** — Create missing 02-VERIFICATION.md and 03-VERIFICATION.md; close CLUS and EXEC orphaned requirements (completed 2026-03-24)
- [x] **Phase 9: Requirements & Roadmap Metadata Closure** — Fix stale REQUIREMENTS.md checkboxes, traceability table, ROADMAP.md plan checkboxes and progress table, and add missing summary frontmatter (completed 2026-03-24)
- [x] **Phase 10: Nyquist Validation Coverage** — Create VALIDATION.md for all 7 phases to achieve Nyquist compliance (completed 2026-03-24)

---

## Overview

| # | Phase | Goal | Requirements | Plans |
|---|-------|------|--------------|-------|
| 1 | Foundation | Bootable MCP server with typed error contract and topology parsing | FOUND-01..04 | 3 plans |
| 2 | Core VM Lifecycle | Full cluster provisioning lifecycle from topology file to running nodes | CLUS-01..07 | 4 plans |
| 3 | SSH Execution | Structured command execution across nodes with automatic reconnection | EXEC-01..03 | 3 plans |
| 4 | Snapshot Management | Atomic cluster-wide snapshot create/revert/list/delete | SNAP-01..04 | 2 plans |
| 5 | Artifact Pipeline | Content-addressed artifact register/sync/install/list/diff | ART-01..05 | 3 plans |
| 6 | Kernel Observability | dmesg marker insertion and classified kernel log collection | OBS-01..03 | 2 plans |
| 7 | Network & Node Fault Injection | Persistent fault injection with iptables/tc netem and node crash | FAULT-01..07 | 5 plans |
| 8 | Retroactive Phase Verification (02 & 03) | Close CLUS and EXEC audit gaps with verification reports | CLUS-01..07, EXEC-01..03 | 2 plans |
| 9 | Requirements & Roadmap Metadata Closure | 3/3 | Complete    | 2026-03-24 |
| 10 | Nyquist Validation Coverage | 3/3 | Complete    | 2026-03-24 |

---

## Phase Details

### Phase 1: Foundation
**Goal:** The MCP server starts, routes every tool call through a structured `ToolResult` error contract, loads config from disk or environment, and parses and validates topology YAML — providing the stable base all subsequent phases build on.
**Depends on:** Nothing (first phase)
**Requirements:** FOUND-01, FOUND-02, FOUND-03, FOUND-04
**Success Criteria** (what must be TRUE):
  1. `python -m vmcluster_mcp.server` starts and completes the MCP initialize handshake without error.
  2. Any tool call that raises a Python exception returns a structured `ToolResult` with `success=False` and a descriptive `error` string — no raw tracebacks reach the agent.
  3. The server reads configuration from `/etc/vmcluster/config.yaml` or `VMCLUSTER_*` environment variables; a startup log or explicit health endpoint confirms which source was used.
  4. Passing a malformed topology YAML returns a `ToolResult` error identifying the invalid field path — valid topology YAML parses and validates without error.
**Plans:** 3 plans
Plans:
- [x] 01-01-PLAN.md — Project scaffold (pyproject.toml, uv setup, module structure)
- [x] 01-02-PLAN.md — Core Pydantic models, `ToolResult[T]`, `@structured_tool_handler` decorator, FastMCP server skeleton
- [x] 01-03-PLAN.md — Config loading (`config.py`) and topology parser (`topology/`)

---

### Phase 2: Core VM Lifecycle
**Goal:** An agent can provision and tear down a named multi-node cluster from a topology file, inspect current node state, and retrieve a `ClusterHandle` for downstream SSH consumers — all through idempotent libvirt/QEMU operations.
**Depends on:** Phase 1
**Requirements:** CLUS-01, CLUS-02, CLUS-03, CLUS-04, CLUS-05, CLUS-06, CLUS-07
**Success Criteria** (what must be TRUE):
  1. `cluster_define` creates libvirt domain definitions and cluster-namespaced overlay qcow2 disks for all topology nodes; calling it a second time on an already-defined cluster returns success without modifying existing domains.
  2. `cluster_start` boots all nodes; with `wait_for_ssh=True`, the call blocks until every node is SSH-reachable and returns time waited.
  3. `cluster_status` returns current domain state (running, shut off, etc.) and SSH reachability for each node in the cluster.
  4. `cluster_stop` and `cluster_destroy` cleanly remove cluster resources; `cluster_destroy` with `remove_overlays=True` also deletes overlay disk images.
  5. `cluster_handle` returns a `ClusterHandle` with node names, IP addresses, and SSH credentials suitable for direct downstream use.
**Plans:** 4/4 plans complete
Plans:
- [x] 02-01-PLAN.md — Thread-safe libvirt client (`libvirt_client.py`) with single-worker executor, domain/network wrappers, MAC generation
- [x] 02-02-PLAN.md — Domain/network XML builders, cloud-init ISO generator, `cluster_define` tool with idempotency
- [x] 02-03-PLAN.md — `cluster_start` with SSH wait loop, `cluster_stop`, `cluster_destroy`
- [x] 02-04-PLAN.md — `cluster_status` and `cluster_handle` query tools

---

### Phase 3: SSH Execution
**Goal:** An agent can run shell commands on individual nodes or all nodes in parallel, receiving structured output with exit codes and durations — and the SSH layer transparently reconnects after any VM reboot.
**Depends on:** Phase 2
**Requirements:** EXEC-01, EXEC-02, EXEC-03
**Success Criteria** (what must be TRUE):
  1. `node_exec` runs a command on a named node and returns a `ToolResult` containing exit code, stdout, stderr, and wall-clock duration.
  2. `node_exec_all` runs the same command across all (or a specified subset of) nodes in parallel and returns per-node results with `all_succeeded` and `failed_nodes` summary fields.
  3. After a VM reboot (via snapshot revert or `node_crash`), the next `node_exec` or `node_exec_all` call succeeds without a manual reconnect — the SSH layer detects the stale connection and re-establishes it transparently.
**Plans:** 3/3 plans complete
Plans:
- [x] 03-01-PLAN.md — SSHConnectionManager with per-node connection caching and transparent reconnect
- [x] 03-02-PLAN.md — `node_exec` tool with structured `ExecResult`, server.py wiring
- [x] 03-03-PLAN.md — `node_exec_all` with `asyncio.gather` parallel fan-out and `require_all` semantics

---

### Phase 4: Snapshot Management
**Goal:** An agent can checkpoint all cluster nodes simultaneously and atomically restore any checkpoint — making test state resets fast, clean, and deterministic.
**Depends on:** Phase 2
**Requirements:** SNAP-01, SNAP-02, SNAP-03, SNAP-04
**Success Criteria** (what must be TRUE):
  1. `snapshot_create` creates an external disk snapshot for every node simultaneously; the named snapshot appears immediately in `snapshot_list` output.
  2. `snapshot_revert` shuts down all nodes, reverts each to the named snapshot, and — if `restart=True` — boots them and waits for SSH reachability; any failure on any node returns `success=False` and leaves no silent partial-revert state.
  3. `snapshot_list` returns all snapshots for a cluster with creation timestamp and per-node disk size.
  4. `snapshot_delete` removes a named snapshot from all nodes and returns per-node confirmation.
**Plans:** 2/2 plans complete
Plans:
- [x] 04-01-PLAN.md — SnapshotManager infrastructure, `snapshot_create` + `snapshot_list` tools, LibvirtClient snapshot methods
- [x] 04-02-PLAN.md — `snapshot_revert` with destroy-all-before-revert + SSH wait, `snapshot_delete` with best-effort per-node

---

### Phase 5: Artifact Pipeline
**Goal:** An agent can register a local build tree, push it to cluster nodes with deduplication, install it using the appropriate method, and compare two artifact versions to see what changed.
**Depends on:** Phase 3
**Requirements:** ART-01, ART-02, ART-03, ART-04, ART-05
**Success Criteria** (what must be TRUE):
  1. `artifact_register` computes a sha256 content hash for a local build tree and stores the entry in the registry via atomic write (`registry.json.tmp` → rename); the artifact is immediately visible in `artifact_list`.
  2. `artifact_sync` transfers a registered artifact to cluster nodes via rsync; nodes where the artifact hash already matches are skipped and reported as such in the result.
  3. `artifact_install` installs a synced artifact on specified nodes using the requested mode (`modprobe_path`, `make_install`, `rpm`, or `deb`) and returns per-node success/failure.
  4. `artifact_list` returns all registered artifacts with hash, source path, and registration timestamp.
  5. `artifact_diff` returns the filenames of `.ko` files and binaries that differ between two named artifacts.
**Plans:** 3/3 plans complete
Plans:
- [x] 05-01-PLAN.md — ArtifactDiff model, ArtifactRegistry with hash_tree + atomic writes, `artifact_register` + `artifact_list` + `artifact_diff` tools
- [x] 05-02-PLAN.md — `artifact_sync` via rsync subprocess with SSH dedup check and asyncio.gather fan-out
- [x] 05-03-PLAN.md — `artifact_install` with four modes (modprobe_path, make_install, rpm, deb)

---

### Phase 6: Kernel Observability
**Goal:** An agent can bracket kernel operations with named dmesg markers and retrieve classified kernel log output, detecting oops, panics, and sanitizer events in structured form.
**Depends on:** Phase 3
**Requirements:** OBS-01, OBS-02, OBS-03
**Success Criteria** (what must be TRUE):
  1. `dmesg_mark` writes a uniquely-identified timestamp marker into the kernel log of one or more nodes; the marker appears in subsequent `dmesg_collect` output from those nodes.
  2. `dmesg_collect` returns structured dmesg lines from a node since a named marker, with optional log-level filtering (e.g., warning and above only).
  3. The `dmesg_collect` result includes a `classifications` field listing any detected oops, kernel panics, KASAN reports, UBSAN reports, or `WARNING:` lines as typed events with originating log lines.
**Plans:** 2/2 plans complete
Plans:
- [x] 06-01-PLAN.md — dmesg_mark tool + classifier + dmesg line parser
- [x] 06-02-PLAN.md — dmesg_collect tool with marker/level filtering and classification

---

### Phase 7: Network & Node Fault Injection
**Goal:** An agent can inject and heal network partitions and traffic impairments between cluster nodes, hard-reset individual nodes to simulate failures, and have all active faults survive an MCP server restart.
**Depends on:** Phase 2, Phase 3
**Requirements:** FAULT-01, FAULT-02, FAULT-03, FAULT-04, FAULT-05, FAULT-06, FAULT-07
**Success Criteria** (what must be TRUE):
  1. `net_partition` installs iptables DROP rules on the libvirt bridge between two node groups; every rule carries a `--comment vmcluster:<cluster>:<handle_id>` tag for reliable identification and cleanup.
  2. `net_impair` applies `tc netem` rules to egress traffic between specific nodes (latency, jitter, loss, or corruption); the impairment is observable in subsequent `node_exec` round-trip measurements.
  3. `net_heal` removes all iptables/tc rules for a specific fault handle; the healed handle no longer appears in `net_fault_list`.
  4. `net_heal_all` removes all active faults for a cluster in a single call; `net_fault_list` returns an empty list immediately after.
  5. `node_crash` executes `virsh destroy` + `virsh start` on the target node and returns after SSH reachability is confirmed.
  6. After an MCP server process restart, `net_fault_list` returns all previously injected faults that were not healed — the registry is written to disk atomically and reconciled against live kernel state on startup.
**Plans:** 5 plans (4 complete + 1 gap closure)
Plans:
- [x] 07-01-PLAN.md — Fault registry with atomic writes and startup reconciliation
- [x] 07-02-PLAN.md — net_partition with tagged iptables rule management
- [x] 07-03-PLAN.md — net_impair, net_heal, net_heal_all with tc netem
- [x] 07-04-PLAN.md — net_fault_list and node_crash tools
- [x] 07-05-PLAN.md — Gap closure: wire startup reconciliation, update impairment truth

---

### Phase 8: Retroactive Phase Verification (02 & 03)
**Goal:** Close the audit gap for Phase 02 (Core VM Lifecycle) and Phase 03 (SSH Execution) by creating the missing verification reports, providing the 3-source evidence required to mark CLUS-01..07 and EXEC-01..03 as satisfied at milestone level.
**Depends on:** Phase 2, Phase 3 (already implemented)
**Requirements:** CLUS-01, CLUS-02, CLUS-03, CLUS-04, CLUS-05, CLUS-06, CLUS-07, EXEC-01, EXEC-02, EXEC-03
**Gap Closure:** Closes gaps from v1.0 milestone audit
**Success Criteria** (what must be TRUE):
  1. `.planning/phases/02-core-vm-lifecycle/02-VERIFICATION.md` exists and confirms all Phase 2 success criteria against the codebase.
  2. `.planning/phases/03-ssh-execution/03-VERIFICATION.md` exists and confirms all Phase 3 success criteria against the codebase.
  3. Re-running audit extract on Phase 02 and 03 summaries returns non-empty `requirements_completed` lists.
  4. CLUS-01..07 and EXEC-01..03 are no longer orphaned in the milestone audit.
**Plans:**
2/2 plans complete
- [x] 08-01-PLAN.md — Audit Phase 02 codebase vs success criteria; write 02-VERIFICATION.md
- [x] 08-02-PLAN.md — Audit Phase 03 codebase vs success criteria; write 03-VERIFICATION.md

---

### Phase 9: Requirements & Roadmap Metadata Closure
**Goal:** Bring all planning metadata into sync with implementation reality — flipping stale checkboxes, fixing traceability statuses, updating ROADMAP.md plan/progress state, and adding missing `requirements_completed` frontmatter to phase summaries — so the milestone audit scores accurately.
**Depends on:** Phase 8
**Requirements:** All v1 requirements (metadata accuracy)
**Gap Closure:** Closes gaps from v1.0 milestone audit
**Success Criteria** (what must be TRUE):
  1. All implemented requirements in REQUIREMENTS.md have `[x]` checkboxes (FOUND-01..04, CLUS-01..07, EXEC-01..03, OBS-02, OBS-03).
  2. Traceability table in REQUIREMENTS.md shows `Complete` for all implemented requirements.
  3. ROADMAP.md plan checkboxes match actual plan completion state (Phases 1–3 plans marked `[x]`).
  4. ROADMAP.md progress table reflects current state (Phases 1–7 showing correct plans-complete counts and dates).
  5. Each phase summary that claims a requirement has `requirements_completed` in its YAML frontmatter.
**Plans:**
3/3 plans complete
- [x] 09-02-PLAN.md — Fix ROADMAP.md plan checkboxes and progress table
- [x] 09-03-PLAN.md — Add `requirements_completed` frontmatter to phase summaries missing it

---

### Phase 10: Nyquist Validation Coverage
**Goal:** Create VALIDATION.md artifacts for all 7 existing phases to achieve full Nyquist compliance, providing structured test-evidence mapping for every phase requirement.
**Depends on:** Phase 9
**Requirements:** All v1 requirements (validation coverage)
**Gap Closure:** Closes tech debt from v1.0 milestone audit
**Success Criteria** (what must be TRUE):
  1. VALIDATION.md exists in each of the 7 phase directories (01–07).
  2. Each VALIDATION.md maps phase requirements to passing test names and assertion evidence.
  3. Nyquist audit discovers no `missing_phases` — `overall` status is `compliant` or `partial`.
**Plans:**
3/3 plans complete
- [x] 10-02-PLAN.md — Create VALIDATION.md for Phases 04, 05
- [x] 10-03-PLAN.md — Create VALIDATION.md for Phases 06, 07

---

## Coverage Validation

| Phase | Requirements | Count |
|-------|-------------|-------|
| 1: Foundation | FOUND-01, FOUND-02, FOUND-03, FOUND-04 | 4 |
| 2: Core VM Lifecycle | CLUS-01, CLUS-02, CLUS-03, CLUS-04, CLUS-05, CLUS-06, CLUS-07 | 7 |
| 3: SSH Execution | EXEC-01, EXEC-02, EXEC-03 | 3 |
| 4: Snapshot Management | SNAP-01, SNAP-02, SNAP-03, SNAP-04 | 4 |
| 5: Artifact Pipeline | ART-01, ART-02, ART-03, ART-04, ART-05 | 5 |
| 6: Kernel Observability | OBS-01, OBS-02, OBS-03 | 3 |
| 7: Network & Node Fault Injection | FAULT-01, FAULT-02, FAULT-03, FAULT-04, FAULT-05, FAULT-06, FAULT-07 | 7 |
| 8: Retroactive Phase Verification (02 & 03) | CLUS-01..07, EXEC-01..03 | — (gap closure) |
| 9: Requirements & Roadmap Metadata Closure | All v1 | — (metadata) |
| 10: Nyquist Validation Coverage | All v1 | — (tech debt) |

- **v1 requirements:** 33
- **Mapped:** 33
- **Unmapped:** 0 ✓

---

## Progress

| Phase | Plans Complete | Status | Completed |
|-------|----------------|--------|-----------|
| 1. Foundation | 3/3 | Complete | 2026-03-22 |
| 2. Core VM Lifecycle | 4/4 | Complete | 2026-03-22 |
| 3. SSH Execution | 3/3 | Complete | 2026-03-23 |
| 4. Snapshot Management | 2/2 | Complete | 2026-03-23 |
| 5. Artifact Pipeline | 3/3 | Complete | 2026-03-23 |
| 6. Kernel Observability | 2/2 | Complete | 2026-03-23 |
| 7. Network & Node Fault Injection | 5/5 | Complete | 2026-03-24 |
| 8. Retroactive Phase Verification (02 & 03) | 2/2 | Complete | 2026-03-24 |
| 9. Requirements & Roadmap Metadata Closure | 0/3 | In Progress | — |
| 10. Nyquist Validation Coverage | 0/3 | Not started | — |

---
*Roadmap created: 2026-03-22*
