# Research Summary: vmcluster-mcp

**Synthesized:** 2026-03-22  
**Sources:** STACK.md, FEATURES.md, ARCHITECTURE.md, PITFALLS.md, PROJECT.md

---

## Executive Summary

vmcluster-mcp is a focused MCP stdio server enabling AI agents to provision, configure, snapshot, and fault-inject multi-VM clusters on libvirt/QEMU — all without human intervention. The recommended implementation uses FastMCP with Pydantic v2 structured outputs, libvirt-python for VM lifecycle, asyncssh for parallel node execution, and a strict module layering that keeps `server.py` as a pure routing layer. The tool's core value loop is: define cluster → start → snapshot clean state → install artifact → test → revert → repeat — this cycle must work end-to-end before observability or fault injection features matter.

The most critical architectural decision is threading discipline for the libvirt C bindings: all `libvirt-python` calls must run through a single dedicated `ThreadPoolExecutor(max_workers=1)` to avoid non-deterministic segfaults. The second most critical is the `ToolResult[T]` error contract — a top-level catch-all decorator in `server.py` must prevent raw Python exceptions from leaking to agents as unstructured MCP protocol errors. Both of these must be established in the earliest phases or all subsequent tools are built on a broken contract.

The feature set has a clean 10-tier build order from foundational Pydantic models up through fault injection. No tier skips are safe — each tier depends on the one below. Within each tier, parallel implementation is possible. External snapshot revert complexity (no native atomic multi-domain revert in libvirt) is the highest-risk implementation challenge and requires explicit "destroy all nodes first" discipline.

---

## Key Findings

### Stack

Use FastMCP (`mcp[cli]>=1.26.0`) as the server framework — it generates MCP tool schemas directly from Python type annotations and auto-serializes Pydantic `BaseModel` returns as `structuredContent`. Pydantic v2 (`>=2.12.5`) is the data contract layer for all models. Use `asyncssh>=2.22.0` (not paramiko) for SSH — it's pure asyncio and makes parallel multi-node execution trivial with `asyncio.gather`. `libvirt-python>=12.1.0` requires system `libvirt-dev` headers and must be isolated to a single-worker executor. `uv` is the mandatory build tool per MCP SDK recommendations.

**Critical version constraint:** `libvirt-python` is NOT pure Python — system `libvirt-dev` + `python3-dev` packages must be installed before `uv sync`.

### Table Stakes Features

- `cluster_define` — topology YAML → libvirt domain XML + overlay qcow2 disks
- `cluster_start` with SSH wait — blocks until all nodes are SSH-reachable
- `cluster_stop` / `cluster_destroy` with `remove_overlays` cleanup
- `cluster_status` — ground truth before any other call
- `node_exec` + `node_exec_all` (parallel) — foundation for all install/verify ops
- `snapshot_create` + `snapshot_revert` — cluster-atomic test reset primitive
- `artifact_register`, `artifact_sync`, `artifact_install` — the core value delivery
- `ToolResult[T]` structured error wrapper — non-negotiable agent contract
- `ClusterHandle` — handoff interface to downstream consumers

### Architecture

The server is organized into 9 module groups with strict dependency layering: `models.py` and `config.py` are pure leaf nodes with no internal dependencies; `server.py` is the only file that imports from all modules and is a pure routing/serialization layer with no domain logic. The critical seams are `topology/` → `cluster/` (TopologySpec contract), `exec/ssh.py` → `artifact/` + `observe/` (ExecResult + NodeDescriptor contracts), and `cluster/libvirt_client.py` → `snapshot/` + `net/bridge.py` (libvirt connection sharing). Between tool calls the server is fully sequential (MCP stdio = one call at a time); within a call, fan-out across nodes uses `asyncio.gather` in `exec/parallel.py`.

**Critical constraint:** All libvirt calls must run through a single `ThreadPoolExecutor(max_workers=1)` — the libvirt C library is not thread-safe with concurrent connection use.

### Top Pitfalls to Avoid

1. **asyncio + libvirt thread safety** (CRITICAL — segfault risk): All `libvirt-python` calls via a single-worker executor; never use a general thread pool for libvirt.
2. **MCP exceptions bypass structured error contract** (HIGH): Implement `@structured_tool_handler` catch-all decorator in Phase 1 before any tool is wired up.
3. **Non-atomic cluster-wide snapshot revert** (HIGH): Destroy all nodes first before reverting external snapshots; treat any partial revert as fatal (`recoverable: False`).
4. **iptables/tc state leakage on server crash** (HIGH): Persist fault registry to disk with atomic writes; startup reconciliation pass to clean orphaned kernel rules; tag all iptables rules with `--comment vmcluster:<cluster>:<handle_id>`.
5. **SSH connections stale after VM reboot** (HIGH): Explicitly invalidate all SSH connections for affected nodes after any VM restart; never reuse pool connections during `wait_for_ssh` polling.
6. **Artifact registry corruption on crash** (HIGH): Always write to `registry.json.tmp`, fsync, then `os.replace()` — never write direct.
7. **Overlay disk path collision / base image mutation** (CRITICAL): Assert in `domain_builder.py` that primary disk path ≠ `topology.base_image`; always namespace overlays as `{overlay_dir}/{cluster_name}/{node_name}.qcow2`.

---

## Recommended Build Order

Built from the 10-tier feature dependency graph in FEATURES.md and Phase groupings in ARCHITECTURE.md:

1. **Foundation** — `models.py`, `config.py`, `topology/`, `server.py` skeleton + `@structured_tool_handler` decorator
2. **Core VM Lifecycle** — `cluster/libvirt_client.py` (single-executor pattern), `cluster/domain_builder.py`, `cluster/manager.py` → tools: `cluster_define`, `cluster_start`, `cluster_stop`, `cluster_destroy`, `cluster_status`, `cluster_handle`
3. **SSH Execution Layer** — `exec/ssh.py`, `exec/parallel.py` → tools: `node_exec`, `node_exec_all` (with `all_succeeded` + `failed_nodes` in result)
4. **Snapshot Management** — `snapshot/manager.py` → tools: `snapshot_create`, `snapshot_revert`, `snapshot_list`, `snapshot_delete`
5. **Artifact Pipeline** — `artifact/registry.py`, `artifact/syncer.py`, `artifact/installer.py`, `artifact/builder.py` → tools: `artifact_register`, `artifact_sync`, `artifact_install`, `artifact_list`, `artifact_diff`, `artifact_build`
6. **Kernel Observability** — `observe/classifier.py`, `observe/dmesg.py` → tools: `dmesg_mark`, `dmesg_collect`
7. **Network Fault Injection** — `net/bridge.py`, `net/fault_registry.py`, `net/fault.py` → tools: `net_partition`, `net_impair`, `net_heal`, `net_heal_all`, `net_fault_list`, `node_crash`

**Minimal viable end-to-end path:**
`models → topology → cluster_define → cluster_start → node_exec → snapshot_create → artifact_register → artifact_sync → artifact_install → cluster_handle`

---

## Critical Decisions

These must be made before implementation starts — changing them mid-build is expensive:

| Decision | Why Early | Recommendation |
|----------|-----------|----------------|
| Single `ThreadPoolExecutor(max_workers=1)` for all libvirt calls | Must be in `libvirt_client.py` from commit 1; retrofitting is unsafe | **Decided: single-worker executor** |
| `@structured_tool_handler` catch-all decorator | Must wrap every tool from the moment tools exist; adding later means testing gap | **Decided: decorator defined in Phase 1, required for registration** |
| External disk-only snapshots (no memory state default) | Affects snapshot XML + revert implementation; internal vs. external are different code paths | **Decided: disk-only, `include_memory=False` default** |
| Destroy-all-before-revert discipline for `snapshot_revert` | Changes the user-visible semantics of `snapshot_revert`; document in docstring immediately | **Decided: all nodes must be shut off before external snapshot revert** |
| Atomic artifact registry writes (write-tmp + rename) | Affects `registry.py` API shape and all callers; not retrofittable without risk | **Decided: atomic write from day one** |
| Overlay path namespace: `{overlay_dir}/{cluster_name}/{node_name}.qcow2` | Baked into `domain_builder.py` disk paths; changing breaks all existing overlays | **Decided: cluster-namespaced overlay paths** |
| `require_all: bool = True` on `node_exec_all` | Determines whether partial node failures propagate as `ToolResult.success=False` | **Decide before SSH exec phase: recommend True by default** |

---

## Phase Implications

How research maps to suggested roadmap phases:

**Phase 1 — Foundation & MCP Scaffolding**
Build all Pydantic models, config loading, topology parser, server skeleton, and the `@structured_tool_handler` catch-all. The libvirt single-executor pattern must be designed here. Nothing else is buildable without stable models and a correct error contract. *Low implementation risk, high architectural importance.*

**Phase 2 — Cluster Lifecycle**
`cluster_define` through `cluster_status` and `cluster_handle`. The overlay disk path + base image assertion must be enforced in `domain_builder.py`. The idempotency hash check (topology hash in libvirt domain metadata) prevents silent topology drift. *Medium risk — libvirt XML generation has edge cases.*

**Phase 3 — SSH Execution**
`node_exec` and `node_exec_all`. Build `exec/ssh.py` with explicit connection invalidation hooks (needed by snapshot revert later). Implement `all_succeeded` + `failed_nodes` in exec results from the start. *Medium risk — connection lifecycle discipline matters.*

**Phase 4 — Snapshot Management** ⚠️ *Needs Phase Research*
`snapshot_create`, `snapshot_revert`, `snapshot_list`, `snapshot_delete`. Highest implementation risk: external snapshot revert requires destroy-all-first discipline and per-node status tracking. SSH reachability pre-check before `snapshot_create`. Treat any partial revert as fatal. *High risk — research recommended before planning.*

**Phase 5 — Artifact Pipeline**
`artifact_register` (atomic writes), `artifact_sync`, `artifact_install`, `artifact_list`. `artifact_build` and `artifact_diff` can be deferred to end of phase or a follow-on. *Medium risk — rsync + install modes have edge cases but patterns are well-understood.*

**Phase 6 — Kernel Observability**
`dmesg_mark`, `dmesg_collect` with structured classification. Low risk — pure consumer of SSH exec layer. Regex classifier is stateless and independently testable. *Low risk.*

**Phase 7 — Network Fault Injection** ⚠️ *Needs Phase Research*
`net_partition`, `net_impair`, `net_heal`, `net_heal_all`, `net_fault_list`, `node_crash`. Highest operational risk (host iptables/tc mutation). Fault registry must be persisted to disk with startup reconciliation. All iptables rules must carry `--comment` tags. `tc netem` interface naming is environment-dependent. *High risk — research recommended before planning.*

---

## Confidence Assessment

| Area | Confidence | Notes |
|------|------------|-------|
| Stack | HIGH | All versions verified from PyPI 2026-03-22; official SDK docs confirm FastMCP + uv recommendations |
| Features | HIGH | Primary source is authoritative design doc (`docs/vmcluster-mcp-design.md`); feature set is stable |
| Architecture | HIGH | Derived directly from design doc; module boundaries and data flow are well-defined |
| Pitfalls | HIGH | libvirt threading from official C API docs; MCP exception behavior from SDK source; SSH staleness from paramiko patterns |

**Overall: HIGH**

**Gaps to address during planning:**
- External snapshot revert implementation specifics (blockcommit vs. blockpull vs. destroy+rebase) — flagged for Phase 4 research
- `tc netem` tap interface naming convention for QEMU/libvirt bridge topology — flagged for Phase 7 research
- SSH connection pool invalidation hook design (how `cluster/manager.py` signals `exec/ssh.py` after reboot) — design decision in Phase 3
- Whether `asyncssh` or paramiko is used (STACK.md recommends asyncssh; ARCHITECTURE.md references paramiko) — **resolve before Phase 3**: asyncssh is the correct choice per stack research

---

## Sources

- `docs/vmcluster-mcp-design.md` — Authoritative design document (HIGH confidence — primary source for features + architecture)
- PyPI live package index (2026-03-22) — Stack versions (HIGH confidence)
- libvirt official C API documentation — Thread safety constraints (HIGH confidence)
- Anthropic MCP Python SDK README + FastMCP source — Exception handling behavior (HIGH confidence)
- Domain analogs: Vagrant, Terraform+libvirt, Ansible, Chaos Mesh — Feature comparison (MEDIUM confidence — indirect)
