# Pitfalls Research: vmcluster-mcp

**Domain:** Python MCP server + libvirt/QEMU cluster orchestration
**Researched:** 2026-03-22
**Confidence:** HIGH (libvirt internals from official docs + MCP SDK from official README)

---

## Critical Pitfalls

### 1. Non-Atomic Cluster-Wide Snapshot Revert

**Category:** libvirt | state management

**What goes wrong:**
`snapshot_revert` issues a separate `virDomainRevertToSnapshot` (or equivalent blockjob + destroy + start sequence for external snapshots) for each node. These calls are sequential or parallel but each can fail independently. If node `controller` reverts successfully but `worker-0` fails mid-revert, the cluster is in a split-brained inconsistent state — one node is at the snapshot baseline, the other is at its current state. No native libvirt API provides multi-domain atomic revert.

**Root cause:**
libvirt external snapshots operate per-domain. External disk-only snapshots (the recommended approach for live VMs) use `blockjob` mechanics for revert, with no cross-domain transaction semantics.

**Consequences:**
- Agents that call `snapshot_revert` and get a partial success will operate on a logically inconsistent cluster, producing non-reproducible test results.
- Subsequent `snapshot_revert` retries may partially succeed again, stranding disks in mid-pivot state.

**Warning signs:**
- `snapshot_revert` returns `ClusterStatus` with some nodes' `domain_state` as `running` and others as `shut off` or `crashed`.
- Revert times vary substantially across nodes (one extremely fast, others time out).

**Prevention:**
- Before reverting, `virsh destroy` all nodes first. Reverting shut-off VMs with external snapshots is simpler and avoids live blockjob mechanics.
- Track per-node revert status in `SnapshotStatus`. If any node fails, automatically attempt to bring all nodes down and retry from a clean power-off baseline.
- Treat a partial revert result as fatal — return `recoverable: False` in `ErrorDetail` and require the agent to call `cluster_destroy` + `cluster_define` to reset.
- Document in `snapshot_revert` docstring: "All nodes must be shut off before reverting external snapshots in v1."

**Phase to address:** Snapshot Management implementation phase.

---

### 2. MCP Tool Exceptions Bypassing Structured Error Contract

**Category:** MCP

**What goes wrong:**
If any unhandled Python exception propagates out of a FastMCP tool handler, the MCP SDK serializes it as a protocol-level error (not a `ToolResult`). The agent receives something like `{"error": {"code": -32603, "message": "Internal error"}}` instead of the designed `ToolResult(success=False, error=ErrorDetail(...))`. Agents coded to inspect `result.success` and `result.error.code` will encounter a different schema and either crash or stall.

**Root cause:**
FastMCP tools that raise exceptions cause the framework to return `CallToolResult(isError=True, content=[TextContent(...)])`. This is structurally different from a Pydantic `ToolResult` model returned as structured data with `success=False`. The agent's parsing logic must handle both paths—but if the design contract says "always `ToolResult`", any exception leakage breaks that contract.

**Consequences:**
- Agent retry logic based on `result.error.recoverable` never fires because the error is not in the expected structure.
- Error `code` values like `"SSH_UNREACHABLE"` are lost; the agent gets a raw exception message instead.
- Agents that parse `result.success` will throw `AttributeError` on the unexpected response shape.

**Warning signs:**
- Any tool that does I/O (SSH, libvirt, disk ops) without a try/except will eventually surface an unhandled exception.
- During development, test exceptions appear as MCP protocol errors in Claude Desktop logs, not in structured `ToolResult` fields.

**Prevention:**
- Every tool handler must have a top-level `try/except Exception as e` that returns `ToolResult(success=False, error=ErrorDetail(code="INTERNAL_ERROR", message=str(e), recoverable=False))` as the fallback.
- Define a decorator `@structured_tool_handler` that wraps any tool with this catch-all, so the contract is enforced at the framework level rather than relying on per-tool discipline.
- Add a note to ARCHITECTURE.md: "Exception leakage from tool handlers is a contract violation. All exits must be `ToolResult`."

**Phase to address:** MCP server scaffolding / core implementation phase.

---

### 3. SSH Connection Pool Staleness After VM Reboot

**Category:** SSH | state management

**What goes wrong:**
After `snapshot_revert` (with `restart_after=True`) or `node_crash` (with `restart_after=True`), the VM's SSH daemon restarts with a new PID and possibly a new host key derived from transient state. Paramiko connections in the pool have underlying TCP sockets that are silently dead: the kernel on the VM side has sent a RST, but depending on timing the local socket may not yet have detected it. The next SSH command on a stale connection raises `paramiko.ssh_exception.SSHException` or `socket.error` and may not be retried automatically.

**Root cause:**
Paramiko `SSHClient` objects are long-lived. A connection pool keyed by IP that was populated before a VM reboot retains dead `SSHClient` instances. The pool has no out-of-band signal to invalidate entries when a VM reboots.

**Consequences:**
- First SSH command after revert fails with a confusing transport error rather than a clear connection error.
- If the pool silently discards the connection and reconnects, it may succeed on the second attempt — making failures non-deterministic and hard to reproduce.
- `wait_for_ssh` polling logic may race against the old (dying) SSH daemon, succeeding on a connection that will immediately be reset.

**Warning signs:**
- Intermittent `Transport not connected` or `EOFError` errors on the first command after revert.
- `wait_for_ssh=True` completes but the first subsequent `node_exec` call fails.
- SSH errors are not reproducible across sequential `snapshot_revert` calls.

**Prevention:**
- After any VM restart (snapshot revert, node_crash, cluster_start), **explicitly invalidate all pool entries** for affected nodes before the `wait_for_ssh` loop.
- In `wait_for_ssh`, open a fresh temporary connection on each poll attempt; do not reuse pool connections during the wait window.
- Use paramiko's `transport.is_active()` check as a lazy validation gate before reusing any pooled connection.
- Set `connect_timeout` and `banner_timeout` explicitly on all paramiko connections; do not rely on system defaults.
- Consider `known_hosts` / `RejectPolicy` vs. `AutoAddPolicy` carefully: for test clusters with static IPs, use `AutoAddPolicy` but log a warning if the host key changes (could indicate a misconfigured revert).

**Phase to address:** SSH executor implementation phase.

---

### 4. Artifact Registry File Corruption on Crash

**Category:** state management

**What goes wrong:**
`artifact_register` reads the JSON registry file, appends a new `BuildArtifact` entry, and writes the file back. If the MCP server process is killed (SIGKILL, OOM), crashes, or loses power between the `open()` and `flush()/close()` calls, the registry file can be:
- Truncated (partial write)
- Empty (file opened for write but nothing flushed)
- Valid JSON for the old content (if OS buffered the position-0 truncation but the new data never landed)

On next startup, `json.load()` raises `JSONDecodeError` and all artifact lookups fail.

**Root cause:**
File write is not atomic. Python `json.dump()` to an open file is a multi-syscall operation (`write()` may be called multiple times for large registries), and there is no fsync unless explicitly called.

**Consequences:**
- After any unexpected process exit during an artifact registration, the entire registry is unreadable.
- With no registry, `artifact_sync` and `artifact_install` cannot function, breaking all downstream workflows.
- Recovery requires manual inspection of the artifact store directory to reconstruct the registry.

**Warning signs:**
- `JSONDecodeError` or `ValueError` when loading `registry.json` at server startup.
- Registry file size is 0 bytes or smaller than expected.
- Last-modified timestamp on registry file is very recent (mid-operation).

**Prevention:**
- **Atomic write**: always write to `registry.json.tmp`, fsync it, then `os.replace()` (POSIX atomic rename) to `registry.json`. This is the standard pattern for safe config/state file updates.
- On startup, validate the registry file: if `registry.json` is missing or corrupt, check for `registry.json.tmp` (a completed tmp that was not renamed), and recover if possible.
- Keep the registry file small: store only `BuildArtifact` metadata, not file contents. Content lives on disk under `artifact_store_dir/{artifact_id}/`.
- For v2, consider SQLite as the registry backend — it provides atomic writes and WAL mode by default.

**Phase to address:** Artifact registry implementation phase.

---

### 5. Parallel SSH Execution: Partial Failures Invisible at Tool Level

**Category:** SSH | MCP

**What goes wrong:**
`node_exec_all(parallel=True)` executes the same command concurrently on N nodes. The return type is `dict[str, ExecResult]`, which correctly captures per-node results. The problem is at the tool call level: the `ToolResult` wrapping this dict reports `success=True` even when 2 of 3 nodes timed out or returned non-zero exit codes, because "the tool itself ran successfully." Agents that check only `result.success` and not each node's `ExecResult.exit_code` or `.timed_out` miss partial failures and proceed as if everything succeeded.

**Root cause:**
There is a semantic gap between "the tool call succeeded" and "all nodes completed the requested operation successfully." The design's `ToolResult` has a single `success` boolean that conflates these two meanings.

**Consequences:**
- An agent that calls `node_exec_all` to install kernel modules proceeds to `snapshot_create` even though one node failed the install.
- Downstream operations (test runs) fail in confusing ways because one node has no modules loaded.
- The error surface is buried in `result.result["worker-0"].exit_code == 1`, which agents may not check.

**Warning signs:**
- Agent proceeds past a `node_exec_all` call that had timeouts, then sees unexpected behavior in subsequent steps.
- `dmesg_collect` shows errors only on specific nodes.
- `cluster_status` shows artifact installed on some nodes but not all.

**Prevention:**
- Add `all_succeeded: bool` and `failed_nodes: list[str]` to the execution result model (or to `ToolResult.error` when `all_succeeded=False`).
- Set `ToolResult.success = False` and populate `ToolResult.error` when any node's `ExecResult` has `exit_code != 0` or `timed_out = True`. Document that `success=True` means **all** nodes succeeded.
- Provide a `require_all: bool = True` parameter on `node_exec_all` — if `True`, any per-node failure causes the tool to return `success=False` with a summary in `ErrorDetail`.

**Phase to address:** Parallel execution implementation phase.

---

### 6. iptables State Leakage on Server Crash

**Category:** state management | libvirt

**What goes wrong:**
`net_partition` and `net_impair` install iptables DROP rules and `tc netem` qdiscs on the bridged tap interfaces. The active fault registry (`net/fault_registry.py`) is in-memory. If the MCP server process crashes or is restarted, the in-memory registry is lost, but the iptables rules and netem qdiscs remain active on the host. The next startup has no knowledge of the active faults:
- `net_fault_list` returns an empty list (no persisted state)
- `net_heal_all` has nothing to clean up
- The cluster continues to experience network partitions or impairments silently

**Root cause:**
The fault registry is described as in-memory (`fault_registry.py`). System network state (iptables, tc) outlives the process that created it.

**Consequences:**
- After a server restart, all new tests run in a silently impaired network environment.
- Faults from previous sessions accumulate: repeated server crashes can stack multiple DROP rules for the same pair.
- Automated test workflows fail non-deterministically with no visible error in vmcluster-mcp — the problem is transparent to the MCP layer.

**Warning signs:**
- Network throughput between VMs is unexpectedly low or zero after a recent server restart.
- `iptables -L FORWARD -n` shows DROP rules for cluster bridge IPs that no active fault session should have installed.
- `tc qdisc show dev <tapX>` shows netem qdiscs that predate the current server session.

**Prevention:**
- **Persist the fault registry to disk** (alongside the artifact registry or in a dedicated `faults.json`). Use the same atomic-write pattern (write-to-tmp, rename).
- On server startup, always run a **reconciliation pass**: read disk registry, verify each rule still exists in iptables/tc, remove orphaned kernel state, update the in-memory registry.
- Implement `net_heal_all` as a cleanup that scans iptables for rules matching a cluster-specific comment tag (e.g., `--comment vmcluster:<cluster_name>:<handle_id>`) rather than relying on in-memory state.
- Tag every iptables rule with `--comment "vmcluster:<cluster>:<handle_id>"` at install time for reliable identification.

**Phase to address:** Network fault injection implementation phase.

---

### 7. cluster_define Non-Idempotency: Domain Name/UUID Conflicts

**Category:** libvirt | state management

**What goes wrong:**
`cluster_define` is documented as "idempotent — safe to call on an already-defined cluster." libvirt's `virDomainDefineXML` will overwrite an existing domain definition if the UUID matches. However:
1. If `cluster_define` crashed after creating some overlays and defining some (but not all) domains, a retry redefines the successful domains (fine) but may recreate failed overlay disks at the same path, potentially creating a conflicting file state.
2. If the topology YAML changes between calls (different base image or vcpu count), `virDomainDefineXML` silently replaces the definition — but the overlay disk was created for the old definition. The domain XML now references a backing file that doesn't match what was used to create the overlay.
3. If two different cluster names happen to map to the same domain name pattern, the second `cluster_define` will destroy the first cluster's domain definition without any warning.

**Root cause:**
libvirt `virDomainDefineXML` is idempotent by UUID, not by cluster concept. The overlay disk creation (`qemu-img create -f qcow2 -b <base>`) is not idempotent by default: calling it on an existing file fails with "file exists" unless explicitly recreating.

**Consequences:**
- Silent topology drift: agent runs tests thinking a 2-vCPU cluster is still 2-vCPU, but it was quietly redefined to 4-vCPU on the second call.
- Overlay disk creation fails on re-call, causing `cluster_define` to fail mid-operation (partial state).
- Domain name collision destroys another user's cluster (in multi-user future expansion).

**Warning signs:**
- `cluster_status` shows unexpected domain specifications after a re-call.
- libvirt `virDomainDefineXML` returns success but `virDomainGetXMLDesc` shows different values than expected.
- Overlay creation fails with `FileExistsError` during a re-define attempt.

**Prevention:**
- On re-call, compare topology hash against the stored topology hash in libvirt domain metadata (`virDomainSetMetadata`). If hash matches, skip redefinition. If hash differs, require explicit `cluster_destroy` first.
- Use `os.path.exists()` before calling `qemu-img create` for overlays. If the overlay exists and the base matches, skip. If base differs, fail explicitly with `TOPOLOGY_CHANGED` error code.
- Store cluster topology hash in domain metadata XML (`virDomainSetMetadata`) so idempotency checks survive server restarts.

**Phase to address:** Cluster lifecycle implementation phase.

---

### 8. Overlay Disk: Base Image Mutation and Path Collisions

**Category:** libvirt | state management

**What goes wrong:**
Two distinct failure modes:

**Mode A — Base image modified directly:**
If the domain XML specifies the base image path as the primary disk (rather than an overlay), QEMU writes to the base image. All future VMs created from that base get the contaminated state. This can happen if `domain_builder.py` has a bug where the overlay path is empty or if the topology `overlay_dir` is misconfigured.

**Mode B — Overlay path collision:**
If two clusters use the same overlay path (e.g., `overlay_dir / "controller.qcow2"` without cluster name in the path), the second `cluster_define` overwrites the first cluster's overlay disk, destroying it silently.

**Root cause:**
QCOW2 overlays require the generated XML to point to the overlay file as the primary disk, with a `<backingStore>` pointing to the base image. The base image backing store is in the overlay metadata, not the domain XML — so if the overlay path in domain XML accidentally equals the base image path, QEMU opens the base for writing.

**Consequences:**
- Mode A: Base image corruption. All subsequent cluster definitions using that base image start with unclean state. Recovery requires restoring the base image from backup.
- Mode B: Silent data loss. `cluster_start` for cluster A after cluster B's `cluster_define` fails or runs on B's disk state.

**Warning signs:**
- Mode A: `qemu-img info <base_image>` shows a `virtual size` that differs from the original, or `actual-size` grows after cluster operations.
- Mode B: `cluster_start` for cluster A fails because its overlay disk has been re-initialized.
- Nodes exhibit unexpected in-VM state (software previously uninstalled is present, or vice versa).

**Prevention:**
- In `domain_builder.py`, assert that the primary disk path in the generated XML is **never** equal to `topology.base_image`. Raise a `ValueError` if this invariant is violated, even in development.
- Overlay filename template must include cluster name: `{overlay_dir}/{cluster_name}/{node_name}.qcow2`. Create the per-cluster subdirectory during `cluster_define`.
- After creating each overlay with `qemu-img create`, verify backing file: `qemu-img info --output=json {overlay}` and assert `"backing-filename"` matches `base_image`.
- Make base image read-only at the OS level as a defense in depth: `chmod a-w {base_image}`.

**Phase to address:** Cluster definition / domain XML generation phase.

---

### 9. asyncio + libvirt Bindings Thread Safety

**Category:** concurrency

**What goes wrong:**
The MCP server uses asyncio (required by the MCP Python SDK stdio transport). libvirt's Python bindings are C extension wrappers around `libvirt.h`. The libvirt C library and its Python bindings assume a **single-threaded event loop**: each `virConnectPtr` has an event loop (`virEventRegisterDefaultImpl`) that expects to be driven from one thread. The Python GIL partially protects CPython-level calls, but:

1. `run_in_executor` dispatches libvirt calls to the `ThreadPoolExecutor`. With multiple threads calling into the same `virConnectPtr` concurrently, libvirt's internal event queue and stream machinery can produce memory corruption or undefined behavior. The C library documentation states connections are not thread-safe without external synchronization.
2. Calling `virEventRunDefaultImpl` from a background thread while concurrent threads execute domain operations on the same connection causes data races in libvirt's internal event dispatch tables.

**Root cause:**
The MCP SDK's asyncio event loop and libvirt's C callback event model are both single-threaded, but naive `run_in_executor` usage creates threads that share a libvirt connection.

**Consequences:**
- Rare, non-deterministic segfaults in libvirt C bindings.
- libvirt connection drop-outs mid-operation, requiring full reconnect.
- Events (domain lifecycle, block job completion) may fire on the wrong thread, causing asyncio callbacks to be invoked from outside the event loop.

**Warning signs:**
- Segfaults with stack traces inside `libvirt.so` or `libvirtmod.cpython-*.so`.
- `virConnectPtr` unexpectedly becomes invalid after concurrent operations.
- libvirt event callbacks raising `RuntimeError: This event loop is already running` or similar asyncio threading violations.

**Prevention:**
- **One libvirt connection, one thread:** Create a single `virConnectPtr` at startup and access it exclusively from a dedicated `asyncio.Lock`. All libvirt calls in `cluster/libvirt_client.py` must acquire the lock before calling into libvirt.
- Alternatively, run all libvirt I/O in a single dedicated `ThreadPoolExecutor(max_workers=1)` thread. Use `asyncio.get_event_loop().run_in_executor(libvirt_executor, fn)` exclusively. This serializes all libvirt calls onto one thread without blocking the asyncio loop.
- **Do not** use a general-purpose `ThreadPoolExecutor` for libvirt calls — the single-thread executor guarantee is essential.
- For SSH (paramiko) calls: paramiko is thread-safe for independent `SSHClient` instances. Each node gets its own `SSHClient`; concurrent `node_exec_all` with a `ThreadPoolExecutor(max_workers=N)` is safe.

**Phase to address:** Core server architecture / libvirt client implementation phase.

---

### 10. Test Isolation Failure: SSH State Not in Snapshot

**Category:** state management | SSH

**What goes wrong:**
`snapshot_create` is intended to capture a stable, clean post-install state for reset. If the snapshot is taken too early — before the SSH daemon is running, before the host's SSH key is written to `authorized_keys`, or before static IP configuration is applied — then `snapshot_revert` restores the VM to a state where SSH is not functional. Every post-revert `wait_for_ssh` will time out.

A subtler variant: `snapshot_create` with `include_memory=False` captures disk state at a point in time. If the VM's network configuration is stored partially in volatile memory (e.g., `ip address add` without persisting to `/etc/network/interfaces`), after revert + reboot the VM has no IP and is unreachable.

**Root cause:**
Snapshot ordering discipline is not enforced by the tool. There is no built-in check that SSH is reachable before `snapshot_create` completes.

**Consequences:**
- All post-revert operations fail with `SSH_UNREACHABLE`.
- The only recovery is `cluster_destroy` + `cluster_define` + `cluster_start` + re-run all setup steps (which may take minutes).
- In automated agentic workflows, a bad snapshot can block an entire test suite run.

**Warning signs:**
- `snapshot_revert` completes but `wait_for_ssh` times out after every revert.
- `snapshot_create` was called during `cluster_start` before the post-boot wait completed.
- Snapshot was taken during `artifact_install` (disk state inconsistent mid-install).

**Prevention:**
- In `snapshot_create`, **verify SSH reachability before taking the snapshot**. If `ssh_reachable=False` on any node, fail with `NODES_NOT_SSH_READY` rather than creating a snapshot that will be unrecoverable.
- The canonical agent workflow should enforce: `cluster_start(wait_for_ssh=True)` → configure nodes → `snapshot_create` only after all setup commands have returned `exit_code=0`.
- Consider a `snapshot_verify` helper that does a dry-run `snapshot_revert` in a test subdomain (future optimization). For v1 minimum: document the invariant and add a pre-check in `snapshot_create`.
- Use **static IPs** (as the topology YAML already specifies with `dhcp: false`). This eliminates the DHCP lease expiry problem. Enforce that `/etc/network/interfaces` (or equivalent) is written before snapshot.

**Phase to address:** Snapshot management implementation phase; also cluster startup sequencing.

---

## Pitfall Priority Matrix

| Pitfall | Likelihood | Impact | Phase |
|---------|-----------|--------|-------|
| 9. asyncio + libvirt thread safety | HIGH | CRITICAL (segfault/data loss) | Architecture |
| 2. MCP exceptions bypass structured errors | HIGH | HIGH (agent contract broken) | MCP scaffolding |
| 3. SSH pool staleness after reboot | HIGH | HIGH (all post-revert ops fail) | SSH executor |
| 6. iptables state leakage on crash | HIGH | HIGH (silent impairment) | Net fault injection |
| 10. Test isolation / bad snapshot | MEDIUM | HIGH (test suite blocked) | Snapshot management |
| 1. Non-atomic cluster-wide snapshot revert | MEDIUM | HIGH (inconsistent cluster state) | Snapshot management |
| 7. cluster_define non-idempotency | MEDIUM | MEDIUM (silent topology drift) | Cluster lifecycle |
| 4. Artifact registry file corruption | LOW | HIGH (complete loss of artifact tracking) | Artifact registry |
| 5. Parallel SSH: partial failures invisible | HIGH | MEDIUM (silent partial success) | Parallel execution |
| 8. Overlay disk mutation / path collision | LOW | CRITICAL (base image corruption) | Domain XML generation |

---

## Phase-Specific Warnings

| Phase Topic | Likely Pitfall | Mitigation |
|-------------|---------------|------------|
| MCP server scaffolding | Exception contract violation (#2) | Implement top-level catch-all decorator before any tool is wired |
| libvirt client wrapper | Thread safety (#9) | Design the single-thread executor pattern first, before any libvirt code |
| Domain XML generation | Base image mutation (#8) | Add assertion: primary disk path ≠ base_image path; test this in CI |
| Snapshot management | Non-atomic revert (#1), bad snapshot state (#10) | shut-off-first revert strategy; SSH pre-check in snapshot_create |
| SSH executor | Pool staleness (#3) | Pool invalidation API on VM restart, not lazy detection |
| Artifact registry | File corruption (#4) | Atomic write pattern in first commit of registry.py |
| Parallel execution | Partial failure invisible (#5) | `all_succeeded` field in result model from day one |
| Net fault injection | iptables leakage (#6) | Persist fault registry to disk; startup reconciliation pass |
| Cluster lifecycle | cluster_define non-idempotency (#7) | Topology hash in libvirt domain metadata |

---

## Sources

- libvirt Python API documentation: https://libvirt.org/html/libvirt-libvirt-domain.html
- MCP Python SDK v1.x README: https://github.com/modelcontextprotocol/python-sdk/blob/main/README.md
- vmcluster-mcp design document: docs/vmcluster-mcp-design.md
- Project requirements: .planning/PROJECT.md
- Confidence: HIGH for libvirt (official C API docs verified), HIGH for MCP SDK (official README verified), HIGH for SSH/asyncio patterns (well-established Python idioms)
