# Features Research: vmcluster-mcp

**Domain:** MCP server for VM cluster orchestration (libvirt/QEMU, agent-driven test workflows)
**Researched:** 2026-03-22
**Confidence:** HIGH — primary source is the authoritative design doc (`docs/vmcluster-mcp-design.md`) supplemented by domain comparison of analogous tools (Vagrant, Terraform+libvirt, Ansible, Chaos tools, Test Kitchen).

---

## Table Stakes

> Features that must exist for the tool to be usable at all — agents cannot perform any meaningful work without these.

| Feature | Why Expected | Complexity | Notes |
|---------|--------------|------------|-------|
| `cluster_define` (topology → VM domains) | Without it, there is no cluster to work with — every other tool has no target | Med | libvirt XML generation + overlay disk creation from topology YAML |
| `cluster_start` + SSH wait | Start is useless if the agent can't know when nodes are ready | Med | Must poll SSH reachability; `wait_for_ssh=True` is the only safe default |
| `cluster_stop` / `cluster_destroy` | Without teardown, disk space and VM slots leak across runs | Low-Med | Graceful vs. hard kill are both necessary; `remove_overlays` on destroy is critical |
| `cluster_status` | Agents need ground truth before calling any other tool | Low | Aggregates libvirt domain state + SSH check + loaded modules |
| Topology-as-data (YAML schema) | Without a declarative topology, every agent run needs imperative construction — fragile and non-reproducible | Med | Cluster shape must be a data artifact, not an agent prompt; schema must validate on load |
| `node_exec` (single-node, typed result) | Foundation of everything downstream (install, verify, observe) | Low-Med | `ExecResult` with `exit_code`, `stdout`, `stderr`, `timed_out` is the minimum viable contract |
| `node_exec_all` (multi-node, parallel option) | Most cluster operations apply to all nodes | Med | Per-node result dict keyed by name; parallel=True is required, not optional |
| `snapshot_create` + `snapshot_revert` | Without snapshot/revert, every test failure requires full reprovisioning — unusable for iteration | Med-High | Must be cluster-atomic (all nodes or none); revert + restart is the "reset to clean" primitive |
| `artifact_sync` + `artifact_install` | Moving build outputs to nodes is the whole point of the tool for test workflows | Med | rsync-based, per-node skip-if-already-there; install modes must include `modprobe_path` |
| `artifact_register` | Artifacts must be registered before sync; provides the content-addressed identity | Med | Recursive sha256 of build tree; scans for `.ko`, userspace bins, test root |
| `ClusterHandle` return | The handoff interface to downstream consumers; without it the boundary doesn't exist | Low | Assembles `NodeDescriptor` list from running cluster state — data-only, no logic |
| `ToolResult[T]` structured error wrapper | Without structured errors, agents parse text or crash silently | Low | `success`, `result`, `error.code`, `error.recoverable` are the minimum fields; machine-readable error codes are non-negotiable |

---

## Differentiators

> Features that make vmcluster-mcp better than simpler alternatives (plain SSH scripts, Vagrant, Terraform, manual `virsh` calls). These do not merely add convenience — they enable agent workflows that are impossible or unreliable without them.

| Feature | Value Proposition | Complexity | Notes |
|---------|-------------------|------------|-------|
| Content-addressed artifacts (`artifact_id` = sha256 of tree) | Agents can call `artifact_sync` idempotently — the server skips nodes that already have the artifact. Eliminates O(n×transfers) waste per run. | Med | SHA256 of `.ko` files + userspace binary mtimes (design decision: fast hashing, not full tree sha256) |
| `dmesg_collect` with structured classification | Raw dmesg is useless to agents — they can't parse oops/panic/KASAN patterns from free text. Structured output with `oops_detected`, `panic_detected`, `warning_count` gives agents actionable signal. | Med | Pattern classification for oops, panic, KASAN, UBSAN, warn is what separates this from `node_exec("dmesg")` |
| `dmesg_mark` (timestamped kernel marker) | Agents can scope dmesg collection to "since I started this test" rather than the full boot log. Without this, every dmesg collection is noise-heavy. | Low | `/dev/kmsg` write returns a marker string; `dmesg_collect(since_marker=...)` filters before that time |
| `net_partition` (iptables-based network split) | Distributed systems tests require the ability to split nodes into communication-isolated groups. No simple SSH-script alternative can do this cleanly. | Med | iptables DROP on bridge; must be reversible via handle ID |
| `net_impair` (`tc netem` — latency, loss, corruption) | Testing degraded networks (not just full partitions) is essential for realistic distributed system fault exercises | High | tap interface detection per source node; netem parameters compose; teardown must be full and clean |
| `NetFaultHandle` + `net_fault_list` (fault inventory) | Agents that inject faults need to track what's active. Without an inventory, compound fault scenarios become unrecoverable. | Low-Med | In-memory active fault registry keyed by `handle_id`; stateless-server constraint means this lives in the fault registry, not persistent storage |
| `net_heal` / `net_heal_all` (targeted and global fault removal) | `net_heal_all` is the emergency reset for fault state, as important as `snapshot_revert` for cluster reset | Low | Paired with fault registry; `net_heal_all` clears everything |
| `node_crash` (hard reset via virsh destroy+start) | Models an unclean node failure — impossible to simulate reliably via SSH scripts without risking data corruption or leaving VM in inconsistent state | Low | `virsh destroy` + `virsh start`; `wait_for_ssh` option closes the loop for the agent |
| `artifact_build` (in-server build execution) | Keeps the agent workflow fully within the MCP boundary — agents don't need a separate build tool if sources are local | Med | Structured compiler errors (`CompilerError` with file/line/column/message) are what makes this valuable vs. shelling out |
| `artifact_diff` (two-artifact comparison) | Agents debugging regressions need to see what changed between builds without reasoning over file trees themselves | Low | Compares `.ko` hashes + userspace binary hashes + git log between SHAs |
| Parallel `node_exec_all` with per-node `ExecResult` | Cluster-wide commands (mount, modprobe, service start) must not serialize across N nodes at N×latency. Per-node result dict lets agents detect partial failures. | Med | asyncio or ThreadPoolExecutor internally; agent doesn't care about the mechanism, only the result contract |
| Stateless server (no in-memory critical state) | Agents in long-running workflows can tolerate MCP server restarts without losing cluster state. All persistent state is in libvirt or on disk. | Med | Exception: `NetFaultHandle` registry is in-memory — this is the only acknowledged "non-stateless" component and it's acceptable because faults are short-lived |
| Structured JSON logs to stderr | The MCP client host can aggregate logs across server and consumer tool calls for a unified audit trail without extra infrastructure | Low | `tool`, `cluster_name`, `node`, `duration_s`, `success`, `error_code` per log entry |
| `snapshot_list` + `snapshot_delete` | Agents managing long test campaigns need to prune old snapshots to avoid disk exhaustion | Low | List returns creation times and sizes; delete must operate on all nodes |

---

## Anti-Features (v1)

> Things to explicitly NOT build in the first milestone, with clear reasoning.

| Anti-Feature | Why Avoid | What to Do Instead |
|--------------|-----------|-------------------|
| SSH connection pooling / ControlMaster multiplexing | Adds state management complexity and a failure mode (stale connections); per-command connections are slower but correct and stateless | Per-command SSH via paramiko; design should accommodate pooling later (open design decision in the doc) |
| Full-tree recursive sha256 artifact hashing | Too slow for large kernel build trees (gigabytes); the SHA256-of-all-content semantics aren't needed | Hash `.ko` files + userspace binary mtimes (faster, sufficient for change detection per open design decision) |
| RDMA / soft-RoCE (o2ib) provisioning | `rxe_cfg` setup is non-trivial; topology schema includes the `rdma:` flag but the implementation is a separate milestone | Leave `rdma: false` as the only supported value in v1; schema is forward-compatible |
| Web UI / dashboard | Zero agent value; adds frontend complexity and a runtime process | Structured JSON logs to stderr give all the observability the human operator needs |
| Multi-user access control / RBAC | v1 is single-developer use; premature RBAC adds API surface and testing complexity | Document single-user assumption; revisit at milestone boundary if team expands |
| Remote libvirt hosts (non-local) | `qemu:///system` on localhost is the intended target; remote libvirt adds connectivity, credential, and TLS complexity | `libvirt_uri` is a config field so remote URIs are possible, but don't build or test for this in v1 |
| Memory-state snapshots as default | Memory snapshots are dramatically slower than disk-only snapshots; for test reset workflows, disk-only is almost always sufficient | `include_memory: bool = False` is the right default; option exists for advanced use but shouldn't be promoted |
| Container / Docker support | Different paradigm, different toolchain; would dilute the focus on VM-based realistic cluster testing | The tool is explicitly for VMs; containers get their own orchestration |
| Plugin system or extensibility hooks | Over-engineering for v1; the only consumer is the project author | Keep the interface flat; extension happens by adding tools in future milestones |
| Application-specific logic | Explicitly out of scope; makes the tool non-general-purpose | `ClusterHandle` is the boundary; downstream consumers implement application setup |
| Automated topology discovery / introspection (reverse-engineering running VMs into topology YAML) | Complex, fragile, not needed for the primary workflow | Topologies are always authored as YAML; the server doesn't infer topology from running state |
| `artifact_build` (v1 maybe, not core) | If sources are not local to the MCP host, this is useless; adds subprocess management complexity and compiler output parsing edge cases | Agents can build externally and call `artifact_register` with the output path; `artifact_build` is a convenience that can land after core features are stable |

---

## Feature Complexity Map

> Complexity estimates account for correctness requirements (idempotency, atomicity, stateless design), not just function call complexity.

| Feature | Complexity | Key Dependencies | Notes |
|---------|------------|-----------------|-------|
| `ToolResult[T]` error model + Pydantic models | Low | None — foundational | Must land first; all other features return these types |
| Topology YAML parser + schema validation | Med | `pydantic`, `pyyaml` | Schema must be strict; bad topologies should fail early with clear messages |
| `cluster_define` | Med | Topology parser, libvirt XML generation, overlay disk creation | libvirt domain XML is verbose; domain_builder needs careful templating |
| `cluster_start` + SSH wait loop | Med | `cluster_define`, `paramiko` | SSH polling with timeout and per-node reachability tracking |
| `cluster_stop` / `cluster_destroy` | Low-Med | libvirt client | Graceful vs. hard kill paths; `remove_overlays` disk cleanup edge cases |
| `cluster_status` | Low | libvirt client, SSH | Aggregation only — no mutation |
| `ClusterHandle` assembly | Low | `cluster_status`, topology | Data-only assembly from running state |
| `node_exec` | Low-Med | `paramiko`, `ExecResult` | Per-command SSH; timeout via asyncio |
| `node_exec_all` | Med | `node_exec` | Parallel execution via `asyncio` or `ThreadPoolExecutor`; result aggregation |
| `snapshot_create` | Med-High | libvirt external snapshots, all nodes running | External disk-only snapshots; must succeed on all nodes or roll back |
| `snapshot_revert` | Med-High | `snapshot_create`, `cluster_start` | Revert must be cluster-atomic; `restart_after` path reuses start logic |
| `snapshot_list` / `snapshot_delete` | Low | libvirt, overlay disk paths | List returns metadata; delete must clean up all per-node snapshots |
| `artifact_register` | Med | Hash implementation, directory scanner | `.ko` + binary scan; JSON registry persistence |
| `artifact_sync` | Med | `artifact_register`, rsync, SSH, per-node skip logic | rsync invocation via SSH; track per-node installed artifact_id |
| `artifact_install` | Med | `artifact_sync`, `node_exec` | Multiple install modes; `modprobe_path` is simplest and safest default |
| `artifact_list` | Low | JSON registry | Read-only |
| `artifact_diff` | Low | `artifact_register` (two artifacts) | Hash comparison + optional git log via subprocess |
| `artifact_build` | Med | subprocess, compiler error parser | Structured `CompilerError` extraction from make output; edge cases in error format parsing |
| `dmesg_mark` | Low | `node_exec`, `/dev/kmsg` | Write a stable marker string; return it for use in `dmesg_collect` |
| `dmesg_collect` + classification | Med | `node_exec`, `since_marker` filter, pattern classifier | Oops/panic/KASAN/UBSAN regex classification; structured `DmesgLine` output |
| `net_partition` | Med | iptables on bridge, `NetFaultHandle` registry | DROP rules between two node sets on bridge; handle-based reversal |
| `net_impair` (`tc netem`) | High | tap interface discovery, `tc` CLI, `NetFaultHandle` registry | tap interface naming per-node is environment-dependent; netem rule composition; teardown must be exact |
| `net_heal` / `net_heal_all` | Low | `NetFaultHandle` registry | Remove iptables/tc rules; registry cleanup |
| `net_fault_list` | Low | `NetFaultHandle` registry | Read-only registry dump |
| `node_crash` | Low | libvirt client, `cluster_start` | `virsh destroy` → `virsh start`; wait logic reuses start path |

---

## Build Order Implications

> Dependencies define what must land before what. This is the strict order — teams can parallelize within a tier but not across tiers.

```
Tier 1 — Foundation (no dependencies)
    ToolResult[T] + all Pydantic models (ClusterStatus, NodeStatus,
    ExecResult, SnapshotInfo, BuildArtifact, NetFaultHandle, ClusterHandle, …)
    → All features return these; must be stable before anything else ships.

Tier 2 — Topology + Infrastructure (depends on Tier 1)
    Topology YAML parser + schema validation
    libvirt client wrapper (thin API layer)
    → cluster_define depends on both; nothing works without VMs.

Tier 3 — Cluster Lifecycle (depends on Tier 2)
    cluster_define, cluster_start, cluster_stop, cluster_destroy, cluster_status
    → Required to have running nodes for every subsequent tier.

Tier 4 — Node Execution (depends on Tier 3)
    node_exec, node_exec_all
    → Used by artifact_install, dmesg tools, and indirectly by all verification logic.

Tier 5 — Snapshot Management (depends on Tier 3, must precede artifact work in workflow)
    snapshot_create, snapshot_revert, snapshot_list, snapshot_delete
    → The "reset to clean state" primitive. Must exist before artifact install
       is meaningful (install → snapshot → test → revert is the canonical loop).

Tier 6 — Artifact Management (depends on Tier 4 + Tier 5)
    artifact_register, artifact_sync, artifact_install, artifact_list
    → Core value delivery. Depends on exec (to install), snapshot (to reset after install).

Tier 7 — ClusterHandle (depends on Tier 6)
    cluster_handle
    → Returns a complete handle for downstream consumers; requires running cluster
       with artifact state populated.

Tier 8 — Observability (depends on Tier 4)
    dmesg_mark, dmesg_collect + classifier
    → Requires node_exec to write markers and collect dmesg output.

Tier 9 — Fault Injection (depends on Tier 3 + network/tap layer)
    net_partition, net_impair, net_heal, net_heal_all, net_fault_list
    node_crash
    → Requires running cluster. Network impairment (netem) is hardest; partition
       (iptables) and node_crash are simpler and can land first within this tier.

Tier 10 — Artifact Conveniences (depends on Tier 6)
    artifact_build   (convenience — build happens before register in primary workflow)
    artifact_diff    (nice-to-have comparison utility)
    → No blocking dependencies on other tiers. Safe to defer or land in parallel
       with Tier 9.
```

**Critical path:**
`models → topology → cluster_define → cluster_start → node_exec → snapshot_create → artifact_register → artifact_sync → artifact_install → cluster_handle`

This is the minimal end-to-end path that validates the core value proposition: agents can provision a cluster, install a build, snapshot the state, and hand off a `ClusterHandle` to a downstream consumer.

---

## Feature Dependencies Graph (summary)

```
cluster_define ──────────────────────────────────────────┐
     │                                                    │
cluster_start ─────────────────────────────┐             │
     │                                     │             │
node_exec ────────────┐                    │             │
     │                │                   │             │
     │          artifact_register          │             │
     │                │                   │             │
     │          artifact_sync ────────────┤             │
     │                │                  │              │
     │          artifact_install          │              │
     │                │                  │              │
     │          snapshot_create ──────────┘              │
     │                │                                   │
     │          snapshot_revert (reuses start)            │
     │                                                    │
     └───── dmesg_mark ──── dmesg_collect                 │
     │                                                    │
     └───── net_partition / net_impair                    │
                  │                                       │
             net_heal / net_heal_all                      │
                                                          │
cluster_handle ───────────────────────────────────────────┘
  (assembled from cluster_status + artifact state)
```

## Sources

- `docs/vmcluster-mcp-design.md` — Authoritative design document (HIGH confidence — primary source)
- `PROJECT.md` — Requirements, constraints, and key decisions (HIGH confidence — authoritative)
- Domain comparison: Vagrant, HashiCorp Nomad, Terraform+libvirt provider, Ansible, Chaos Monkey, tc-netem tooling, Test Kitchen (MEDIUM confidence — well-known ecosystem tools, no web lookup required for feature inventory)
