# MCP Server Design: VM Cluster MCP

**Status:** Design — Ready for Architectural Planning  
**Scope:** General-purpose MCP server for agentic multi-VM cluster orchestration, build artifact distribution, snapshotting, node execution, and fault injection.

-----

## 1. Overview

VM Cluster MCP (`vmcluster-mcp`) is general-purpose. It manages the lifecycle of multi-VM clusters using libvirt/QEMU. It knows about nodes, networks, snapshots, build artifacts, and fault injection. It has no knowledge of any specific distributed filesystem or application stack.

### Handoff Interface

The handoff boundary is the `ClusterHandle` — a lightweight descriptor the VM Cluster MCP produces for downstream consumers:

```python
class ClusterHandle(BaseModel):
    cluster_name: str
    nodes: list[NodeDescriptor]
    artifact_path: str        # Absolute path on nodes where build tree is installed
    kernel_version: str       # uname -r of running kernel on nodes

class NodeDescriptor(BaseModel):
    name: str
    role: str                 # Interpreted by downstream consumers
    ip: str
    ssh_port: int = 22
    ssh_user: str = "root"
    ssh_key_path: str
```

Downstream consumers do not call back into the VM Cluster MCP at runtime. They SSH directly into nodes using the handle. This means the VM Cluster MCP is equally usable for any software stack deployed onto VMs, physical hardware, or cloud instances — the `ClusterHandle` is the only coupling.

### Canonical Agent Workflow

```
[VM Cluster MCP]
  cluster_define("example-3node")
  cluster_start("example-3node")
  artifact_sync("example-3node", artifact_id)
  artifact_install("example-3node", artifact_id)
  snapshot_create("example-3node", "post-install-clean")
  → returns ClusterHandle

[Downstream consumer]
  use ClusterHandle to connect to nodes directly and perform application-specific configuration and testing

[VM Cluster MCP]  (on failure or completion)
  snapshot_revert("example-3node", "post-install-clean")
```

-----

## 2. VM Cluster MCP (`vmcluster-mcp`)

### 2.1 Design Goals

- **Topology-as-data:** Cluster shape is declared in a YAML file, not constructed imperatively by the agent.
- **Atomic cluster operations:** Snapshot revert, bring-up, and teardown operate on the whole cluster or named node sets, not individual VMs.
- **Structured outputs:** All tools return typed Pydantic models serialized as JSON. Agents never parse free-text to determine success or failure.
- **Content-addressed artifacts:** Build artifacts are identified by hash, not filename. Nodes track which artifact is currently installed and skip redundant transfers.
- **Stateless server:** No critical runtime state lives in the server process. All persistent state lives in libvirt and on disk. The server can restart freely.
- **Fault injection as first-class capability:** Network partitions and node crashes are explicit tools, not workarounds.

### 2.2 Topology File Schema

Topology files are YAML, stored in a configured directory on the host. The agent references topologies by name. The server resolves names to files.

```yaml
# /etc/vmcluster/topologies/example-3node.yaml

cluster_name: example-3node
base_image: /var/lib/vmcluster/images/ubuntu-6.8-base.qcow2
overlay_dir: /var/lib/vmcluster/overlays/

network:
  name: clusternet-example
  bridge: virbr-example0
  subnet: 192.168.100.0/24
  dhcp: false         # Static IPs only; server writes /etc/hosts on each node
  rdma: false         # If true, provisions soft-RoCE (rxe) on the bridge

nodes:
  - name: controller
    role: control
    vcpus: 2
    memory_mb: 2048
    ip: 192.168.100.10
    extra_disks:
      - path: /var/lib/vmcluster/disks/data0.qcow2
        size_gb: 20
        bus: virtio

  - name: worker-0
    role: worker
    vcpus: 2
    memory_mb: 2048
    ip: 192.168.100.11
    extra_disks:
      - path: /var/lib/vmcluster/disks/data1.qcow2
        size_gb: 50
        bus: virtio

  - name: client-0
    role: client
    vcpus: 2
    memory_mb: 1024
    ip: 192.168.100.20

ssh:
  key_path: /etc/vmcluster/ssh/vmcluster_id_ed25519
  user: root
  connect_timeout_s: 30

snapshots:
  baseline: clean-boot     # Snapshot to use for snapshot_revert("baseline")
```

### 2.3 Build Artifact Model

```python
class BuildArtifact(BaseModel):
    artifact_id: str              # sha256 of the build tree content
    source_path: str              # Local path on the MCP host
    build_type: Literal["raw_tree", "make_install", "rpm", "deb"]
    kernel_version: str           # Kernel this was built against
    modules: list[str]            # Relative paths to .ko files within the tree
    userspace_bins: list[str]     # Relative paths to userspace tools
    test_framework_path: str | None  # Relative path to test scripts root (e.g. "tests")
    git_sha: str | None
    git_branch: str | None
    build_timestamp: str          # ISO 8601
    metadata: dict[str, str]      # Arbitrary key/value for agent annotation
```

Artifacts are registered once per build and referenced by `artifact_id` for all subsequent operations. The server maintains a JSON registry file on disk.

### 2.4 Tool Definitions

#### Topology & Cluster Lifecycle

```
cluster_define(topology_name: str) -> ClusterStatus
```

Instantiate VMs and network from a topology file. Creates libvirt domain definitions and per-node overlay disk images from the base image. Does not boot VMs. Idempotent — safe to call on an already-defined cluster.

```
cluster_start(
    cluster_name: str,
    wait_for_ssh: bool = True,
    ssh_timeout_s: int = 120
) -> ClusterStatus
```

Boot all nodes in the cluster. If `wait_for_ssh=True`, blocks until all nodes are reachable via SSH or timeout is reached. Returns per-node reachability status.

```
cluster_stop(
    cluster_name: str,
    mode: Literal["shutdown", "destroy"] = "shutdown"
) -> ClusterStatus
```

Gracefully shut down (`virsh shutdown`) or hard-kill (`virsh destroy`) all nodes.

```
cluster_destroy(
    cluster_name: str,
    remove_overlays: bool = True
) -> bool
```

Undefine all VM domains and optionally delete overlay disk images. Base images and topology files are never touched.

```
cluster_status(cluster_name: str) -> ClusterStatus
```

Return current state of all nodes: domain state (running/shut off/paused), SSH reachability, loaded kernel modules visible via SSH, current artifact_id installed (if any).

```
cluster_handle(cluster_name: str) -> ClusterHandle
```

Return the `ClusterHandle` descriptor for the running cluster. This is what gets passed to downstream consumers.

```python
class ClusterStatus(BaseModel):
    cluster_name: str
    network_active: bool
    nodes: list[NodeStatus]

class NodeStatus(BaseModel):
    name: str
    role: str
    ip: str
    domain_state: Literal["running", "shut off", "paused", "crashed", "unknown"]
    ssh_reachable: bool
    artifact_id_installed: str | None
    loaded_modules: list[str]
```

#### Snapshot Management

```
snapshot_create(
    cluster_name: str,
    snapshot_name: str,
    include_memory: bool = False
) -> SnapshotStatus
```

Create a named disk snapshot of all nodes simultaneously using libvirt’s external snapshot mechanism. `include_memory=True` adds a memory dump (slower but captures in-flight state). Returns per-node snapshot paths and sizes.

```
snapshot_revert(
    cluster_name: str,
    snapshot_name: str,
    restart_after: bool = True,
    wait_for_ssh: bool = True
) -> ClusterStatus
```

Revert all nodes to a named snapshot atomically. If `restart_after=True`, reboots all nodes after revert and waits for SSH. This is the canonical “reset to clean state” operation.

```
snapshot_list(cluster_name: str) -> list[SnapshotInfo]
```

List all named snapshots for the cluster with creation time and size.

```
snapshot_delete(cluster_name: str, snapshot_name: str) -> bool
```

Delete a named snapshot from all nodes.

#### Artifact Management

```
artifact_register(
    source_path: str,
    build_type: Literal["raw_tree", "make_install", "rpm", "deb"],
    kernel_version: str | None = None,
    metadata: dict[str, str] | None = None
) -> BuildArtifact
```

Register a local build tree as a named artifact. Computes `artifact_id` from a recursive sha256 of the tree content. Scans for `.ko` files, userspace binaries, and a test framework root. Does not transfer any files.

```
artifact_build(
    source_path: str,
    kernel_src: str,
    configure_args: list[str] | None = None,
    jobs: int = 8,
    metadata: dict[str, str] | None = None
) -> ArtifactBuildResult
```

Run `configure` + `make` in `source_path` against kernel headers at `kernel_src`. Registers and returns the artifact on success. Returns structured compiler errors (file, line, message) extracted from make output rather than raw stdout.

```python
class ArtifactBuildResult(BaseModel):
    success: bool
    artifact: BuildArtifact | None
    errors: list[CompilerError]
    warnings: list[CompilerWarning]
    duration_s: float
    stdout_tail: str      # Last 50 lines of make output for context

class CompilerError(BaseModel):
    file: str
    line: int
    column: int | None
    message: str
```

```
artifact_list() -> list[BuildArtifact]
```

List all registered artifacts.

```
artifact_diff(artifact_id_a: str, artifact_id_b: str) -> ArtifactDiff
```

Compare two artifacts: which `.ko` files changed (by hash), which userspace binaries changed, git log between their SHAs if available.

```
artifact_sync(
    cluster_name: str,
    artifact_id: str,
    nodes: list[str] | Literal["all"] = "all",
    force: bool = False,
    dest_base: str = "/opt/vmcluster/artifacts"
) -> SyncStatus
```

rsync the artifact tree to `{dest_base}/{artifact_id}/` on each target node. Skips nodes where this `artifact_id` is already present unless `force=True`. Returns per-node: bytes transferred, duration, rsync exit code.

```
artifact_install(
    cluster_name: str,
    artifact_id: str,
    nodes: list[str] | Literal["all"] = "all",
    install_mode: Literal["make_install", "rpm", "deb", "modprobe_path"] = "modprobe_path"
) -> InstallStatus
```

Run installation on each node after sync.

- `make_install`: Runs `make install` in synced tree. Writes to system paths.
- `rpm` / `deb`: Installs packages found in artifact tree.
- `modprobe_path`: Writes `/etc/modprobe.d/vmcluster-test.conf` pointing at synced module paths. Does not touch system paths. Trivially reverted by deleting the conf file. **Recommended default for test clusters.**

Returns per-node: exit code, installed module paths, `depmod` output.

#### Node Execution & Observation

```
node_exec(
    cluster_name: str,
    node: str,
    command: str,
    timeout_s: int = 30,
    env: dict[str, str] | None = None
) -> ExecResult
```

Run a shell command on a single node via SSH.

```
node_exec_all(
    cluster_name: str,
    command: str,
    nodes: list[str] | Literal["all"] = "all",
    timeout_s: int = 30,
    parallel: bool = True
) -> dict[str, ExecResult]
```

Run the same command on multiple nodes, optionally in parallel. Returns a dict keyed by node name.

```python
class ExecResult(BaseModel):
    exit_code: int
    stdout: str
    stderr: str
    duration_s: float
    timed_out: bool
```

```
dmesg_mark(
    cluster_name: str,
    nodes: list[str] | Literal["all"] = "all"
) -> dict[str, str]
```

Write a timestamped marker into dmesg on each node via `/dev/kmsg`. Returns the marker string per node. Use before any operation so subsequent `dmesg_collect` calls return only lines from after the marker.

```
dmesg_collect(
    cluster_name: str,
    nodes: list[str] | Literal["all"] = "all",
    since_marker: str | None = None,
    filter_level: Literal["all", "warn+", "err+"] = "all"
) -> dict[str, DmesgResult]
```

Collect and classify dmesg from nodes. Returns structured output, not raw strings.

```python
class DmesgResult(BaseModel):
    lines: list[DmesgLine]
    oops_detected: bool
    panic_detected: bool
    warning_count: int
    error_count: int

class DmesgLine(BaseModel):
    timestamp: float
    level: Literal["debug", "info", "notice", "warning", "err", "crit", "alert", "emerg"]
    facility: str
    text: str
    classified_as: Literal["oops", "panic", "warn", "kasan", "ubsan", "normal"] | None
```

#### Network Fault Injection

```
net_partition(
    cluster_name: str,
    partition_a: list[str],
    partition_b: list[str]
) -> NetFaultHandle
```

Install iptables DROP rules on the bridge to block traffic between `partition_a` and `partition_b` node sets. Nodes within the same partition are unaffected. Returns a handle for later removal.

```
net_impair(
    cluster_name: str,
    source_node: str,
    target_node: str | Literal["all"],
    latency_ms: int = 0,
    jitter_ms: int = 0,
    loss_pct: float = 0.0,
    corrupt_pct: float = 0.0,
    reorder_pct: float = 0.0
) -> NetFaultHandle
```

Apply `tc netem` impairment on the tap interface for `source_node`’s egress traffic toward `target_node(s)`. Returns a handle for later removal.

```
net_heal(cluster_name: str, fault_handle: str) -> bool
```

Remove a specific network fault by handle ID.

```
net_heal_all(cluster_name: str) -> int
```

Remove all active network faults for the cluster. Returns count of faults removed.

```
net_fault_list(cluster_name: str) -> list[NetFaultInfo]
```

List all currently active network faults and their parameters.

```
node_crash(
    cluster_name: str,
    node: str,
    restart_after: bool = True,
    wait_for_ssh: bool = True
) -> NodeStatus
```

Hard-reset a node via `virsh destroy` then `virsh start`. Models an unclean node failure. If `restart_after=True`, starts the node and optionally waits for SSH.

```python
class NetFaultHandle(BaseModel):
    handle_id: str
    fault_type: Literal["partition", "impairment"]
    affected_nodes: list[str]
    created_at: str
```

### 2.5 Internal Module Structure

```text
vmcluster_mcp/
├── server.py              # MCP server entrypoint, tool registration, stdio transport
├── config.py              # Server configuration: topology_dir, overlay_dir, ssh settings
├── models.py              # All Pydantic models shared across modules
│
├── topology/
│   ├── parser.py          # YAML topology loading and validation
│   └── schema.py          # Topology dataclasses
│
├── cluster/
│   ├── manager.py         # cluster_define, cluster_start, cluster_stop, cluster_destroy
│   ├── libvirt_client.py  # Thin wrapper around libvirt Python API
│   └── domain_builder.py  # Generates libvirt XML domain definitions from topology
│
├── snapshot/
│   └── manager.py         # snapshot_create, snapshot_revert, snapshot_list, snapshot_delete
│
├── artifact/
│   ├── registry.py        # Artifact registration, content hashing, JSON registry persistence
│   ├── builder.py         # Local build execution, compiler error extraction
│   ├── syncer.py          # rsync management, per-node sync state tracking
│   └── installer.py       # make_install, rpm, deb, modprobe_path installation modes
│
├── exec/
│   ├── ssh.py             # SSH connection pool, command execution, SCP/rsync wrappers
│   └── parallel.py        # Parallel execution across nodes with result aggregation
│
├── observe/
│   ├── dmesg.py           # dmesg collection, marker insertion, classification
│   └── classifier.py      # Oops/panic/KASAN/UBSAN pattern detection
│
└── net/
    ├── fault.py           # net_partition, net_impair, net_heal, net_heal_all
    ├── bridge.py          # Bridge and tap interface management
    └── fault_registry.py  # In-memory active fault tracking (keyed by handle_id)
```

### 2.6 Configuration

The server is configured via environment variables or a config file at `/etc/vmcluster/config.yaml`:

```yaml
topology_dir: /etc/vmcluster/topologies
overlay_dir: /var/lib/vmcluster/overlays
artifact_registry: /var/lib/vmcluster/artifacts/registry.json
artifact_store_dir: /var/lib/vmcluster/artifacts/trees
ssh_key_path: /etc/vmcluster/ssh/vmcluster_id_ed25519
ssh_user: root
libvirt_uri: qemu:///system
log_level: INFO
```

### 2.7 Server Entrypoint

```bash
# Install
pip install vmcluster-mcp

# Run (stdio mode for MCP client consumption)
python -m vmcluster_mcp.server

# MCP client configuration (e.g. Claude Desktop, Cursor)
{
  "mcpServers": {
    "vmcluster": {
      "command": "python",
      "args": ["-m", "vmcluster_mcp.server"],
      "env": {
        "VMCLUSTER_CONFIG": "/etc/vmcluster/config.yaml"
      }
    }
  }
}
```

### 2.8 Error Model

All tools return structured errors rather than raising exceptions to the agent. The agent can inspect `success` and `error` fields without try/catch logic.

```python
class ToolResult(BaseModel, Generic[T]):
    success: bool
    result: T | None
    error: ErrorDetail | None

class ErrorDetail(BaseModel):
    code: str           # Machine-readable: "SSH_UNREACHABLE", "SNAPSHOT_NOT_FOUND", etc.
    message: str        # Human-readable description
    node: str | None    # Which node caused the error, if applicable
    recoverable: bool   # Whether retrying or reverting to snapshot may resolve this
```

-----

## 3. Cross-Cutting Concerns

### 3.1 Shared SSH Executor

The VM Cluster MCP needs SSH execution. Options:

1. **Duplicate the module** in each package (simple, no shared dependency).
1. **Extract to a shared library** `vmcluster-ssh` installed as a dependency of multiple consumers (cleaner, adds coordination overhead).
1. **Downstream consumers reuse the SSH config from ClusterHandle** (SSH key path, user, port are in `NodeDescriptor`) and maintain their own SSH pool.

Recommendation: Option 3. The `ClusterHandle` carries all SSH configuration. Each server or consumer maintains its own SSH connection pool keyed by IP. No shared library needed.

### 3.2 Logging and Observability

The server should write structured JSON logs to stderr (captured by the MCP client). Log entries include:

- `tool`: Tool name being executed
- `cluster_name`: Cluster being acted on
- `node`: Node being acted on (where applicable)
- `duration_s`: Execution time
- `success`: Boolean outcome
- `error_code`: On failure

This allows the agent’s host environment to aggregate logs across infrastructure and application-specific test sessions for a unified audit trail.

### 3.3 Concurrency Model

The server handles one tool call at a time per MCP connection (stdio transport is inherently sequential). Parallelism within a tool call (e.g., `node_exec_all`) is managed internally using `asyncio` or `ThreadPoolExecutor`. No inter-tool concurrency is expected or required.

### 3.4 Dependency Summary

**VM Cluster MCP:**

```text
libvirt-python    # VM and network management
paramiko          # SSH execution
pydantic          # Model validation and serialization
pyyaml            # Topology file parsing
mcp               # MCP Python SDK (stdio transport)
```

-----

## 4. Open Design Decisions

The following items are intentionally left for detailed architectural planning:

- **Libvirt snapshot strategy:** External disk-only snapshots vs. internal snapshots vs. full memory snapshots. Performance and revert-time tradeoffs need benchmarking against realistic cluster images.
  - Optimize for speed first.
- **SSH connection pooling:** Whether to use persistent multiplexed SSH connections (`ControlMaster`) or per-command connections. Multiplexed connections are faster but add state management complexity.
  - start with per-command. Architecture should accommodate connection pooling in the future
- **Artifact content hashing strategy:** Full recursive sha256 is correct but slow for large build trees. A faster alternative (hash of `.ko` files + binary mtimes) may be acceptable.
  - hash of `.ko` + binary m times.
- **rdma/o2ib support in VM Cluster MCP:** The topology schema includes `rdma: true/false` but the implementation of soft-RoCE provisioning via `rxe_cfg` is non-trivial and should be treated as a separate implementation milestone.
  - Not part of initial implementation
