# Architecture Research: vmcluster-mcp

**Project:** vmcluster-mcp  
**Researched:** 2026-03-22  
**Source confidence:** HIGH — derived directly from the authoritative design doc (`docs/vmcluster-mcp-design.md`)

---

## Component Map

### `server.py` — MCP Entrypoint
The single entry point for the stdio MCP server. Registers all tool handlers by name, dispatches incoming calls to the appropriate module, and wraps results in `ToolResult[T]` before serializing to stdout. This is the only file the MCP client interacts with. It owns no domain logic — it is a pure routing and serialization layer.

### `config.py` — Server Configuration
Loads and validates server configuration from `/etc/vmcluster/config.yaml` or environment variables. Provides a singleton `Config` object to all other modules. Contains: `topology_dir`, `overlay_dir`, `artifact_registry`, `artifact_store_dir`, `ssh_key_path`, `ssh_user`, `libvirt_uri`, `log_level`.

### `models.py` — Shared Pydantic Models
Defines all structured types used across module boundaries: `ClusterHandle`, `ClusterStatus`, `NodeStatus`, `NodeDescriptor`, `BuildArtifact`, `ExecResult`, `SnapshotInfo`, `NetFaultHandle`, `DmesgResult`, `DmesgLine`, `ToolResult[T]`, `ErrorDetail`, and all input/output models for each tool. This is the single source of truth for the data contract between modules and the agent.

### `topology/` — Topology Loading & Validation
- `parser.py`: Reads YAML topology files from `Config.topology_dir`, validates schema, returns typed `TopologySpec` objects.
- `schema.py`: Dataclass definitions for topology structure (nodes, network, ssh config, snapshot defaults).

**Boundary:** Strictly read-only. Produces `TopologySpec`, consumes nothing from other domain modules.

### `cluster/` — VM Lifecycle Management
- `manager.py`: Implements `cluster_define`, `cluster_start`, `cluster_stop`, `cluster_destroy`, `cluster_status`, `cluster_handle`. Orchestrates the libvirt layer and calls `topology/` to resolve cluster shape.
- `libvirt_client.py`: Thin wrapper around `libvirt-python`. Abstracts domain lookup, start, shutdown, destroy, and undefine. Single point of libvirt interaction.
- `domain_builder.py`: Translates a `TopologySpec` node entry into a libvirt XML domain definition string. Also generates overlay qcow2 disk creation commands.

**Boundary:** Reads `TopologySpec` from `topology/`. Writes to libvirt and disk. Returns `ClusterStatus` and `ClusterHandle`.

### `snapshot/` — Snapshot Lifecycle
- `manager.py`: Implements `snapshot_create`, `snapshot_revert`, `snapshot_list`, `snapshot_delete` using libvirt's external snapshot mechanism. Operates on all nodes of a cluster simultaneously.

**Boundary:** Requires the cluster to exist in libvirt (depends on `cluster/` having run `cluster_define`). Does not use SSH. Returns `SnapshotStatus` / `list[SnapshotInfo]`.

### `exec/` — SSH Execution Layer
- `ssh.py`: Per-command SSH connections via `paramiko`. Executes shell commands, handles timeouts, and wraps output in `ExecResult`. Also provides rsync wrappers used by `artifact/syncer.py`. The connection architecture is per-command in v1 with the structure designed to accommodate `ControlMaster`-style pooling later.
- `parallel.py`: Runs the same `ssh.py` operation across a list of nodes using `asyncio` or `ThreadPoolExecutor`, collects results into `dict[str, ExecResult]`.

**Boundary:** Requires `NodeDescriptor` (ip, ssh_port, ssh_user, ssh_key_path). Consumed by `artifact/`, `observe/`, and indirectly by `cluster/` (for SSH-ready waiting).

### `artifact/` — Build Artifact Lifecycle
- `registry.py`: Content-addressed registration. Hashes `.ko` files + binary mtimes. Persists artifacts as JSON at `Config.artifact_registry`. Provides lookup by `artifact_id`.
- `builder.py`: Shells out `configure` + `make` against kernel headers. Parses compiler stderr into `list[CompilerError]`. Calls `registry.py` on success.
- `syncer.py`: rsync of artifact tree to `{dest_base}/{artifact_id}/` on each target node via `exec/ssh.py`. Tracks which node already has which artifact_id to skip redundant transfers.
- `installer.py`: Runs install mode (`make_install`, `rpm`, `deb`, `modprobe_path`) on each target node via `exec/ssh.py`. Updates `NodeStatus.artifact_id_installed`.

**Boundary:** Depends on `exec/` for all remote operations. Returns `BuildArtifact`, `SyncStatus`, `InstallStatus`.

### `observe/` — Kernel Observability
- `dmesg.py`: Implements `dmesg_mark` (writes to `/dev/kmsg`) and `dmesg_collect` (retrieves and filters dmesg lines) via `exec/ssh.py`. Returns `DmesgResult`.
- `classifier.py`: Stateless classifier — takes raw dmesg text and classifies each line as oops, panic, warn, KASAN, UBSAN, or normal using regex patterns.

**Boundary:** Depends on `exec/` for SSH. No dependency on `cluster/`, `snapshot/`, or `artifact/`. Returns `dict[str, DmesgResult]`.

### `net/` — Network Fault Injection
- `fault.py`: Implements `net_partition` (iptables DROP rules on bridge), `net_impair` (tc netem on tap interface), `net_heal`, `net_heal_all`, `net_fault_list`. Shells out to `iptables` and `tc` on the host directly (not via SSH).
- `bridge.py`: Bridge and tap interface name resolution — maps node names to tap device names on the host by querying libvirt domain XML.
- `fault_registry.py`: In-memory `dict[str, NetFaultHandle]` keyed by `handle_id`. The one module with intentional runtime state. Not persisted — faults do not survive server restart, which is acceptable since the topology reset path (snapshot_revert) also heals network state.

**Boundary:** Requires libvirt (via `bridge.py` for tap name lookup) but no dependency on `exec/`. Host-side iptables/tc operations only. Returns `NetFaultHandle`, `NetFaultInfo`.

---

## Data Flow

### Standard Tool Call Path

```
MCP Client (agent)
  │
  │ JSON-RPC over stdio
  ▼
server.py  ──→  deserializes args, validates input types
  │
  │ calls module function with typed args
  ▼
Module handler (e.g. cluster/manager.py)
  │
  ├── reads config from config.py
  ├── loads topology via topology/parser.py (if cluster-scoped)
  │
  ├──[VM operations]──→ cluster/libvirt_client.py ──→ libvirt (qemu:///system)
  ├──[SSH operations]──→ exec/ssh.py ──→ paramiko ──→ guest node
  ├──[disk operations]──→ file I/O on host filesystem
  │
  │ returns typed Pydantic model (e.g. ClusterStatus)
  ▼
server.py  ──→  wraps in ToolResult[T]{success, result, error}
  │
  │ serializes to JSON
  ▼
MCP Client (agent)
```

### Error Propagation Path

Exceptions from `libvirt_client.py`, `exec/ssh.py`, or OS calls are caught at the module boundary and converted to `ErrorDetail(code, message, node, recoverable)`. The module returns `ToolResult(success=False, result=None, error=ErrorDetail(...))`. The agent receives structured JSON — never a raw Python exception or unstructured traceback.

### Artifact Workflow (multi-step data flow)

```
artifact_register(source_path)
  └─→ artifact/registry.py: hash .ko + binaries → BuildArtifact → write registry.json

artifact_sync(cluster_name, artifact_id)
  └─→ artifact/syncer.py → exec/ssh.py → rsync to nodes → SyncStatus

artifact_install(cluster_name, artifact_id, install_mode)
  └─→ artifact/installer.py → exec/ssh.py → install commands on nodes → InstallStatus
```

### Snapshot Workflow

```
snapshot_create(cluster_name, snapshot_name)
  └─→ snapshot/manager.py → cluster/libvirt_client.py → libvirt external snapshot API
                                                         (disk-only, all nodes parallel)

snapshot_revert(cluster_name, snapshot_name, restart_after=True)
  └─→ snapshot/manager.py → libvirt revert → cluster/manager.py cluster_start (if restart_after)
                                           → exec/ssh.py wait for SSH (if wait_for_ssh)
```

---

## Component Dependencies

```
models.py         ← no dependencies (leaf node, imported everywhere)
config.py         ← no dependencies (leaf node, imported everywhere)

topology/parser   ← config.py, models.py
topology/schema   ← models.py

cluster/libvirt_client  ← config.py, models.py
cluster/domain_builder  ← topology/schema, models.py
cluster/manager         ← topology/parser, cluster/libvirt_client, cluster/domain_builder,
                          exec/ssh (for wait-for-SSH), models.py

exec/ssh          ← config.py, models.py
exec/parallel     ← exec/ssh, models.py

snapshot/manager  ← cluster/libvirt_client, models.py

artifact/registry ← config.py, models.py
artifact/builder  ← artifact/registry, models.py
artifact/syncer   ← exec/ssh, artifact/registry, models.py
artifact/installer← exec/ssh, artifact/registry, models.py

observe/classifier← models.py
observe/dmesg     ← exec/ssh, observe/classifier, models.py

net/fault_registry← models.py
net/bridge        ← cluster/libvirt_client, models.py
net/fault         ← net/bridge, net/fault_registry, models.py

server.py         ← ALL modules (routing layer)
```

**Critical invariant:** `models.py` and `config.py` must have zero internal cross-dependencies. They are the foundation all other modules stand on.

---

## Suggested Phase Groupings

Phase groupings follow the dependency graph — each phase only requires what prior phases established.

### Phase 1: Foundation
**Modules:** `models.py`, `config.py`, `topology/`, `server.py` (skeleton)  
**Deliverables:** Server boots and accepts MCP connections. Topology files load and validate. Config resolves from file/env.  
**Why first:** Every downstream module imports these. No integration surface area — pure data shapes and config.

### Phase 2: Core VM Lifecycle
**Modules:** `cluster/libvirt_client.py`, `cluster/domain_builder.py`, `cluster/manager.py`  
**Tools unlocked:** `cluster_define`, `cluster_start`, `cluster_stop`, `cluster_destroy`, `cluster_status`, `cluster_handle`  
**Why here:** Cluster operations are the core product value. Snapshot, exec, and artifact all require a running cluster. `exec/` (SSH) is needed only for `wait_for_ssh` in `cluster_start` — this can be a thin blocking loop without the full SSH module initially, then replaced.

### Phase 3: SSH Execution Layer
**Modules:** `exec/ssh.py`, `exec/parallel.py`  
**Tools unlocked:** `node_exec`, `node_exec_all`  
**Why here:** SSH is the prerequisite for artifact sync/install and observe. Building it as a standalone phase with its own `node_exec` tools verifies it works before higher-level callers depend on it.

### Phase 4: Snapshots
**Modules:** `snapshot/manager.py`  
**Tools unlocked:** `snapshot_create`, `snapshot_revert`, `snapshot_list`, `snapshot_delete`  
**Why here:** Snapshots only require libvirt (already established in Phase 2). No SSH dependency. Unblocks the canonical workflow: define → start → snapshot → reset loop. This is the "reliable test reset" capability and is high value early.

### Phase 5: Artifact Pipeline
**Modules:** `artifact/registry.py`, `artifact/syncer.py`, `artifact/installer.py`, `artifact/builder.py`  
**Tools unlocked:** `artifact_register`, `artifact_sync`, `artifact_install`, `artifact_list`, `artifact_diff`, `artifact_build`  
**Why here:** Requires `exec/` (Phase 3). The order within phase: registry first (no SSH), then syncer and installer (need SSH), then builder (optional, needs host build toolchain). `artifact_build` can ship last or be deferred.

### Phase 6: Kernel Observability
**Modules:** `observe/classifier.py`, `observe/dmesg.py`  
**Tools unlocked:** `dmesg_mark`, `dmesg_collect`  
**Why here:** Pure consumer of `exec/ssh`. Low dependency surface. The classifier is a stateless regex engine testable in isolation.

### Phase 7: Network Fault Injection
**Modules:** `net/bridge.py`, `net/fault_registry.py`, `net/fault.py`  
**Tools unlocked:** `net_partition`, `net_impair`, `net_heal`, `net_heal_all`, `net_fault_list`, `node_crash`  
**Why last:** Highest operational risk (host iptables/tc mutation). Requires `cluster/libvirt_client` for tap name resolution. Deliberately deferred until all other core capabilities are validated. `fault_registry.py` in-memory state is isolated to this phase.

---

## Integration Points

These are the seams between modules where interface design matters most:

### `topology/` → `cluster/manager.py`
`TopologySpec` is the contract. Any topology schema change breaks `domain_builder.py`. Keep topology parsing strictly encapsulated — `cluster/` must never reach into YAML directly.

### `exec/ssh.py` → `artifact/syncer.py`, `artifact/installer.py`, `observe/dmesg.py`
The `ExecResult` model and the rsync wrapper signatures are the shared contract. All three callers are affected if `ssh.py` changes its return shape. Version this interface carefully. The `NodeDescriptor` (from `ClusterHandle`) is the sole credential source — no module should independently manage SSH credentials.

### `cluster/manager.py` → `snapshot/manager.py`
Snapshot operations require the cluster to exist as libvirt domains. `snapshot/manager.py` should call `cluster/libvirt_client.py` directly rather than going through `cluster/manager.py` — avoids a circular dependency if `snapshot_revert` triggers `cluster_start` (revert can call both independently).

### `net/bridge.py` → `cluster/libvirt_client.py`
Tap device name resolution requires parsing libvirt domain XML. This is the only point where `net/` couples to `cluster/`. Keep it narrow: `bridge.py` only needs one function — "given node IP, return host tap device name."

### `server.py` → all modules
`server.py` is intentionally the only file that imports from all modules. It is the integration surface. Every new tool must be explicitly registered here. This is a feature: it makes the MCP tool surface auditable in one place.

### `ToolResult[T]` boundary
The wrapping in `ToolResult` happens in `server.py` — individual module functions return their typed result OR raise a domain exception. `server.py` catches domain exceptions and converts them. This keeps modules clean and testable without MCP scaffolding.

---

## Concurrency Model

### Between Tool Calls: Sequential
MCP stdio transport delivers one tool call at a time per connection. No inter-tool concurrency. The server does not need a request queue, mutex, or session state.

**Implication:** No locking required for `Config`, topology caching, or artifact registry reads. The `net/fault_registry.py` in-memory dict is the single mutable shared object — it is safe without locks because only one tool call can mutate it at a time.

### Within a Tool Call: Parallelism Opt-In
Tools that fan out across nodes (`node_exec_all`, `artifact_sync`, `artifact_install`, `snapshot_create`, `dmesg_collect`) use internal parallelism:

- **Model:** `asyncio.gather` or `concurrent.futures.ThreadPoolExecutor` within `exec/parallel.py`
- **Granularity:** Per-node operations are the unit. Each node gets an independent SSH connection and result slot.
- **Result aggregation:** `dict[str, ExecResult]` — all nodes complete (or timeout) before the tool returns. No partial streaming.
- **Timeout handling:** Per-node `ssh_timeout_s` enforced independently. A hung node does not block healthy nodes.

### Libvirt Calls: Synchronous
All libvirt operations are synchronous (the `libvirt-python` API is blocking). Long-running operations (`cluster_start` with `wait_for_ssh`, `snapshot_revert`) block the tool call until completion. Timeout values are explicit in the tool signatures.

### Artifact Builder: Subprocess
`artifact/builder.py` shells out to `make` as a subprocess. This blocks the tool call. The agent should treat `artifact_build` as a potentially long-running synchronous call. No need for async here in v1.

### Design Invariant
No background threads or tasks persist between tool calls. The server has no event loop running independently of incoming MCP requests. This upholds the stateless-server constraint and ensures safe restart behavior.
