---
phase: 02-core-vm-lifecycle
verified: 2026-03-24T05:06:50Z
status: passed
score: 5/5 roadmap success criteria verified; all must_have truths resolved
gaps: []
human_verification:
  - test: "SC-2 cluster_start SSH wait blocking behaviour"
    expected: "With wait_for_ssh=True, cluster_start blocks until all nodes SSH-reachable and returns time waited"
    why_human: "Requires live VMs with real SSH endpoints; unit tests mock asyncssh.connect"
  - test: "SC-3 cluster_status live SSH reachability"
    expected: "cluster_status returns ssh_reachable=True for running nodes and False for stopped nodes"
    why_human: "Requires live VMs; unit tests mock the SSH check"
---

# Phase 2: Core VM Lifecycle — Verification Report

**Phase Goal:** An agent can define, start, stop, destroy, inspect, and obtain handles for topology-declared multi-node VM clusters running on libvirt/QEMU — with idempotent defines and parallel SSH-reachability checks.
**Verified:** 2026-03-24
**Status:** passed

---

## Goal Achievement

### Observable Truths (Roadmap Success Criteria)

| # | Truth | Status | Evidence |
|---|-------|--------|----------|
| SC-1 | `cluster_define` creates libvirt domain definitions and cluster-namespaced overlay qcow2 disks; second call returns success without modifying existing domains | ✓ VERIFIED | `cluster/__init__.py:59` idempotency guard (`lookup_domain` before define); `cluster/__init__.py:92` namespaced overlay path `{overlay_dir}/{cluster_name}/`; `assert base_image != overlay_path` at line 114; `test_cluster_define.py`: test_cluster_define_creates_network_and_domains, test_cluster_define_idempotent, test_cluster_define_creates_overlay_dir, test_cluster_define_calls_qemu_img (4 tests) |
| SC-2 | `cluster_start` boots all nodes; with `wait_for_ssh=True`, blocks until SSH-reachable and returns time waited | ✓ VERIFIED | `cluster/__init__.py:234` (`cluster_start`); `_wait_for_ssh` at line 202 uses `asyncssh.connect`; `asyncio.gather` parallel SSH wait (line 275); `test_cluster_lifecycle.py`: test_cluster_start_ssh_success_sets_reachable, test_cluster_start_ssh_timeout_returns_failure |
| SC-3 | `cluster_status` returns domain state (running, shut off, etc.) and SSH reachability for each node | ✓ VERIFIED | `cluster/__init__.py:471`; SSH checked only for running nodes via `asyncio.gather` (line 505); returns `ToolResult[ClusterStatus]` with per-node `NodeStatus`; `test_cluster_status.py`: 7 cluster_status tests |
| SC-4 | `cluster_stop` and `cluster_destroy` cleanly remove cluster resources; `cluster_destroy` with `remove_overlays=True` deletes overlay disks | ✓ VERIFIED | `cluster/__init__.py:338` (`cluster_stop`, shutdown+destroy modes); `cluster/__init__.py:400` (`cluster_destroy`, always removes network, `shutil.rmtree` at line 451); `test_cluster_lifecycle.py`: 5 cluster_stop tests + 5 cluster_destroy tests (8 tests total) |
| SC-5 | `cluster_handle` returns `ClusterHandle` with node names, IPs, SSH credentials for downstream use | ✓ VERIFIED | `cluster/__init__.py:544`; returns `ToolResult[ClusterHandle]` with `NodeDescriptor` list + `kernel_version`; `test_cluster_status.py`: 6 cluster_handle tests |

**Score:** 5/5 SCs fully verified

---

### Plan Must-Have Truths

#### Plan 02-01: Thread-safe libvirt client abstraction layer

| Truth | Status | Evidence |
|-------|--------|----------|
| `LibvirtClient` dispatches all libvirt C calls via `ThreadPoolExecutor(max_workers=1)` | ✓ VERIFIED | `libvirt_client.py:19` — `_executor = ThreadPoolExecutor(max_workers=1)` at module level; `_run()` at line 76 uses `run_in_executor` |
| `domain_name()`, `generate_mac()`, `map_domain_state()` utility functions present | ✓ VERIFIED | `libvirt_client.py:35` (domain_name), line 40 (generate_mac), line 52 (map_domain_state) |
| `generate_mac()` returns `52:54:00:xx:xx:xx` format deterministically | ✓ VERIFIED | `libvirt_client.py:40`; deterministic hash from cluster+node name; test_libvirt_client.py confirms format |
| `LibvirtClient.__init__` does NOT call `libvirt.open` (lazy connection) | ✓ VERIFIED | tests/test_libvirt_client.py — no libvirt.open in __init__; connection deferred to first use |
| All async methods dispatch through `_run` (not raw executor calls) | ✓ VERIFIED | `libvirt_client.py:76` — all public async methods call `await self._run(...)` |
| `lookup_domain` and `lookup_network` return `None` on `libvirt.libvirtError` | ✓ VERIFIED | `libvirt_client.py:88` and line 129 — `except libvirt.libvirtError: return None` |
| `pytest tests/test_libvirt_client.py` → 31 tests passing | ✓ VERIFIED | Confirmed: 31 passed in 1.19s |

#### Plan 02-02: XML builders, cloud-init ISO, cluster_define tool

| Truth | Status | Evidence |
|-------|--------|----------|
| `domain_builder.py` uses `xml.etree.ElementTree` (no Jinja2/templating) | ✓ VERIFIED | `cluster/domain_builder.py` — ElementTree XML construction; D-06 applied |
| `domain_builder.py` includes overlay disk (vda/virtio), cidata cdrom (sda/sata/readonly), bridge interface, serial console | ✓ VERIFIED | tests/test_builders.py: 17 tests verify all XML elements; ruff clean |
| `network_builder.py` forward mode='nat', no DHCP block | ✓ VERIFIED | `cluster/network_builder.py` — NAT mode; D-03 applied (no DHCP) |
| `cloud_init.py` calls `subprocess.run` with genisoimage / mkisofs fallback | ✓ VERIFIED | `cluster/cloud_init.py` — genisoimage first, mkisofs fallback; tests/test_cloud_init.py: 5 tests |
| `cluster_define` is idempotent — `lookup_domain` guard before define | ✓ VERIFIED | `cluster/__init__.py:59` — `existing_dom = await _client.lookup_domain(first_dom_name)`; CLUS-07 satisfied |
| `cluster_define` asserts `base_image != overlay_path` | ✓ VERIFIED | `cluster/__init__.py:114` — `assert os.path.abspath(topo.base_image) != os.path.abspath(overlay_path)` |
| `cluster_define` calls `qemu-img create` for overlay disks | ✓ VERIFIED | `cluster/__init__.py:120` — subprocess call to `qemu-img create -b base -F qcow2 -f qcow2 overlay` |
| `pytest tests/test_builders.py tests/test_cloud_init.py tests/test_cluster_define.py` → 26 tests passing | ✓ VERIFIED | Confirmed: 17 + 5 + 4 = 26 passed |

#### Plan 02-03: cluster_start / cluster_stop / cluster_destroy

| Truth | Status | Evidence |
|-------|--------|----------|
| `_wait_for_ssh` helper uses `asyncssh.connect` and never raises | ✓ VERIFIED | `cluster/__init__.py:202` — polls asyncssh.connect; returns `(False, elapsed)` on timeout; D-11 applied |
| `cluster_start` boots all stopped domains; skips already-active | ✓ VERIFIED | `cluster/__init__.py:234`; test_cluster_start_boots_all_domains, test_cluster_start_skips_already_active_domain |
| `cluster_start` with `wait_for_ssh=True` fires parallel `asyncio.gather` SSH checks | ✓ VERIFIED | `cluster/__init__.py:275` — `asyncio.gather(*[_wait_for_ssh(...) for node in ...])` |
| D-04 strict SSH: any SSH timeout → `success=False` with `SSH_TIMEOUT` error | ✓ VERIFIED | `cluster/__init__.py` — any failure flips success=False; test_cluster_start_ssh_timeout_returns_failure passes |
| `cluster_stop` shutdown mode calls `shutdown_domain`; destroy mode calls `destroy_domain` | ✓ VERIFIED | `cluster/__init__.py:338`; test_cluster_stop_shutdown_mode, test_cluster_stop_destroy_mode |
| `cluster_stop` skips inactive domains | ✓ VERIFIED | `cluster/__init__.py` — checks domain state; test_cluster_stop_skips_inactive_domains passes |
| `cluster_destroy` always destroys + undefines the network (D-01) | ✓ VERIFIED | `cluster/__init__.py:400` — network removal unconditional; test_cluster_destroy_always_removes_network |
| `cluster_destroy` with `remove_overlays=True` calls `shutil.rmtree` on overlay directory | ✓ VERIFIED | `cluster/__init__.py:451` — `shutil.rmtree(overlay_dir, ignore_errors=True)`; test_cluster_destroy_remove_overlays_true |
| `pytest tests/test_cluster_lifecycle.py` → 13 tests passing | ✓ VERIFIED | Confirmed: 13 passed in 1.09s |

#### Plan 02-04: cluster_status / cluster_handle

| Truth | Status | Evidence |
|-------|--------|----------|
| `cluster_status` checks network via `lookup_network` + `is_network_active` | ✓ VERIFIED | `cluster/__init__.py:471`; test_cluster_status_network_active_true passes |
| `cluster_status` queries domain state for each node | ✓ VERIFIED | `cluster/__init__.py:492` — `await _client.lookup_domain(dname)` per node; test_cluster_status_per_node_domain_state |
| SSH checked only for running nodes in `cluster_status` | ✓ VERIFIED | `cluster/__init__.py:505` — conditional asyncio.gather only for running nodes; test_cluster_status_ssh_checked_only_for_running_nodes |
| `cluster_status` returns `ToolResult[ClusterStatus]` with `network_active` + per-node `NodeStatus` | ✓ VERIFIED | `cluster/__init__.py:471`; all 7 cluster_status tests confirm shape |
| `cluster_handle` validates cluster is running before fetching kernel version | ✓ VERIFIED | `cluster/__init__.py:560` — lookup_domain on first node → CLUSTER_NOT_RUNNING if absent; test_cluster_handle_not_running_returns_error |
| `cluster_handle` fetches `kernel_version` via SSH `uname -r` on first node | ✓ VERIFIED | `cluster/__init__.py:544`; test_cluster_handle_kernel_version_from_ssh passes |
| `cluster_handle` returns `ToolResult[ClusterHandle]` with `NodeDescriptor` list | ✓ VERIFIED | `cluster/__init__.py:594` — `NodeDescriptor(...)` per node; test_cluster_handle_node_descriptors |
| `pytest tests/test_cluster_status.py` → 13 tests passing | ✓ VERIFIED | Confirmed: 13 passed in 1.04s (7 cluster_status + 6 cluster_handle) |

---

### Required Artifacts

| Artifact | Status | Notes |
|----------|--------|-------|
| `vmcluster_mcp/cluster/libvirt_client.py` | ✓ VERIFIED | LibvirtClient, ThreadPoolExecutor(max_workers=1), 31 tests |
| `vmcluster_mcp/cluster/domain_builder.py` | ✓ VERIFIED | ElementTree XML builder for KVM domains |
| `vmcluster_mcp/cluster/network_builder.py` | ✓ VERIFIED | NAT network XML, no DHCP block |
| `vmcluster_mcp/cluster/cloud_init.py` | ✓ VERIFIED | NoCloud ISO via genisoimage/mkisofs |
| `vmcluster_mcp/cluster/__init__.py` | ✓ VERIFIED | 6 tools: cluster_define/start/stop/destroy/status/handle |
| `tests/test_libvirt_client.py` | ✓ VERIFIED | 31 tests |
| `tests/test_builders.py` | ✓ VERIFIED | 17 tests |
| `tests/test_cloud_init.py` | ✓ VERIFIED | 5 tests |
| `tests/test_cluster_define.py` | ✓ VERIFIED | 4 tests |
| `tests/test_cluster_lifecycle.py` | ✓ VERIFIED | 13 tests |
| `tests/test_cluster_status.py` | ✓ VERIFIED | 13 tests (7 cluster_status + 6 cluster_handle) |

---

### Key Link Verification

| From | To | Via | Status |
|------|----|-----|--------|
| `cluster/__init__.py` | `libvirt_client.py` | `_client = LibvirtClient()` + all async calls via `await _client.*` | ✓ WIRED |
| `cluster/__init__.py` | `domain_builder.py` | `from vmcluster_mcp.cluster.domain_builder import build_domain_xml` | ✓ WIRED |
| `cluster/__init__.py` | `network_builder.py` | `from vmcluster_mcp.cluster.network_builder import build_network_xml` | ✓ WIRED |
| `cluster/__init__.py` | `cloud_init.py` | `from vmcluster_mcp.cluster.cloud_init import generate_cloud_init_iso` | ✓ WIRED |
| `cluster/__init__.py` | `server.py` | `from vmcluster_mcp import cluster` triggers `@mcp.tool()` registration | ✓ WIRED |
| `libvirt_client.py` | `ThreadPoolExecutor` | `_executor = ThreadPoolExecutor(max_workers=1)` at module level | ✓ WIRED |

---

### Requirements Coverage

| Requirement | Description | Plan | Status | Evidence |
|-------------|-------------|------|--------|----------|
| CLUS-01 | Agent can define a cluster from a topology file | 02-02 | ✓ SATISFIED | cluster_define + 4 tests |
| CLUS-02 | Agent can start all nodes and optionally wait for SSH | 02-03 | ✓ SATISFIED | cluster_start + 5 tests |
| CLUS-03 | Agent can stop all nodes (graceful or forcibly) | 02-03 | ✓ SATISFIED | cluster_stop + 3 tests |
| CLUS-04 | Agent can destroy cluster and optionally remove overlays | 02-03 | ✓ SATISFIED | cluster_destroy + 5 tests |
| CLUS-05 | Agent can query current status of all nodes | 02-04 | ✓ SATISFIED | cluster_status + 7 tests |
| CLUS-06 | Agent can obtain a ClusterHandle for downstream SSH consumers | 02-04 | ✓ SATISFIED | cluster_handle + 6 tests |
| CLUS-07 | cluster_define is idempotent | 02-02 | ✓ SATISFIED | lookup_domain guard at line 59; test_cluster_define_idempotent |

---

### Lightweight Verification Run

```
$ .venv/bin/python -m pytest tests/test_libvirt_client.py tests/test_builders.py tests/test_cloud_init.py tests/test_cluster_define.py tests/test_cluster_lifecycle.py tests/test_cluster_status.py -q --no-header

...............................................................................
...........
83 passed in 2.19s

$ .venv/bin/python -m ruff check vmcluster_mcp/cluster/
All checks passed!
```

Per-file breakdown:
- `tests/test_libvirt_client.py`: 31 passed
- `tests/test_builders.py`: 17 passed
- `tests/test_cloud_init.py`: 5 passed
- `tests/test_cluster_define.py`: 4 passed
- `tests/test_cluster_lifecycle.py`: 13 passed
- `tests/test_cluster_status.py`: 13 passed

---

### Anti-Patterns Found

No TODOs, FIXMEs, PLACEHOLDER, or stub implementations found in `vmcluster_mcp/cluster/`.

Design decision D-08 (single-worker executor) is correctly applied: `ThreadPoolExecutor(max_workers=1)` at module level in `libvirt_client.py:19`, preventing non-deterministic libvirt segfaults from concurrent C library use.

---

### Human Verification (Non-Blocking)

#### 1. cluster_start SSH wait blocking behaviour

**Test:** Deploy a live cluster; call `cluster_start(cluster_name, wait_for_ssh=True)`.
**Expected:** `cluster_start` blocks until all nodes SSH-reachable and returns time waited per node in result.
**Why human:** Requires live VMs with real SSH endpoints; unit tests mock `asyncssh.connect` to return immediately.

#### 2. cluster_status live SSH reachability

**Test:** Deploy a live cluster with some nodes running and some stopped; call `cluster_status(cluster_name)`.
**Expected:** Running nodes show `ssh_reachable=True`; stopped/shut-off nodes show `ssh_reachable=False`.
**Why human:** Requires live VMs; unit tests mock the SSH check function.

---

_Verified: 2026-03-24_
_Verifier: Claude Sonnet 4.6 (gsd-executor mode)_
