---
phase: 02-core-vm-lifecycle
plan: 03
type: execute
wave: 3
depends_on: ["02-02"]
files_modified:
  - vmcluster_mcp/cluster/__init__.py
  - tests/test_cluster_lifecycle.py
autonomous: true
requirements:
  - CLUS-02
  - CLUS-03
  - CLUS-04

must_haves:
  truths:
    - "cluster_start boots all nodes and optionally blocks until every node is SSH-reachable"
    - "cluster_start with wait_for_ssh=True returns success=False if any node fails SSH within timeout"
    - "cluster_start returns per-node status including time_waited"
    - "cluster_stop gracefully shuts down all nodes (shutdown mode) or force-kills (destroy mode)"
    - "cluster_destroy undefines all domains and removes the libvirt network"
    - "cluster_destroy with remove_overlays=True deletes overlay disk files and cloud-init ISOs"
    - "cluster_destroy with remove_overlays=False leaves overlay files intact"
  artifacts:
    - path: "vmcluster_mcp/cluster/__init__.py"
      provides: "cluster_start, cluster_stop, cluster_destroy tool registrations"
      contains: "cluster_start"
    - path: "tests/test_cluster_lifecycle.py"
      provides: "Lifecycle tool unit tests"
      min_lines: 80
  key_links:
    - from: "vmcluster_mcp/cluster/__init__.py"
      to: "vmcluster_mcp/cluster/libvirt_client.py"
      via: "LibvirtClient for domain create/shutdown/destroy/undefine"
      pattern: "_client\\."
    - from: "vmcluster_mcp/cluster/__init__.py"
      to: "asyncssh"
      via: "SSH wait loop in cluster_start"
      pattern: "asyncssh\\.connect"
    - from: "vmcluster_mcp/cluster/__init__.py"
      to: "vmcluster_mcp/topology/parser.py"
      via: "load_topology to discover cluster nodes"
      pattern: "load_topology"
---

<objective>
Add cluster_start (with SSH wait), cluster_stop, and cluster_destroy tools to complete the lifecycle operations.

Purpose: After cluster_define creates the resources, the agent needs to boot nodes, shut them down, and fully clean up. cluster_start's SSH wait loop is critical — it provides the guarantee that nodes are usable after the call returns.

Output: Three new tools in cluster/__init__.py plus comprehensive tests.
</objective>

<execution_context>
@~/.copilot/get-shit-done/workflows/execute-plan.md
@~/.copilot/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/02-core-vm-lifecycle/02-CONTEXT.md
@.planning/phases/02-core-vm-lifecycle/02-RESEARCH.md
@.planning/phases/02-core-vm-lifecycle/02-01-SUMMARY.md
@.planning/phases/02-core-vm-lifecycle/02-02-SUMMARY.md

<interfaces>
<!-- From Plans 01 and 02 -->

From vmcluster_mcp/cluster/libvirt_client.py:
```python
def domain_name(cluster_name: str, node_name: str) -> str: ...
def map_domain_state(state_int: int) -> str: ...

class LibvirtClient:
    async def lookup_domain(self, name: str) -> virDomain | None: ...
    async def start_domain(self, dom) -> None: ...
    async def shutdown_domain(self, dom) -> None: ...
    async def destroy_domain(self, dom) -> None: ...
    async def undefine_domain(self, dom) -> None: ...
    async def get_domain_state(self, dom) -> str: ...
    async def is_domain_active(self, dom) -> bool: ...
    async def lookup_network(self, name: str) -> virNetwork | None: ...
    async def destroy_network(self, net) -> None: ...
    async def undefine_network(self, net) -> None: ...
    async def is_network_active(self, net) -> bool: ...
```

From vmcluster_mcp/cluster/__init__.py (Plan 02 output):
```python
_client = LibvirtClient(config.libvirt_uri)
# cluster_define already registered
```

From vmcluster_mcp/topology/schema.py:
```python
class TopologySpec(BaseModel):
    cluster_name: str; base_image: str; overlay_dir: str
    network: NetworkSpec; nodes: list[NodeSpec]; ssh: SshSpec

class SshSpec(BaseModel):
    key_path: str; user: str = "root"; connect_timeout_s: int = 30
```

From vmcluster_mcp/models.py:
```python
class ClusterStatus(BaseModel):
    cluster_name: str; network_active: bool; nodes: list[NodeStatus]
class NodeStatus(BaseModel):
    name: str; role: str; ip: str
    domain_state: Literal["running", "shut off", "paused", "crashed", "unknown"]
    ssh_reachable: bool; artifact_id_installed: str | None = None
class ToolResult(BaseModel, Generic[T]):
    success: bool; result: T | None = None; error: ErrorDetail | None = None
```

Design doc cluster_start signature:
```
cluster_start(cluster_name: str, wait_for_ssh: bool = True, ssh_timeout_s: int = 120) -> ClusterStatus
cluster_stop(cluster_name: str, mode: Literal["shutdown", "destroy"] = "shutdown") -> ClusterStatus
cluster_destroy(cluster_name: str, remove_overlays: bool = True) -> bool
```
</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Add cluster_start tool with SSH wait loop</name>
  <files>vmcluster_mcp/cluster/__init__.py, tests/test_cluster_lifecycle.py</files>
  <read_first>
    - vmcluster_mcp/cluster/__init__.py (current state from Plan 02 — cluster_define, _client, imports)
    - vmcluster_mcp/cluster/libvirt_client.py (start_domain, lookup_domain, get_domain_state, is_domain_active)
    - vmcluster_mcp/topology/schema.py (SshSpec.connect_timeout_s, SshSpec.key_path, SshSpec.user)
    - vmcluster_mcp/config.py (config.ssh_key_path, config.ssh_user)
    - .planning/phases/02-core-vm-lifecycle/02-CONTEXT.md (D-04: strict all-or-nothing SSH wait, D-11: asyncssh)
    - .planning/phases/02-core-vm-lifecycle/02-RESEARCH.md (§7: SSH wait loop pattern)
  </read_first>
  <behavior>
    - Test: cluster_start boots all domains via start_domain
    - Test: cluster_start with wait_for_ssh=False returns immediately after booting
    - Test: cluster_start with wait_for_ssh=True polls SSH on each node
    - Test: cluster_start returns ClusterStatus with domain_state="running" for all nodes
    - Test: cluster_start with SSH success sets ssh_reachable=True for all nodes
    - Test: cluster_start with any SSH timeout returns success=False (per D-04)
    - Test: cluster_start returns time_waited or per-node timing
    - Test: cluster_start on unknown cluster returns error
  </behavior>
  <action>
    Add to `vmcluster_mcp/cluster/__init__.py` (append after cluster_define):

    Add `import asyncssh` and `import time` to imports.

    **Helper function (module-level, not a tool):**
    ```python
    async def _wait_for_ssh(ip: str, port: int, user: str, key_path: str, timeout_s: int) -> tuple[bool, float]:
        """Poll SSH until reachable or timeout. Returns (reachable, time_waited)."""
        start = time.monotonic()
        deadline = start + timeout_s
        while time.monotonic() < deadline:
            try:
                async with asyncssh.connect(
                    ip, port=port, username=user,
                    client_keys=[key_path],
                    known_hosts=None,
                    login_timeout=5,
                ) as conn:
                    await conn.run("echo ok", check=True)
                    return True, time.monotonic() - start
            except (OSError, asyncssh.Error):
                await asyncio.sleep(2)
        return False, time.monotonic() - start
    ```

    **cluster_start tool:**
    `@mcp.tool()` + `@structured_tool_handler`
    `async def cluster_start(cluster_name: str, wait_for_ssh: bool = True, ssh_timeout_s: int = 120) -> ToolResult[ClusterStatus]:`

    1. `topo = load_topology(cluster_name, config.topology_dir)`
    2. For each node: lookup domain via `_client.lookup_domain(domain_name(topo.cluster_name, node.name))`. If not found, return error `CLUSTER_NOT_DEFINED`.
    3. For each domain: `await _client.start_domain(dom)` (skip if already active via `is_domain_active`)
    4. If `wait_for_ssh`:
       - Use `asyncio.gather` to run `_wait_for_ssh` for all nodes in parallel
       - Each call uses: `node.ip`, port=22, `topo.ssh.user`, `topo.ssh.key_path`, timeout=`ssh_timeout_s`
       - If ANY node returns `reachable=False` → return `ToolResult(success=False, error=ErrorDetail(code="SSH_TIMEOUT", message="Nodes failed SSH: {failed_names}", recoverable=True))` with per-node status
    5. Build ClusterStatus with domain_state, ssh_reachable per node
    6. Return `ToolResult(success=True, result=status)`
  </action>
  <verify>
    <automated>cd /home/hornc/repos/vmcluster-mcp && python -m pytest tests/test_cluster_lifecycle.py -x -v -k "start" 2>&1 | tail -30</automated>
  </verify>
  <acceptance_criteria>
    - cluster/__init__.py contains `async def cluster_start(`
    - cluster/__init__.py contains `@mcp.tool()` decorator on cluster_start
    - cluster/__init__.py contains `async def _wait_for_ssh(`
    - cluster/__init__.py imports `asyncssh`
    - cluster/__init__.py contains `asyncio.gather` for parallel SSH wait
    - SSH wait uses `known_hosts=None` (no host key verification for test VMs)
    - D-04 enforced: if any node fails SSH, ToolResult.success=False
    - Tests pass: `pytest tests/test_cluster_lifecycle.py -k start` exits 0
  </acceptance_criteria>
  <done>cluster_start tool boots domains and optionally waits for SSH with strict all-or-nothing semantics, tests pass</done>
</task>

<task type="auto" tdd="true">
  <name>Task 2: Add cluster_stop and cluster_destroy tools</name>
  <files>vmcluster_mcp/cluster/__init__.py, tests/test_cluster_lifecycle.py</files>
  <read_first>
    - vmcluster_mcp/cluster/__init__.py (current state — cluster_define + cluster_start)
    - vmcluster_mcp/cluster/libvirt_client.py (shutdown_domain, destroy_domain, undefine_domain, destroy_network, undefine_network)
    - vmcluster_mcp/config.py (config.overlay_dir)
    - .planning/phases/02-core-vm-lifecycle/02-CONTEXT.md (D-01: network always torn down, D-02: remove_overlays only controls disk deletion)
    - docs/vmcluster-mcp-design.md §2.4 (cluster_stop and cluster_destroy signatures)
  </read_first>
  <behavior>
    - Test: cluster_stop mode="shutdown" calls shutdown_domain on all nodes
    - Test: cluster_stop mode="destroy" calls destroy_domain on all nodes
    - Test: cluster_stop returns ClusterStatus with domain_state="shut off"
    - Test: cluster_stop on unknown cluster returns error
    - Test: cluster_destroy undefines all domains
    - Test: cluster_destroy always destroys and undefines the network (per D-01)
    - Test: cluster_destroy with remove_overlays=True deletes overlay dir
    - Test: cluster_destroy with remove_overlays=False leaves overlay files
    - Test: cluster_destroy on unknown cluster returns error
  </behavior>
  <action>
    Append to `vmcluster_mcp/cluster/__init__.py`:

    Add `import shutil` to imports.

    **cluster_stop:**
    `@mcp.tool()` + `@structured_tool_handler`
    `async def cluster_stop(cluster_name: str, mode: str = "shutdown") -> ToolResult[ClusterStatus]:`

    1. `topo = load_topology(cluster_name, config.topology_dir)`
    2. For each node:
       - `dom = await _client.lookup_domain(domain_name(topo.cluster_name, node.name))`
       - If not found, skip (already cleaned up)
       - If active: call `_client.shutdown_domain(dom)` if mode=="shutdown", or `_client.destroy_domain(dom)` if mode=="destroy"
    3. Build ClusterStatus with updated domain states
    4. Return `ToolResult(success=True, result=status)`

    **cluster_destroy:**
    `@mcp.tool()` + `@structured_tool_handler`
    `async def cluster_destroy(cluster_name: str, remove_overlays: bool = True) -> ToolResult[bool]:`

    1. `topo = load_topology(cluster_name, config.topology_dir)`
    2. For each node:
       - `dom = await _client.lookup_domain(domain_name(topo.cluster_name, node.name))`
       - If found and active: `await _client.destroy_domain(dom)` (force stop first)
       - If found: `await _client.undefine_domain(dom)`
    3. Network cleanup (per D-01 — always torn down):
       - `net = await _client.lookup_network(topo.network.name)`
       - If found and active: `await _client.destroy_network(net)`
       - If found: `await _client.undefine_network(net)`
    4. If `remove_overlays` (per D-02 — only controls disk files):
       - `overlay_dir = os.path.join(config.overlay_dir, topo.cluster_name)`
       - `shutil.rmtree(overlay_dir, ignore_errors=True)` — removes overlays + cloud-init ISOs
    5. Return `ToolResult(success=True, result=True)`

    **Add lifecycle tests to tests/test_cluster_lifecycle.py** covering stop/destroy behavior with mocked libvirt client.
  </action>
  <verify>
    <automated>cd /home/hornc/repos/vmcluster-mcp && python -m pytest tests/test_cluster_lifecycle.py -x -v 2>&1 | tail -30</automated>
  </verify>
  <acceptance_criteria>
    - cluster/__init__.py contains `async def cluster_stop(` with `@mcp.tool()`
    - cluster/__init__.py contains `async def cluster_destroy(` with `@mcp.tool()`
    - cluster_stop supports mode="shutdown" (calls shutdown_domain) and mode="destroy" (calls destroy_domain)
    - cluster_destroy always tears down network regardless of remove_overlays (per D-01)
    - cluster_destroy with remove_overlays=True calls shutil.rmtree on overlay dir
    - cluster_destroy with remove_overlays=False does NOT delete overlay dir
    - cluster_destroy force-stops active domains before undefining
    - All tests pass: `pytest tests/test_cluster_lifecycle.py` exits 0
    - ruff check passes
  </acceptance_criteria>
  <done>cluster_stop and cluster_destroy tools registered, network always torn down on destroy, overlays optionally removed, all tests pass</done>
</task>

</tasks>

<verification>
```bash
# All lifecycle tests pass
python -m pytest tests/test_cluster_lifecycle.py -x -v

# All Phase 2 tests pass so far
python -m pytest tests/ -x -v

# Ruff clean
ruff check vmcluster_mcp/cluster/__init__.py

# Tools registered
python -c "
from vmcluster_mcp.cluster import cluster_start, cluster_stop, cluster_destroy
print('All lifecycle tools registered')
"
```
</verification>

<success_criteria>
- cluster_start boots all domains and optionally waits for SSH reachability with asyncio.gather parallel polling
- cluster_start with wait_for_ssh=True is strict all-or-nothing (D-04)
- cluster_stop supports both graceful shutdown and force destroy modes
- cluster_destroy undefines domains + network, optionally removes overlays
- Network is always torn down on destroy regardless of remove_overlays (D-01, D-02)
- All tests pass, ruff clean
</success_criteria>

<output>
After completion, create `.planning/phases/02-core-vm-lifecycle/02-03-SUMMARY.md`
</output>
