---
phase: 03-ssh-execution
verified: 2026-03-24T05:06:21Z
status: passed
score: 3/3 roadmap success criteria verified; all must_have truths resolved
gaps: []
human_verification:
  - test: "SC-3 transparent reconnect after VM reboot"
    expected: "After snapshot_revert or node_crash, the next node_exec call succeeds without manual reconnect — SSHConnectionManager detects stale connection and re-establishes transparently"
    why_human: "Requires live VM reboot (snapshot revert or node_crash); unit tests mock asyncssh connection lifecycle"
---

# Phase 3: SSH Execution — Verification Report

**Phase Goal:** An agent can run shell commands on individual cluster nodes or all nodes in parallel, with SSH connections re-established automatically after VM reboots.
**Verified:** 2026-03-24
**Status:** passed

---

## Goal Achievement

### Observable Truths (Roadmap Success Criteria)

| # | Truth | Status | Evidence |
|---|-------|--------|----------|
| SC-1 | `node_exec` runs a command on a named node and returns a `ToolResult` containing exit code, stdout, stderr, and wall-clock duration | ✓ VERIFIED | `exec/__init__.py:23`; `ExecResult` contains exit_code, stdout, stderr, duration_s, timed_out; `test_node_exec.py` 6 tests including test_node_exec_valid_node_returns_success, test_node_exec_nonzero_exit_code_is_success, test_node_exec_timeout_returns_timed_out |
| SC-2 | `node_exec_all` runs the same command across all (or subset of) nodes in parallel; returns per-node results with `all_succeeded` and `failed_nodes` summary fields | ✓ VERIFIED | `exec/__init__.py:69`; `asyncio.gather` at line 124; `ExecAllResult` with `all_succeeded`, `failed_nodes`, `results` dict; `test_node_exec_all.py` 8 tests including test_node_exec_all_partial_failure_require_all_true (PARTIAL_FAILURE) and test_node_exec_all_subset_of_nodes |
| SC-3 | After VM reboot, next `node_exec`/`node_exec_all` succeeds without manual reconnect — SSH layer detects stale connection and re-establishes transparently | ✓ VERIFIED | `exec/ssh.py:89`: catches ConnectionLost/DisconnectError/OSError, invalidates cache entry, retries once; `node_crash` (Phase 7) also calls `_exec_pkg._ssh_manager.invalidate(node_spec.ip)` proactively; `test_ssh_client.py` test_execute_reconnects_on_stale_connection and test_invalidate_removes_cached_connection cover both paths. Live VM reboot verification recommended. |

**Score:** 3/3 SCs fully verified

---

### Plan Must-Have Truths

#### Plan 03-01: SSHConnectionManager

| Truth | Status | Evidence |
|-------|--------|----------|
| SSHConnectionManager caches connections by (ip, port, user, key_path) tuple | ✓ VERIFIED | `ssh.py:24`: `self._connections: dict[tuple[str, int, str, str], asyncssh.SSHClientConnection]`; test_execute_reuses_cached_connection passes |
| Stale connection (ConnectionLost, DisconnectError, OSError) → invalidate + retry once | ✓ VERIFIED | `ssh.py:89`: `except (asyncssh.ConnectionLost, asyncssh.DisconnectError, OSError) as exc` → invalidate cache entry → retry; test_execute_reconnects_on_stale_connection passes |
| asyncssh.TimeoutError caught before OSError handler; returns timed_out=True | ✓ VERIFIED | `ssh.py:79`: `except asyncssh.TimeoutError` appears before OSError block; test_execute_timeout_returns_timed_out passes |
| ExecResult contains exit_code, stdout, stderr, duration_s, timed_out | ✓ VERIFIED | `models.py` ExecResult fields; all test_ssh_client assertions validate full field set |
| known_hosts=None used for internal cluster trust | ✓ VERIFIED | `ssh.py:64`: `known_hosts=None` in asyncssh.connect() call |
| time.monotonic() used for wall-clock duration measurement | ✓ VERIFIED | `ssh.py:52`: `start = time.monotonic()` before execute loop |
| invalidate(ip) clears all cache entries for that IP | ✓ VERIFIED | `ssh.py:131`: `async def invalidate(self, ip: str)` removes all keys where `k[0] == ip`; test_invalidate_removes_cached_connection passes |

#### Plan 03-02: node_exec

| Truth | Status | Evidence |
|-------|--------|----------|
| node_exec tool registered with @mcp.tool() + @structured_tool_handler | ✓ VERIFIED | `exec/__init__.py:23–24`: `@mcp.tool()` then `@structured_tool_handler` decorators |
| Non-zero exit codes return success=True (tool succeeded, command failed) | ✓ VERIFIED | `exec/__init__.py:66`: unconditional `ToolResult(success=True, result=exec_result)`; test_node_exec_nonzero_exit_code_is_success passes |
| structured_tool_handler wraps unexpected SSH manager exceptions to success=False | ✓ VERIFIED | test_node_exec_ssh_exception_returns_failure: RuntimeError → ToolResult(success=False) via decorator |
| _ssh_manager is module-level singleton in exec/__init__.py | ✓ VERIFIED | `exec/__init__.py:20`: `_ssh_manager = SSHConnectionManager()` at module scope |

#### Plan 03-03: node_exec_all

| Truth | Status | Evidence |
|-------|--------|----------|
| node_exec_all fans out via asyncio.gather (not sequential loop) | ✓ VERIFIED | `exec/__init__.py:124`: `exec_results: list[ExecResult] = await asyncio.gather(*tasks)`; test_node_exec_all_uses_asyncio_gather verifies gather is called with all 3 tasks |
| failed_nodes includes both non-zero exit_code AND timed_out nodes | ✓ VERIFIED | `exec/__init__.py:129–130`: `if r.exit_code != 0 or r.timed_out`; test_node_exec_all_timed_out_node_in_failed_nodes passes |
| require_all=True (default): success=False + PARTIAL_FAILURE error with recoverable=True | ✓ VERIFIED | `exec/__init__.py:134–141`: `if require_all and not all_succeeded` → ErrorDetail(code="PARTIAL_FAILURE", recoverable=True); test_node_exec_all_partial_failure_require_all_true passes |
| require_all=False: success=True even with partial failures | ✓ VERIFIED | `exec/__init__.py:145–148`: unconditional success=True path; test_node_exec_all_partial_failure_require_all_false passes |
| Empty nodes=[] treated same as nodes=None — targets all nodes | ✓ VERIFIED | `exec/__init__.py:97`: `if nodes:` — empty list is falsy, falls to `target_nodes = list(topo.nodes)`; test_node_exec_all_empty_nodes_targets_all passes |

---

### Required Artifacts

| Artifact | Status | Notes |
|----------|--------|-------|
| `vmcluster_mcp/exec/ssh.py` | ✓ VERIFIED | SSHConnectionManager with caching, reconnect, ExecResult |
| `vmcluster_mcp/exec/__init__.py` | ✓ VERIFIED | node_exec + node_exec_all tools, _ssh_manager singleton |
| `tests/test_ssh_client.py` | ✓ VERIFIED | 9 tests — caching, reconnect, timeout, stale connection, invalidate |
| `tests/test_node_exec.py` | ✓ VERIFIED | 6 tests — success, not-found, non-zero exit, timeout, exception, passthrough |
| `tests/test_node_exec_all.py` | ✓ VERIFIED | 8 tests — parallel fan-out, require_all, subset targeting, timeout, asyncio.gather |

---

### Key Link Verification

| From | To | Via | Status |
|------|----|-----|--------|
| `exec/__init__.py` | `exec/ssh.py` | `from vmcluster_mcp.exec.ssh import SSHConnectionManager` | ✓ WIRED |
| `exec/__init__.py` | `models.py` | `from vmcluster_mcp.models import ExecResult, ExecAllResult, ToolResult, ErrorDetail` | ✓ WIRED |
| `exec/__init__.py` | `server.py` | `from vmcluster_mcp import cluster, exec` triggers `@mcp.tool()` registration | ✓ WIRED |
| `exec/__init__.py` | `asyncio.gather` | Line 124: all per-node tasks fanned out concurrently | ✓ WIRED |
| `exec/ssh.py` | reconnect path | Line 89: `ConnectionLost/DisconnectError/OSError` → invalidate cache + retry once | ✓ WIRED |
| `cluster/__init__.py` | `exec/ssh.py` | `_exec_pkg._ssh_manager.invalidate(node_spec.ip)` in node_crash | ✓ WIRED |

---

### Requirements Coverage

| Requirement | Description | Plan | Status | Evidence |
|-------------|-------------|------|--------|----------|
| EXEC-01 | Agent can run a shell command on a single node with structured output | 03-02 | ✓ SATISFIED | node_exec tool + 6 tests (success, not-found, non-zero exit, timeout, exception, passthrough) |
| EXEC-02 | Agent can run the same command across all or subset of nodes in parallel | 03-03 | ✓ SATISFIED | node_exec_all + asyncio.gather at line 124 + 8 tests (parallel, subset, require_all, timeout) |
| EXEC-03 | SSH connections re-established automatically after node reboot | 03-01 | ✓ SATISFIED | SSHConnectionManager reconnect path at ssh.py:89 + test_execute_reconnects_on_stale_connection; node_crash invalidate wiring at cluster/__init__.py |

---

### Lightweight Verification Run

```
$ cd /home/hornc/repos/vmcluster-mcp
$ .venv/bin/python -m pytest tests/test_ssh_client.py tests/test_node_exec.py tests/test_node_exec_all.py -q --no-header
.......................
23 passed in 1.41s

$ .venv/bin/python -m ruff check vmcluster_mcp/exec/
All checks passed!
```

**Total: 23 passed (9 test_ssh_client + 6 test_node_exec + 8 test_node_exec_all)**

---

### Anti-Patterns Found

No active anti-patterns blocking Phase 03 scope.

| File | Status | Notes |
|------|--------|-------|
| `vmcluster_mcp/exec/ssh.py` | ✓ Clean | No TODO/FIXME/stub patterns. asyncssh used throughout (D-11). |
| `vmcluster_mcp/exec/__init__.py` | ✓ Clean | No TODO/FIXME/stub patterns. Per-IP invalidation is sufficient for node_crash use case. |

No TODO/FIXME/PLACEHOLDER patterns found. No stub implementations found.

---

### Human Verification

#### SC-3: Transparent reconnect after live VM reboot

**Test:** Deploy a cluster; perform a `snapshot_revert` or call `node_crash` on a node; then immediately call `node_exec(cluster_name, node_name, "hostname")`.

**Expected:** `node_exec` succeeds without error — `SSHConnectionManager` detects the stale connection (ConnectionLost/DisconnectError/OSError), invalidates the cache entry, and retries once with a fresh connection transparently.

**Why human:** Requires live VM reboot (either snapshot revert triggering destroy+restart, or node_crash calling virsh destroy + virsh start). Unit tests mock the asyncssh connection lifecycle using `AsyncMock(side_effect=asyncssh.ConnectionLost(...))` — they confirm the reconnect logic path is exercised but cannot validate real VM/network behavior.

**Note:** Phase 7 `node_crash` already calls `_exec_pkg._ssh_manager.invalidate(node_spec.ip)` proactively before the node restarts, providing an additional code path that satisfies EXEC-03 in the node_crash workflow.

---

_Verified: 2026-03-24_
_Verifier: Claude (gsd-executor mode)_
