---
phase: 08-retroactive-phase-verification
plan: 02
type: execute
wave: 1
depends_on: []
files_modified:
  - .planning/phases/03-ssh-execution/03-VERIFICATION.md
autonomous: true
requirements:
  - EXEC-01
  - EXEC-02
  - EXEC-03

must_haves:
  truths:
    - "03-VERIFICATION.md exists in .planning/phases/03-ssh-execution/"
    - "YAML frontmatter includes phase, verified, status, score, and gaps fields"
    - "All 3 roadmap success criteria for Phase 3 have a row with status VERIFIED or human_needed"
    - "Each SC row cites an implementation file (with line reference), a test file, and a test name"
    - "Requirements coverage table maps EXEC-01 through EXEC-03 to plans and test counts"
    - "Per-plan must-have truth tables are present (one section per plan 03-01 through 03-03)"
  artifacts:
    - path: ".planning/phases/03-ssh-execution/03-VERIFICATION.md"
      provides: "Phase 3 verification report with 3-source evidence"
      contains: "EXEC-01"
  key_links:
    - from: "03-VERIFICATION.md SC table"
      to: "vmcluster_mcp/exec/ssh.py"
      via: "file:line evidence citations"
      pattern: "exec/ssh"
    - from: "03-VERIFICATION.md requirements table"
      to: "EXEC-01..03"
      via: "requirements coverage section"
      pattern: "EXEC-0[1-3]"
---

<objective>
Audit the Phase 03 (SSH Execution) codebase against the roadmap success criteria and write the missing 03-VERIFICATION.md report. The report provides the 3-source evidence required to mark EXEC-01 through EXEC-03 as satisfied at milestone level.

Purpose: Close the audit gap that leaves EXEC-01..03 orphaned in the v1.0 milestone audit. Phase 3 was fully implemented and tested but no verification report was written.
Output: `.planning/phases/03-ssh-execution/03-VERIFICATION.md`
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md

@.planning/phases/03-ssh-execution/03-01-SUMMARY.md
@.planning/phases/03-ssh-execution/03-02-SUMMARY.md
@.planning/phases/03-ssh-execution/03-03-SUMMARY.md

<!-- Canonical VERIFICATION.md format reference -->
@.planning/phases/07-network-node-fault-injection/07-VERIFICATION.md

<interfaces>
<!-- Key source locations confirmed in codebase. Use these as evidence citations. -->

vmcluster_mcp/exec/ssh.py:
  - Line 15: `class SSHConnectionManager`
  - Line 37: docstring notes stale connection retry: ConnectionLost, DisconnectError, OSError → invalidate + retry once
  - Line 52: `start = time.monotonic()` — wall-clock duration measurement
  - Line 64: `known_hosts=None` — internal cluster trust
  - Line 79: `except asyncssh.TimeoutError:` — catches timeout before OSError
  - Line 89: `except (asyncssh.ConnectionLost, asyncssh.DisconnectError, OSError) as exc:` — reconnect logic
  - Line 131: `async def invalidate(self, ip: str) -> None` — cache invalidation

vmcluster_mcp/exec/__init__.py:
  - Line 20: `_ssh_manager = SSHConnectionManager()` — module-level singleton
  - Line 23–30: `@mcp.tool()` / `@structured_tool_handler` / `async def node_exec`
  - Line 69–77: `@mcp.tool()` / `@structured_tool_handler` / `async def node_exec_all`
  - Line 112: comment `# Fan out via asyncio.gather for concurrent execution`
  - Line 124: `exec_results: list[ExecResult] = await asyncio.gather(*tasks)`
  - Line 129: `failed_nodes = [...]` — non-zero exit or timed_out
  - Line 132: `all_succeeded = len(failed_nodes) == 0`
  - Line 134: `if require_all and not all_succeeded:` — PARTIAL_FAILURE gate
  - Line 139: `code="PARTIAL_FAILURE"`

Test files and counts (all currently passing):
  - tests/test_ssh_client.py — 9 tests
  - tests/test_node_exec.py — 6 tests
  - tests/test_node_exec_all.py — 8 tests

Key test names from 03-03-SUMMARY.md:
  - test_node_exec_all: require_all semantics, subset targeting, timeout handling, asyncio.gather verification
  - test_node_exec: success, not-found, non-zero exit, timeout, exception, timeout passthrough
  - test_ssh_client: connection caching, reconnect on stale connection, TimeoutError handling

Note on EXEC-03 (transparent reconnect): SSHConnectionManager.run() at line 89 invalidates the stale cache
entry and retries the connection once on ConnectionLost/DisconnectError/OSError. node_crash in Phase 7 also
calls `_exec_pkg._ssh_manager.invalidate(node_spec.ip)` to proactively clear the cache. Both paths satisfy
EXEC-03. The unit test `test_ssh_client.py` covers the stale-connection retry path. Live verification
requires an actual VM reboot.
</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Audit Phase 03 codebase and write 03-VERIFICATION.md</name>
  <files>.planning/phases/03-ssh-execution/03-VERIFICATION.md</files>
  <read_first>
    - .planning/phases/07-network-node-fault-injection/07-VERIFICATION.md (canonical format — replicate YAML frontmatter schema, table structure, section headings exactly)
    - .planning/phases/03-ssh-execution/03-01-SUMMARY.md
    - .planning/phases/03-ssh-execution/03-02-SUMMARY.md
    - .planning/phases/03-ssh-execution/03-03-SUMMARY.md
    - vmcluster_mcp/exec/ssh.py (confirm line numbers for reconnect logic)
    - vmcluster_mcp/exec/__init__.py (confirm asyncio.gather and PARTIAL_FAILURE locations)
    - tests/test_ssh_client.py (confirm reconnect test names)
    - tests/test_node_exec.py (confirm node_exec test names)
    - tests/test_node_exec_all.py (confirm node_exec_all test names)
  </read_first>
  <action>
Run these verification commands first and record the outputs:

```
cd /home/hornc/repos/vmcluster-mcp
.venv/bin/python -m pytest tests/test_ssh_client.py tests/test_node_exec.py tests/test_node_exec_all.py -q --no-header
.venv/bin/python -m ruff check vmcluster_mcp/exec/
```

Then write `.planning/phases/03-ssh-execution/03-VERIFICATION.md` using the exact schema from 07-VERIFICATION.md:

**YAML frontmatter** (required fields, same keys as 07):
```yaml
---
phase: 03-ssh-execution
verified: 2026-03-23T00:00:00Z
status: passed
score: 3/3 roadmap success criteria verified; all must_have truths resolved
gaps: []
human_verification:
  - test: "SC-3 transparent reconnect after VM reboot"
    expected: "After snapshot_revert or node_crash, the next node_exec call succeeds without manual reconnect — SSHConnectionManager detects stale connection and re-establishes transparently"
    why_human: "Requires live VM reboot (snapshot revert or node_crash); unit tests mock asyncssh connection lifecycle"
---
```

**Phase Goal** header line and Verified/Status metadata identical to 07 format.

**Observable Truths table** — map each roadmap SC to code evidence + test evidence:

| # | Truth | Status | Evidence |
|---|-------|--------|----------|
| SC-1 | `node_exec` runs a command on a named node and returns a `ToolResult` containing exit code, stdout, stderr, and wall-clock duration | ✓ VERIFIED | `exec/__init__.py:23`; `ExecResult` contains exit_code, stdout, stderr, duration_s, timed_out; `test_node_exec.py` 6 tests including test_node_exec_success, test_node_exec_nonzero_exit, test_node_exec_timeout |
| SC-2 | `node_exec_all` runs the same command across all (or subset of) nodes in parallel; returns per-node results with `all_succeeded` and `failed_nodes` summary fields | ✓ VERIFIED | `exec/__init__.py:69`; `asyncio.gather` at line 124; `ExecAllResult` with `all_succeeded`, `failed_nodes`, `results` dict; `test_node_exec_all.py` 8 tests including require_all semantics and subset targeting |
| SC-3 | After VM reboot, next `node_exec`/`node_exec_all` succeeds without manual reconnect — SSH layer detects stale connection and re-establishes transparently | ✓ VERIFIED | `exec/ssh.py:89`: catches ConnectionLost/DisconnectError/OSError, invalidates cache, retries once; `node_crash` (Phase 7) also calls `_ssh_manager.invalidate()`; `test_ssh_client.py` covers stale-connection retry path. Live VM reboot verification recommended. |

**Required Artifacts table**:

| Artifact | Status | Notes |
|----------|--------|-------|
| `vmcluster_mcp/exec/ssh.py` | ✓ VERIFIED | SSHConnectionManager with caching, reconnect, ExecResult |
| `vmcluster_mcp/exec/__init__.py` | ✓ VERIFIED | node_exec + node_exec_all tools, _ssh_manager singleton |
| `tests/test_ssh_client.py` | ✓ VERIFIED | 9 tests — caching, reconnect, timeout, stale connection |
| `tests/test_node_exec.py` | ✓ VERIFIED | 6 tests — success, not-found, non-zero exit, timeout |
| `tests/test_node_exec_all.py` | ✓ VERIFIED | 8 tests — parallel fan-out, require_all, subset targeting |

**Key Link Verification table**:

| From | To | Via | Status |
|------|----|-----|--------|
| `exec/__init__.py` | `exec/ssh.py` | `from vmcluster_mcp.exec.ssh import SSHConnectionManager` | ✓ WIRED |
| `exec/__init__.py` | `models.py` | `from vmcluster_mcp.models import ExecResult, ExecAllResult, ToolResult, ErrorDetail` | ✓ WIRED |
| `exec/__init__.py` | `server.py` | `from vmcluster_mcp import cluster, exec` triggers `@mcp.tool()` registration | ✓ WIRED |
| `exec/__init__.py` | `asyncio.gather` | Line 124: all per-node tasks fanned out concurrently | ✓ WIRED |
| `exec/ssh.py` | reconnect path | Line 89: `ConnectionLost/DisconnectError/OSError` → invalidate + retry | ✓ WIRED |
| `cluster/__init__.py` | `exec/ssh.py` | `_exec_pkg._ssh_manager.invalidate(node_spec.ip)` in node_crash | ✓ WIRED |

**Per-Plan Must-Have Truth tables** — one section for each of plans 03-01 through 03-03, replicating the 07 format. Derive truths from each plan's SUMMARY.md self-check/decisions list.

For plan 03-01 (SSHConnectionManager), truths to include:
- SSHConnectionManager caches connections by (ip, port, user, key_path) tuple
- Stale connection (ConnectionLost, DisconnectError, OSError) → invalidate + retry once
- asyncssh.TimeoutError caught before OSError handler; returns timed_out=True
- ExecResult contains exit_code, stdout, stderr, duration_s, timed_out
- known_hosts=None used for internal cluster trust
- time.monotonic() used for wall-clock duration measurement
- invalidate(ip) clears all cache entries for that IP

For plan 03-02 (node_exec), truths to include:
- node_exec tool registered with @mcp.tool() + @structured_tool_handler
- Non-zero exit codes return success=True (tool succeeded, command failed)
- structured_tool_handler wraps unexpected SSH manager exceptions to success=False
- _ssh_manager is module-level singleton in exec/__init__.py

For plan 03-03 (node_exec_all), truths to include:
- node_exec_all fans out via asyncio.gather (not sequential loop)
- failed_nodes includes both non-zero exit_code AND timed_out nodes
- require_all=True (default): success=False + PARTIAL_FAILURE error with recoverable=True
- require_all=False: success=True even with partial failures
- Empty nodes=[] treated same as nodes=None — targets all nodes

**Requirements Coverage table**:

| Requirement | Description | Plan | Status | Evidence |
|-------------|-------------|------|--------|----------|
| EXEC-01 | Agent can run a shell command on a single node with structured output | 03-02 | ✓ SATISFIED | node_exec + 6 tests |
| EXEC-02 | Agent can run the same command across all or subset of nodes in parallel | 03-03 | ✓ SATISFIED | node_exec_all + asyncio.gather + 8 tests |
| EXEC-03 | SSH connections re-established automatically after node reboot | 03-01 | ✓ SATISFIED | SSHConnectionManager reconnect path + test_ssh_client stale-connection test; node_crash invalidate wiring |

**Lightweight Verification Run** section — paste the actual pytest output from the commands run at start.

**Anti-Patterns Found** section — note no TODOs/FIXMEs/stubs; note asyncssh (D-11) used throughout; note per-IP invalidation is sufficient for node_crash use case.

**Human Verification** section — one entry for SC-3 explaining what live-VM testing would confirm: after snapshot_revert or node_crash, the next node_exec succeeds without error.
  </action>
  <verify>
    <automated>grep -l "EXEC-01" /home/hornc/repos/vmcluster-mcp/.planning/phases/03-ssh-execution/03-VERIFICATION.md && grep "phase: 03-ssh-execution" /home/hornc/repos/vmcluster-mcp/.planning/phases/03-ssh-execution/03-VERIFICATION.md && grep "EXEC-03" /home/hornc/repos/vmcluster-mcp/.planning/phases/03-ssh-execution/03-VERIFICATION.md && grep "SC-1\|SC-2\|SC-3" /home/hornc/repos/vmcluster-mcp/.planning/phases/03-ssh-execution/03-VERIFICATION.md | grep "VERIFIED" | wc -l</automated>
  </verify>
  <acceptance_criteria>
    - `.planning/phases/03-ssh-execution/03-VERIFICATION.md` exists
    - File contains `phase: 03-ssh-execution` in YAML frontmatter
    - File contains `status:` field in YAML frontmatter
    - File contains `score:` field in YAML frontmatter
    - File contains `EXEC-01`, `EXEC-02`, and `EXEC-03` — all 3 requirement IDs present
    - File contains at least 3 rows with `VERIFIED` in the SC table
    - File contains `human_verification:` block in frontmatter
    - `grep "03-01\|03-02\|03-03" 03-VERIFICATION.md` returns at least 3 matches (per-plan sections)
    - File contains `asyncio.gather` as evidence citation
    - File contains `test_ssh_client` as evidence citation
    - File contains `PARTIAL_FAILURE` as evidence citation
  </acceptance_criteria>
  <done>03-VERIFICATION.md exists, all 3 roadmap SCs are VERIFIED with 3-source evidence (code location + test file + test name), EXEC-01..03 requirements table is present and all marked SATISFIED, per-plan must-have truth tables are present for all 3 plans.</done>
</task>

</tasks>

<verification>
After completion:
- `ls .planning/phases/03-ssh-execution/03-VERIFICATION.md` — file exists
- `grep "EXEC-0[1-3]" .planning/phases/03-ssh-execution/03-VERIFICATION.md | wc -l` — returns 3 or more
- `grep "SATISFIED" .planning/phases/03-ssh-execution/03-VERIFICATION.md | wc -l` — returns 3
- `grep "VERIFIED" .planning/phases/03-ssh-execution/03-VERIFICATION.md | wc -l` — returns at least 3 (SC rows)
</verification>

<success_criteria>
- 03-VERIFICATION.md written and follows the exact schema of 07-VERIFICATION.md
- All 3 Phase 3 roadmap success criteria verified with 3-source evidence
- All EXEC-01 through EXEC-03 requirements marked SATISFIED in requirements coverage table
- Per-plan must-have truth tables present for all 3 plans (03-01 through 03-03)
- Human verification section documents SC-3 live-VM test requirement
</success_criteria>

<output>
After completion, create `.planning/phases/08-retroactive-phase-verification/08-02-SUMMARY.md`
</output>
