---
phase: 03-ssh-execution
plan: 03
type: execute
wave: 3
depends_on: ["03-02"]
files_modified:
  - vmcluster_mcp/exec/__init__.py
  - tests/test_node_exec_all.py
autonomous: true
requirements:
  - EXEC-01
  - EXEC-02

must_haves:
  truths:
    - "Agent can run the same command across all nodes in parallel and receive per-node results"
    - "Agent can target a subset of nodes by passing a node list"
    - "require_all=True (default) causes ToolResult.success=False when any node has non-zero exit code"
    - "require_all=False returns success=True even when some nodes fail, with failed_nodes populated"
    - "Results include all_succeeded and failed_nodes summary fields"
  artifacts:
    - path: "vmcluster_mcp/exec/__init__.py"
      provides: "node_exec_all MCP tool (appended to existing file)"
      exports: ["node_exec_all"]
    - path: "tests/test_node_exec_all.py"
      provides: "Unit tests for node_exec_all tool"
      min_lines: 50
  key_links:
    - from: "vmcluster_mcp/exec/__init__.py"
      to: "vmcluster_mcp/exec/ssh.py"
      via: "SSHConnectionManager.execute() called per-node via asyncio.gather"
      pattern: "asyncio\\.gather"
    - from: "vmcluster_mcp/exec/__init__.py"
      to: "vmcluster_mcp/models.py"
      via: "ExecAllResult for aggregated per-node output"
      pattern: "ExecAllResult"
---

<objective>
Create the `node_exec_all` MCP tool that runs a command across all (or a subset of) cluster nodes in parallel via `asyncio.gather`, returning aggregated per-node results.

Purpose: EXEC-02 — agent's interface for parallel command execution with `require_all` failure semantics.

Output: `node_exec_all` tool added to `vmcluster_mcp/exec/__init__.py` + unit tests.
</objective>

<execution_context>
@~/.copilot/get-shit-done/workflows/execute-plan.md
@~/.copilot/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/REQUIREMENTS.md
@.planning/phases/03-ssh-execution/03-01-PLAN.md
@.planning/phases/03-ssh-execution/03-02-PLAN.md

<interfaces>
<!-- Key types and contracts the executor needs. Extracted from codebase. -->

From vmcluster_mcp/models.py:
```python
class ExecResult(BaseModel):
    exit_code: int
    stdout: str
    stderr: str
    duration_s: float
    timed_out: bool

class ExecAllResult(BaseModel):
    results: dict[str, ExecResult]
    all_succeeded: bool
    failed_nodes: list[str]
```

From vmcluster_mcp/exec/__init__.py (created in 03-02):
```python
_ssh_manager = SSHConnectionManager()

@mcp.tool()
@structured_tool_handler
async def node_exec(cluster_name: str, node_name: str, command: str, timeout_s: int = 30) -> ToolResult[ExecResult]:
    ...
```

From vmcluster_mcp/exec/ssh.py (created in 03-01):
```python
class SSHConnectionManager:
    async def execute(self, ip, port, user, key_path, command, timeout_s=30) -> ExecResult: ...
```

From vmcluster_mcp/topology/schema.py:
```python
class TopologySpec(BaseModel):
    cluster_name: str
    nodes: list[NodeSpec]
    ssh: SshSpec
```

From STATE.md decisions:
- `require_all=True` default on `node_exec_all` — partial failures propagate as ToolResult.success=False
</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Add node_exec_all tool with asyncio.gather parallel fan-out</name>
  <files>vmcluster_mcp/exec/__init__.py, tests/test_node_exec_all.py</files>
  <read_first>
    - vmcluster_mcp/exec/__init__.py (current state after 03-02 — node_exec tool, _ssh_manager, imports)
    - vmcluster_mcp/models.py (ExecAllResult definition — results dict, all_succeeded, failed_nodes)
    - vmcluster_mcp/topology/schema.py (TopologySpec.nodes, NodeSpec.name, NodeSpec.ip)
    - vmcluster_mcp/cluster/__init__.py (asyncio.gather pattern used in cluster_start for parallel SSH checks)
  </read_first>
  <behavior>
    - Test 1: node_exec_all with 3 nodes, all succeed → ToolResult.success=True, ExecAllResult.all_succeeded=True, failed_nodes=[], results has 3 entries
    - Test 2: node_exec_all with 3 nodes, one has exit_code=1, require_all=True (default) → ToolResult.success=False, all_succeeded=False, failed_nodes=["failing_node"], error.code="PARTIAL_FAILURE"
    - Test 3: node_exec_all with 3 nodes, one has exit_code=1, require_all=False → ToolResult.success=True, all_succeeded=False, failed_nodes=["failing_node"]
    - Test 4: node_exec_all with nodes=["node1", "node2"] subset → only those 2 nodes executed (not all 3 in topology)
    - Test 5: node_exec_all with nodes=[] or nodes=None → executes on ALL nodes in topology
    - Test 6: node_exec_all with an invalid node name in the subset → returns ToolResult with success=False, error.code="NODE_NOT_FOUND"
    - Test 7: node_exec_all with one node timing out → that node appears in failed_nodes with timed_out=True in its ExecResult
    - Test 8: Verify asyncio.gather is used (commands run concurrently, not sequentially)
  </behavior>
  <action>
    Append `node_exec_all` tool to `vmcluster_mcp/exec/__init__.py`:

    1. Add `ExecAllResult` to the models import line (add to existing `from vmcluster_mcp.models import ...`).

    2. Add `import asyncio` at the top of the file.

    **`node_exec_all` tool:**
    - Signature: `async def node_exec_all(cluster_name: str, command: str, nodes: list[str] | None = None, require_all: bool = True, timeout_s: int = 30) -> ToolResult[ExecAllResult]`
    - Decorated with `@mcp.tool()` then `@structured_tool_handler`
    - Load topology via `load_topology(cluster_name, config.topology_dir)`
    - If `nodes` is None or empty list: target ALL nodes from topology
    - If `nodes` is provided: validate each name exists in topology — if any not found, return `ToolResult(success=False, error=ErrorDetail(code="NODE_NOT_FOUND", message=f"Nodes not found: {missing}", recoverable=False))`
    - Create async tasks: for each target node, call `_ssh_manager.execute(ip=node.ip, port=22, user=topo.ssh.user, key_path=topo.ssh.key_path, command=command, timeout_s=timeout_s)`
    - Fan out with `asyncio.gather(*tasks)` — all nodes execute concurrently
    - Build `results: dict[str, ExecResult]` mapping node_name → ExecResult
    - Compute `failed_nodes`: nodes where `exit_code != 0` OR `timed_out is True`
    - Compute `all_succeeded = len(failed_nodes) == 0`
    - If `require_all` and not `all_succeeded`: return `ToolResult(success=False, result=ExecAllResult(results=results, all_succeeded=False, failed_nodes=failed_nodes), error=ErrorDetail(code="PARTIAL_FAILURE", message=f"Command failed on nodes: {', '.join(failed_nodes)}", recoverable=True))`
    - Otherwise: return `ToolResult(success=True, result=ExecAllResult(results=results, all_succeeded=all_succeeded, failed_nodes=failed_nodes))`

    **Docstring** must describe: args (cluster_name, command, nodes, require_all, timeout_s), returns (ToolResult[ExecAllResult] with per-node results, all_succeeded, failed_nodes).
  </action>
  <acceptance_criteria>
    - vmcluster_mcp/exec/__init__.py contains `async def node_exec_all(cluster_name: str, command: str, nodes: list[str] | None = None, require_all: bool = True, timeout_s: int = 30)`
    - vmcluster_mcp/exec/__init__.py contains `@mcp.tool()` before node_exec_all
    - vmcluster_mcp/exec/__init__.py contains `@structured_tool_handler` before node_exec_all
    - vmcluster_mcp/exec/__init__.py contains `asyncio.gather(`
    - vmcluster_mcp/exec/__init__.py contains `ExecAllResult` in imports
    - vmcluster_mcp/exec/__init__.py contains `PARTIAL_FAILURE` error code
    - vmcluster_mcp/exec/__init__.py contains `require_all` logic
    - vmcluster_mcp/exec/__init__.py contains `failed_nodes` computation
    - tests/test_node_exec_all.py exists with at least 7 test functions
    - `pytest tests/test_node_exec_all.py -x` exits 0
    - `ruff check vmcluster_mcp/exec/__init__.py` exits 0
    - `pytest tests/ -x` exits 0 (full suite still passes)
  </acceptance_criteria>
  <verify>
    <automated>pytest tests/test_node_exec_all.py -x && ruff check vmcluster_mcp/exec/__init__.py && pytest tests/ -x</automated>
  </verify>
  <done>node_exec_all tool registered on FastMCP, executes commands on all/subset of nodes in parallel via asyncio.gather, returns ExecAllResult with per-node results and require_all failure semantics. Full test suite passes.</done>
</task>

</tasks>

<verification>
- `pytest tests/ -x` — full test suite passes (all prior + new tests)
- `ruff check vmcluster_mcp/exec/` — no lint errors in exec module
- `grep "node_exec_all" vmcluster_mcp/exec/__init__.py` confirms tool exists
- `grep "asyncio.gather" vmcluster_mcp/exec/__init__.py` confirms parallel execution
- `grep "require_all" vmcluster_mcp/exec/__init__.py` confirms failure semantics
- `grep "PARTIAL_FAILURE" vmcluster_mcp/exec/__init__.py` confirms error code
</verification>

<success_criteria>
`node_exec_all` tool is registered on FastMCP, accepts cluster_name/command/nodes/require_all/timeout_s, fans out via asyncio.gather, returns ToolResult[ExecAllResult] with per-node ExecResult, all_succeeded, and failed_nodes. Default `require_all=True` causes success=False on any node failure. Full test suite (124 prior + new) passes.
</success_criteria>

<output>
After completion, create `.planning/phases/03-ssh-execution/03-03-SUMMARY.md`
</output>
