---
phase: 07-network-node-fault-injection
plan: 04
type: execute
wave: 4
depends_on: ["07-03"]
files_modified:
  - vmcluster_mcp/net/__init__.py
  - vmcluster_mcp/cluster/__init__.py
  - vmcluster_mcp/cluster/libvirt_client.py
  - tests/test_net_fault_list.py
  - tests/test_node_crash.py
autonomous: true
requirements:
  - FAULT-05
  - FAULT-06

must_haves:
  truths:
    - "net_fault_list returns all active NetFaultInfo entries for a cluster from the reconciled registry"
    - "net_fault_list returns empty list when no active faults exist"
    - "net_fault_list includes parameters dict with fault-specific details (partition groups, impairment values)"
    - "node_crash performs virsh destroy + virsh start via LibvirtClient (D-02, D-15)"
    - "node_crash with restart_after=True boots the node and waits for SSH reachability (D-12)"
    - "node_crash with restart_after=False leaves the node shut off (D-13)"
    - "node_crash invalidates the SSH connection cache for the crashed node (D-12)"
    - "node_crash returns NodeStatus with current domain_state and ssh_reachable"
  artifacts:
    - path: "vmcluster_mcp/net/__init__.py"
      provides: "net_fault_list tool registration"
      exports: ["net_partition", "net_impair", "net_heal", "net_heal_all", "net_fault_list"]
    - path: "vmcluster_mcp/cluster/__init__.py"
      provides: "node_crash tool registration"
      contains: "async def node_crash"
    - path: "tests/test_net_fault_list.py"
      provides: "Unit tests for net_fault_list tool"
      min_lines: 25
    - path: "tests/test_node_crash.py"
      provides: "Unit tests for node_crash tool"
      min_lines: 40
  key_links:
    - from: "vmcluster_mcp/net/__init__.py"
      to: "vmcluster_mcp/net/fault_registry.py"
      via: "_registry.list_faults for net_fault_list"
      pattern: "_registry\\.list_faults"
    - from: "vmcluster_mcp/cluster/__init__.py"
      to: "vmcluster_mcp/cluster/libvirt_client.py"
      via: "_client.destroy_domain + _client.start_domain_by_name for node_crash"
      pattern: "_client\\.destroy_domain|_client\\.start_domain_by_name"
    - from: "vmcluster_mcp/cluster/__init__.py"
      to: "vmcluster_mcp/exec/ssh.py"
      via: "SSHConnectionManager.invalidate after crash"
      pattern: "_ssh_manager\\.invalidate"
---

<objective>
Implement the `net_fault_list` and `node_crash` MCP tools — the final Phase 7 tools for fault visibility and node hard-reset simulation.

Purpose: `net_fault_list` gives agents visibility into all active faults (completing the inject-list-heal cycle). `node_crash` simulates unclean node failures via libvirt destroy+start, exercising SSH reconnect semantics from Phase 3. Together these complete the full fault injection feature set.

Output: `net_fault_list` tool in `net/__init__.py`, `node_crash` tool in `cluster/__init__.py`, comprehensive tests.
</objective>

<execution_context>
@~/.copilot/get-shit-done/workflows/execute-plan.md
@~/.copilot/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/07-network-node-fault-injection/07-CONTEXT.md
@.planning/phases/07-network-node-fault-injection/07-03-SUMMARY.md

<interfaces>
<!-- Key types and contracts from prior plans + existing codebase -->

From vmcluster_mcp/models.py:
```python
class NetFaultInfo(BaseModel):
    handle_id: str
    fault_type: Literal["partition", "impairment"]
    affected_nodes: list[str]
    created_at: str
    parameters: dict[str, str | int | float]

class NodeStatus(BaseModel):
    name: str
    role: str
    ip: str
    domain_state: Literal["running", "shut off", "paused", "crashed", "unknown"]
    ssh_reachable: bool
    artifact_id_installed: str | None = None
    loaded_modules: list[str] = []

class ToolResult(BaseModel, Generic[T]):
    success: bool
    result: T | None = None
    error: ErrorDetail | None = None
```

From vmcluster_mcp/net/fault_registry.py (created in 07-01):
```python
class FaultRegistry:
    def list_faults(self, cluster_name: str) -> list[NetFaultInfo]: ...
    async def reconcile(self) -> int: ...
```

From vmcluster_mcp/net/__init__.py (created in 07-02, extended in 07-03):
```python
_registry = FaultRegistry(config.fault_registry)
_registry.load()

# net_partition, net_impair, net_heal, net_heal_all already registered
```

From vmcluster_mcp/cluster/libvirt_client.py:
```python
class LibvirtClient:
    async def lookup_domain(self, name: str) -> libvirt.virDomain | None: ...
    async def destroy_domain(self, dom: libvirt.virDomain) -> None: ...
    async def start_domain_by_name(self, name: str) -> None: ...
    async def get_domain_state(self, dom: libvirt.virDomain) -> str: ...
    async def is_domain_active(self, dom: libvirt.virDomain) -> bool: ...

def domain_name(cluster_name: str, node_name: str) -> str:
    return f"{cluster_name}_{node_name}"
```

From vmcluster_mcp/cluster/__init__.py:
```python
_client = LibvirtClient(config.libvirt_uri)

async def _wait_for_ssh(ip, port, user, key_path, timeout_s) -> tuple[bool, float]: ...

# Existing tools: cluster_define, cluster_start, cluster_stop, cluster_destroy,
#                 cluster_status, cluster_handle
```

From vmcluster_mcp/exec/ssh.py:
```python
class SSHConnectionManager:
    async def invalidate(self, ip: str) -> None: ...
```

From vmcluster_mcp/exec/__init__.py:
```python
_ssh_manager = SSHConnectionManager()
```
</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: net_fault_list tool</name>
  <files>vmcluster_mcp/net/__init__.py, tests/test_net_fault_list.py</files>
  <read_first>
    - vmcluster_mcp/net/__init__.py (current state — add net_fault_list after existing tools)
    - vmcluster_mcp/net/fault_registry.py (FaultRegistry.list_faults returns list[NetFaultInfo])
    - vmcluster_mcp/models.py (NetFaultInfo fields)
    - .planning/phases/07-network-node-fault-injection/07-CONTEXT.md (D-04, D-11)
  </read_first>
  <behavior>
    - Test: net_fault_list with active partition + impairment faults returns both as NetFaultInfo
    - Test: net_fault_list with no active faults returns empty list
    - Test: net_fault_list filters by cluster_name (faults from other clusters not returned)
    - Test: net_fault_list includes parameters dict with fault-specific details
    - Test: net_fault_list reflects reconciled state (if reconcile() was called, stale entries gone)
  </behavior>
  <action>
1. **Add `net_fault_list` tool to `vmcluster_mcp/net/__init__.py`:**
   - `@mcp.tool()` + `@structured_tool_handler`
   - `async def net_fault_list(cluster_name: str) -> ToolResult[list[NetFaultInfo]]:`
   - Call `_registry.list_faults(cluster_name)` to get all active faults for the cluster
   - Return `ToolResult(success=True, result=faults)`
   - This is deliberately simple — the registry owns all complexity. The tool is a thin wrapper with the structured_tool_handler error contract.

2. **Create `tests/test_net_fault_list.py`** — mock FaultRegistry. Pre-populate with test entries, verify filtering and return shape.
  </action>
  <verify>
    <automated>cd /home/hornc/repos/vmcluster-mcp && python -m pytest tests/test_net_fault_list.py -x -v && ruff check vmcluster_mcp/net/__init__.py</automated>
  </verify>
  <done>net_fault_list returns filtered NetFaultInfo entries from reconciled registry, all tests pass.</done>
</task>

<task type="auto" tdd="true">
  <name>Task 2: node_crash tool with libvirt destroy/start and SSH invalidation</name>
  <files>vmcluster_mcp/cluster/__init__.py, tests/test_node_crash.py</files>
  <read_first>
    - vmcluster_mcp/cluster/__init__.py (existing tools — add node_crash following same pattern)
    - vmcluster_mcp/cluster/libvirt_client.py (destroy_domain, start_domain_by_name, get_domain_state, domain_name)
    - vmcluster_mcp/exec/ssh.py (SSHConnectionManager.invalidate)
    - vmcluster_mcp/exec/__init__.py (_ssh_manager instance)
    - .planning/phases/07-network-node-fault-injection/07-CONTEXT.md (D-02, D-12, D-13, D-15)
    - .planning/phases/04-snapshot-management/04-CONTEXT.md (destroy/restart/wait patterns)
  </read_first>
  <behavior>
    - Test: node_crash calls _client.destroy_domain on the target domain (D-02, D-15)
    - Test: node_crash with restart_after=True calls _client.start_domain_by_name after destroy
    - Test: node_crash with restart_after=True and wait_for_ssh=True uses _wait_for_ssh (D-12)
    - Test: node_crash with restart_after=False does NOT start or wait for SSH (D-13)
    - Test: node_crash invalidates SSH cache for the crashed node's IP (D-12)
    - Test: node_crash returns NodeStatus with domain_state="running" and ssh_reachable=True after restart+wait
    - Test: node_crash returns NodeStatus with domain_state="shut off" when restart_after=False
    - Test: node_crash on unknown node returns ToolResult(success=False) with NODE_NOT_FOUND error
    - Test: node_crash on already shut-off node still performs destroy attempt (best-effort) then starts if restart_after=True
    - Test: SSH wait timeout returns ToolResult with success=False and SSH_TIMEOUT error code
  </behavior>
  <action>
1. **Add `node_crash` tool to `vmcluster_mcp/cluster/__init__.py`:**
   - Import `SSHConnectionManager` and get access to the exec module's `_ssh_manager`
     (import from `vmcluster_mcp.exec` — the same instance used by node_exec)
   - `@mcp.tool()` + `@structured_tool_handler`
   - `async def node_crash(cluster_name: str, node: str, restart_after: bool = True, wait_for_ssh: bool = True, ssh_timeout_s: int = 120) -> ToolResult[NodeStatus]:`

   Implementation:
   a. Load topology: `topo = load_topology(cluster_name, config.topology_dir)`
   b. Find node in topology. If not found, return `ToolResult(success=False, error=ErrorDetail(code="NODE_NOT_FOUND", ...))`
   c. Look up domain: `dom = await _client.lookup_domain(domain_name(cluster_name, node))`
   d. **Destroy** (D-02): If domain is active, `await _client.destroy_domain(dom)`. Catch libvirtError if already shut off.
   e. **Invalidate SSH** (D-12): `await _ssh_manager.invalidate(node_spec.ip)` — clears cached connection so next exec re-establishes.
   f. If `restart_after=True`:
      - `await _client.start_domain_by_name(domain_name(cluster_name, node))`
      - If `wait_for_ssh=True`:
        - `reachable, elapsed = await _wait_for_ssh(node_spec.ip, 22, topo.ssh.user, topo.ssh.key_path, ssh_timeout_s)`
        - If not reachable: return `ToolResult(success=False, error=ErrorDetail(code="SSH_TIMEOUT", message=f"Node {node} not SSH-reachable after {ssh_timeout_s}s", node=node, recoverable=True))`
      - Get domain state: `state = await _client.get_domain_state(dom_new)`
      - Return `ToolResult(success=True, result=NodeStatus(name=node, role=node_spec.role, ip=node_spec.ip, domain_state=state, ssh_reachable=reachable if wait_for_ssh else False))`
   g. If `restart_after=False` (D-13):
      - Return `ToolResult(success=True, result=NodeStatus(name=node, role=node_spec.role, ip=node_spec.ip, domain_state="shut off", ssh_reachable=False))`

2. **Import `_ssh_manager`:** Add `from vmcluster_mcp.exec import _ssh_manager` at the appropriate location in `cluster/__init__.py` (after the module-level imports, being careful not to create circular imports — the exec module imports from server.py not from cluster, so this is safe).

3. **Create `tests/test_node_crash.py`** — mock LibvirtClient, _wait_for_ssh, _ssh_manager. Test all behaviors above.
  </action>
  <verify>
    <automated>cd /home/hornc/repos/vmcluster-mcp && python -m pytest tests/test_node_crash.py -x -v && ruff check vmcluster_mcp/cluster/__init__.py</automated>
  </verify>
  <done>node_crash destroys via LibvirtClient, invalidates SSH cache, optionally restarts with SSH wait, returns NodeStatus, all tests pass.</done>
</task>

</tasks>

<verification>
- `python -m pytest tests/test_net_fault_list.py tests/test_node_crash.py -x -v` — all tests pass
- `ruff check vmcluster_mcp/net/ vmcluster_mcp/cluster/` — no lint errors
- net_fault_list returns correctly filtered, reconciled fault information
- node_crash performs destroy+start via LibvirtClient single-worker executor
- SSH cache invalidation occurs before restart attempt
</verification>

<success_criteria>
- net_fault_list returns all active NetFaultInfo for a cluster, empty list when none
- node_crash performs destroy → SSH invalidate → optional start → optional SSH wait
- node_crash returns NodeStatus with correct domain_state and ssh_reachable
- All Phase 7 tools are registered: net_partition, net_impair, net_heal, net_heal_all, net_fault_list, node_crash
- All tests pass, ruff clean
</success_criteria>

<output>
After completion, create `.planning/phases/07-network-node-fault-injection/07-04-SUMMARY.md`
</output>
