---
phase: 07-network-node-fault-injection
plan: "04"
subsystem: net-fault/cluster-crash
tags: [net_fault_list, node_crash, fault-injection, libvirt, ssh-cache]
dependency_graph:
  requires: ["07-03"]
  provides: [net_fault_list, node_crash]
  affects: [vmcluster_mcp/net/__init__.py, vmcluster_mcp/cluster/__init__.py, vmcluster_mcp/models.py]
tech_stack:
  added: []
  patterns: [tdd-red-green, lazy-inline-import, thin-tool-wrapper]
key_files:
  created:
    - tests/test_net_fault_list.py
    - tests/test_node_crash.py
  modified:
    - vmcluster_mcp/net/__init__.py
    - vmcluster_mcp/cluster/__init__.py
    - vmcluster_mcp/models.py
decisions:
  - "Lazy inline imports (import libvirt, import vmcluster_mcp.exec) inside node_crash body to avoid circular import chain: exec->server->cluster->exec"
  - "Patch vmcluster_mcp.exec._ssh_manager in tests (not vmcluster_mcp.cluster._ssh_manager) because exec module owns the instance"
  - "NetFaultInfo.parameters widened from dict[str, str|int|float] to dict[str, Any] — net_partition stores list values for partition_a/partition_b"
metrics:
  duration_s: 482
  completed_date: "2026-03-23"
  tasks: 2
  files_changed: 5
requirements:
  - FAULT-05
  - FAULT-06
requirements-completed: [FAULT-05, FAULT-06]
---

# Phase 07 Plan 04: net_fault_list + node_crash Summary

**One-liner:** `net_fault_list` thin registry wrapper + `node_crash` virsh-destroy/restart with SSH cache invalidation completing Phase 7 fault injection toolset.

## What Was Built

### net_fault_list (Task 1)

Registered `net_fault_list` MCP tool in `vmcluster_mcp/net/__init__.py`:

```python
async def net_fault_list(cluster_name: str) -> ToolResult[list[NetFaultInfo]]:
    faults = _registry.list_faults(cluster_name)
    return ToolResult(success=True, result=faults)
```

Deliberately thin — the FaultRegistry owns all filtering and reconciliation logic. The tool is a `@structured_tool_handler`-wrapped MCP endpoint that gives agents full visibility into active faults (partition + impairment) for a cluster, completing the inject→list→heal cycle.

### node_crash (Task 2)

Registered `node_crash` MCP tool in `vmcluster_mcp/cluster/__init__.py`:

- **Destroy**: Calls `_client.destroy_domain(dom)` — force-off equivalent to `virsh destroy`. Best-effort: libvirtError caught and logged if domain is already shut off.
- **SSH cache invalidation**: Calls `_exec_pkg._ssh_manager.invalidate(node_spec.ip)` — clears the cached asyncssh connection so the next `node_exec` call reconnects cleanly.
- **Conditional restart**: If `restart_after=True`, calls `_client.start_domain_by_name(dname)`.
- **SSH wait**: If `wait_for_ssh=True`, polls via `_wait_for_ssh`; returns `SSH_TIMEOUT` error on timeout.
- **NodeStatus return**: Returns `NodeStatus(domain_state, ssh_reachable)` reflecting post-operation state.

### Phase 7 Tool Inventory (complete)

| Tool | Module | Purpose |
|------|--------|---------|
| `net_partition` | `net/__init__.py` | iptables DROP between node groups |
| `net_impair` | `net/__init__.py` | tc netem latency/loss/jitter |
| `net_heal` | `net/__init__.py` | Remove specific fault by handle |
| `net_heal_all` | `net/__init__.py` | Remove all cluster faults |
| `net_fault_list` | `net/__init__.py` | List active faults for cluster |
| `node_crash` | `cluster/__init__.py` | Force destroy + optional restart |

## Tests

| File | Tests | Coverage |
|------|-------|---------|
| `tests/test_net_fault_list.py` | 6 | Both fault types, empty list, cluster filter, parameters, reconciled state, model instances |
| `tests/test_node_crash.py` | 10 | Unknown node, destroy call, best-effort on already-off, SSH invalidation (with/without restart), restart=False semantics, restart=True with start_domain_by_name, wait_for_ssh, full NodeStatus, SSH timeout |

**Total new tests:** 16  
**Full suite:** 361 passed

## Deviations from Plan

### Auto-fixed Issues

**1. [Rule 1 - Bug] Fixed `NetFaultInfo.parameters` type too narrow**
- **Found during:** Task 1 (TDD RED → first test run)
- **Issue:** `NetFaultInfo.parameters` typed as `dict[str, str | int | float]`. But `net_partition` stores `partition_a: list[str]` and `partition_b: list[str]` in parameters, so `list_faults()` would fail Pydantic validation when reconstructing entries from the registry.
- **Fix:** Widened to `dict[str, Any]` in `vmcluster_mcp/models.py`
- **Files modified:** `vmcluster_mcp/models.py`
- **Commit:** `0f3af2b` (included in Task 1 commit)

**2. [Rule 3 - Blocking] Fixed circular import: `cluster` → `exec` at module load time**
- **Found during:** Task 2 (full test suite after implementing node_crash)
- **Issue:** Adding `from vmcluster_mcp.exec import _ssh_manager` at cluster module level created a circular dependency: `exec/__init__.py` imports `server.py`, `server.py` imports `cluster`, `cluster` tries to import `exec` — but `exec` is still mid-initialization at that point, so `_ssh_manager` is not yet defined.
- **Fix:** Used lazy inline imports inside `node_crash` function body (`import libvirt` and `import vmcluster_mcp.exec as _exec_pkg`). Tests patched at `vmcluster_mcp.exec._ssh_manager` (where the instance actually lives).
- **Files modified:** `vmcluster_mcp/cluster/__init__.py`, `tests/test_node_crash.py`
- **Commit:** `6213b08`

## Self-Check: PASSED

| Check | Result |
|-------|--------|
| `vmcluster_mcp/net/__init__.py` exists | ✓ |
| `vmcluster_mcp/cluster/__init__.py` exists | ✓ |
| `tests/test_net_fault_list.py` exists | ✓ |
| `tests/test_node_crash.py` exists | ✓ |
| Commit `0f3af2b` (net_fault_list) | ✓ |
| Commit `6213b08` (node_crash) | ✓ |
| `_registry.list_faults` pattern in net/__init__.py | ✓ |
| `_client.destroy_domain` / `_client.start_domain_by_name` in cluster/__init__.py | ✓ |
| `_exec_pkg._ssh_manager.invalidate` in cluster/__init__.py | ✓ |
