---
phase: 07-network-node-fault-injection
plan: 03
type: execute
wave: 3
depends_on: ["07-02"]
files_modified:
  - vmcluster_mcp/net/fault.py
  - vmcluster_mcp/net/__init__.py
  - tests/test_net_impair.py
  - tests/test_net_heal.py
autonomous: true
requirements:
  - FAULT-02
  - FAULT-03
  - FAULT-04

must_haves:
  truths:
    - "net_impair applies tc netem qdisc on the source node's egress tap interface"
    - "Impairment is scoped to traffic destined for specific target node(s) via tc filter"
    - "A single impairment handle can combine latency, jitter, loss, corruption, and reordering"
    - "Zero-valued parameters are omitted from the effective tc command (D-09)"
    - "net_heal removes all host-side iptables/tc state for a specific handle and removes the registry entry"
    - "net_heal returns success=True after cleanup even if live rules were already absent"
    - "net_heal_all removes all active faults for a cluster and returns the count of handles removed"
    - "net_fault_list returns empty list immediately after net_heal_all"
  artifacts:
    - path: "vmcluster_mcp/net/fault.py"
      provides: "tc netem command builder, apply, and cleanup functions; tap interface discovery"
      contains: "build_impairment_cmd"
    - path: "vmcluster_mcp/net/__init__.py"
      provides: "net_impair, net_heal, net_heal_all tool registrations"
      exports: ["net_impair", "net_heal", "net_heal_all"]
    - path: "tests/test_net_impair.py"
      provides: "Unit tests for tc netem rule generation and impairment tool behavior"
      min_lines: 50
    - path: "tests/test_net_heal.py"
      provides: "Unit tests for net_heal + net_heal_all cleanup behavior"
      min_lines: 40
  key_links:
    - from: "vmcluster_mcp/net/fault.py"
      to: "vmcluster_mcp/cluster/libvirt_client.py"
      via: "get_domain_xml for tap interface discovery"
      pattern: "get_domain_xml"
    - from: "vmcluster_mcp/net/__init__.py"
      to: "vmcluster_mcp/net/fault.py"
      via: "import apply_impairment, remove_impairment, remove_rules_by_comment"
      pattern: "from.*fault.*import"
    - from: "vmcluster_mcp/net/__init__.py"
      to: "vmcluster_mcp/net/fault_registry.py"
      via: "FaultRegistry for handle tracking during heal/heal_all"
      pattern: "_registry\\."
---

<objective>
Implement `net_impair` (tc netem traffic degradation), `net_heal` (handle-specific fault removal), and `net_heal_all` (cluster-wide fault cleanup) MCP tools.

Purpose: Completes the fault injection/healing cycle — agents can now degrade traffic with latency/loss/corruption and clean up faults individually or in bulk. `net_heal` must handle both partition (iptables) and impairment (tc) fault types generically via the registry's fault_type and rule_identifiers metadata.

Output: Extended `net/fault.py` with tc netem helpers and tap discovery, three new tool registrations in `net/__init__.py`, tests.
</objective>

<execution_context>
@~/.copilot/get-shit-done/workflows/execute-plan.md
@~/.copilot/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/07-network-node-fault-injection/07-CONTEXT.md
@.planning/phases/07-network-node-fault-injection/07-01-SUMMARY.md
@.planning/phases/07-network-node-fault-injection/07-02-SUMMARY.md

<interfaces>
<!-- Key types and contracts from prior plans + existing codebase -->

From vmcluster_mcp/models.py:
```python
class NetFaultHandle(BaseModel):
    handle_id: str
    fault_type: Literal["partition", "impairment"]
    affected_nodes: list[str]
    created_at: str

class NetFaultInfo(BaseModel):
    handle_id: str
    fault_type: Literal["partition", "impairment"]
    affected_nodes: list[str]
    created_at: str
    parameters: dict[str, str | int | float]

class ToolResult(BaseModel, Generic[T]):
    success: bool
    result: T | None = None
    error: ErrorDetail | None = None
```

From vmcluster_mcp/net/fault_registry.py (created in 07-01):
```python
class FaultRegistry:
    def register(self, handle_id, cluster_name, fault_type, affected_nodes, parameters, rule_identifiers) -> NetFaultHandle: ...
    def get(self, handle_id: str) -> dict | None: ...
    def remove(self, handle_id: str) -> bool: ...
    def list_faults(self, cluster_name: str) -> list[NetFaultInfo]: ...

async def _run_host_cmd(cmd: list[str]) -> tuple[int, str]: ...
```

From vmcluster_mcp/net/fault.py (created in 07-02):
```python
async def apply_partition_rules(rules: list[list[str]]) -> list[str]: ...
async def remove_rules_by_comment(comment_tag: str) -> int: ...
```

From vmcluster_mcp/net/__init__.py (created in 07-02):
```python
from vmcluster_mcp.net.fault_registry import FaultRegistry
_registry = FaultRegistry(config.fault_registry)
_registry.load()

@mcp.tool()
@structured_tool_handler
async def net_partition(...) -> ToolResult[NetFaultHandle]: ...
```

From vmcluster_mcp/cluster/libvirt_client.py:
```python
class LibvirtClient:
    async def lookup_domain(self, name: str) -> libvirt.virDomain | None: ...
    async def get_domain_xml(self, dom: libvirt.virDomain, flags: int = 0) -> str: ...
```

From vmcluster_mcp/cluster/__init__.py:
```python
_client = LibvirtClient(config.libvirt_uri)
```

From vmcluster_mcp/cluster/libvirt_client.py:
```python
def domain_name(cluster_name: str, node_name: str) -> str:
    return f"{cluster_name}_{node_name}"
```
</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: tc netem helpers + tap interface discovery + net_impair tool</name>
  <files>vmcluster_mcp/net/fault.py, vmcluster_mcp/net/__init__.py, tests/test_net_impair.py</files>
  <read_first>
    - vmcluster_mcp/net/fault.py (current state from 07-02 — iptables functions)
    - vmcluster_mcp/cluster/libvirt_client.py (get_domain_xml for interface discovery, domain_name)
    - vmcluster_mcp/net/fault_registry.py (_run_host_cmd helper)
    - vmcluster_mcp/topology/schema.py (NodeSpec, NetworkSpec)
    - .planning/phases/07-network-node-fault-injection/07-CONTEXT.md (D-08, D-09)
  </read_first>
  <behavior>
    - Test: discover_tap_interface parses domain XML to find the tap device name for a given bridge
    - Test: build_impairment_cmd with latency_ms=100, jitter_ms=10 generates "tc qdisc add dev {tap} root netem delay 100ms 10ms"
    - Test: build_impairment_cmd with loss_pct=5.0 generates "tc qdisc add dev {tap} root netem loss 5%"
    - Test: build_impairment_cmd with latency_ms=100, loss_pct=5.0, corrupt_pct=1.0 combines all in one tc command
    - Test: build_impairment_cmd omits zero-valued parameters from the effective command (D-09)
    - Test: apply_impairment calls _run_host_cmd with the generated tc command
    - Test: remove_impairment calls "tc qdisc del dev {tap} root" to remove netem qdisc
    - Test: net_impair resolves source node to tap interface, applies tc, registers in FaultRegistry
    - Test: net_impair with all zeroparam (latency=0, loss=0, etc.) returns error (no-op impairment)
  </behavior>
  <action>
1. **Add to `vmcluster_mcp/net/fault.py`:**

   **`async def discover_tap_interface(libvirt_client, cluster_name, node_name, bridge) -> str`:**
   - Look up domain via `libvirt_client.lookup_domain(domain_name(cluster_name, node_name))`
   - Get domain XML via `libvirt_client.get_domain_xml(dom)`
   - Parse XML to find `<interface type='bridge'>` element where `<source bridge='{bridge}'/>`
   - Extract `<target dev='{tap_name}'/>` from that interface element
   - Return tap device name (e.g., `vnet3`)
   - Raise RuntimeError if domain or interface not found

   **`def build_impairment_cmd(tap_dev: str, latency_ms: int, jitter_ms: int, loss_pct: float, corrupt_pct: float, reorder_pct: float) -> list[str]`:**
   - Start with `["tc", "qdisc", "add", "dev", tap_dev, "root", "netem"]`
   - Append `"delay", f"{latency_ms}ms"` if latency_ms > 0; append `f"{jitter_ms}ms"` if jitter_ms > 0
   - Append `"loss", f"{loss_pct}%"` if loss_pct > 0
   - Append `"corrupt", f"{corrupt_pct}%"` if corrupt_pct > 0
   - Append `"reorder", f"{reorder_pct}%"` if reorder_pct > 0
   - Return the command list (D-09: zero-valued params omitted)

   **`async def apply_impairment(cmd: list[str]) -> None`:**
   - Call `_run_host_cmd(cmd)`. Raise RuntimeError on non-zero exit.

   **`async def remove_impairment(tap_dev: str) -> bool`:**
   - Call `_run_host_cmd(["tc", "qdisc", "del", "dev", tap_dev, "root"])`.
   - Return True if exit_code == 0, False if rule already absent (exit_code == 2).

2. **Add `net_impair` tool to `vmcluster_mcp/net/__init__.py`:**
   - Import `discover_tap_interface`, `build_impairment_cmd`, `apply_impairment` from `fault`
   - Import `LibvirtClient`, `domain_name` from `cluster.libvirt_client`
   - Create module-level `_libvirt_client = LibvirtClient(config.libvirt_uri)` (or reuse from cluster)
   - `@mcp.tool()` + `@structured_tool_handler`
   - `async def net_impair(cluster_name: str, source_node: str, target_node: str = "all", latency_ms: int = 0, jitter_ms: int = 0, loss_pct: float = 0.0, corrupt_pct: float = 0.0, reorder_pct: float = 0.0) -> ToolResult[NetFaultHandle]:`
   - Validate at least one parameter is non-zero; return error otherwise
   - Load topology, resolve source_node IP
   - Discover tap interface for source_node
   - Build and apply tc command
   - Generate handle_id, register in FaultRegistry with rule_identifiers containing the tap device name
   - Return `ToolResult(success=True, result=handle)`

3. **Create `tests/test_net_impair.py`** — mock `_run_host_cmd`, `discover_tap_interface`. Test all behaviors above.
  </action>
  <verify>
    <automated>cd /home/hornc/repos/vmcluster-mcp && python -m pytest tests/test_net_impair.py -x -v && ruff check vmcluster_mcp/net/fault.py vmcluster_mcp/net/__init__.py</automated>
  </verify>
  <done>tc netem command builder with zero-param omission, tap interface discovery from domain XML, net_impair tool registered with FaultRegistry persistence, all tests pass.</done>
</task>

<task type="auto" tdd="true">
  <name>Task 2: net_heal and net_heal_all tools</name>
  <files>vmcluster_mcp/net/__init__.py, tests/test_net_heal.py</files>
  <read_first>
    - vmcluster_mcp/net/__init__.py (current state — net_partition + net_impair registered, _registry instance)
    - vmcluster_mcp/net/fault.py (remove_rules_by_comment for partition cleanup, remove_impairment for tc cleanup)
    - vmcluster_mcp/net/fault_registry.py (FaultRegistry.get, .remove, .list_faults)
    - .planning/phases/07-network-node-fault-injection/07-CONTEXT.md (D-10, D-11)
  </read_first>
  <behavior>
    - Test: net_heal with partition handle calls remove_rules_by_comment with the comment tag, then removes registry entry
    - Test: net_heal with impairment handle calls remove_impairment with the tap device, then removes registry entry
    - Test: net_heal with unknown handle_id returns ToolResult(success=False, error.code="HANDLE_NOT_FOUND")
    - Test: net_heal succeeds even if live rules already absent (D-10: cleanup confirmed absent is success)
    - Test: net_heal removes registry entry only after cleanup succeeds or live state is confirmed absent (D-10)
    - Test: net_heal_all removes all active faults for a cluster, returns count of handles removed
    - Test: net_heal_all is best-effort — failure on one handle does not abort others (D-11)
    - Test: net_heal_all on cluster with no active faults returns count=0
  </behavior>
  <action>
1. **Add `net_heal` tool to `vmcluster_mcp/net/__init__.py`:**
   - `@mcp.tool()` + `@structured_tool_handler`
   - `async def net_heal(cluster_name: str, fault_handle: str) -> ToolResult[bool]:`
   - Look up handle in registry: `entry = _registry.get(fault_handle)`
   - If not found, return `ToolResult(success=False, error=ErrorDetail(code="HANDLE_NOT_FOUND", message=f"No active fault with handle {fault_handle}", recoverable=True))`
   - If `entry["fault_type"] == "partition"`:
     - Call `remove_rules_by_comment(entry["rule_identifiers"][0])` (the comment tag)
   - If `entry["fault_type"] == "impairment"`:
     - Call `remove_impairment(entry["rule_identifiers"][0])` (the tap device)
   - Remove from registry: `_registry.remove(fault_handle)` (D-10: only after cleanup)
   - Return `ToolResult(success=True, result=True)`

2. **Add `net_heal_all` tool to `vmcluster_mcp/net/__init__.py`:**
   - `@mcp.tool()` + `@structured_tool_handler`
   - `async def net_heal_all(cluster_name: str) -> ToolResult[int]:`
   - Get all faults for cluster: `faults = _registry.list_faults(cluster_name)`
   - For each fault, attempt heal (same logic as net_heal). Catch exceptions per-handle and continue (D-11 best-effort).
   - Return `ToolResult(success=True, result=count_removed)`

3. **Create `tests/test_net_heal.py`** — mock `remove_rules_by_comment`, `remove_impairment`, FaultRegistry. Test all behaviors above including best-effort semantics.
  </action>
  <verify>
    <automated>cd /home/hornc/repos/vmcluster-mcp && python -m pytest tests/test_net_heal.py tests/test_net_impair.py -x -v && ruff check vmcluster_mcp/net/__init__.py</automated>
  </verify>
  <done>net_heal removes host-side state by handle type then removes registry entry; net_heal_all is best-effort cluster-wide cleanup; all tests pass.</done>
</task>

</tasks>

<verification>
- `python -m pytest tests/test_net_impair.py tests/test_net_heal.py -x -v` — all tests pass
- `ruff check vmcluster_mcp/net/` — no lint errors
- tc netem commands correctly omit zero-valued parameters
- net_heal correctly dispatches to iptables or tc cleanup based on fault_type
- net_heal_all is best-effort across all handles
</verification>

<success_criteria>
- net_impair applies tc netem on the correct tap interface with combined parameters
- net_heal removes partition OR impairment host state based on registry fault_type
- net_heal_all cleans up all cluster faults best-effort and returns removed count
- Registry entries are removed only after host-side cleanup succeeds or is confirmed absent
- All tests pass, ruff clean
</success_criteria>

<output>
After completion, create `.planning/phases/07-network-node-fault-injection/07-03-SUMMARY.md`
</output>
