# Phase 7: Network & Node Fault Injection - Context

**Gathered:** 2026-03-23
**Status:** Ready for planning

<domain>
## Phase Boundary

Implement `net_partition`, `net_impair`, `net_heal`, `net_heal_all`,
`net_fault_list`, and `node_crash` so an agent can inject and heal host-side
network faults between topology nodes, hard-reset a node to model an unclean
failure, and recover active fault state after an MCP server restart.

Requirements: FAULT-01, FAULT-02, FAULT-03, FAULT-04, FAULT-05, FAULT-06,
FAULT-07.

This phase does NOT add scheduled chaos scenarios, app-level failure
orchestration, or broader observability beyond listing active faults.

</domain>

<decisions>
## Implementation Decisions

### Host-Side Execution Boundary
- **D-01:** Network fault injection runs on the MCP host, not inside guests.
  `iptables` and `tc` operations target the libvirt bridge and tap interfaces via
  host-side subprocess execution. Guest SSH is not used to install or heal
  network faults.
- **D-02:** `node_crash` reuses the existing libvirt API path rather than shelling
  out to `virsh`. The behavior must remain equivalent to `virsh destroy` +
  optional `virsh start`, but implementation stays inside
  `LibvirtClient` so the single-worker executor guarantee from Phase 2 remains
  intact.

### Persistent Fault Registry
- **D-03:** Active network faults are stored in a disk-backed registry keyed by
  `handle_id`. Registry writes use the established atomic pattern:
  write-to-`.tmp` then `os.replace()`.
- **D-04:** Startup includes a reconciliation pass: persisted registry entries are
  checked against live `iptables` and `tc` state before any tool returns data.
  `net_fault_list` must reflect reconciled reality immediately after server
  restart.
- **D-05:** Each registry entry persists enough metadata to support handle-based
  healing and restart recovery: `cluster_name`, `fault_type`, `affected_nodes`,
  creation time, user-visible parameters, and host-side rule identifiers needed
  for cleanup.

### Partition Rule Design
- **D-06:** `net_partition` installs symmetric, host-side `iptables` DROP rules
  between the two node groups using topology IP addresses as the traffic match.
  Nodes within the same partition remain connected.
- **D-07:** Every partition rule carries the exact comment tag format required by
  the roadmap: `vmcluster:<cluster>:<handle_id>`. Cleanup and reconciliation are
  keyed off this tag, not off positional rule numbers.

### Impairment Targeting
- **D-08:** `net_impair` applies `tc netem` on the source node's egress tap path,
  scoped to traffic destined for either one target node or all other cluster
  nodes when `target_node="all"`. The default implementation target is
  source-specific and destination-scoped, not whole-bridge degradation.
- **D-09:** A single impairment handle may combine latency, jitter, loss,
  corruption, and reordering parameters. Zero-valued parameters are omitted from
  the effective `tc` command rather than creating separate rule families.

### Healing and Listing Semantics
- **D-10:** `net_heal` is handle-specific. It removes all host-side state for the
  selected handle, then removes the registry entry only after cleanup succeeds or
  the live state is confirmed absent.
- **D-11:** `net_heal_all` operates cluster-wide and is best-effort across active
  handles, returning the count of handles removed. `net_fault_list` reads from the
  reconciled registry so healed handles disappear immediately after successful
  cleanup.

### Node Crash Recovery
- **D-12:** `node_crash` models an unclean reset as destroy-first, then optional
  restart. With `restart_after=True` and `wait_for_ssh=True` (both defaults), it
  invalidates any cached SSH connection for the node and reuses the existing
  `_wait_for_ssh` polling helper before returning.
- **D-13:** With `restart_after=False`, `node_crash` returns the node's post-crash
  `NodeStatus` without an SSH wait. The tool remains focused on reset semantics,
  not broader remediation.

### Carried Forward (from prior phases)
- **D-14:** `@structured_tool_handler` wraps every tool handler — no undecorated
  tools.
- **D-15:** All libvirt operations continue to flow through the single
  `ThreadPoolExecutor(max_workers=1)` in `cluster/libvirt_client.py`.
- **D-16:** Distributed operations use `asyncio.gather` and typed Pydantic return
  models; restart-safe disk state is preferred over critical in-memory state.

### the agent's Discretion
- Exact on-disk registry path and whether it becomes a new config field or a
  derived default under the existing VMCluster data directory.
- Exact host command wrapper shape for `iptables`/`tc` execution
  (`asyncio.create_subprocess_exec` helper vs. localized subprocess logic in the
  `net/` package).
- Exact interface-discovery strategy for mapping libvirt domains to tap devices,
  subject to research and testability.
- Exact error contract when reconciliation finds drift between persisted registry
  state and host kernel state.

</decisions>

<canonical_refs>
## Canonical References

**Downstream agents MUST read these before planning or implementing.**

### Design Specification
- `docs/vmcluster-mcp-design.md` §2.1 — Stateless-server and fault-injection
  design goals that govern persistence and host-side control boundaries.
- `docs/vmcluster-mcp-design.md` §2.4 — `net_partition`, `net_impair`,
  `net_heal`, `net_heal_all`, `net_fault_list`, and `node_crash` tool signatures
  plus `NetFaultHandle` contract.
- `docs/vmcluster-mcp-design.md` §2.5 — Intended `net/` module split
  (`fault.py`, `bridge.py`, `fault_registry.py`).
- `docs/vmcluster-mcp-design.md` §2.6 — Existing config surface; confirms fault
  registry persistence is new work for this phase.

### Project Constraints
- `.planning/PROJECT.md` — Stateless-server requirement, Linux/libvirt target
  environment, and infrastructure-only scope.
- `.planning/REQUIREMENTS.md` — FAULT-01 through FAULT-07 acceptance criteria for
  this phase.
- `.planning/STATE.md` — Carry-forward decision that the Phase 7 fault registry
  must persist to disk with atomic writes and startup reconciliation.

### Existing Models And Infrastructure
- `vmcluster_mcp/models.py` — `ToolResult`, `ErrorDetail`, `NodeStatus`,
  `NetFaultHandle`, and `NetFaultInfo` contracts already available.
- `vmcluster_mcp/config.py` — Current config model; reference point for any new
  registry-path configuration.
- `vmcluster_mcp/server.py` — FastMCP instance and universal
  `@structured_tool_handler` pattern.
- `vmcluster_mcp/cluster/libvirt_client.py` — Single-worker libvirt wrapper and
  domain XML access for node reset and interface-discovery work.
- `vmcluster_mcp/cluster/__init__.py` — `_wait_for_ssh` helper and existing tool
  registration conventions.
- `vmcluster_mcp/exec/ssh.py` — SSH connection invalidation and reconnect behavior
  that `node_crash` must cooperate with.
- `vmcluster_mcp/artifact/registry.py` — Atomic JSON persistence pattern to reuse
  for the fault registry.

### Prior Phase Context
- `.planning/phases/04-snapshot-management/04-CONTEXT.md` — Destroy/restart/wait
  semantics and strict cleanup expectations for libvirt-driven resets.
- `.planning/phases/05-artifact-pipeline/05-CONTEXT.md` — Atomic disk registry
  pattern and best-effort multi-target semantics.
- `.planning/phases/06-kernel-observability/06-CONTEXT.md` — Current tool
  registration, fan-out, and structured-result conventions.

</canonical_refs>

<code_context>
## Existing Code Insights

### Reusable Assets
- `NetFaultHandle` / `NetFaultInfo` (`models.py`): phase 7 data contracts already
  exist and should be reused rather than redesigned.
- `LibvirtClient` (`cluster/libvirt_client.py`): already provides
  `lookup_domain`, `destroy_domain`, `start_domain_by_name`, and `get_domain_xml`
  for `node_crash` and host-interface discovery.
- `_wait_for_ssh` (`cluster/__init__.py`): existing readiness loop for
  restart-after-reset flows.
- `SSHConnectionManager.invalidate()` (`exec/ssh.py`): existing cache invalidation
  hook needed after `node_crash`.
- `ArtifactRegistry.save()` (`artifact/registry.py`): concrete atomic tmp-rename
  persistence pattern to mirror in `fault_registry.py`.
- `load_topology()` (`topology/parser.py`): canonical source for cluster node
  names and IP addresses used by `iptables` and `tc` rule generation.

### Established Patterns
- Tool registration lives in package `__init__.py` files via `@mcp.tool()` plus
  `@structured_tool_handler`.
- Distributed operations fan out with `asyncio.gather` and return typed per-node
  or per-handle results.
- Critical state is persisted on disk with atomic writes, then reloaded on demand.
- Libvirt access stays behind the single-worker executor; raw host commands are
  only used where no existing Python abstraction exists.

### Integration Points
- `vmcluster_mcp/net/__init__.py` should register the phase 7 tools and be added
  to the import block in `vmcluster_mcp/server.py`.
- `vmcluster_mcp/net/fault_registry.py` should own persistence and startup
  reconciliation for active handles.
- `vmcluster_mcp/net/fault.py` should own host-side `iptables` and `tc` command
  construction plus handle-based cleanup.
- `node_crash` must coordinate libvirt domain control with SSH cache invalidation
  and the existing topology-derived node metadata.

</code_context>

<specifics>
## Specific Ideas

- No specific user requirements were added during discussion — use standard
  Linux bridge, `iptables`, and `tc netem` approaches consistent with the design
  doc and existing codebase patterns.
- Planning research should explicitly confirm the best strategy for discovering
  the correct tap interface(s) for a libvirt domain in a bridged QEMU topology.

</specifics>

<deferred>
## Deferred Ideas

None — discussion stayed within phase scope.

</deferred>

---

*Phase: 07-network-node-fault-injection*
*Context gathered: 2026-03-23*
*Mode: --auto (all gray areas auto-selected, recommended defaults applied)*