# Phase 7: Network & Node Fault Injection - Discussion Log

> **Audit trail only.** Do not use as input to planning, research, or execution agents.
> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered.

**Date:** 2026-03-23
**Phase:** 07-network-node-fault-injection
**Areas discussed:** Host-side execution boundary, persistent fault registry,
partition rule design, impairment targeting, healing/list behavior, node crash
recovery

---

## Host-Side Execution Boundary

| Option | Description | Selected |
|--------|-------------|----------|
| Reuse existing libvirt API for node resets; use host subprocesses only for `iptables` and `tc` | Keeps libvirt calls inside the single-worker executor and runs host-network mutations at the correct layer | ✓ |
| Shell out to `virsh`, `iptables`, and `tc` for all operations | Matches CLI behavior but duplicates existing libvirt abstractions and weakens reuse | |
| SSH into guests to inject faults | Incorrect layer for bridge/tap manipulation; cannot guarantee cluster-wide isolation | |

**User's choice:** Reuse existing libvirt API for node resets; use host subprocesses only for `iptables` and `tc`
**Notes:** [auto] Selected recommended default based on `cluster/libvirt_client.py`
and the phase goal's host-side bridge/tap requirements.

---

## Persistent Fault Registry

| Option | Description | Selected |
|--------|-------------|----------|
| Persist active faults on disk with atomic writes and startup reconciliation | Survives MCP restarts and matches the roadmap plus prior atomic-registry patterns | ✓ |
| Persist on disk without reconciliation | Keeps handles across restarts but risks stale or drifted fault state | |
| Keep fault state in memory only | Violates FAULT-07 and the project's stateless-server rule | |

**User's choice:** Persist active faults on disk with atomic writes and startup reconciliation
**Notes:** [auto] Selected recommended default from ROADMAP.md, STATE.md, and the
existing atomic write pattern in `artifact/registry.py`.

---

## Partition Rule Design

| Option | Description | Selected |
|--------|-------------|----------|
| Symmetric IP-based `iptables` DROP rules tagged with `vmcluster:<cluster>:<handle_id>` | Matches FAULT-01 exactly and gives stable cleanup keys across restart and heal operations | ✓ |
| Untagged DROP rules matched only by IP pairs | Simpler initially but brittle for cleanup and reconciliation | |
| Whole-bridge blocking without partition-specific node sets | Over-broad; does not preserve intra-partition connectivity | |

**User's choice:** Symmetric IP-based `iptables` DROP rules tagged with `vmcluster:<cluster>:<handle_id>`
**Notes:** [auto] Selected recommended default because the roadmap makes the
comment tag part of the contract.

---

## Impairment Targeting

| Option | Description | Selected |
|--------|-------------|----------|
| Apply `tc netem` on the source tap egress path and scope it to target node IPs | Preserves the design doc's source-to-target impairment semantics without degrading unrelated traffic | ✓ |
| Apply `tc netem` to all egress traffic from the source node | Easier to implement but too broad when only one target should be impaired | |
| Apply `tc netem` at the whole-bridge level | Simplest topology-wise but loses source/target precision entirely | |

**User's choice:** Apply `tc netem` on the source tap egress path and scope it to target node IPs
**Notes:** [auto] Selected recommended default because the design doc defines
`net_impair` as source-node egress impairment toward specific targets.

---

## Healing / Listing Behavior

| Option | Description | Selected |
|--------|-------------|----------|
| Registry-driven handle cleanup with reconciled `net_fault_list` and best-effort `net_heal_all` | Preserves handle semantics, restart visibility, and practical cluster-wide cleanup behavior | ✓ |
| Stateless scan of live kernel rules only | Avoids stored state but makes handles and restart guarantees harder to honor cleanly | |
| Support `net_heal_all` only, without per-handle cleanup or listing | Too coarse for the roadmap's explicit handle-based API | |

**User's choice:** Registry-driven handle cleanup with reconciled `net_fault_list` and best-effort `net_heal_all`
**Notes:** [auto] Selected recommended default because the phase API is
handle-oriented and restart-safe.

---

## Node Crash Recovery

| Option | Description | Selected |
|--------|-------------|----------|
| Destroy, optionally restart, invalidate SSH cache, and wait for SSH by default | Matches the phase goal and reuses the existing reconnect/wait patterns from prior phases | ✓ |
| Destroy and restart without any SSH cache invalidation or readiness wait | Faster, but leaves stale connections and weak post-reset guarantees | |
| Reboot over SSH from inside the guest | Models a cleaner reboot, not the unclean failure this phase is supposed to simulate | |

**User's choice:** Destroy, optionally restart, invalidate SSH cache, and wait for SSH by default
**Notes:** [auto] Selected recommended default from FAULT-06 and existing
`_wait_for_ssh` plus `SSHConnectionManager.invalidate()` patterns.

---

## the agent's Discretion

- Exact registry path and config plumbing.
- Exact tap-interface discovery mechanism for libvirt domains.
- Exact subprocess wrapper structure inside the `net/` package.
- Exact reconciliation error-reporting details.

## Deferred Ideas

None.
