---
phase: 7
slug: network-node-fault-injection
status: complete
nyquist_compliant: true
wave_0_complete: true
created: 2026-03-24
---

# Phase 7 — Validation Strategy

> Per-phase validation contract for feedback sampling during execution.

---

## Test Infrastructure

| Property | Value |
|----------|-------|
| **Framework** | pytest 8.x (with pytest-asyncio) |
| **Config file** | pyproject.toml |
| **Quick run command** | `.venv/bin/pytest tests/test_fault_registry.py tests/test_net_partition.py tests/test_net_impair.py tests/test_net_heal.py tests/test_net_fault_list.py tests/test_node_crash.py -x -q` |
| **Full suite command** | `.venv/bin/pytest tests/ -q` |
| **Estimated runtime** | ~12 seconds |

---

## Sampling Rate

- **After every task commit:** Run `.venv/bin/pytest tests/test_fault_registry.py tests/test_net_partition.py tests/test_net_impair.py tests/test_net_heal.py tests/test_net_fault_list.py tests/test_node_crash.py -x -q`
- **After every plan wave:** Run `.venv/bin/pytest tests/ -q`
- **Before `/gsd:verify-work`:** Full suite must be green
- **Max feedback latency:** 20 seconds

---

## Per-Task Verification Map

| Task ID | Plan | Wave | Requirement | Test Type | Automated Command | File Exists | Status |
|---------|------|------|-------------|-----------|-------------------|-------------|--------|
| 07-01-01 | 01 | 1 | FAULT-07 | unit | `.venv/bin/pytest tests/test_fault_registry.py -x -q` | ✅ | ✅ green |
| 07-02-01 | 02 | 1 | FAULT-01 | unit | `.venv/bin/pytest tests/test_net_partition.py -x -q` | ✅ | ✅ green |
| 07-03-01 | 03 | 1 | FAULT-02 | unit | `.venv/bin/pytest tests/test_net_impair.py -x -q` | ✅ | ✅ green |
| 07-03-02 | 03 | 1 | FAULT-03 | unit | `.venv/bin/pytest tests/test_net_heal.py -k "heal" -x -q` | ✅ | ✅ green |
| 07-03-03 | 03 | 1 | FAULT-04 | unit | `.venv/bin/pytest tests/test_net_heal.py -k "heal_all" -x -q` | ✅ | ✅ green |
| 07-04-01 | 04 | 1 | FAULT-05 | unit | `.venv/bin/pytest tests/test_net_fault_list.py -x -q` | ✅ | ✅ green |
| 07-04-02 | 04 | 1 | FAULT-06 | unit | `.venv/bin/pytest tests/test_node_crash.py -x -q` | ✅ | ✅ green |
| 07-05-01 | 05 | 1 | FAULT-07 | unit | `.venv/bin/pytest tests/test_net_fault_list.py::test_net_fault_list_calls_reconcile_on_first_call tests/test_fault_registry.py -k "reconcile" -x -q` | ✅ | ✅ green |

*Status: ⬜ pending · ✅ green · ❌ red · ⚠️ flaky*

---

## Wave 0 Requirements

Existing infrastructure covers all phase requirements.

---

## Manual-Only Verifications

All phase behaviors have automated verification.

---

## Requirement Coverage

| Requirement | Description | Covered By | Test Names |
|-------------|-------------|------------|------------|
| FAULT-01 | Partition cluster into two groups via iptables DROP rules on bridge; rules carry vmcluster:<cluster>:<handle_id> comment tag | tests/test_net_partition.py | `test_build_partition_rules_1_vs_1_produces_two_rules`, `test_build_partition_rules_2_vs_2_produces_eight_rules`, `test_build_partition_rules_symmetric`, `test_build_partition_rules_comment_tag_format`, `test_build_partition_rules_targets_bridge_interface`, `test_build_partition_rules_action_is_drop`, `test_apply_partition_rules_calls_run_host_cmd_for_each_rule`, `test_apply_partition_rules_returns_comment_tags`, `test_apply_partition_rules_raises_on_failure` |
| FAULT-02 | Apply tc netem impairment to egress traffic (latency, jitter, loss, corruption) | tests/test_net_impair.py | `test_build_impairment_cmd_starts_with_tc_netem`, `test_build_impairment_cmd_latency_and_jitter`, `test_build_impairment_cmd_loss`, `test_build_impairment_cmd_corrupt`, `test_build_impairment_cmd_reorder`, `test_build_impairment_cmd_combined`, `test_apply_impairment_calls_run_host_cmd`, `test_net_impair_registers_fault_and_returns_handle`, `test_net_impair_stores_tap_as_rule_identifier` |
| FAULT-03 | Heal a specific network fault by handle ID; healed handle no longer appears in net_fault_list | tests/test_net_heal.py | `test_net_heal_returns_error_for_unknown_handle`, `test_net_heal_partition_calls_remove_rules_by_comment`, `test_net_heal_partition_removes_registry_entry_after_cleanup`, `test_net_heal_partition_succeeds_even_if_rules_already_absent`, `test_net_heal_impairment_calls_remove_impairment`, `test_net_heal_impairment_succeeds_when_tap_already_absent`, `test_net_heal_impairment_removes_registry_entry_after_cleanup` |
| FAULT-04 | Heal all active faults for a cluster in one call; net_fault_list returns empty list after | tests/test_net_heal.py | `test_net_heal_all_returns_zero_for_empty_cluster`, `test_net_heal_all_removes_all_cluster_faults`, `test_net_heal_all_continues_after_per_handle_failure`, `test_net_heal_all_net_fault_list_empty_after_success` |
| FAULT-05 | List all currently active network faults for a cluster | tests/test_net_fault_list.py | `test_net_fault_list_returns_both_fault_types`, `test_net_fault_list_returns_empty_list_when_no_active_faults`, `test_net_fault_list_filters_by_cluster_name`, `test_net_fault_list_includes_parameters_dict`, `test_net_fault_list_returns_netfaultinfo_instances` |
| FAULT-06 | Hard-reset a node (virsh destroy + start) to simulate unclean failure; returns after SSH reachability confirmed | tests/test_node_crash.py | `test_node_crash_unknown_node_returns_node_not_found`, `test_node_crash_calls_destroy_domain`, `test_node_crash_best_effort_if_domain_already_off`, `test_node_crash_invalidates_ssh_cache_before_restart`, `test_node_crash_restart_false_does_not_start_or_wait`, `test_node_crash_restart_true_calls_start_domain_by_name`, `test_node_crash_restart_true_wait_ssh_calls_wait_for_ssh`, `test_node_crash_returns_running_nodestatus_after_restart_and_wait`, `test_node_crash_ssh_timeout_returns_failure` |
| FAULT-07 | Active fault state survives MCP server restart — persisted to disk atomically, reconciled against live kernel state on startup | tests/test_fault_registry.py, tests/test_net_fault_list.py | `test_save_uses_atomic_write`, `test_save_creates_parent_dirs`, `test_register_saves_immediately`, `test_round_trip_persistence`, `test_reconcile_retains_partition_with_matching_tag`, `test_reconcile_prunes_partition_with_missing_tag`, `test_reconcile_retains_impairment_with_existing_qdisc`, `test_reconcile_prunes_impairment_with_missing_qdisc`, `test_reconcile_saves_after_pruning`, `test_net_fault_list_calls_reconcile_on_first_call`, `test_net_fault_list_reflects_reconciled_state`, `test_ensure_reconciled_sets_flag` |

---

## Validation Sign-Off

- [x] All tasks have `<automated>` verify or Wave 0 dependencies
- [x] Sampling continuity: no 3 consecutive tasks without automated verify
- [x] Wave 0 covers all MISSING references
- [x] No watch-mode flags
- [x] Feedback latency < 20s
- [x] `nyquist_compliant: true` set in frontmatter

**Approval:** approved 2026-03-24
