# Project Retrospective

*A living document updated after each milestone. Lessons feed forward into future planning.*

---

## Milestone: v1.0 — MVP

**Shipped:** 2026-03-24
**Phases:** 10 | **Plans:** 30 | **Timeline:** 3 days (2026-03-21 → 2026-03-24)

### What Was Built

- Typed MCP server skeleton with `ToolResult[T]` error contract and Pydantic v2 models — safe foundation for all tool additions
- Full cluster lifecycle: topology-driven libvirt/QEMU define/start/stop/destroy with idempotency and SSH wait
- Parallel SSH execution via asyncssh with transparent reconnect after VM reboot
- Atomic cluster-wide snapshot create/revert/delete (188 tests)
- Content-addressed artifact pipeline: register/sync/install/diff with sha256 dedup (228 tests)
- dmesg marker + oops/panic/KASAN/UBSAN classifier for kernel observability
- Persistent fault injection: iptables partitions + tc netem impairments + node_crash, fault registry survives MCP restart (361 tests)

### What Worked

- **Typed error contract first:** Establishing `ToolResult[T]` and `@structured_tool_handler` in Phase 1 before any domain code eliminated an entire class of error-handling retrofitting. Every subsequent phase got error handling for free.
- **Single-executor libvirt pattern:** Establishing `ThreadPoolExecutor(max_workers=1)` in `libvirt_client.py` from Phase 2 day-one prevented threading bugs that would have been nearly impossible to diagnose later.
- **asyncssh over paramiko:** The pure-asyncio choice paid off immediately in Phase 3 — parallel `node_exec_all` via `asyncio.gather` with zero threading complexity.
- **Atomic writes everywhere:** The tmp-rename pattern for fault registry and artifact registry caught a class of corruption bugs at design time.
- **Gap-closure phase pattern:** Adding Phases 8–10 as dedicated gap-closure phases (retroactive verification, metadata repair, Nyquist coverage) was more effective than trying to fix gaps mid-milestone. The work was scoped and bounded.

### What Was Inefficient

- **Missing VERIFICATION.md for Phases 2 & 3:** The original milestone design didn't enforce verification artifacts per-phase, requiring retroactive Phases 8 to create them. A per-phase verification gate from the start would have avoided this.
- **Summary frontmatter gaps:** Many early phase summaries were missing `requirements_completed` YAML frontmatter, requiring Phase 9 metadata repair. The summary template should enforce this field.
- **Nyquist after the fact:** Creating VALIDATION.md for 7 phases retroactively in Phase 10 was significant rework. If VALIDATION.md had been part of the definition-of-done for each phase, it would have added ~1 plan per phase at the time of execution.
- **Stale audit:** The milestone audit ran before Phases 8–10 were added, so the `gaps_found` status was stale by completion time. Re-running audit after gap closure would have produced a cleaner close.

### Patterns Established

- **`ToolResult[T]` + `@structured_tool_handler`**: Apply to every tool from first commit — retrofitting is not safe.
- **Single `ThreadPoolExecutor(max_workers=1)` for thread-unsafe C libraries**: Must be in the client module from phase 1.
- **Destroy-all-before-revert discipline**: Any operation with no atomic multi-entity primitive must be all-or-nothing with explicit failure handling.
- **Gap-closure phases**: When an audit reveals gaps, add explicit phases to close them rather than trying to fix ad-hoc. Keeps scope clean.
- **Atomic writes for any registry**: All persistent state that can be partially written during a crash must use write-tmp + os.replace.

### Key Lessons

1. **Define-of-done must include verification artifacts.** Phase VERIFICATION.md and VALIDATION.md should be required deliverables, not retroactive additions.
2. **The error contract is load-bearing infrastructure.** `ToolResult[T]` and the catch-all decorator are more important than any individual tool — they must come first and be applied universally.
3. **Circular imports surface at module load time.** Lazy inline imports are the correct fix for cross-module circular dependencies in Python — not restructuring unless the cycle is a design error.
4. **Type narrowing matters for nested structures.** `dict[str, Any]` is the correct type for heterogeneous parameter maps; avoid premature concrete types for fields that store runtime-shaped data.
5. **A 3-day v1.0 is achievable with GSD when the architecture is right.** The stateless server + typed contract pattern compressed the entire build to sequential, independently-verifiable phases with minimal integration surprises.

### Cost Observations

- Model: Sonnet 4.6 throughout
- Sessions: ~1 multi-day session
- Notable: All 33 requirements mapped and shipped in 3 calendar days; gap-closure phases added 3 days of work but closed all audit findings before milestone archive

---

## Cross-Milestone Trends

### Process Evolution

| Milestone | Phases | Plans | Key Change |
|-----------|--------|-------|------------|
| v1.0 | 10 | 30 | First milestone — baseline established |

### Cumulative Quality

| Milestone | Tests | LOC | Notes |
|-----------|-------|-----|-------|
| v1.0 | ~800+ | ~11,972 py | Foundation + full feature set |

### Top Lessons (Verified Across Milestones)

1. Error contracts must be established before domain code — retrofitting is unsafe and creates testing gaps.
2. Retroactive verification work (audit gap closure) is expensive — enforce verification artifacts per-phase from the start.
