# Investigation: DISTILL-to-DELIVER Handoff Gap

**Date**: 2026-03-07
**Investigator**: Rex (nw-troubleshooter)
**Methodology**: Toyota 5 Whys with multi-causal branching

## Problem Statement

Acceptance tests created during `/nw:distill` (with `@skip`, `it.skip`, `[Ignore]`, `xit`, `.skip` markers) are ignored during `/nw:deliver`. The crafter writes NEW acceptance tests instead of unskipping and using the pre-existing distilled ones.

**Scope**: DISTILL wave output -> DELIVER wave consumption -> EXECUTE prompt -> crafter behavior.
**Boundary**: Workflow orchestration files only. Not application code, not DES runtime.

---

## Evidence Collection

### Track 1: What DISTILL Produces

**E1.1** `nWave/tasks/nw/distill.md:75` -- Success criteria includes:
> "One-at-a-time implementation strategy established (@skip/@pending tags)"

**E1.2** `nWave/tasks/nw/distill.md:87` -- Example describes:
> "Quinn creates Given-When-Then acceptance tests from requirements and architecture, establishes walking skeleton first, then milestone features with @skip tags for one-at-a-time implementation."

**E1.3** `nWave/tasks/nw/distill.md:92-109` -- Expected outputs:
```
tests/{test-type-path}/{feature-id}/acceptance/
  walking-skeleton.feature
  milestone-{N}-{description}.feature
  steps/
    conftest.py
    {domain}_steps.py

docs/feature/{feature-id}/distill/
  test-scenarios.md
  walking-skeleton.md
  acceptance-review.md
```

**E1.4** `nWave/agents/nw-acceptance-designer.md:107-108` -- Handoff package:
> "Hands off to DELIVER: acceptance test suite|walking skeleton identification|one-at-a-time implementation sequence|mandate compliance evidence (CM-A/B/C)|peer review approval."

**E1.5** `nWave/agents/nw-acceptance-designer.md:83-84` -- Phase 3:
> "Mark all scenarios except first with skip/ignore"
> "Verify first scenario runs (fails for business logic reason)"

**Finding 1**: DISTILL produces acceptance tests as FILES on disk with skip markers. The output is well-defined -- `.feature` files with `@skip`/`@ignore` tags in a known directory structure. The handoff is described in prose but has no machine-readable manifest linking test files to roadmap steps.

### Track 2: How DELIVER/EXECUTE Consumes DISTILL Output

**E2.1** `nWave/tasks/nw/execute.md:67` -- TASK_CONTEXT section lists expected fields:
> "{step context from roadmap - name|description|acceptance_criteria|test_file|scenario_line|acceptance_test_scenario|quality_gates|implementation_notes|dependencies|estimated_hours|deliverables}"

This reference to `test_file`, `scenario_line`, and `acceptance_test_scenario` is CRITICAL -- the template EXPECTS these fields to exist in the roadmap step.

**E2.2** `nWave/templates/step-tdd-cycle-schema.json:8-27` -- The TDD cycle schema defines:
```json
"acceptance_test": {
    "scenario_name": "",
    "test_file": "",
    "test_file_format": "feature",
    "scenario_index": 0,
    "initially_ignored": true,
    ...
    "mapped_scenario": {
        "description": "The specific acceptance test scenario this step makes pass (RED -> GREEN)",
        "scenario_function": "",
        "scenario_description": "",
        "mapping_type": "feature"
    }
}
```

This schema explicitly models `test_file`, `scenario_name`, and `initially_ignored: true`. The infrastructure EXISTS in the schema.

**E2.3** `nWave/templates/step-tdd-cycle-schema.json:318` -- Mandatory phase description:
> "PREPARE - Remove @skip, verify only 1 scenario enabled"

**E2.4** `nWave/agents/nw-software-crafter.md:174` -- Phase 0: PREPARE:
> "Remove @skip from target acceptance test. Verify exactly ONE scenario enabled. Gate: one acceptance test active."

**E2.5** `nWave/agents/nw-software-crafter.md:177-178` -- Phase 1: RED (Acceptance):
> "Run acceptance test -- must fail for valid reason (business logic not implemented|missing endpoint)."

**E2.6** `nWave/tasks/nw/execute.md:71-72` -- TDD_PHASES section:
```
0. PREPARE - Load context, verify prerequisites
1. RED_ACCEPTANCE - Write failing acceptance test
```

Note: the execute template says "Write failing acceptance test" -- it does NOT say "unskip pre-existing acceptance test." This is the crux of the problem.

**E2.7** `nWave/skills/software-crafter/tdd-methodology.md` -- No mention of "distill", "pre-existing", "unskip", or "acceptance-designer" anywhere in the skill. The only related text is about E2E test management (line 133-139) which describes "Enable ONE E2E test at a time" but this is generic methodology, not DISTILL-aware.

**E2.8** `nWave/skills/software-crafter/collaboration-and-handoffs.md:11` -- Receives from section:
> "acceptance_designer (DISTILL): E2E acceptance tests | business validation requirements | production integration patterns"

This acknowledges the handoff EXISTS conceptually but provides no operational instructions on HOW to consume pre-existing tests.

**Finding 2**: The execute.md template EXPECTS `test_file`/`scenario_line`/`acceptance_test_scenario` fields to be present in roadmap step context. The TDD cycle schema models them. The crafter's PREPARE phase says "Remove @skip from target acceptance test." BUT: the RED_ACCEPTANCE phase says "Write failing acceptance test" -- contradicting the PREPARE instruction. And no skill file explains how to locate or consume pre-existing distilled tests.

### Track 3: Roadmap Schema and Step Population

**E3.1** `nWave/templates/roadmap-schema.json:6-11` -- Required and optional step fields:
```json
"required_fields": {
    "step": ["id", "name", "criteria"]
},
"optional_fields": {
    "step": ["agent", "deps", "scenario", "files_to_modify", "implementation_notes"]
}
```

`test_file`, `scenario_line`, and `acceptance_test_scenario` are NOT in the roadmap schema -- neither required nor optional. The `scenario` field exists as optional but is undocumented.

**E3.2** `src/des/cli/roadmap.py:52-58` -- Roadmap skeleton contains:
```python
steps.append({
    "id": step_id,
    "name": "TODO: step name",
    "criteria": "TODO: acceptance criteria",
})
```

No `test_file`, `scenario_line`, or `acceptance_test_scenario` fields are scaffolded.

**E3.3** `nWave/skills/solution-architect/roadmap-design.md:81-86` -- Step-to-Scenario Mapping:
> "For roadmaps feeding DELIVER wave:"
> "1. Read acceptance tests (tests/acceptance/test_*.py) before creating roadmap"
> "2. Count scenarios (def test_*) | 3. ~1 step per scenario"
> "4. Mark infra steps: type: infrastructure | 5. Principle: 1 Scenario = 1 Step = 1 TDD Cycle"

The architect is instructed to READ acceptance tests and map 1:1 to steps, but the roadmap schema has no field to record which test file/scenario maps to which step.

**E3.4** `nWave/tasks/nw/deliver.md:85-86` -- Roadmap creation:
> "@nw-solution-architect creates roadmap.json"

The solution architect creates the roadmap but has no mechanism to populate `test_file`/`scenario_line` because those fields don't exist in the schema.

**Finding 3**: The roadmap schema is the bottleneck. The execute.md template REFERENCES `test_file|scenario_line|acceptance_test_scenario` fields, but the roadmap schema does NOT define them, the CLI does NOT scaffold them, and the architect skill instructs mapping but has no fields to record the mapping. The `scenario` optional field exists but is undocumented and unused for this purpose.

### Track 4: The Full Causal Chain

Combining Findings 1-3, the full causal chain is:

```
DISTILL produces .feature files with @skip markers
  -> Architect creates roadmap, reads tests, maps 1:1 to steps
    -> BUT roadmap schema has no test_file/scenario_line fields
      -> Roadmap steps contain no reference to pre-existing tests
        -> Execute template says "Write failing acceptance test"
          -> Crafter has no information about pre-existing tests
            -> Crafter writes NEW acceptance tests
              -> Distilled tests remain skipped, ignored
```

---

## Toyota 5 Whys Analysis

### Branch A: Crafter writes new tests instead of unskipping existing ones

**WHY 1A**: Crafter writes new acceptance tests during RED_ACCEPTANCE phase.
[Evidence: `execute.md:72` says "Write failing acceptance test" -- directive is CREATE not UNSKIP]

**WHY 2A**: The execute.md template instructs the crafter to "write" rather than "unskip/find" a pre-existing acceptance test.
[Evidence: `execute.md:71-72` -- "1. RED_ACCEPTANCE - Write failing acceptance test". Contrast with crafter agent at line 174: "Remove @skip from target acceptance test" -- the contradiction exists within nWave's own files]

**WHY 3A**: The execute.md template was written as a generic TDD instruction, not as a DISTILL-aware instruction. It does not distinguish between "fresh" (no prior DISTILL) and "distill-preceded" workflows.
[Evidence: `execute.md` has zero references to "distill", "pre-existing", "unskip", or "feature file". The word "skip" only appears in the context of DES phase skipping, not test marker removal]

**WHY 4A**: The nWave workflow defines DISTILL -> DELIVER as a sequential handoff but never specified the exact consumption protocol for the crafter.
[Evidence: `collaboration-and-handoffs.md:11` acknowledges the handoff: "acceptance_designer (DISTILL): E2E acceptance tests" but provides zero operational detail on how to use them. No skill file bridges DISTILL output to crafter behavior.]

**WHY 5A**: The DISTILL -> DELIVER handoff protocol was designed at the wave level (acceptance-designer hands off to deliver) but never translated into step-level execution instructions. The execute.md template operates at step level and has no awareness of wave-level context.
-> **ROOT CAUSE A**: Missing handoff protocol between DISTILL wave output and EXECUTE step-level instructions. The execute.md RED_ACCEPTANCE phase lacks instructions to search for and unskip pre-existing distilled acceptance tests before attempting to write new ones.

### Branch B: Roadmap steps don't reference distilled test files

**WHY 1B**: Roadmap steps contain no `test_file` or `scenario_line` references to distilled acceptance tests.
[Evidence: `roadmap-schema.json:6-7` -- required step fields are only `id`, `name`, `criteria`]

**WHY 2B**: The roadmap schema defines no field for test file/scenario references.
[Evidence: `roadmap-schema.json:8-11` -- optional fields include `scenario` but no `test_file` or `scenario_line`]

**WHY 3B**: The roadmap CLI scaffolds steps with only `id`, `name`, `criteria` and the architect fills content without test mapping fields.
[Evidence: `src/des/cli/roadmap.py:52-58` -- skeleton has three fields only. `roadmap-design.md:84-86` instructs architect to "Read acceptance tests" and map 1:1 but provides no field to record the mapping]

**WHY 4B**: The roadmap schema was designed for generic step planning, not specifically for the DISTILL -> DELIVER pipeline. The step-tdd-cycle-schema.json (which defines `test_file`, `scenario_name`, `mapped_scenario`) operates at a different level than the roadmap schema.
[Evidence: `step-tdd-cycle-schema.json:8-27` has the full acceptance_test structure with test_file|scenario_name|mapped_scenario, but this schema is not integrated into roadmap.json]

**WHY 5B**: Two schemas exist that should work together but are disconnected: roadmap-schema.json (planning) and step-tdd-cycle-schema.json (execution). The TDD schema models the test-to-step mapping precisely, but the roadmap schema (which feeds the execute template) has no place to carry this information.
-> **ROOT CAUSE B**: Schema disconnection between roadmap-schema.json (planning, no test reference fields) and step-tdd-cycle-schema.json (execution, has full test reference model). The roadmap is the vehicle that carries step context to the crafter, but it cannot carry test file references because its schema doesn't support them.

### Branch C: Architect skill instructs mapping but mapping has nowhere to go

**WHY 1C**: The architect's roadmap-design skill instructs "Read acceptance tests before creating roadmap" and "1 Scenario = 1 Step = 1 TDD Cycle" but the mapping is not recorded.
[Evidence: `roadmap-design.md:83-86`]

**WHY 2C**: The optional `scenario` field in the roadmap schema exists but is undocumented and unused for test file referencing.
[Evidence: `roadmap-schema.json:10` lists `scenario` as optional step field with no further definition]

**WHY 3C**: The architect has knowledge (read acceptance tests) but no place to store it (no schema field), so the knowledge is lost between roadmap creation and step execution.
[Evidence: Architect skill has the right instructions; roadmap schema lacks the right fields. Gap is structural.]

**WHY 4C**: The `scenario` field was likely intended for this purpose but was never formally connected to the execute template or the crafter's behavior.
[Evidence: Circumstantial -- `scenario` is an optional field name suggestive of test scenario mapping, but no documentation explains its purpose or expected format]

**WHY 5C**: The roadmap infrastructure was built incrementally -- skeleton CLI first, then validation, then architect skill -- without an end-to-end integration test verifying that DISTILL output reaches the crafter through the roadmap pipeline.
-> **ROOT CAUSE C**: The `scenario` optional field in roadmap-schema.json was never formally documented, connected to the execute template, or used by the architect to store test file/scenario references. It represents an incomplete implementation of the mapping concept.

---

## Cross-Validation

### Forward trace: Root Cause A -> Symptom
If execute.md says "Write failing acceptance test" without mentioning pre-existing distilled tests, does the crafter write new tests? **YES** -- confirmed by user report.

### Forward trace: Root Cause B -> Symptom
If roadmap steps have no `test_file`/`scenario_line` fields, can the crafter know which distilled test to unskip? **NO** -- even if execute.md said "find pre-existing test," there is no pointer in the step context.

### Forward trace: Root Cause C -> Symptom
If the architect reads distilled tests but has no schema field to record the mapping, is the knowledge lost? **YES** -- the mapping exists in the architect's reasoning during roadmap creation but is not persisted.

### Root Cause A + B + C consistency
All three are consistent and non-contradictory. They describe different layers of the same gap:
- **C** = schema-level missing field (data layer)
- **B** = schema disconnection (architecture layer)
- **A** = missing instruction in execute template (behavior layer)

### Completeness check
Do all three root causes, addressed together, explain the full symptom?
- C provides the data vehicle (field in roadmap to carry test references)
- B provides the data flow (roadmap -> execute template -> crafter)
- A provides the behavioral instruction (crafter knows to look for and use pre-existing tests)
All three are necessary; no single one is sufficient.

---

## Hypothesis Evaluation

### H1: "Just add unskip instructions to execute.md RED_ACCEPTANCE" (user's initial suggestion)
**PARTIALLY CONFIRMED**. This addresses Root Cause A but NOT B or C. Without `test_file`/`scenario_line` in roadmap steps, telling the crafter to "find the test" is vague -- what file? what scenario? The crafter would need to search the codebase, which is unreliable.
**Verdict**: Necessary but insufficient. A behavioral instruction without data linkage produces inconsistent results.

### H2: "The issue is in the DELIVER wave roadmap creation, not linking steps to distilled test files"
**CONFIRMED**. This is Root Cause B/C. The roadmap is the transport mechanism but has no fields for test references.

### H3: "Maybe the crafter DOES know about distilled tests but the roadmap doesn't link them"
**PARTIALLY CONFIRMED**. The crafter agent (`nw-software-crafter.md:174`) says "Remove @skip from target acceptance test" in PREPARE phase. So the crafter has GENERIC knowledge about unskipping. But it lacks SPECIFIC knowledge: which file, which scenario. And the execute.md RED_ACCEPTANCE instruction ("Write failing acceptance test") contradicts the PREPARE instruction.

### H4: "Maybe the distill output format varies by framework and the crafter can't reliably find skip markers"
**REFUTED**. `step-tdd-cycle-schema.json:283-313` defines `test_file_formats` with explicit `ignore_syntax` per framework: `@ignore` (feature), `[Ignore("...")]` (NUnit), `@pytest.mark.skip` (pytest), `test.skip()` (Jest), `@Disabled` (JUnit 5). The schema knows all the markers. The issue is not framework variability -- it is information flow.

### H5: "Maybe this is a DESIGN gap -- the workflow never specified how DISTILL hands off to DELIVER"
**CONFIRMED**. This is the fundamental issue. The wave-level handoff is described in prose ("Handoff To: nw-software-crafter") but never translated into a concrete data protocol. DISTILL produces files; DELIVER consumes a roadmap. There is no bridge linking specific test files to specific roadmap steps.

---

## Scope Assessment

### Files Requiring Changes

| File | Change | Complexity | Root Cause |
|------|--------|------------|------------|
| `nWave/templates/roadmap-schema.json` | Add `test_file` and `scenario_name` to optional step fields | Low | B, C |
| `src/des/cli/roadmap.py` | Scaffold `test_file`/`scenario_name` placeholders in step skeleton | Low | B |
| `nWave/skills/solution-architect/roadmap-design.md` | Add explicit instruction to populate `test_file`/`scenario_name` fields from distilled tests | Medium | C |
| `nWave/tasks/nw/execute.md` | Modify RED_ACCEPTANCE phase: "If test_file provided, unskip and run the pre-existing distilled test. Only write a new acceptance test if no test_file is specified." | Medium | A |
| `nWave/agents/nw-software-crafter.md` | Align Phase 1: RED (Acceptance) with new execute.md language | Low | A |
| `nWave/agents/nw-functional-software-crafter.md` | Same alignment for functional crafter | Low | A |

### Files NOT Requiring Changes (Verified)

| File | Reason |
|------|--------|
| `nWave/templates/step-tdd-cycle-schema.json` | Already has `test_file`, `scenario_name`, `mapped_scenario` -- no changes needed |
| `nWave/agents/nw-acceptance-designer.md` | DISTILL output is correct; the gap is in consumption, not production |
| `nWave/tasks/nw/distill.md` | DISTILL task is correct; produces files with skip markers as designed |
| `nWave/skills/software-crafter/tdd-methodology.md` | Generic methodology is correct; the specific DISTILL-awareness belongs in execute.md |
| `nWave/tasks/nw/deliver.md` | Orchestrator passes roadmap context to execute; if roadmap has the fields, they flow through |

### Tests

No existing tests validate the DISTILL -> DELIVER handoff. The roadmap validator (`src/des/domain/roadmap_validator.py`) validates structure but not semantic completeness of step-to-test mapping.

### Risk: Breaking workflows where DISTILL is skipped

When DISTILL is skipped (common for quick fixes, spikes, or workflows starting at DELIVER), `test_file` will be empty. The execute.md instruction must handle this gracefully: "If test_file is present, unskip. If absent, write new." This is a conditional, not a breaking change.

---

## Solution Design

### Immediate Mitigation (restore correct behavior in current sessions)

**M1**: Modify `execute.md` RED_ACCEPTANCE instruction. Change from:
```
1. RED_ACCEPTANCE - Write failing acceptance test
```
To:
```
1. RED_ACCEPTANCE - Activate acceptance test
   If test_file is provided in TASK_CONTEXT: locate the file, remove @skip/@ignore marker from the target scenario, run it -- must fail for business logic reason.
   If no test_file provided: write a new failing acceptance test.
```
This produces correct behavior even without schema changes, because the crafter's PREPARE phase already says "Remove @skip from target acceptance test."

### Permanent Fix (prevent recurrence via data linkage)

**P1**: Add `test_file` and `scenario_name` to `roadmap-schema.json` optional step fields.

**P2**: Update roadmap CLI skeleton to include `test_file` and `scenario_name` as empty-string placeholders (visible to the architect as fillable fields).

**P3**: Update `roadmap-design.md` Step-to-Scenario Mapping section to explicitly instruct: "For each step mapped to a distilled scenario, populate `test_file` with the path and `scenario_name` with the scenario title from the .feature file."

**P4**: Update `execute.md` TASK_CONTEXT section to document the new fields and their use.

### Early Detection (catch this class of gap faster)

**D1**: Add a deliver-wave pre-flight check: if `docs/feature/{feature-id}/distill/` exists (DISTILL was run), verify that roadmap steps reference test files. Warn if distill artifacts exist but no roadmap step has `test_file` populated.

---

## Confidence Assessment

| Finding | Confidence | Reasoning |
|---------|------------|-----------|
| Root Cause A (execute.md instruction) | **HIGH** | Direct textual evidence: "Write" vs "Remove @skip" contradiction |
| Root Cause B (schema disconnection) | **HIGH** | Schema fields verified: roadmap has no test_file, TDD schema does |
| Root Cause C (undocumented scenario field) | **MEDIUM** | Circumstantial: field exists, purpose inferred, no documentation found |
| Mitigation M1 sufficiency | **MEDIUM** | Addresses crafter behavior but relies on crafter searching for files without precise pointers |
| Permanent fix P1-P4 sufficiency | **HIGH** | Data linkage + instruction = both information and behavior aligned |

---

## Summary

This is a **three-layer workflow integration gap**, not a one-line fix:

1. **Data layer** (Root Cause B/C): Roadmap schema lacks fields to carry test file references from DISTILL to DELIVER
2. **Instruction layer** (Root Cause A): execute.md says "Write" when it should say "Activate (unskip if exists, write if not)"
3. **Knowledge layer** (Root Cause C): Architect skill instructs 1:1 mapping but has nowhere to record it

The minimum viable fix is updating `execute.md` alone (Root Cause A), which will improve behavior for cases where the crafter can discover distilled tests by searching the codebase. The complete fix requires schema + CLI + skill + template changes (6 files, estimated low-to-medium complexity per file).

Note: the `step-tdd-cycle-schema.json` already models the full acceptance test reference structure (`test_file`, `scenario_name`, `mapped_scenario`). The infrastructure was designed; it was never connected to the roadmap pipeline.
