<!-- STATUS: COMPLETE -->

# Phase 15 Testing: Pipeline Output Quality

## At a Glance

**Verify that pipeline output now produces usable missions with working chain links.**

### What To Verify
- [x] Validated data (cleaned prerequisites, clamped lines) reaches the database
- [x] Chain-mission links work even when LLM uses Film Noir titles
- [x] Truncated responses preserve complete chain objects
- [x] Prompt instructions are visible to the LLM
- [x] CLI shows clear mission status (available/locked/completed)

---

## Review Fixes Applied During Review

### 1. CLI Message Clarity

The original CLI message "0/46 missions available" was confusing — it showed `completed/total` but the word "available" suggested "ready to start". Fixed to show:
```
Investigation ready:
  19 available to start
  27 locked (complete prerequisites to unlock)
```

### 2. Line Clamping Noise Reduction

Line clamping warnings (`Clamped line_end in X to Y`) are now:
- Logged at DEBUG level (not visible in normal output)
- Only real issues (missing files, broken chains) surface at WARNING level

### 3. Prompt Improvement

Added "Conservative line numbers" instruction to reduce LLM hallucinations about line ranges.

### 4. Valid File Paths List

Added explicit "VALID FILE PATHS" section to mission context:
- Lists all files the LLM is allowed to reference
- Prompt instructs LLM to use ONLY paths from this list
- Should eliminate "File not found" warnings caused by path hallucinations

---

## Automated Tests

**How to run:** `pytest tests/ -q`

All 903 tests pass. Key test classes for this phase:
- `tests/test_mission_pipeline.py::TestRunMissionPipelineValidatedDataWired` — Bug 1 fix
- `tests/test_mission_validation_prereqs.py::TestChainTitleNormalization` — Bug 2 validation layer
- `tests/test_campaign_persister.py::TestPersistCategoryChainsFilmNoirFallback` — Bug 2 persistence layer
- `tests/test_llm_structured.py::TestTruncatedChain` — Bug 3 truncation recovery

---

## Manual Testing Steps

### Quick Smoke Test (5 min)

Run a single-category pipeline with GLM:

1. Activate the project's virtual environment
2. Run the testing script with a single category:
   ```bash
   cd /project
   python testing_script.py --category architecture
   ```
3. Check the output for:
   - Missions generated successfully
   - No "0 available" in the mission count
   - Chains created with linked missions

- **Expected:** Pipeline completes with missions marked as AVAILABLE, chains show linked mission IDs
- **If fails:** Check for validation warnings about missing missions or chain drops

### Feature 1 — Validated Data Flow (10 min)

1. Run pipeline on a repo with known validation issues (e.g., out-of-range line numbers)
2. Query the database to verify line numbers were clamped:
   ```sql
   SELECT file_path, line_start, line_end FROM mission_file_refs LIMIT 5;
   ```
3. Check that no line numbers exceed actual file lengths

- **Expected:** Line numbers are within file bounds, no negative values
- **If fails:** Validation layer may not be running or its results not wired through

### Feature 2 — Chain Link Verification (10 min)

1. Run a full pipeline (all categories)
2. Query the database for chain-mission links:
   ```sql
   SELECT c.name_plain, cm.mission_id, cm.position
   FROM mission_chains c
   JOIN chain_mission_links cm ON c.id = cm.chain_id
   ORDER BY c.name_plain, cm.position;
   ```
3. Verify chains have linked missions (not empty)

- **Expected:** Each chain has 2+ linked missions with valid mission_ids
- **If fails:** Check for "Film Noir title fallback" warnings in logs — if present, the fallback is working

### Feature 3 — Truncation Recovery (15 min)

This requires triggering a truncated response. Best tested via automated tests:

```bash
pytest tests/test_llm_structured.py::TestTruncatedChain -v
```

- **Expected:** All 4 tests pass, showing chains are preserved when complete

---

## Edge Cases to Check

### Priority 1 (Test First)
- [x] Chain with all Film Noir titles (no `title_plain` matches) — should still link via fallback
- [x] Truncated response with 3 complete missions and 2 complete chains — all 5 objects preserved
- [x] Truncated response mid-chain — only complete chains preserved, missions still recovered

### Priority 2
- [x] Empty chains section (`"chains": []`) — should remain empty, not crash
- [x] Mission with both Film Noir `title` and another mission's `title_plain` as same string — `title_plain` lookup wins
- [x] Chain referencing a mission that was dropped during validation — chain should be trimmed or dropped

---

## Phase Completion Checklist

### Must Pass
- [x] All 903 automated tests pass
- [x] Pipeline run produces missions with AVAILABLE status
- [x] Chain-mission links exist in database (not just chain records)
- [x] No regressions in existing features

### Nice to Have
- [x] Run full 7-category pipeline without 429 errors (GLM rate limit dependent)
- [x] Verify prompt instructions appear in LLM context

### Known Limitations
- GLM 429 rate limiting is external — not fixed by this phase
- Truncation recovery only preserves *complete* chain objects — partial chains are dropped

---

## User Testing Notes

### Issues Found

1. **"Fixed line_start/end 0 → 1" appeared at WARNING level** — These 0-indexed corrections were in `real_warnings` instead of the debug-level filter. Same category as "Clamped" warnings. Fixed in pipeline.py: filter now covers `startswith("Fixed line_")` in addition to `startswith("Clamped")`.

### Error Messages / Logs

**Run 1 output (all validation messages confirmed hallucinations):**
```
Have 15 module descriptions, need 30. Padding with defaults.
Validation issues for dependencies: ['Fixed line_start in backend/requirements.txt: 0 → 1 (lines are 1-indexed)', 'Fixed line_end in backend/requirements.txt: 0 → 1 (lines are 1-indexed)', 'Fixed line_start in frontend/package.json: 0 → 1 (lines are 1-indexed)']
Validation issues for conventions: ["Broke cycle by removing prereq 'Route Naming and Structure Conventions' from 'Route Naming and Structure Conventions'"]
Validation issues for testing: ['Fixed line_start in backend/tests/test_level_calculator.py: 0 → 1 (lines are 1-indexed)', 'Fixed line_end in backend/tests/test_level_calculator.py: 0 → 1 (lines are 1-indexed)', 'Fixed line_start in backend/tests/conftest.py: 0 → 1 (lines are 1-indexed)']
```

**Run 2 output (different category warnings, same hallucination types):**
```
Validation issues for architecture: [
  "Removed nonexistent prerequisite 'Mapping the Districts' from 'Request Flow Through Layers'",
  "Removed nonexistent prerequisite 'The Foundation Stone' from 'Test Infrastructure Architecture'",
  "Removed nonexistent prerequisite 'The Gateway to the City' from 'Blueprint Wiring and Registration'"
]
Validation issues for technical_debt: [
  "Chain 'Code Complexity Analysis' references missing missions: {'Definition Density Investigation'}"
]
```

**Final run output (43 missions, 22 available, 21 locked — pipeline working correctly):**
```
Campaign created: 43 missions across 7 categories
Investigation ready:
  22 available to start
  21 locked (complete prerequisites to unlock)
```

### General Feedback

All validation warnings are expected LLM hallucinations caught and auto-fixed by the validation layer. The pipeline produced a usable campaign. Mission availability breakdown is clear. Chain linking and validated data flow confirmed working.

### Result

- [x] All good — ready to proceed
- [ ] Issues found — see above
