# Phase Shift Requirements: GLM Pipeline Stabilization & Test Reset

**Created:** 2026-03-13
**Trigger:** Pipeline fails unpredictably with GLM; recent changes happened outside formal build process; tests may be out of sync with implementation

---

## Problem Statement

The pipeline works end-to-end with `glm-4.7-Flash` but is unreliable:
1. **Rate limits** — GLM uses sliding window; 45-min gap still hits 429 errors
2. **JSON truncation** — 7+ missions in one category exceeds 4095 max_tokens
3. **No scaling path** — Current architecture can't handle 7 categories × 5-8 missions

Additionally, significant code changes happened outside the formal build process:
- Commits `5e48e7e`, `ff3bc06`, `899fe8b` added/modified code without formal plans
- New module `llm/structured.py` (273+ lines) has no interface contracts
- Tests were written during fixing sessions — likely test "what code does" not "what it should do"
- The 13 fixes documented in `test_pipeline_issue.md` may or may not be complete

---

## Phase Insertion Plan

**Three phases to be inserted:**

| Phase | Name | Purpose |
|-------|------|---------|
| 10 | Pipeline Stabilization | Multi-call mission generation, retry delays, verify 13 fixes |
| 11 | Interface Contract Generation | Fresh context, analyze LLM module, write contracts |
| 12 | Test Reset & Coverage | New session, test against contracts, full audit |

**Renumbering:**
- Old Phase 10 (Campaign/CLI) → Phase 13
- Old Phase 11 (TUI layout) → Phase 14
- Old Phase 12 (TUI screens) → Phase 15
- Old Phase 13 (Theme) → Phase 16
- Old Phase 14 (Tests) → Phase 17 (may be partially redundant after Phase 12)
- Old Phase 15 (Polish) → Phase 18

---

## Phase 10: Pipeline Stabilization

### Scope

**Primary goal:** Make the pipeline reliable enough to scale to 7 categories

**Tasks:**

1. **Verify 13 fixes from test_pipeline_issue.md**
   - Check each fix is actually implemented
   - If missing, add to this phase's scope
   - Document verification in phase summary

2. **Multi-call mission generation**
   - Current: 1 call per category, request 3-8 missions
   - New: 2-3 calls per category, request 2-3 missions each
   - Changes: `missions/generator.py`, `missions/pipeline.py`
   - Preserve full mission quality (no artificial limits on schema)

3. **Truncation resume detection**
   - Current: JSON recovery extracts complete missions from truncated response, discards incomplete
   - New: Detect truncation, recover complete missions, then **resume** generation from the broken mission
   - Approach:
     - Parse partial response to identify last complete mission
     - Construct follow-up prompt with: "Continue from mission X, generating Y more missions"
     - Merge results from both calls
   - Changes: `structured.py` (resume detection), `missions/generator.py` (resume logic)
   - Handles edge case: Provider hits max_tokens mid-mission

4. **Provider-specific retry delays**
   - Current: 1s, 2s, 4s, 8s, 16s (too aggressive for GLM)
   - New: Provider-aware base delay (30-60s for GLM, keep current for Anthropic)
   - Changes: `retry.py`, `providers/factory.py`

5. **Fix tokens_used bug**
   - `missions/context.py`: Counter initialized to 0 but never incremented
   - Security findings section always passes budget check
   - Changes: `missions/context.py`

### Files Affected

```
src/unfamiliar/llm/
├── missions/
│   ├── generator.py    # Multi-call logic + resume
│   ├── pipeline.py     # Orchestration changes
│   └── context.py      # tokens_used fix
├── providers/
│   └── factory.py      # Provider config for retry delays
├── retry.py            # Provider-aware backoff
└── structured.py       # Truncation resume detection
```

### Verification Criteria

- [ ] Pipeline can generate missions for all 7 categories without truncation
- [ ] Truncation resume: incomplete response triggers follow-up call to continue
- [ ] Rate limit recovery succeeds with 60s+ gaps between calls
- [ ] `tokens_used` counter accurately reflects context size
- [ ] All 13 fixes from test_pipeline_issue.md verified present
- [ ] 723+ tests still pass

### Out of Scope

- Anthropic provider testing (too expensive for now)
- Tiered context architecture (Option A from test_pipeline_issue.md)
- Two-pass generation (Option C)
- LLM-optional briefing (Option D)

---

## Phase 11: Interface Contract Generation

### Context

This phase runs in a **new session** with fresh context. The agent should NOT have access to implementation details from Phase 10 — only the current code state.

### Scope

**Primary goal:** Document what the LLM module *should* do, not what it *happens* to do

**Tasks:**

1. **Analyze entire `llm/` module**
   - Read all source files
   - Understand current architecture
   - Identify public interfaces

2. **Generate interface contracts for:**
   - `llm/__init__.py` — Public API exports
   - `llm/exceptions.py` — Exception hierarchy
   - `llm/retry.py` — Retry utilities
   - `llm/context.py` — Briefing context builder
   - `llm/structured.py` — JSON parsing/recovery (NEW, no contracts exist)
   - `llm/providers/__init__.py` — Provider package exports
   - `llm/providers/factory.py` — Provider factory
   - `llm/briefing/__init__.py` — Briefing exports
   - `llm/briefing/schemas.py` — Briefing Pydantic models
   - `llm/briefing/prompts.py` — Briefing prompts
   - `llm/briefing/generator.py` — Briefing orchestrator
   - `llm/missions/__init__.py` — Mission exports
   - `llm/missions/schemas.py` — Mission Pydantic models
   - `llm/missions/prompts.py` — Mission prompts
   - `llm/missions/context.py` — Mission context builder
   - `llm/missions/generator.py` — Mission generator
   - `llm/missions/validation.py` — Validation logic
   - `llm/missions/pipeline.py` — LangGraph pipeline

3. **Create `project-requirements/interface_contracts/` directory**
   - One contract file per module
   - Include: function signatures, expected behavior, edge cases, error handling
   - Reference: Load `my-style/testing.md` for contract format guidance

### Output

```
_planning/project-requirements/
└── interface_contracts/
    ├── llm_core.md           # __init__, exceptions, retry, context
    ├── llm_structured.md     # JSON parsing/recovery
    ├── llm_providers.md      # Factory, provider abstraction
    ├── llm_briefing.md       # Briefing pipeline
    └── llm_missions.md       # Mission pipeline
```

### Verification Criteria

- [ ] All public functions documented with signatures
- [ ] Expected behavior described (not implementation details)
- [ ] Edge cases and error handling documented
- [ ] Contracts reviewed against `my-style` standards

---

## Phase 12: Test Reset & Coverage

### Context

This phase runs in a **new session** (test-writing mode). The agent receives interface contracts but NOT implementation conversation context.

### Scope

**Primary goal:** Ensure tests verify correct behavior, not just "code runs"

**Tasks:**

1. **Audit existing tests in `tests/llm/`**
   - Compare against interface contracts
   - Flag tests that verify implementation details instead of behavior
   - Identify tests with wrong mock targets
   - Find tests testing stale interfaces

2. **Fix drifting tests**
   - Update assertions to match contract expectations
   - Fix mock targets (e.g., patch at correct module level)
   - Remove tests for deprecated behavior

3. **Add missing coverage**
   - `llm/structured.py` — New module, likely undertested
   - Provider-specific retry delays
   - Multi-call mission generation
   - Truncation resume detection
   - `tokens_used` counter
   - JSON truncation recovery

4. **Full suite verification**
   - Run `pytest --tb=short -q`
   - Confirm 723+ tests pass
   - Document any environment-related skips

5. **Coverage report**
   - Run `coverage report` on `src/unfamiliar/llm/`
   - Identify gaps below 80% threshold
   - Add tests to close gaps

### Files Affected

```
tests/
├── test_llm_context.py
├── test_llm_provider.py
├── test_llm_retry.py
├── test_llm_structured.py      # May need creation
├── test_briefing_generator.py
├── test_mission_*.py
└── conftest.py                  # Shared fixtures
```

### Verification Criteria

- [ ] All tests pass
- [ ] No tests flagged as "testing implementation details"
- [ ] Coverage ≥80% on `src/unfamiliar/llm/`
- [ ] New modules have test files
- [ ] Mock targets verified correct

---

## Architectural Decisions (Pre-Approved)

1. **Multi-call mission generation** — Split large requests into smaller calls rather than limiting mission quality
2. **Truncation resume** — When response is truncated, recover complete missions and resume from the broken one (not just discard incomplete)
3. **Provider-aware retry** — Different base delays per provider, not one-size-fits-all
4. **No Anthropic testing yet** — GLM is primary target; Anthropic compatibility via code review only
5. **Full LLM module audit** — Not just changed files, to catch drift across the board

---

## Risks

| Risk | Mitigation |
|------|------------|
| Phase 10 breaks existing tests | Run full suite after each task |
| Interface contracts miss edge cases | Phase 12 can flag gaps for iteration |
| Multi-call increases rate limit exposure | Provider-aware delays + cooldown periods |
| Resume prompts confuse LLM | Clear instruction format, test with GLM |
| Test rewrite is extensive | Prioritize by coverage gap size |

---

## Success Criteria

After Phase 12:
- Pipeline reliably generates 7 categories × 5-8 missions without truncation
- Truncation resume works: incomplete response triggers follow-up call, results merged
- Rate limit recovery succeeds with appropriate delays
- All tests pass and verify contract-specified behavior
- Coverage ≥80% on LLM module
- Foundation is solid for Campaign/CLI phases
