<!-- STATUS: IN_PROGRESS -->

# Phase 10: Pipeline Stabilization

## Goal

Make the LLM pipeline reliable enough to scale to 7 categories with the GLM provider by implementing multi-call mission generation, provider-aware retry delays, and truncation resume logic. Also verify all 13 fixes from the debugging session are in place.

## Dependencies

- **Phase 9** (Analysis filtering) — Reduces non-code files at discovery phase
- **Phase 8.5** (Phase 8 fixes) — chmod removal, mock fixes established

## Context

The pipeline works end-to-end with `glm-4.7-Flash` but is unreliable:
1. **Rate limits** — GLM uses sliding window; 45-min gap still hits 429 errors
2. **JSON truncation** — 7+ missions in one category exceeds 4095 max_tokens
3. **No scaling path** — Current architecture can't handle 7 categories × 5-8 missions

Additionally, significant code changes happened outside the formal build process (commits 5e48e7e, ff3bc06, 899fe8b). This phase verifies those fixes and adds missing functionality.

---

## Research Notes

### Current Retry Behavior (`llm/retry.py`)

- Base delay: 1.0 second
- Exponential backoff: `base_delay * 2^(attempt-1) + jitter`
- Delays: 1s → 2s → 4s → 8s → 16s
- **Problem:** GLM rate limits require 30-60 second gaps, not 1-16 seconds

### Current Mission Generation (`llm/missions/generator.py`)

- Single LLM call per category via `generate_category_missions()`
- Requests 3-8 missions in one call
- No truncation resume — just recovery of complete missions from partial JSON

### Current Truncation Handling (`llm/structured.py`)

- `_recover_truncated_json()` extracts complete missions from truncated response
- **Missing:** No follow-up call to continue generation from the broken mission

### Provider Configuration (`llm/providers/factory.py`)

- `ProviderConfig` has: `langchain_id`, `default_model`, `max_token_budget`
- **Missing:** No retry delay configuration per provider

### Tokens Used Bug (`llm/missions/context.py`)

- Line 175: `tokens_used = 0`
- Line 192: `if tokens_used + security_tokens < token_budget:`
- **Bug:** `tokens_used` is never incremented, so security section always passes budget check

---

## Verification: 13 Fixes Status

| # | Fix | File | Status |
|---|-----|------|--------|
| 1 | Auth detection reorder | `retry.py` | ✅ Verified (lines 65-79) |
| 2 | Provider token budget cap | `factory.py` | ✅ Verified (line 47: `max_token_budget=8_000`) |
| 3 | JSON parsing fallback | `structured.py` | ✅ Verified (ChatZhipuAI in denylist) |
| 4 | Model env var | `factory.py` | ✅ Verified (`UNFAMILIAR_MODEL`) |
| 5 | Example-based prompt | `structured.py` | ✅ Verified (`_build_example_structure`) |
| 6 | Module output cap | `briefing/generator.py` | ✅ Verified (`_MAX_LLM_MODULES = 15`) |
| 7 | Singleton filtering | `missions/context.py` | ✅ Verified (line 99-101) |
| 8 | Testing script | `_planning/testing_script.py` | ✅ Verified (file exists) |
| 9 | max_tokens for GLM | `factory.py` | ✅ Verified (passed to ChatZhipuAI) |
| 10 | Mission logging | `testing_script.py` | ⚠️ Not verified (testing script only) |
| 11 | Test expectation | `test_llm_provider.py` | ✅ Verified (4095 in tests) |
| 12 | Configurable max_tokens | `factory.py` | ✅ Verified (`UNFAMILIAR_MAX_TOKENS`) |
| 13 | Truncated JSON recovery | `structured.py` | ✅ Verified (`_recover_truncated_json`) |

**Result:** All 13 fixes verified present. No restoration needed.

---

## Tasks

### Task 0: Audit testing script for logic that masks app issues ✅ DONE

**File:** `_planning/testing_script.py`

**Action:**
1. Review script for any logic that duplicates app functionality
2. Remove or relocate:
   - `_PHASE_COOLDOWN_SECONDS` and `time.sleep()` (Task 8 adds cooldown to pipeline)
   - Duplicate `CHARS_PER_TOKEN` constant (import from app)
   - Independent token counting (relies on app's `tokens_used` after Task 1)
3. Ensure script is a thin wrapper that only:
   - Loads config
   - Calls app functions (`analyze_repository`, `generate_briefing`, `generate_category_missions`)
   - Logs results

**Verify:**
- Script imports constants from app instead of defining its own
- No duplicate cooldown logic (uses pipeline's cooldown)
- No independent token counting (uses values from app)

**Done when:**
- Script contains no logic that could mask issues in app code
- Script behavior unchanged from user perspective (still produces same output)

---

### Task 1: Fix `tokens_used` counter bug in `missions/context.py` ✅ DONE

**File:** `src/unfamiliar/llm/missions/context.py`

**Action:**
1. Add `tokens_used += len(section_text) // CHARS_PER_TOKEN` after each section is built
2. Update the security section budget check to use the incremented value

**Verify:**
- `tokens_used` is incremented for each section added
- Security section respects actual token usage

**Done when:**
- Unit test confirms `tokens_used` reflects actual character count
- Security section is skipped when budget exhausted

---

### Task 2: Add provider-specific retry delays to `ProviderConfig` ✅ DONE

**File:** `src/unfamiliar/llm/providers/factory.py`

**Action:**
1. Add `retry_base_delay: float` field to `ProviderConfig` dataclass
2. Set `retry_base_delay=1.0` for Anthropic (current default)
3. Set `retry_base_delay=30.0` for GLM (conservative for rate limits)
4. Add `get_retry_delay(provider_name: str) -> float` function

**Verify:**
- `ProviderConfig` has `retry_base_delay` field
- Anthropic returns 1.0, GLM returns 30.0

**Done when:**
- `get_retry_delay("anthropic") == 1.0`
- `get_retry_delay("glm") == 30.0`

---

### Task 3: Update `with_llm_retry` to accept provider-aware delay ✅ DONE

**File:** `src/unfamiliar/llm/retry.py`

**Action:**
1. Add optional `provider_name: str | None = None` parameter
2. Import `get_retry_delay` from factory
3. If `provider_name` provided, use `get_retry_delay(provider_name)` as base
4. Otherwise use `_DEFAULT_BASE_DELAY`

**Verify:**
- `with_llm_retry(fn, provider_name="glm")` uses 30s base delay
- `with_llm_retry(fn, provider_name="anthropic")` uses 1s base delay
- Backward compatible: `with_llm_retry(fn)` uses 1s default

**Done when:**
- Unit test confirms different delays for different providers
- Existing tests still pass

---

### Task 4: Implement truncation resume detection in `structured.py` ✅ DONE

**File:** `src/unfamiliar/llm/structured.py`

**Action:**
1. Add `TruncationResult` dataclass with fields:
   - `recovered_data: BaseModel | None`
   - `is_truncated: bool`
   - `last_complete_index: int | None` (index of last complete item in array)
   - `resume_hint: str | None` (description of where to resume)
2. Add `detect_truncation(text: str, schema: type[T]) -> TruncationResult` function
3. Modify `_recover_truncated_json` to return `TruncationResult` with resume info

**Verify:**
- `detect_truncation` returns `is_truncated=True` for incomplete JSON
- `last_complete_index` identifies the last complete mission
- `resume_hint` provides context for continuation prompt

**Done when:**
- Unit tests cover: complete response, truncated mid-mission, truncated mid-field
- Resume hint is human-readable for continuation prompt

---

### Task 5: Add multi-call mission generation to `missions/generator.py` ✅ DONE

**File:** `src/unfamiliar/llm/missions/generator.py`

**Action:**
1. Add constants:
   - `_MISSIONS_PER_CALL = 3` (request 3 missions per call)
   - `_MAX_CALLS_PER_CATEGORY = 3` (max 3 calls per category)
2. Add `generate_category_missions_multi()` function that:
   - Makes 2-3 calls per category requesting 2-3 missions each
   - Uses truncation detection after each call
   - If truncated, continues from the last complete mission
   - Merges results from all calls
3. Add `continue_from_mission` parameter to prompt for resume calls
4. Update `MISSION_PROMPT` or add `MISSION_CONTINUE_PROMPT` for resume scenarios

**Verify:**
- Multiple calls are made when truncation detected
- Missions from all calls are merged into single response
- No duplicate missions in merged result

**Done when:**
- Integration test with mock confirms multi-call behavior
- Resumed calls include "continue from mission X" context

---

### Task 6: Wire provider-aware retry into mission generator ✅ DONE

**File:** `src/unfamiliar/llm/missions/generator.py`

**Action:**
1. Accept `provider_name: str` parameter in `generate_category_missions`
2. Pass to `with_llm_retry(invoke_chain, provider_name=provider_name)`
3. Default to `None` for backward compatibility

**Verify:**
- GLM provider gets 30s base delay
- Anthropic provider gets 1s base delay
- Missing provider_name falls back to 1s

**Done when:**
- Unit test confirms delay selection based on provider
- Existing tests pass

---

### Task 7: Update pipeline to pass provider info ✅ DONE

**File:** `src/unfamiliar/llm/missions/pipeline.py`

**Action:**
1. Add `provider_name: str` parameter to `run_mission_pipeline`
2. Pass to `generate_category_missions` calls
3. Extract provider name from config or model instance

**Verify:**
- Provider name flows from config → pipeline → generator → retry

**Done when:**
- End-to-end test confirms provider-specific delays used

---

### Task 8: Add cooldown period between category calls ✅ DONE

**File:** `src/unfamiliar/llm/missions/pipeline.py`

**Action:**
1. Add `category_cooldown: float = 0.0` parameter to `run_mission_pipeline`
2. Add `time.sleep(category_cooldown)` between categories
3. GLM recommended: 60 seconds between categories

**Verify:**
- Cooldown is applied between category calls
- Zero cooldown (default) has no delay

**Done when:**
- Unit test confirms sleep called with correct duration
- No performance impact for providers that don't need cooldown

---

## Interface Contracts

### `llm/providers/factory.py`

#### `get_retry_delay(provider_name: str) -> float`

**Purpose:** Return the appropriate base delay for retry backoff based on provider.

**Signature:**
```python
def get_retry_delay(provider_name: str) -> float
```

**Invariants:**
- Always returns a positive float
- Unknown providers return default (1.0)

**Valid inputs:**
- `provider_name`: Any non-empty string (case-insensitive)

**Expected returns:**
- `"anthropic"` → `1.0`
- `"glm"` → `30.0`
- Unknown provider → `1.0`

**Edge cases:**
- Empty string → `1.0` (default)
- None → TypeError (parameter required)

---

### `llm/retry.py`

#### `with_llm_retry` (updated signature)

**Purpose:** Execute function with exponential backoff, using provider-specific delays.

**Signature:**
```python
def with_llm_retry(
    fn: Callable[..., T],
    *args: object,
    max_attempts: int = 5,
    base_delay: float | None = None,
    provider_name: str | None = None,
    **kwargs: object,
) -> T
```

**Invariants:**
- If both `base_delay` and `provider_name` provided, `base_delay` takes precedence
- If neither provided, uses `_DEFAULT_BASE_DELAY` (1.0)
- Delays always include jitter (0-1 second added)

**Valid inputs:**
- `provider_name`: "anthropic", "glm", or None
- `base_delay`: Positive float or None

**Error conditions:**
- AuthenticationError on 401/403 (no retry)
- Original exception after max_attempts exhausted

---

### `llm/structured.py`

#### `TruncationResult` (dataclass)

```python
@dataclass
class TruncationResult:
    recovered_data: BaseModel | None
    is_truncated: bool
    last_complete_index: int | None
    resume_hint: str | None
```

**Purpose:** Encapsulate results of truncated JSON parsing with resume context.

**Fields:**
- `recovered_data`: Parsed model if any complete data recovered, else None
- `is_truncated`: True if response was incomplete
- `last_complete_index`: Index of last complete item in array (0-based)
- `resume_hint`: Human-readable description for continuation prompt

---

#### `detect_truncation`

**Signature:**
```python
def detect_truncation(
    text: str,
    schema: type[T],
) -> TruncationResult
```

**Purpose:** Parse LLM response and detect if JSON was truncated.

**Invariants:**
- Never raises exception - always returns TruncationResult
- `is_truncated=False` implies `recovered_data` is valid

**Valid inputs:**
- `text`: Any string (including empty)
- `schema`: Pydantic model class with array field

**Edge cases:**
- Empty text → `TruncationResult(None, False, None, None)`
- Complete JSON → `TruncationResult(parsed, False, None, None)`
- Truncated mid-object → `TruncationResult(partial, True, index, hint)`

---

### `llm/missions/generator.py`

#### `generate_category_missions_multi` (new function)

**Signature:**
```python
def generate_category_missions_multi(
    analysis: AnalysisResult,
    briefing_result: BriefingResult,
    category: InvestigationCategory,
    model: BaseChatModel,
    previous_missions: list[str] | None = None,
    token_budget: int = DEFAULT_TOKEN_BUDGET,
    provider_name: str | None = None,
    missions_per_call: int = 3,
    max_calls: int = 3,
) -> CategoryMissionsResponse
```

**Purpose:** Generate missions with multi-call strategy and truncation resume.

**Invariants:**
- Returns at least one mission (or raises MissionGenerationError)
- Total missions ≤ `missions_per_call * max_calls`
- No duplicate mission titles in result

**Valid inputs:**
- `missions_per_call`: 1-5 (reasonable range)
- `max_calls`: 1-5 (reasonable range)

**Error conditions:**
- MissionGenerationError if all calls fail
- MissionGenerationError if no missions generated

---

## Verify Criteria

### Agent Verification (Unit Tests with Mocks)

- [x] Testing script audited — no duplicate logic that masks app issues
- [x] Unit test: `tokens_used` counter reflects actual character count
- [x] Unit test: `get_retry_delay("glm")` returns 30.0, `get_retry_delay("anthropic")` returns 1.0
- [x] Unit test: `with_llm_retry` uses provider-specific base delay
- [x] Unit test: `detect_truncation` correctly identifies truncated vs complete JSON
- [x] Unit test: `generate_category_missions_multi` makes multiple calls when truncation detected
- [x] Unit test: Merged missions contain no duplicates
- [x] 723+ tests still pass after changes

### User Verification (Manual Testing with Real LLM)

- [x] Run `_planning/testing_script.py` with GLM — all 7 categories complete
- [x] Observe truncation resume in logs (follow-up call triggered)
- [x] Confirm rate limit recovery with 30s+ delays
- [ ] Review generated missions for quality

**Note:** Agent does NOT make real LLM calls during build. All LLM behavior is tested via mocks.

---

## Out of Scope

- Anthropic provider testing (too expensive for now)
- Tiered context architecture (Option A from test_pipeline_issue.md)
- Two-pass generation (Option C)
- LLM-optional briefing (Option D)
- Phase 11 (interface contracts) and Phase 12 (test reset)

---

## Post-Build Fix (2026-03-14)

### Issue Found During User Testing

Output 6 showed 429 rate limit errors during mission generation despite 60s cooldown.

**Root cause:** `provider_name` was not being passed through the call chain:
1. `generate_briefing()` didn't accept `provider_name` parameter
2. Testing script didn't pass `provider_name` to either function

**Result:** Retry delays used default 1s instead of GLM's 30s.

### Fix Applied

1. **`src/unfamiliar/llm/briefing/generator.py`:**
   - Added `provider_name: str | None = None` parameter
   - Pass to `with_llm_retry()` call

2. **`_planning/testing_script.py`:**
   - Pass `provider_name=config.provider` to `generate_briefing()`
   - Pass `provider_name=config.provider` to `generate_category_missions()`

**Verification:** 723 tests still pass.

---

### Issue 2: Schema Validation Errors Not Retried (Output 7)

Output 7 showed GLM returned valid JSON but with missing `completion_condition` fields.

**Root cause:** `with_llm_retry` didn't treat schema validation errors as transient.

**Result:** Errors like "LLM response did not match expected schema" failed immediately without retry.

### Fix Applied

1. **`src/unfamiliar/llm/retry.py`:**
   - Added "did not match expected schema" and "validation error" to transient patterns

2. **`tests/test_llm_retry.py`:**
   - Added `test_schema_validation_error_retried`
   - Added `test_pydantic_validation_error_retried`

**Verification:** 725 tests pass (2 new tests added).
