# Phase Shift Requirements: Provider-Agnostic LLM Factory

**Created:** 2026-03-18
**Trigger:** GLM 429 errors caused by `ChatZhipuAI` legacy JWT authentication; need to validate pipeline end-to-end before building TUI

---

## Problem Statement

The pipeline has never completed a full end-to-end run. GLM returns HTTP 429 on the **first request**, even with a paid account, exponential backoff up to 7.5 minutes, and different API keys. The user's agentic coding tools (Claude Code, Open Code) use the same GLM account without issues.

### Root Cause

`langchain-community`'s `ChatZhipuAI` class uses **legacy JWT-based authentication**:

```python
# zhipuai.py line 552 — what ChatZhipuAI sends
headers = {"Authorization": _get_jwt_token(self.zhipuai_api_key)}
```

This splits the API key into `id.secret`, generates a short-lived JWT signed with HMAC-SHA256, and sends it as a bare token (no `Bearer` prefix). This is ZhipuAI's deprecated auth method.

Agentic coding tools use ZhipuAI's **OpenAI-compatible endpoint** with standard `Authorization: Bearer <api_key>` auth. This routes to the paid-tier rate limits. The JWT path likely routes to a different (stricter) tier, explaining why:

- 429 hits on the very first request (not a burst issue)
- Retries with any delay don't help (tier mapping, not cooldown)
- Different API keys don't help (same JWT code path)
- The same account works fine through other tools

### Why `ChatZhipuAI` Was Used

At the time of Phase 6, `init_chat_model` did not support ZhipuAI, so `ChatZhipuAI` was the recommended LangChain integration. The JWT auth being legacy is an internal implementation detail not visible from LangChain's docs. It was the correct decision with available information.

### Secondary Issue: GLM-Specific `max_tokens` Logic

The factory contains GLM-specific `_get_default_max_tokens()` and `_max_tokens_for_glm_model()` functions that were added while troubleshooting this issue. `max_tokens` is **optional** in the OpenAI API spec — when omitted, the model generates until it naturally completes (EOS token). For structured output (Pydantic models), the model stops when the JSON is complete.

Setting `max_tokens=4095` may have been **causing** truncation issues — the model hits the ceiling before finishing the JSON, triggering the truncation recovery pipeline. Removing the low ceiling should reduce truncation frequency (though the recovery logic stays as a safety net).

---

## Solution: OpenAI-Compatible Provider Factory

### Verified Approach

LangChain's `init_chat_model` officially supports OpenAI-compatible providers via `base_url`:

```python
model = init_chat_model(
    model="glm-5",
    model_provider="openai",
    base_url="https://open.bigmodel.cn/api/paas/v4/",
    api_key=api_key,
)
```

This uses `langchain-openai`'s `ChatOpenAI` under the hood, which sends standard `Bearer` auth — the same method that works for the user's agentic tools.

### Design Goals

1. **Provider-agnostic**: Adding a new OpenAI-compatible provider = config entry, not code change
2. **Config-driven**: Provider, model, base_url, api_key env var all configurable in `config.toml`
3. **Sensible defaults**: Built-in registry handles common providers; config overrides when needed
4. **Single code path**: All providers go through `init_chat_model`; no provider-specific branches

### Provider Registry Design

Built-in registry with config overrides (option C from interview):

```python
# Built-in defaults — user doesn't need to know these for common providers
_BUILTIN_PROVIDERS = {
    "anthropic": ProviderConfig(
        langchain_provider="anthropic",
        default_model="claude-sonnet-4-5-2025-929",
        api_key_env="ANTHROPIC_API_KEY",
    ),
    "glm": ProviderConfig(
        langchain_provider="openai",  # OpenAI-compatible
        default_model="glm-5",
        base_url="https://open.bigmodel.cn/api/paas/v4/",
        api_key_env="ZHIPUAI_API_KEY",
    ),
}
```

Config file can override any field or define entirely new providers:

```toml
[general]
provider = "glm"

[llm]
model = "glm-5"
max_tokens = 16384  # optional safety ceiling, generous default

# Optional overrides — most users won't need these
# base_url = "https://custom-endpoint.example.com/v1/"
# api_key_env = "MY_CUSTOM_KEY"
```

### `max_tokens` Strategy

- **Drop all GLM-specific max_tokens logic** (`_get_default_max_tokens`, `_max_tokens_for_glm_model`)
- **Global configurable default**: generous ceiling (16384) as safety net against runaway generation
- **Optional config override**: users can set `max_tokens` in `[llm]` section
- **Omit entirely if not set**: let model decide when no ceiling is configured
- **Keep truncation recovery**: `detect_truncation` and `generate_category_missions_multi` stay as-is — they cost nothing when not triggered and catch real problems

### API Key Resolution

Lookup order (unchanged pattern, extended):

1. Config-specified env var name (`api_key_env` override in config)
2. Provider registry's default env var name (e.g., `ZHIPUAI_API_KEY` for GLM)
3. Generic `UNFAMILIAR_API_KEY` fallback

---

## Phase Insertion Plan

**One phase inserted as Phase 14.**

**Renumbering:**

| Old | New | Name |
|-----|-----|------|
| 14 | 15 | TUI — layout, widgets, and command routing |
| 15 | 16 | TUI — screens and interaction flows |
| 16 | 17 | Film Noir theme and narrative |
| 17 | 18 | Tests for phases 13-17 (campaign, CLI, TUI, theme, provider factory) |
| 18 | 19 | Polish — error handling, accessibility, and edge cases |

---

## Phase 14: Provider-Agnostic LLM Factory

### Scope

**Primary goal:** Replace `ChatZhipuAI` with OpenAI-compatible `init_chat_model` path; make provider/model selection config-driven.

### Tasks

1. **Extend config schema**
   - Add `[llm]` section to `UnfamiliarConfig`: `model`, `base_url`, `api_key_env`, `max_tokens` (all optional, with defaults)
   - Update `load_config()` to read `[llm]` section
   - Update `_DEFAULT_CONFIG_CONTENT` template
   - Backwards compatible: existing `config.toml` files with only `[general]` still work

2. **Build provider registry**
   - New `ProviderConfig` dataclass: `langchain_provider`, `default_model`, `base_url` (optional), `api_key_env`, `retry_base_delay`
   - Built-in entries for `anthropic` and `glm`
   - Merge logic: config overrides registry defaults per-field

3. **Rewrite factory to single code path**
   - All providers go through `init_chat_model` with `model_provider`, `model`, `api_key`, and optional `base_url`
   - Delete `_create_glm_model()` function
   - Delete `_get_default_max_tokens()` and `_max_tokens_for_glm_model()`
   - Remove `ChatZhipuAI` import
   - Pass `max_tokens` only when explicitly configured (otherwise omit to let model decide)

4. **Update API key resolution**
   - `resolve_api_key()` checks config-specified `api_key_env` first, then registry default, then generic fallback
   - Remove hardcoded `_PROVIDER_ENV_VARS` dict from `config.py` (move to registry)

5. **Update dependencies**
   - Add `langchain-openai` to `pyproject.toml`
   - Remove `pyjwt` from `pyproject.toml`

6. **Update existing tests**
   - `test_llm_provider.py`: Update factory tests for new single code path
   - `test_config.py`: Add tests for new `[llm]` config fields
   - Remove tests for deleted GLM-specific functions
   - Verify 888+ tests still pass

7. **Write interface contracts for inter-phase code**
   - `generate_category_missions_multi()` — updated signature with `functools.partial`
   - `run_mission_pipeline()` — new `categories` parameter
   - `_create_checkpointer()` — simplified SQLite-only
   - `_save_checkpoint()` — narrowed exception handling
   - These contracts feed into Phase 18 (test batch)

8. **Verify with GLM end-to-end**
   - User runs `unfamiliar investigate` against a real repo with GLM + `glm-5`
   - Confirm: no 429, briefing generates, missions generate for all 7 categories
   - This is user-side verification, not automated test

### Files Affected

```
src/unfamiliar/
├── config.py              # Extended config schema, API key resolution
├── constants.py           # DEFAULT_MAX_TOKENS addition
└── llm/
    └── providers/
        └── factory.py     # Full rewrite — single code path, provider registry

pyproject.toml             # Dependency swap: +langchain-openai, -pyjwt

tests/
├── test_config.py         # New [llm] section tests
└── test_llm_provider.py   # Factory test updates
```

### What Stays Unchanged

- `llm/retry.py` — error handling and retry logic untouched
- `llm/structured.py` — truncation detection/recovery stays as safety net
- `llm/missions/generator.py` — multi-call logic untouched
- `llm/missions/pipeline.py` — pipeline orchestration untouched
- `llm/briefing/` — briefing pipeline untouched

### Verification Criteria

- [ ] `init_chat_model` creates GLM model via OpenAI-compatible path (no `ChatZhipuAI`)
- [ ] `init_chat_model` creates Anthropic model via native path
- [ ] Config with only `[general]` section still loads (backwards compatible)
- [ ] Config with `[llm]` section overrides provider defaults
- [ ] API key resolution: config env var > registry default > generic fallback
- [ ] `max_tokens` omitted from API call when not configured
- [ ] `max_tokens` passed through when user sets it in config
- [ ] `pyjwt` removed from dependencies
- [ ] `langchain-openai` added to dependencies
- [ ] 888+ existing tests pass
- [ ] User verification: GLM pipeline completes without 429 errors

### Out of Scope

- Retry logic changes (keep current exponential backoff as-is)
- Truncation recovery changes (keep `detect_truncation` and multi-call as-is)
- Anthropic end-to-end testing (user may test if token cost is acceptable)
- TUI work (deferred to Phase 15+)
- Adding providers beyond Anthropic and GLM (architecture supports it; not needed now)

---

## Interface Contracts for Inter-Phase Code

These functions were modified in commits `bb4077d` through `f409239` (between phases 12 and 13) and lack formal contracts. Contracts will be written during Phase 14 task 7 and tested during Phase 18.

### `generate_category_missions_multi()`

**Location:** `llm/missions/generator.py`

**Signature:**
```python
def generate_category_missions_multi(
    analysis: AnalysisResult,
    briefing_result: BriefingResponse,
    category: InvestigationCategory,
    model: BaseChatModel,
    token_budget: int = DEFAULT_TOKEN_BUDGET,
    provider_name: str | None = None,
    max_calls: int = 3,
) -> CategoryMissionsResponse:
```

**Behavior:**
- Makes up to `max_calls` LLM calls per category, accumulating missions across calls
- Uses `functools.partial(chain.invoke, input_data)` for retry-safe invocation
- Detects truncation via `detect_truncation()` and resumes from last complete mission
- Deduplicates missions by title across calls
- Each call wrapped in `with_llm_retry()` with provider-specific delays
- Type-checks raw result — raises `TypeError` if not `CategoryMissionsResponse`

**Error behavior:**
- `AuthenticationError`: propagates immediately (no retry)
- `MissionGenerationError`: raised after all retries exhausted, includes `category` and `attempts` (call count, not retry count)
- Zero missions after all calls: raises `MissionGenerationError`

### `run_mission_pipeline()`

**Location:** `llm/missions/pipeline.py`

**Signature:**
```python
def run_mission_pipeline(
    analysis: AnalysisResult,
    briefing_result: BriefingResponse,
    model: BaseChatModel,
    campaign_dir: Path,
    token_budget: int = DEFAULT_TOKEN_BUDGET,
    provider_name: str | None = None,
    category_cooldown: float = 0.0,
    categories: list[InvestigationCategory] | None = None,
) -> PipelineResult:
```

**Behavior:**
- Processes categories in `CATEGORY_ORDER` sequence
- When `categories` is provided, filters to only those categories but preserves canonical order
- When `categories` is `None`, processes all 7 categories
- Calls `generate_category_missions_multi()` (not `generate_category_missions()`)
- Checkpoints after each successful category via SQLite
- Sleeps `category_cooldown` seconds between categories (when > 0 and not first)

**Error behavior:**
- `AuthenticationError`: stops pipeline, appends error, returns partial `PipelineResult`
- `MissionGenerationError`: logs error, appends to `result.errors`, continues to next category
- `TimeoutError`: logs error, appends to `result.errors`, continues to next category

### `_create_checkpointer()`

**Location:** `llm/missions/pipeline.py`

**Signature:**
```python
def _create_checkpointer(campaign_dir: Path) -> SqliteSaver:
```

**Behavior:**
- Creates `campaign_dir` if it doesn't exist (`mkdir(parents=True, exist_ok=True)`)
- Opens SQLite connection at `campaign_dir/mission_checkpoint.db`
- Returns `SqliteSaver` instance (no in-memory fallback)

### `_save_checkpoint()`

**Location:** `llm/missions/pipeline.py`

**Signature:**
```python
def _save_checkpoint(
    checkpointer: SqliteSaver,
    thread_id: str,
    result: PipelineResult,
) -> None:
```

**Behavior:**
- Serializes `PipelineResult` as checkpoint config value
- Commits via `checkpointer.conn.commit()` (guarded by `hasattr`)
- Catches only `OSError | sqlite3.Error` on commit failure (logs warning, does not raise)

---

## Architectural Decisions (Pre-Approved)

1. **OpenAI-compatible endpoint for GLM** — Use `init_chat_model` with `model_provider="openai"` and `base_url` instead of `ChatZhipuAI` with JWT auth
2. **Built-in registry + config overrides** — Common providers ship with defaults; config can override any field or define new providers
3. **Drop `max_tokens` default** — Let model decide output length; optional user-configurable ceiling for cost control
4. **Keep truncation recovery** — `detect_truncation` and multi-call stay as safety nets
5. **Keep retry logic unchanged** — Exponential backoff with provider-specific delays remains
6. **Single `init_chat_model` code path** — No provider-specific factory branches

---

## Risks

| Risk | Mitigation |
|------|------------|
| `langchain-openai` not installed in user's venv | User installs on host before phase build |
| `init_chat_model` doesn't pass `base_url` correctly to `ChatOpenAI` | Verified in LangChain docs — officially documented pattern |
| GLM rejects requests via OpenAI-compatible path | Unlikely — this is how all agentic tools connect; user can verify quickly |
| Config schema change breaks existing configs | Backwards compatible — new `[llm]` section is entirely optional |
| Removing `max_tokens` causes runaway generation | Structured output stops at JSON completion; optional ceiling available |
| `glm-5` model name incorrect | User confirms correct model name before end-to-end test |

---

## Success Criteria

After Phase 14:
- GLM pipeline completes full investigation without 429 errors
- Provider/model selection is config-driven, not code-driven
- Factory has one code path for all providers
- `pyjwt` dependency removed
- Existing tests pass
- Interface contracts documented for inter-phase code
- Foundation is solid for TUI phases (15-16)
