# Phase 14: Provider-Agnostic LLM Factory

<!-- STATUS: DONE -->

## Goal

Replace `ChatZhipuAI` (legacy JWT auth) with `init_chat_model` using `model_provider="openai"` and `base_url` for OpenAI-compatible providers. Make provider/model selection config-driven. Drop GLM-specific `max_tokens` logic. This unblocks the pipeline end-to-end — `ChatZhipuAI`'s JWT auth routes requests to the wrong rate-limit tier, causing 429 errors on every call despite a paid account.

## Dependencies

- Phase 13: Campaign orchestration and CLI (complete)
- User must install `langchain-openai` on host before build

## Research Notes

### `init_chat_model` with OpenAI-compatible providers

**Verified** via LangChain docs (Context7, 2026-03-18):

```python
model = init_chat_model(
    model="MODEL_NAME",
    model_provider="openai",
    base_url="BASE_URL",
    api_key="YOUR_API_KEY",
)
```

This creates a `ChatOpenAI` instance under the hood, which sends standard `Authorization: Bearer <api_key>` headers — the same auth method used by the user's working agentic tools (Claude Code, Open Code).

**Key parameters supported by `init_chat_model`:**
- `model` — model name string
- `model_provider` — LangChain provider identifier (e.g., `"anthropic"`, `"openai"`)
- `api_key` — API key string
- `base_url` — custom endpoint URL (used for OpenAI-compatible providers)
- `temperature`, `max_tokens`, `max_retries` — standard model params passed through as kwargs

### `_NO_TOOL_CALLING_MODELS` in structured.py

`structured.py` line 30 denylists `"ChatZhipuAI"` from tool-calling-based structured output. After this phase, GLM will go through `ChatOpenAI`, which **does** support tool calling. The denylist entry must be updated — the check should be by class name, and `ChatOpenAI` supports tool calling, so the entry can be removed.

### API key env var naming

Currently `config.py` maps `"glm"` → `"GLM_API_KEY"`. The phase shift requirements specify `"ZHIPUAI_API_KEY"`. The provider registry should use `"ZHIPUAI_API_KEY"` to match ZhipuAI's documented convention and what the user's other tools expect. We'll keep backward compatibility by checking `GLM_API_KEY` as a fallback in the registry.

### Deferred item: Static testing mission data

`deferred.md` has an open item about generating static mission data for TUI iteration. This is relevant to Phase 15 (TUI), not Phase 14. No action needed here — just note it for the next planning session.

---

## Tasks

### Task 1: Extend config schema with `[llm]` section [DONE]

**Files:** `src/unfamiliar/config.py`

**Action:**
- Add optional `LLMConfig` Pydantic model with fields: `model: str | None`, `base_url: str | None`, `api_key_env: str | None`, `max_tokens: int | None` (all default `None`)
- Add `llm: LLMConfig` field to `UnfamiliarConfig` (default empty `LLMConfig()`)
- Update `load_config()` to read `[llm]` section from TOML and populate `LLMConfig`
- Update `_DEFAULT_CONFIG_CONTENT` to show commented-out `[llm]` section as documentation
- Move `_PROVIDER_ENV_VARS` dict to the provider registry (Task 2) — it no longer belongs in config

**Verify:** `pytest tests/test_config.py -v` — all existing tests pass. Config with only `[general]` section still loads. Config with `[llm]` section populates the new fields.

**Done when:** `UnfamiliarConfig` has `llm` field; `load_config()` reads `[llm]` section; existing configs still work.

### Task 2: Build provider registry with config override merge [DONE]

**Files:** `src/unfamiliar/llm/providers/factory.py`

**Action:**
- Redesign `ProviderConfig` dataclass: add `base_url: str | None = None`, add `api_key_env: str`, rename `langchain_id` → `langchain_provider`
- Remove `max_token_budget` from `ProviderConfig` (no longer used)
- Update `_PROVIDERS` → `_BUILTIN_PROVIDERS`:
  - `"anthropic"`: `langchain_provider="anthropic"`, `default_model="claude-sonnet-4-5-2025-929"`, `api_key_env="ANTHROPIC_API_KEY"`
  - `"glm"`: `langchain_provider="openai"`, `default_model="glm-4-flash"`, `base_url="https://api.z.ai/api/coding/paas/v4"`, `api_key_env="ZHIPUAI_API_KEY"`, `retry_base_delay=2.0`
- Add `get_provider_config(provider_name: str, config: UnfamiliarConfig) -> ProviderConfig` function that merges config overrides onto registry defaults:
  - Looks up provider in `_BUILTIN_PROVIDERS`
  - Overlays non-None fields from `config.llm` (model, base_url, api_key_env)
  - Returns merged `ProviderConfig`
  - Raises `ProviderError` if provider is not in registry

**Verify:** Unit tests for `get_provider_config()` — returns defaults, applies overrides, raises for unknown providers.

**Done when:** Provider registry exists with `anthropic` and `glm` entries; merge logic produces correct `ProviderConfig` for both default and overridden cases.

### Task 3: Rewrite factory to single `init_chat_model` code path [DONE]

**Files:** `src/unfamiliar/llm/providers/factory.py`

**Action:**
- Rewrite `create_chat_model()`:
  - Call `get_provider_config()` to get merged config
  - Resolve API key using registry's `api_key_env` (see Task 4)
  - Call `init_chat_model(model=..., model_provider=..., api_key=..., temperature=0.2, max_retries=2)` with optional `base_url` and `max_tokens` kwargs
  - `base_url`: pass only if non-None in provider config
  - `max_tokens`: pass only if explicitly set in `config.llm.max_tokens` or `UNFAMILIAR_MAX_TOKENS` env var; **omit** otherwise (let model decide)
  - Apply `UNFAMILIAR_MODEL` env var override for model name (existing behavior)
- Delete `_create_glm_model()` function
- Delete `_create_standard_model()` function
- Delete `_get_default_max_tokens()` function
- Delete `_max_tokens_for_glm_model()` function
- Delete `get_effective_token_budget()` function (unused since max_token_budget removed)
- Remove `ChatZhipuAI` import

**Verify:** `pytest tests/test_llm_provider.py -v` — tests updated in Task 6.

**Done when:** Factory has one code path; all providers go through `init_chat_model`; no `ChatZhipuAI` references remain in factory.

### Task 4: Update API key resolution [DONE]

**Files:** `src/unfamiliar/config.py`, `src/unfamiliar/llm/providers/factory.py`

**Action:**
- Update `resolve_api_key()` in `config.py` to accept an optional `api_key_env_override: str | None` parameter:
  1. If `api_key_env_override` is provided, check that env var first
  2. Fall back to `_PROVIDER_ENV_VARS` (kept as backward compat — `GLM_API_KEY` still works)
  3. Fall back to `UNFAMILIAR_API_KEY`
- In `create_chat_model()`: pass `provider_config.api_key_env` as the override to `resolve_api_key()`
- The lookup chain becomes: config-specified env var (`api_key_env` from registry/config) → legacy provider env var → generic fallback

**Verify:** Existing `test_config.py` API key tests still pass. New test: `ZHIPUAI_API_KEY` resolves for GLM provider.

**Done when:** GLM resolves key from `ZHIPUAI_API_KEY` by default; `GLM_API_KEY` still works as fallback; config `api_key_env` override takes precedence.

### Task 5: Update structured.py denylist [DONE]

**Files:** `src/unfamiliar/llm/structured.py`

**Action:**
- Remove `"ChatZhipuAI"` from `_NO_TOOL_CALLING_MODELS` set
- GLM now goes through `ChatOpenAI` which supports tool-calling-based structured output
- If the set becomes empty, keep the empty set and the check logic (zero runtime cost, provides extension point for future providers)

**Verify:** `grep -r "ChatZhipuAI" src/` returns no hits.

**Done when:** No references to `ChatZhipuAI` remain in the codebase.

### Task 6: Update dependencies [DONE]

**Files:** `pyproject.toml`

**Action:**
- Add `"langchain-openai>=0.3"` to dependencies
- Remove `"pyjwt>=2.0"` from dependencies
- Remove the `# Required for GLM/Zhipu provider JWT tokens` comment

**Verify:** `python -c "import langchain_openai"` succeeds (user must have installed it).

**Done when:** `pyproject.toml` has `langchain-openai`, no `pyjwt`.

### Task 7: Update existing tests [DONE]

**Files:** `tests/test_llm_provider.py`, `tests/test_config.py`

**Action:**
- **test_llm_provider.py:**
  - `TestValidGlmProvider`: GLM now goes through `init_chat_model` (not `_create_glm_model`). Update to mock `init_chat_model` and verify it's called with `model_provider="openai"`, `base_url="https://api.z.ai/api/coding/paas/v4"`, model name, and API key
  - `TestProviderCaseInsensitive`: GLM tests should verify `init_chat_model` called (not `_create_glm_model`)
  - `TestMaxTokensForGlmModel`: Delete entire class (functions no longer exist)
  - `TestCreateGlmModelDirectly`: Delete entire class (function no longer exists)
  - `TestGetEffectiveTokenBudget`: Delete entire class (function no longer exists)
  - `TestCreateStandardModelDirectly`: Update — `_create_standard_model` no longer exists; test `create_chat_model` directly or mock `init_chat_model`
  - `TestInitChatModelImportError`: Keep, update mock targets if needed
  - Add new test: `max_tokens` omitted when not configured
  - Add new test: `max_tokens` passed when set in config `llm.max_tokens`
  - Add new test: `UNFAMILIAR_MAX_TOKENS` env var override works
  - Add new test: `base_url` passed for GLM, not passed for Anthropic
- **test_config.py:**
  - Add tests for `[llm]` section parsing: model, base_url, api_key_env, max_tokens
  - Add test: config without `[llm]` section loads with defaults (backwards compat)
  - Add test: `ZHIPUAI_API_KEY` env var resolves for GLM

**Verify:** `pytest tests/test_llm_provider.py tests/test_config.py -v` — all pass.

**Done when:** All tests updated for new factory design; no references to deleted functions; new behavior covered.

### Task 8: Run full test suite [DONE]

**Files:** None (verification only)

**Action:**
- Run `pytest` — all 888+ tests pass
- Run `ruff check src/unfamiliar/` — no linting errors
- Verify no remaining references: `grep -r "ChatZhipuAI\|_create_glm_model\|_create_standard_model\|_get_default_max_tokens\|_max_tokens_for_glm_model\|get_effective_token_budget" src/ tests/`

**Verify:** Clean test run, clean lint, no stale references.

**Done when:** Full suite green; no references to deleted code.

---

## Security Checklist

- [x] **API keys from env vars only** — never in config file, never logged
- [x] **API keys not logged** — provider errors include env var name but not actual key value
- [x] **base_url not logged** — only used to construct model, passed to `init_chat_model`
- [x] **Provider config not exposed in TUI** — LLMConfig fields are not displayed
- [x] **max_tokens not logged** — only used to construct model
- [x] **No secrets in checkpoints** — config files don't store secrets
- [x] **Input validation** — API key env var names validated, provider names normalized/whitespace-stripped

## Issues Discovered During Verification Stage

None. All tasks completed without issues.

---

## Interface Contracts

These contracts describe the **new** public interface after the refactor. They feed into Phase 18 (test batch).

### `ProviderConfig` (dataclass)

**Location:** `llm/providers/factory.py`

```python
@dataclass(frozen=True)
class ProviderConfig:
    langchain_provider: str       # e.g., "anthropic", "openai"
    default_model: str            # e.g., "claude-sonnet-4-5-2025-929", "glm-5"
    api_key_env: str              # e.g., "ANTHROPIC_API_KEY"
    base_url: str | None = None   # e.g., "https://api.z.ai/api/coding/paas/v4"
    retry_base_delay: float = 1.0
    supports_tool_calling: bool = True
```

**Invariants:**
- `langchain_provider` is always a LangChain-recognized provider string
- `api_key_env` is always a non-empty string
- `base_url` is `None` for native providers (Anthropic), set for OpenAI-compatible providers (GLM)
- `supports_tool_calling` is `False` for providers that rate-limit or don't support the Structured Outputs API (`response_format` with JSON schema). When `False`, `create_structured_chain` uses JSON text fallback instead.

### `LLMConfig` (Pydantic model)

**Location:** `config.py`

```python
class LLMConfig(BaseModel):
    model: str | None = None
    base_url: str | None = None
    api_key_env: str | None = None
    max_tokens: int | None = Field(default=None, gt=0)
```

**Invariants:**
- All fields default to `None` (no overrides)
- `max_tokens` must be > 0 if set
- When all fields are `None`, factory uses registry defaults

### `get_provider_config(provider_name, config) -> ProviderConfig`

**Location:** `llm/providers/factory.py`

**Signature:**
```python
def get_provider_config(
    provider_name: str,
    config: UnfamiliarConfig,
) -> ProviderConfig:
```

**Purpose:** Merge config overrides onto built-in provider defaults.

**Invariants:**
- Provider name is case-insensitive, whitespace-stripped
- Config `llm.model` overrides `default_model` when non-None
- Config `llm.base_url` overrides `base_url` when non-None
- Config `llm.api_key_env` overrides `api_key_env` when non-None
- `retry_base_delay` comes from registry only (not user-configurable)
- `supports_tool_calling` comes from registry only (not user-configurable)

**Error conditions:**
- `ProviderError` if `provider_name` (after normalization) is not in `_BUILTIN_PROVIDERS`

**Edge cases:**
- `"  GLM  "` → normalizes to `"glm"` → returns GLM config
- `"unknown"` → raises `ProviderError` with message listing supported providers
- Config with `llm.model="glm-5"` + provider `"glm"` → returns GLM config with `default_model="glm-5"`

### `provider_supports_tool_calling(provider_name) -> bool`

**Location:** `llm/providers/factory.py`

**Signature:**
```python
def provider_supports_tool_calling(provider_name: str) -> bool:
```

**Purpose:** Check whether a provider supports tool-calling-based structured output (the OpenAI Structured Outputs API with `response_format` JSON schema). Used by `create_structured_chain` to decide between `with_structured_output` and JSON text fallback.

**Invariants:**
- Provider name is case-insensitive, whitespace-stripped
- Returns `True` for unknown providers (optimistic default)
- Returns `False` for GLM (rate-limits structured outputs endpoint)
- Returns `True` for Anthropic

**Edge cases:**
- `"unknown_provider"` → `True` (optimistic default)
- `"  GLM  "` → normalizes to `"glm"` → `False`

### `create_chat_model(config, max_tokens?) -> BaseChatModel`

**Location:** `llm/providers/factory.py`

**Signature:**
```python
def create_chat_model(
    config: UnfamiliarConfig,
    max_tokens: int | None = None,
) -> BaseChatModel:
```

**Purpose:** Create a LangChain chat model from user configuration via a single `init_chat_model` code path.

**Invariants:**
- All providers go through `init_chat_model` — no provider-specific branches
- `temperature` is always 0.2
- `max_retries` is always 2
- `base_url` is passed only when non-None in provider config
- `max_tokens` priority: `max_tokens` param > `config.llm.max_tokens` > `UNFAMILIAR_MAX_TOKENS` env var > **omitted** (model decides)
- `UNFAMILIAR_MODEL` env var overrides model name (existing behavior)

**Error conditions:**
- `ProviderError`: unknown provider, missing API key, missing provider package (ImportError)

**Edge cases:**
- No `max_tokens` anywhere → kwarg omitted from `init_chat_model` call → model decides output length
- `UNFAMILIAR_MAX_TOKENS=invalid` → logs warning, treats as unset
- `UNFAMILIAR_MODEL=glm-5` → overrides default model name

### `resolve_api_key(provider, config, api_key_env_override?) -> str`

**Location:** `config.py`

**Signature:**
```python
def resolve_api_key(
    provider: str,
    config: UnfamiliarConfig,
    api_key_env_override: str | None = None,
) -> str:
```

**Purpose:** Resolve API key from environment variables.

**Invariants:**
- Lookup order: `api_key_env_override` env var → legacy `_PROVIDER_ENV_VARS` env var → `UNFAMILIAR_API_KEY`
- Never returns empty string (raises `APIKeyError` instead)

**Error conditions:**
- `APIKeyError`: no key found in any source

**Edge cases:**
- `api_key_env_override="ZHIPUAI_API_KEY"` with `ZHIPUAI_API_KEY` set → returns that key
- `api_key_env_override="ZHIPUAI_API_KEY"` not set, `GLM_API_KEY` set → falls back to legacy mapping
- No env vars set → `APIKeyError`

### `get_retry_delay(provider_name) -> float`

**Location:** `llm/providers/factory.py`

**Signature unchanged.** Uses `_BUILTIN_PROVIDERS` instead of `_PROVIDERS`.

**Edge cases:** Same as before — case-insensitive, whitespace-stripped, 1.0 default for unknown.

### `persist_briefing_data(db, campaign_id, briefing_result, analysis?) -> list[Module]`

**Location:** `campaign/persister.py`

**Signature:**
```python
def persist_briefing_data(
    db: CampaignDB,
    campaign_id: int,
    briefing_result: BriefingResult,
    analysis: AnalysisResult | None = None,
) -> list[Module]:
```

**Purpose:** Persist the briefing and module descriptions to the database. Bridges two data sources: LLM-generated `ModuleDescription` (name, description, security_relevant) and analysis-generated `DetectedModule` (primary_path, file_count, category_hints).

**Invariants:**
- Creates one `Briefing` record and N `Module` records
- Modules are returned in the same order as `briefing_result.modules` (index-based linking for missions)
- When `analysis` is provided, structural data (path, file_count, category_relevance) is pulled from `analysis.modules[idx]` by matching index
- When `analysis` is `None` or index is out of range, defaults are used: `path=""`, `file_count=0`, `category_relevance=[]`
- LLM data (name, description, security_relevant) always comes from `ModuleDescription`

**Error conditions:**
- `DatabaseError` if campaign doesn't exist or briefing already exists

**Edge cases:**
- `analysis=None` → all modules get default structural data
- More `ModuleDescription`s than `DetectedModule`s (padding from briefing generator) → excess modules get default structural data
- Fewer `ModuleDescription`s than `DetectedModule`s → only described modules are created

### `CampaignRepository.delete(campaign_id) -> None`

**Location:** `db/repositories/campaign.py`

**Signature:**
```python
def delete(self, campaign_id: int) -> None:
```

**Purpose:** Delete a campaign and all associated data. All related records (briefing, modules, missions, chains, file refs, prerequisites, links) are deleted automatically via `ON DELETE CASCADE` foreign keys.

**Invariants:**
- Cascades to all child tables (briefing, module, mission, mission_chain, etc.)
- Commits immediately
- No-op if campaign_id doesn't exist (DELETE WHERE id=? affects 0 rows)

**Error conditions:**
- `DatabaseError` on SQLite failure

### `create_campaign(repo_path, config, progress_callback?) -> Campaign`

**Location:** `campaign/orchestrator.py`

**Signature:**
```python
def create_campaign(
    repo_path: Path,
    config: UnfamiliarConfig,
    progress_callback: Callable[[str], None] | None = None,
) -> Campaign:
```

**Purpose:** Orchestrate full campaign creation. If a campaign already exists for this repo, deletes it and starts fresh (overwrite semantics).

**Invariants:**
- Always creates a new campaign — never resumes a partial one
- Existing campaign for same `repo_path` is deleted before creation (CASCADE cleans all data)
- Returns campaign with `status=ACTIVE` on success
- On partial failure (some categories failed), campaign is still `ACTIVE` with partial missions

**Error conditions:**
- `CampaignOrchestrationError` if path is not a git repo
- `AuthenticationError` propagated from LLM layer
- `AnalysisError` if repository analysis fails

### `create_structured_chain(model, prompt, schema, provider_name?) -> Runnable`

**Location:** `llm/structured.py`

**Signature:**
```python
def create_structured_chain(
    model: BaseChatModel,
    prompt: ChatPromptTemplate,
    schema: type[T],
    provider_name: str | None = None,
) -> Runnable:
```

**Purpose:** Build a chain that returns validated structured output. Selects between tool-calling structured output (`with_structured_output`) and JSON text fallback based on provider capabilities.

**Invariants:**
- When `provider_name` is set and provider has `supports_tool_calling=False`, always uses JSON text fallback (never sends `response_format` to the API)
- When `provider_name` is `None`, falls back to class-name denylist (`_NO_TOOL_CALLING_MODELS`)
- JSON fallback augments the prompt with schema instructions and parses text response
- Both paths return a Runnable whose `.invoke()` returns a validated schema instance

**Edge cases:**
- `provider_name="glm"` → JSON fallback (GLM rate-limits structured outputs)
- `provider_name="anthropic"` → tool-calling structured output
- `provider_name=None` with `ChatOpenAI` model → tool-calling (not in denylist)

---

## Verification Criteria

- [x] `init_chat_model` creates GLM model via `model_provider="openai"` with `base_url` (no `ChatZhipuAI`)
- [x] `init_chat_model` creates Anthropic model via `model_provider="anthropic"` (no `base_url`)
- [x] Config with only `[general]` section still loads (backwards compatible)
- [x] Config with `[llm]` section overrides provider defaults
- [x] API key resolution: config env var > legacy env var > generic fallback
- [x] `max_tokens` omitted from API call when not configured anywhere
- [x] `max_tokens` passed through when user sets it in config or env var
- [x] `pyjwt` removed from dependencies
- [x] `langchain-openai` added to dependencies
- [x] No references to `ChatZhipuAI` remain in codebase
- [x] `_NO_TOOL_CALLING_MODELS` denylist updated
- [x] GLM uses `supports_tool_calling=False` → JSON text fallback (not Structured Outputs API)
- [x] GLM `base_url` points to Z.AI Coding Plan endpoint (`/api/coding/paas/v4`)
- [x] `persist_briefing_data` pulls structural data from `AnalysisResult`, not `ModuleDescription`
- [x] 888+ existing tests pass
- [ ] User verification: GLM pipeline completes end-to-end
