# Phase 6: LLM Integration — Provider Abstraction and Briefing Generation

## Goal

Build the LLM integration layer: a provider-agnostic factory for creating chat models, retry/error handling utilities, a context builder that transforms `AnalysisResult` into LLM-consumable prompts, and a briefing generator that produces structured codebase summaries with module naming — all through a single LLM call.

## Dependencies

- **Phase 3–4**: `AnalysisResult` from `analyze_repository()` (static analysis output consumed as input)
- **Phase 2**: Pydantic models (`Briefing`, `Module`), DB repositories (`CampaignRepository`)
- **Phase 1**: Config system (`UnfamiliarConfig`, `resolve_api_key()`)

## Research Notes

### LangChain Provider Architecture
- All chat models inherit from `BaseChatModel` in `langchain-core`
- Provider-specific packages: `langchain-anthropic` (ChatAnthropic), `langchain-community` or `langchain-zhipuai` (ChatZhipuAI)
- `.with_structured_output(PydanticModel)` works across all providers via tool-calling — the portable approach
- `ChatPromptTemplate.from_messages([("system", "..."), ("human", "...")])` for prompt construction
- Pipe operator `prompt | model` creates `RunnableSequence` (LCEL pattern)

### ChatAnthropic Constructor
- Key params: `model` (str), `temperature` (float), `max_tokens` (int), `max_retries` (int), `api_key` (str|None)
- Defaults to `ANTHROPIC_API_KEY` env var if `api_key` not passed
- Default model for briefing: `claude-sonnet-4-5-20250929` (good balance of quality and cost)

### GLM (Zhipu) Integration
- **Needs verification**: Check whether `langchain-zhipuai` or `langchain-community` provides `ChatZhipuAI`
- GLM API key env var: `ZHIPUAI_API_KEY` (LangChain convention) — our config maps `GLM_API_KEY` to this
- Default model: `glm-4` or latest available

### Token Estimation
- Rough heuristic: 1 token ≈ 4 characters (works for English text and code)
- No need for tiktoken — approximate budget enforcement is sufficient for key file selection
- The token budget (default 50,000) controls how many key file contents are included in context

### Retry Strategy (per requirements)
- Transient failures (429, 5xx, network errors): exponential backoff with jitter, max 5 retries
- Auth errors (401/403): immediate stop with clear message
- LangChain's `max_retries` on ChatModel handles basic retries, but we need higher-level retry for structured output parsing failures

### Structured Output Strategy
- Use `.with_structured_output(PydanticModel)` (tool-calling based) for provider portability
- Single LLM call produces both briefing content AND module naming
- Pydantic `Field(description=...)` guides the model's output

---

## Tasks

### Task 1: Update dependencies in pyproject.toml

**Files:** `pyproject.toml`
**Action:** Add `langchain-zhipuai` (or verified GLM package) to dependencies. Verify existing LangChain deps are sufficient: `langchain-core>=0.3`, `langchain-anthropic>=0.3`, `langgraph>=0.4`. Add `langchain>=0.3` if `init_chat_model` is used.
**Verify:** `pip install -e .` succeeds; new packages are importable.
**Done when:** All required LangChain packages are declared and installable.

### Task 2: Create provider factory

**Files:** `src/unfamiliar/llm/providers/factory.py`, `src/unfamiliar/llm/providers/__init__.py`
**Action:** Implement `create_chat_model(config: UnfamiliarConfig) -> BaseChatModel` that:
- Reads `config.provider` to select provider
- Calls `resolve_api_key(provider, config)` for the API key
- Returns a configured `ChatAnthropic` or `ChatZhipuAI` instance
- Raises `ProviderError` (custom exception) for unknown provider or missing API key
- Includes sensible defaults: `temperature=0.2`, `max_tokens=4096`, `max_retries=2`
**Verify:** `ruff check`, `mypy` pass on new files.
**Done when:** Factory returns a working `BaseChatModel` for both provider strings.

### Task 3: Create retry utilities

**Files:** `src/unfamiliar/llm/retry.py`
**Action:** Implement `with_llm_retry()` — an async-compatible retry wrapper that:
- Retries on transient HTTP errors (429, 500, 502, 503, 504) and connection errors
- Uses exponential backoff: `base_delay * 2^attempt + jitter` (jitter = random 0–1s)
- Max 5 attempts (configurable)
- Immediately raises on auth errors (401, 403) with a clear "check your API key" message
- Logs each retry attempt at WARNING level
- Returns the successful result or raises the last error after exhausting retries
**Verify:** `ruff check`, `mypy` pass.
**Done when:** Retry wrapper correctly distinguishes transient vs permanent errors and applies backoff.

### Task 4: Create context builder

**Files:** `src/unfamiliar/llm/context.py`
**Action:** Implement `build_briefing_context(analysis: AnalysisResult, token_budget: int) -> str` that:
- Produces a structured text document from `AnalysisResult` for LLM consumption
- Sections: repository overview (languages, file count), entry points, module structure (clusters from `DetectedModule`), import graph summary (hub files, connectivity), test coverage summary, security findings, key file contents
- Key file selection: iterate `scored_files` in priority order, include full file contents until `token_budget` is reached (estimate: len(content) / 4 tokens)
- Remaining files listed as signatures only (path + definition names)
- Each section clearly labeled with headers for LLM parsing
**Verify:** Run against a real `AnalysisResult` — output is human-readable, stays within budget.
**Done when:** Context builder produces complete, well-structured LLM context from analysis data.

### Task 5: Create briefing response schemas

**Files:** `src/unfamiliar/llm/briefing/schemas.py`
**Action:** Define Pydantic models for LLM structured output:
- `ModuleDescription(BaseModel)`: `name: str`, `description: str`, `security_relevant: bool` — with Field descriptions guiding output
- `BriefingResponse(BaseModel)`: `summary: str`, `tech_stack: list[str]`, `security_posture: str`, `modules: list[ModuleDescription]`
- Field descriptions should be explicit about what the LLM should produce (e.g., summary = "Plain-language overview of what the codebase does, its purpose, and how it is organized. 3-5 paragraphs.")
**Verify:** `ruff check`, `mypy` pass. Models can be instantiated with sample data.
**Done when:** Schema models are defined with clear field descriptions for LLM guidance.

### Task 6: Create briefing prompt template

**Files:** `src/unfamiliar/llm/briefing/prompts.py`
**Action:** Define `BRIEFING_PROMPT` as a `ChatPromptTemplate.from_messages()` with:
- System message: role definition (expert code analyst), output expectations (structured briefing), module naming instructions (name each detected module cluster based on file contents and purpose)
- Human message: `{context}` variable for the built context string
- Module count variable `{module_count}` so the LLM knows how many modules to name
- Prompt should emphasize: plain language (no theme), accuracy over creativity, identify all 7 investigation categories and note relative complexity
**Verify:** Template renders without error when invoked with sample context.
**Done when:** Prompt template is complete and produces correct message structure.

### Task 7: Create briefing generator

**Files:** `src/unfamiliar/llm/briefing/generator.py`, `src/unfamiliar/llm/briefing/__init__.py`
**Action:** Implement `generate_briefing(analysis: AnalysisResult, model: BaseChatModel, token_budget: int) -> BriefingResult` that:
- Calls `build_briefing_context()` to prepare the context
- Creates the chain: `BRIEFING_PROMPT | model.with_structured_output(BriefingResponse)`
- Invokes the chain with the context
- Wraps the call with `with_llm_retry()` for transient error handling
- Returns a `BriefingResult` dataclass containing: `briefing: BriefingResponse`, `modules: list[ModuleDescription]`, `context_tokens_estimate: int`
- Raises `BriefingGenerationError` on permanent failure
**Verify:** `ruff check`, `mypy` pass. Unit test with mocked model returns expected structure.
**Done when:** Generator produces a valid `BriefingResult` from analysis data via LLM call.

### Task 8: Create custom exceptions

**Files:** `src/unfamiliar/llm/exceptions.py`
**Action:** Define exception hierarchy:
- `LLMError(Exception)` — base for all LLM errors
- `ProviderError(LLMError)` — unknown provider, missing API key
- `AuthenticationError(LLMError)` — 401/403 from provider
- `BriefingGenerationError(LLMError)` — briefing generation failed after retries
- `TokenBudgetExceededError(LLMError)` — context too large (future-proofing)
**Verify:** `ruff check`, `mypy` pass.
**Done when:** Exception classes are defined and importable.

### Task 9: Wire up public API and exports

**Files:** `src/unfamiliar/llm/__init__.py`, `src/unfamiliar/llm/providers/__init__.py`, `src/unfamiliar/llm/briefing/__init__.py`
**Action:** Update all `__init__.py` files to export the public API:
- `llm/__init__.py`: export `create_chat_model`, `generate_briefing`, `BriefingResult`, exceptions
- `llm/providers/__init__.py`: export `create_chat_model`
- `llm/briefing/__init__.py`: export `generate_briefing`, `BriefingResult`, `BriefingResponse`
**Verify:** `from unfamiliar.llm import create_chat_model, generate_briefing` works.
**Done when:** Clean public API surface with all exports working.

### Task 10: Lint and type-check all new files

**Files:** All files created in tasks 1–9
**Action:** Run `ruff check src/unfamiliar/llm/` and `mypy src/unfamiliar/llm/` — fix any issues.
**Verify:** Zero errors from both tools.
**Done when:** All new code passes linting and type checking.

---

## Interface Contracts

### `create_chat_model`

```python
def create_chat_model(config: UnfamiliarConfig) -> BaseChatModel:
```

**Purpose:** Create a LangChain chat model from user configuration.

**Invariants:**
- Always returns a valid `BaseChatModel` subclass instance
- API key is resolved before model creation (never passed as empty string)
- Provider name is case-insensitive

**Valid inputs:**
- `config.provider` in `{"anthropic", "glm"}` (MVP)
- API key available via env var or config file

**Error conditions:**
- `ProviderError("Unknown provider: {name}")` — unrecognized provider string
- `ProviderError("No API key found for provider '{name}'. Set {ENV_VAR} or add to config.")` — no API key resolved
- `ProviderError("Failed to initialize {provider}: {detail}")` — provider package not installed or import error

**Edge cases:**
- Provider string with different casing ("Anthropic", "ANTHROPIC") should work
- Config with API key in file but wrong permissions → `resolve_api_key()` returns None → ProviderError

---

### `build_briefing_context`

```python
def build_briefing_context(
    analysis: AnalysisResult,
    token_budget: int = DEFAULT_TOKEN_BUDGET,
) -> str:
```

**Purpose:** Transform static analysis results into a structured text context for the LLM briefing call.

**Invariants:**
- Output is always valid UTF-8 text
- Key file contents are included in priority order (highest score first)
- Total estimated tokens for key file contents ≤ `token_budget`
- Every section has a labeled header
- Files not included as full content are listed as signatures

**Valid inputs:**
- `analysis`: A complete `AnalysisResult` from `analyze_repository()`
- `token_budget`: Positive integer (default 50,000)

**Error conditions:**
- None anticipated — gracefully handles empty file lists, zero modules, etc.

**Edge cases:**
- Empty `scored_files` → no key file contents section, only structural summary
- Token budget of 0 → only structural summary, no file contents
- Very large files → individual files exceeding budget are truncated
- Binary or non-UTF-8 files → should have been filtered by analysis pipeline, but skip gracefully if encountered

---

### `generate_briefing`

```python
def generate_briefing(
    analysis: AnalysisResult,
    model: BaseChatModel,
    token_budget: int = DEFAULT_TOKEN_BUDGET,
) -> BriefingResult:
```

**Purpose:** Generate a structured codebase briefing by calling the LLM with analysis context.

**Invariants:**
- Makes exactly 1 LLM call (plus retries on transient failure)
- Returns both briefing content and module descriptions in a single result
- `context_tokens_estimate` reflects the actual context size sent
- Module count in response matches module count in analysis input

**Valid inputs:**
- `analysis`: Complete `AnalysisResult`
- `model`: Any `BaseChatModel` instance (from `create_chat_model`)
- `token_budget`: Positive integer

**Error conditions:**
- `AuthenticationError` — 401/403 from provider (no retry, immediate)
- `BriefingGenerationError` — LLM call failed after max retries
- `BriefingGenerationError` — structured output parsing failed (model returned invalid JSON)

**Edge cases:**
- Model returns fewer modules than expected → pad with default names based on directory paths
- Model returns empty `tech_stack` → log warning, proceed with empty list
- Very small codebase (< 5 files) → context is small, briefing should still be generated
- No security findings → security_posture should note "no issues detected"

---

### `BriefingResult`

```python
@dataclass
class BriefingResult:
    briefing: BriefingResponse     # summary, tech_stack, security_posture
    modules: list[ModuleDescription]  # name, description, security_relevant per module
    context_tokens_estimate: int   # approximate tokens in the context sent
```

**Purpose:** Container for the complete output of briefing generation.

**Invariants:**
- `briefing` is always a valid `BriefingResponse` with non-empty `summary`
- `modules` length matches the number of `DetectedModule` entries in the input `AnalysisResult`
- `context_tokens_estimate` > 0

---

### `with_llm_retry`

```python
def with_llm_retry(
    fn: Callable[..., T],
    *args: Any,
    max_attempts: int = 5,
    base_delay: float = 1.0,
    **kwargs: Any,
) -> T:
```

**Purpose:** Execute a function with exponential backoff retry on transient LLM errors.

**Invariants:**
- Never retries on auth errors (401, 403)
- Retries on 429, 5xx, and connection errors
- Delay doubles each attempt: `base_delay * 2^attempt + random(0, 1)`
- Logs each retry at WARNING level
- Returns the function's return value on success

**Valid inputs:**
- `fn`: Any callable (typically an LLM chain `.invoke()`)
- `max_attempts`: 1–10 (1 = no retries)
- `base_delay`: Positive float in seconds

**Error conditions:**
- `AuthenticationError` — raised immediately on 401/403 (no retry)
- Re-raises the last caught exception after `max_attempts` exhausted

**Edge cases:**
- `max_attempts=1` → single attempt, no retries
- Function succeeds on first try → no delay, immediate return
- Non-HTTP errors (e.g., JSON parse error) → retry (model may produce valid output on second attempt)

---

### Exception Hierarchy

```python
class LLMError(Exception): ...
class ProviderError(LLMError): ...
class AuthenticationError(LLMError): ...
class BriefingGenerationError(LLMError): ...
class TokenBudgetExceededError(LLMError): ...
```

**Invariants:**
- All LLM-related errors inherit from `LLMError`
- `AuthenticationError` always includes the provider name and action to take
- `ProviderError` always includes which provider failed and why
- `BriefingGenerationError` includes the number of retry attempts made

---

## Property-Based Testing Candidates (for Phase 8)

1. **`build_briefing_context` token budget**: For any `AnalysisResult` and `token_budget`, the estimated token count of included key file contents never exceeds `token_budget`.

2. **`build_briefing_context` completeness**: Every file in `scored_files` either appears as full content or as a signature — no files are silently dropped.

3. **`create_chat_model` provider roundtrip**: For all valid provider strings (case variations of "anthropic", "glm"), `create_chat_model` returns a `BaseChatModel` without error (given a valid API key).

4. **Retry backoff timing**: For any `max_attempts` and `base_delay`, the delay between attempt N and N+1 is always >= `base_delay * 2^N` and <= `base_delay * 2^N + 1.0`.

5. **`BriefingResponse` module count**: For any input `AnalysisResult` with N modules, the `BriefingResult.modules` list has exactly N entries.

---

## Security Checklist

### API Key Handling
- [x] API keys are never logged or exposed in error messages
- [x] API keys resolved via `resolve_api_key()` which checks file permissions
- [x] API keys are read from environment variables first, then config file (only if strict permissions)
- [x] GLM API key is mapped to `ZHIPUAI_API_KEY` environment variable for LangChain compatibility

### Input Validation
- [x] Provider name is validated against known providers list
- [x] Provider name is normalized to lowercase and stripped before lookup
- [x] Token budget must be positive (validated by default value)
- [x] AnalysisResult input is typed but not validated (assumed valid from analysis pipeline)

### Error Handling
- [x] Auth errors (401/403) fail immediately with actionable message
- [x] Transient errors are retried with exponential backoff
- [x] Provider initialization failures raise `ProviderError` with clear message
- [x] Briefing generation failures raise `BriefingGenerationError` with attempt count
- [x] No sensitive data (API keys, file contents) in exception messages

### File Access
- [x] File contents read with explicit UTF-8 encoding
- [x] `UnicodeDecodeError` is caught and logged, file is skipped
- [x] `OSError` on file read is caught and logged, file is skipped
- [x] Binary files filtered upstream by analysis pipeline

### Logging
- [x] Sensitive data is never logged
- [x] API key mapping is logged at DEBUG level only
- [x] Retry attempts logged at WARNING level
- [x] Errors logged at ERROR level

---

## Issues Discovered During Verification Stage

None. All tasks completed successfully. Ruff linting passed after fixing line length issues in prompts and schemas (intentionally long for LLM guidance).
