# LLM Integration Notes

**Created:** 2026-03-18
**Status:** Active Investigation

## User's LLM Setup

- **GLM (Zhipu AI):** Paid tier account
- **Anthropic:** Paid tier account

## Cost Considerations

Anthropic is very expensive to use. User prefers GLM for test data generation, especially while the pipeline stability is still being validated. Once the pipeline is confirmed stable, Anthropic can be considered for production use.

## The Mystery: GLM 429 Errors

### What's Happening

When running `unfamiliar investigate`, the app receives HTTP 429 (Too Many Requests) errors from GLM's API, even though:

1. User has a **paid GLM tier**
2. User uses GLM agents for coding tasks regularly
3. User has **never hit rate limits** when using GLM for agentic coding
4. The calls made by `unfamiliar` are **not larger or more frequent** than agentic coding calls

### Why This Doesn't Make Sense

Agentic coding sessions involve:
- Long-running conversations
- Multiple tool uses per message
- Large context windows
- Frequent back-and-forth requests

In contrast, `unfamiliar investigate` makes:
- One briefing call (~4000 tokens)
- Then mission generation calls (one per category)

If agentic coding works fine, `unfamiliar` should work fine too.

### Possible Causes

1. **GLM bug/outage** - GLM may be returning 429 erroneously during certain conditions
	1. User note: this is unlikely because the 429 error is hit consistently during all testing over multiple days in the past week.
2. **Different API endpoints** - Agentic coding might use different endpoints with different rate limits
3. **Request format issue** - Something about how LangChain formats requests to GLM might trigger a different code path
4. **Account-level issue** - Something specific to the user's account configuration
	1. Further context: user has generated a separate API key to use with unfamiliar. It does not work, nor does using the same API key used for agentic coding. The API key for agentic coding continues to work. 
5. **App-level bug** - The app might be making more calls than expected (e.g., retries counting against limits)

### Diagnostic Information Added

As of 2026-03-18, the retry logic now logs:
- Rate limit headers from 429 responses (`Retry-After`, `X-RateLimit-Reset`, etc.)
- Response body from failed requests

This should help identify what GLM is actually saying when it returns 429.

### Next Steps for Investigation

1. **Check GLM dashboard** - https://open.bigmodel.cn/ for usage stats during 429 errors
2. **Compare request formats** - Look at what agentic coding tools send vs. what LangChain sends
3. **Test with different model** - Try `glm-4` instead of `glm-4-flash` to see if behavior differs
4. **Add request logging** - Log the actual HTTP request being sent to GLM
5. **Investigate other provider-agnostic applications** - Open Code (CLI application) is language-agnostic, as is Claude Code. If these tools can use any provider, unfamiliar should be able to as well if it implements similar patterns


### Fix Applied (2026-03-18)

Added `supports_tool_calling: bool = True` to `ProviderConfig`, set `False` for GLM. `create_structured_chain` now accepts `provider_name` and checks the flag via `provider_supports_tool_calling()`. When `False`, GLM uses JSON text fallback (prompt augmentation + text parsing) instead of `response_format` / Structured Outputs API.

Also updated GLM default model from `glm-4-flash` to `glm-5`.

**Result:** Error changed from code `1302` (rate limit) to code `1113` (insufficient balance). The structured output fix worked — traceback confirmed `.create()` path, not `.parse()`. But the 1113 error revealed a **third issue**: wrong API endpoint.

### Endpoint fix (2026-03-18)

Changed GLM `base_url` from `https://open.bigmodel.cn/api/paas/v4/` to `https://api.z.ai/api/paas/v4/`.

Zhipu operates two platforms:
- **BigModel** (`open.bigmodel.cn`) — mainland China
- **Z.AI** (`api.z.ai`) — international

User's account and credits are on Z.AI. The BigModel endpoint accepted the API key but returned "insufficient balance" because the credits don't exist on that platform.

**Result:** Same 1113 error on `api.z.ai/api/paas/v4/`. Error message switched to English (confirming endpoint change) but "no resource package" persisted.

### Coding Plan endpoint fix (2026-03-18)

Changed GLM `base_url` from `https://api.z.ai/api/paas/v4/` to `https://api.z.ai/api/coding/paas/v4`.

User's Z.AI account is a **Coding Plan**. Coding tools (Claude Code, Open Code) use `https://api.z.ai/api/anthropic` — a coding-plan-specific Anthropic-compatible endpoint. The general `/api/paas/v4/` path doesn't authorize Coding Plan credits.

The OpenAI-compatible equivalent for Coding Plan subscribers is `/api/coding/paas/v4`. Since we use LangChain's `ChatOpenAI`, this is the correct endpoint.

**Needs UA testing:** Run `python3 -m unfamiliar investigate <repo>` with GLM to confirm pipeline completes.

### Current Best Theory (2026-03-18)

Phase 14 switched from JWT auth (`ChatZhipuAI`) to Bearer auth (`ChatOpenAI` via `init_chat_model`). This is confirmed working — the traceback now goes through `langchain_openai`, not `langchain_community/zhipuai`.

**But the 429 persists.** New clue from the post-phase-14 traceback:

```
langchain_openai/chat_models/base.py:1464
    raw_response = self.root_client.chat.completions.with_raw_response.parse(**payload)
```

This is the **OpenAI Structured Outputs API** — `response_format` with a JSON schema. This is a newer OpenAI API feature, distinct from regular chat completions. GLM likely has **separate rate limit quotas** for it.

Before phase 14, `ChatZhipuAI` was in `_NO_TOOL_CALLING_MODELS`, which forced the **JSON text fallback path** (no tool calling, no `response_format` — just a regular chat completion that parses the text response). Phase 14 removed `ChatZhipuAI` from the denylist, so GLM now uses `with_structured_output(schema)` → Structured Outputs API.

**Key question:** Does GLM rate-limit Structured Outputs differently from regular chat completions?

- Regular chat (agentic coding tools): works fine
- Structured output (`response_format` JSON schema): 429 immediately

This would explain why the same API key works for agentic coding but not for unfamiliar.

**Investigation plan:**
1. Try forcing JSON fallback: temporarily add `"ChatOpenAI"` to `_NO_TOOL_CALLING_MODELS` in `structured.py` and re-run. If 429 stops → structured output quota is the issue.
2. Try correct model name: `UNFAMILIAR_MODEL=glm-4.7-Flash` (from memory, this is the working model name; registry default is `glm-4-flash`)
3. Check GLM dashboard at https://open.bigmodel.cn/ for quota breakdown by API feature

**If structured output is the trigger:** Add `supports_tool_calling: bool = True` to `ProviderConfig`, set it `False` for GLM, and use that flag in `_supports_tool_calling()` in `structured.py`. This lets GLM use the JSON fallback path without affecting other providers.

### Not a Solution

- "Increase retry delays" — Doesn't help if the underlying issue is something else
	- Previous retry delays increased exponentially for a total of 7.5 minute waits, and it still did not reliably generate missions for every category
- "Switch to Anthropic" — Avoids the problem, doesn't solve it
	- I have never tested the pipeline with anthropic because of the expense. If we can get a better idea of the tokens being used when running the pipeline and determine that it's a reasonable amount, I may be willing to try
- "It's a rate limit" — By definition yes, but WHY is the real question
- "It's the JWT auth tier" — Was the leading theory for phase 14, but the 429 persists even with Bearer auth. JWT auth was still worth fixing (it was wrong regardless), but it wasn't the root cause of the 429

## Technical Details

### Current GLM Configuration

```python
# From factory.py
"glm": ProviderConfig(
    langchain_provider="openai",
    default_model="glm-5",
    api_key_env="ZHIPUAI_API_KEY",
    base_url="https://api.z.ai/api/coding/paas/v4",
    retry_base_delay=2.0,
    supports_tool_calling=False,
)
```

### Retry Behavior

- Max attempts: 5
- Base delay for GLM: 2.0 seconds
- Exponential backoff with jitter: `base * (2 ** (attempt - 1)) * jitter(0.5, 1.5)`
- Actual delays observed: 2.88s, 4.70s, 8.92s, 16.20s
	- Previous delays started at 30s and increased exponentially for 5 iterations. The effect was the same.

### LangChain Integration

**After Phase 14:** The app uses `init_chat_model(model_provider="openai", base_url=...)` via `langchain_openai`'s `ChatOpenAI`:
```python
model = init_chat_model(
    model="glm-4-flash",
    model_provider="openai",
    base_url="https://api.z.ai/api/coding/paas/v4",
    api_key=api_key,
    temperature=0.2,
    max_retries=2,
)
```

This uses Bearer auth (`Authorization: Bearer <api_key>`) — same as agentic coding tools.

**Structured output:** Since `ChatZhipuAI` was removed from `_NO_TOOL_CALLING_MODELS`, GLM now uses `with_structured_output(schema)` which sends `response_format` JSON schema (OpenAI Structured Outputs API). This is a newer feature with potentially separate quotas on GLM.

---
## Current Error Messages
Error messages after phase 15 2026-03-18:
```bash
(.venv) ➜  unfamiliar git:(007-MVP) ✗ python3 -m unfamiliar investigate ~/Documents/source/lifeLevel
Starting investigation for: /home/erinmeaker/Documents/source/lifeLevel
  Checking repository: /home/erinmeaker/Documents/source/lifeLevel
  Campaign directory: /home/erinmeaker/.unfamiliar/campaigns/lifeLevel-1fb492
  Analyzing repository...
  Connecting to LLM provider...
  Generating codebase briefing...
Have 15 module descriptions, need 30. Padding with defaults.
  Saving briefing data...
  Generating investigation missions...
Validation issues for dependencies: ['Fixed line_start in backend/requirements.txt: 0 → 1 (lines are 1-indexed)', 'Fixed line_end in backend/requirements.txt: 0 → 1 (lines are 1-indexed)', 'Fixed line_start in frontend/package.json: 0 → 1 (lines are 1-indexed)']
Validation issues for conventions: ["Broke cycle by removing prereq 'Route Naming and Structure Conventions' from 'Route Naming and Structure Conventions'"]
Validation issues for testing: ['Fixed line_start in backend/tests/test_level_calculator.py: 0 → 1 (lines are 1-indexed)', 'Fixed line_end in backend/tests/test_level_calculator.py: 0 → 1 (lines are 1-indexed)', 'Fixed line_start in backend/tests/conftest.py: 0 → 1 (lines are 1-indexed)']
  Saving mission data...
  Campaign created: 43 missions across 7 categories

Investigation ready:                                                                                                                               
  22 available to start
  21 locked (complete prerequisites to unlock)

TUI coming in Phase 14 — use 'unfamiliar campaigns' to view progress      
```

### Analysis (2026-03-18 Review)

**These are NOT bugs** — they are expected behaviors:

1. **Transient JSON error** — LLM returned malformed JSON, retry logic caught it and succeeded on retry 2
2. **Padding module descriptions** — Briefing returned 15 modules instead of 30, padding kicks in (expected)
3. **Validation warnings** — Line clamping and missing chain references are logged but handled gracefully
4. **"0/46 missions available"** — **This was confusing UX, not a bug.** The message showed `completed/total` but the word "available" suggested "ready to start". Fixed in review to show clear breakdown:
   - `19 available to start`
   - `27 locked (complete prerequisites to unlock)`

The pipeline is working correctly: 46 missions created, 19 available (root missions), 27 locked (have prerequisites).

### Line Clamping Warnings — Reduced Noise (2026-03-18)

Line clamping warnings (`Clamped line_end in X to Y`) are informational — the validation layer auto-fixes LLM hallucinations about line numbers. These are now:

1. **Logged at DEBUG level** — Not visible in normal output
2. **Prompt improved** — Added "Conservative line numbers" instruction to reduce hallucinations
3. **Valid file paths list added** — Context now includes explicit list of all valid paths the LLM may reference

Only real issues (missing files, broken chain references) now surface at WARNING level.

### LLM Hallucination Types (2026-03-18)

The validation layer catches several types of LLM hallucinations:

| Type | Example | Auto-Fixed? | Root Cause |
|------|---------|-------------|------------|
| **File path hallucination** | `backend/src/app.py` (doesn't exist) | Yes - removed from mission | LLM invents paths not in context |
| **Line number hallucination** | `line_end: 500` (file has 78 lines) | Yes - clamped to actual length | LLM guesses line counts |
| **Prerequisite hallucination** | References "The Foundation Stone" (mission doesn't exist) | Yes - removed from prereq list | LLM references missions it didn't generate |
| **Chain reference hallucination** | Chain lists "Definition Density Investigation" (doesn't exist) | Yes - chain dropped or trimmed | LLM invents mission titles for chains |

**Current errors (latest run):**
```
Validation issues for architecture: [
  "Removed nonexistent prerequisite 'Mapping the Districts' from 'Request Flow Through Layers'",
  "Removed nonexistent prerequisite 'The Foundation Stone' from 'Test Infrastructure Architecture'",
  "Removed nonexistent prerequisite 'The Gateway to the City' from 'Blueprint Wiring and Registration'"
]
Validation issues for technical_debt: [
  "Chain 'Code Complexity Analysis' references missing missions: {'Definition Density Investigation'}"
]
```

**Why this happens:** The LLM generates missions and their prerequisites/chains in a single response. It may:
1. Plan to create a mission but forget to include it
2. Reference a mission by Film Noir title that it named differently
3. Invent "logical" prerequisites that seem like they should exist

**Mitigation in place:**
- Validation removes bad references automatically
- Root missions are ensured (at least 1 per category)
- Chains are dropped if they reference missing missions
- **Self-consistency instructions added** (2026-03-18): Prompt now explicitly requires LLM to verify every prerequisite/chain reference matches a mission it created

**Potential improvements (future):**
1. Two-pass generation: missions first, then prerequisites/chains
2. Add generated mission list to prompt for prerequisite validation
3. Stricter schema validation on title_plain consistency

---

## Complete Validation Warning Reference

All warnings are auto-fixable. The pipeline continues even when warnings occur.

### File Reference Warnings (validate_file_refs)

| Warning | Cause | Auto-Fix |
|---------|-------|----------|
| `File not found: {path} (removed from mission)` | LLM hallucinated a path that doesn't exist | Remove file ref from mission |
| `Path traversal detected: {path}` | LLM tried to reference file outside repo | Remove file ref |
| `Could not read file {path}: {error}` | Binary or unreadable file | Remove file ref |
| `Fixed line_start in {path}: 0 → 1` | LLM used 0-indexed lines | Correct to 1-indexed |
| `Fixed line_end in {path}: 0 → 1` | LLM used 0-indexed lines | Correct to 1-indexed |
| `Swapped line range in {path}` | LLM had start > end | Swap values |
| `Clamped line range in {path} to {n}` | LLM start > file length | Clamp to last line |
| `Clamped line_end in {path} to {n}` | LLM end > file length | Clamp to last line |

### Prerequisite Warnings (validate_prerequisites)

| Warning | Cause | Auto-Fix |
|---------|-------|----------|
| `Removed nonexistent prerequisite '{title}' from '{mission}'` | LLM referenced mission that doesn't exist | Remove from prereq list |
| `Broke cycle by removing prereq '{title}' from '{mission}'` | Circular dependency detected | Remove one edge |
| `Trimmed prereq '{title}' from '{mission}' (depth N > max M)` | Prerequisite chain too deep | Remove deepest prereq |
| `Created root mission '{mission}' by removing prereqs: [...]` | Not enough root missions | Clear prereqs |
| `Made unreachable mission '{mission}' a root` | Mission not reachable from any root | Clear prereqs |

### Chain Warnings (_validate_chains)

| Warning | Cause | Auto-Fix |
|---------|-------|----------|
| `Normalized Film Noir title '{title}' to title_plain '{plain}'` | Chain used Film Noir title | Convert to title_plain |
| `Chain '{chain}' references missing missions: {set}` | Chain lists non-existent missions | Drop chain or trim refs |
| `Dropped chain '{chain}' (needs 2+ valid missions)` | Chain has < 2 valid missions after cleanup | Drop chain entirely |

### Log Levels

- **DEBUG:** Line clamping (`Clamped line_end in X to Y`) and 0-index corrections (`Fixed line_start/end in X: 0 → 1`) — all auto-corrected, not user-facing. Pipeline filter covers `startswith("Clamped")` and `startswith("Fixed line_")`.
- **WARNING:** Missing files, broken chains, nonexistent prerequisites, cycles — shown but auto-handled

---

## Changelog

### 2026-03-18 (UA Testing — Phase 15)
- Confirmed all validation warnings across multiple pipeline runs are LLM hallucinations, not bugs
- **Log level fix (pipeline.py):** `Fixed line_start/end 0 → 1` warnings were appearing at WARNING level despite being trivial auto-corrections like "Clamped". Extended the debug-level filter to include `startswith("Fixed line_")`. All auto-corrected number format issues now logged at DEBUG.
- Updated llm_integration_notes log level documentation to clarify both prefixes
- Updated ua_testing.md with testing results

### 2026-03-18
- Created document to capture user's context
- Added rate limit header logging to retry logic (introduced `str.append()` bug — see below)
- Fixed `_resume_initialization` missing `campaign_dir` parameter
- **Phase 14 built:** Switched from `ChatZhipuAI` (JWT) to `ChatOpenAI` via `init_chat_model` (Bearer). Confirmed working via traceback — now goes through `langchain_openai`.
- **Bug found and fixed in review:** `retry.py` exhausted-retries branch used `str.append()` — crashes with `AttributeError` on any real HTTP error that has a response body (all GLM 429s). Fixed to `+=`.
- **New theory documented:** 429 may be caused by GLM's separate quota for Structured Outputs API (`response_format`), which is now used since `ChatZhipuAI` was removed from `_NO_TOOL_CALLING_MODELS`. Investigation steps added above.
- **Structured output fix:** Added `supports_tool_calling=False` to GLM ProviderConfig, routes through JSON text fallback. Confirmed working — traceback shows `.create()` not `.parse()`.
- **Endpoint fix:** Changed `base_url` to `https://api.z.ai/api/coding/paas/v4` (user is on Z.AI Coding Plan).
- **Persister bug fixed:** `persist_briefing_data` tried to access `category_relevance`, `path`, `file_count` on `ModuleDescription` (LLM schema) — those fields don't exist. Fixed to pull structural data from `AnalysisResult.modules` (DetectedModule) instead. Added `analysis` parameter to `persist_briefing_data`.
- **Campaign overwrite fix:** Re-running `investigate` on same repo hit `UNIQUE constraint failed: briefing.campaign_id` because `_resume_initialization` re-ran full pipeline without cleaning up prior data. Fixed: `create_campaign` now deletes existing campaign (CASCADE cleans all related data) before creating fresh. Removed `_resume_initialization`. Added `CampaignRepository.delete()` method.
