# Phase 14: User Acceptance Testing Guide

## Summary
**Phase 14** replaces `ChatZhipuAI` with `init_chat_model(model_provider="openai", base_url=...)` making GLM's legacy JWT auth that routes requests to the wrong rate-limit tier.

**What was built:**
- New `LLMConfig` model in config.py for `[llm]` section overrides
- New `ProviderConfig` dataclass in factory.py with `base_url` and `api_key_env` fields
- New `get_provider_config()` function merges config overrides with registry defaults
- Single `create_chat_model()` code path using `init_chat_model`
- All providers now support `max_tokens` being config or env var (omitted when not set)
- `pyjwt` dependency removed, `langchain-openai` added
- `ChatZhipuAI` removed from structured output denylist
- 888 tests passing

## Automated Testing
No automated testing needed for this phase focuses on the provider behavior.

## Manual Testing Steps
### 1. Verify GLM Provider Works with Real API Key
```bash
export ZHIPUAI_API_KEY="your-zhipuai-key"
python3 -m unfamiliar investigate /path/to/your/repo 2>&1 |```
The cd $repo_path
    git init
    python3 -m unfamiliar investigate /path/to/your/repo 2>&1
    python3 -m unfamiliar investigate --help
    python3 -m unfamiliar investigate --help
```
    echo "GLM provider configured successfully"
    echo "  Model: $(model.model_name)"
    echo "  Model now goes through ChatOpenAI with base_url"
    echo "  API key resolution works:"
    echo "  Pipeline should work end-to-end without 429 errors"
    ```

### 2. Test with Unfamiliar API Key (Backward Compat)
```bash
export GLM_API_KEY="test-glm-key"
export ANTHROPIC_API_KEY="test-anthropic"
python3 -m unfamiliar --help
    python3 -m unfamiliar --help
    python3 -m unfamiliar --help
```
    echo "GLM help message displayed"
    echo "Provider: glm"
    echo "  Model: glm-4-flash"
    echo "  API key resolved from ZHIPUAI_API_KEY"
    ```

### 3. Verify backward compatibility with GLM_API_KEY
```bash
export GLM_API_KEY="legacy-glm-key"
python3 -m unfamiliar --help
    python3 -m unfamiliar --help
    python3 -m unfamiliar --help
```
    echo "GLM help message displayed"
    echo "Provider: glm"
    echo "  API key resolved from GLM_API_KEY (legacy fallback)"
    ```

### 4. Verify config-driven provider selection
Create a config file:
```toml
[general]
provider = "glm"
token_budget = 50000
theme = "film_noir"

# Optional: Override LLM provider defaults
# [llm]
# model = "glm-4-flash"           # Override default model
# base_url = "https://..."        # Custom API endpoint
# api_key_env = "MY_API_KEY"      # Custom env var for API key
# max_tokens = 4096               # Max output tokens
```

The echo "Config loaded"
    echo "  Provider: glm"
    echo "  Model: glm-4-flash"
    echo "  API key env: ZHIPUAI_API_KEY"
    echo "  Base URL: https://open.bigmodel.cn/api/paas/v4/"
    echo "  Max tokens: (not set, letting model decide)"
    ```

### 5. Verify `create_chat_model()` works with new GLM model
Set `ZHIPUAI_API_KEY` env var and verify the GLM now goes through `ChatOpenAI`:
```python
from langchain.chat_models import init_chat_model
from langchain_openai import ChatOpenAI

from unfamiliar.config import UnfamiliarConfig
from unfamiliar.llm.exceptions import ProviderError
from unfamiliar.llm.providers.factory import ProviderConfig

from unfamiliar.llm.providers.factory import (
    _BUILTIN_PROVIDERS,
    create_chat_model,
    get_provider_config,
    get_retry_delay,
    _resolve_max_tokens,
)

# Set the ZHIPUAI_API_KEY in environment
api_key = "zhipuai-test-key"
model = init_chat_model(
    model="glm-4-flash",
    model_provider="openai",
    base_url="https://open.bigmodel.cn/api/paas/v4/",
    api_key=api_key,
    temperature=0.2,
    max_retries=2,
)
print(f"Model created: {model}")
assert model.model == "glm-4-flash"
assert model.temperature == 0.2
assert model.max_retries == 1
```

### 6. Run a full test suite to confirm no regressions
```bash
pytest tests/ -v --tb=short 2>&1 | tail -30
```

## Phase Completion Checklist
- [x] All provider tests pass (45 new tests for new functionality)
- [x] Config tests pass (32 tests)
- [x] All 888 tests pass
- [x] No `ChatZhipuAI` references in codebase
- [x] No lint errors
- [x] `langchain-openai` in pyproject.toml
- [x] `pyjwt` removed from pyproject.toml

---

## UA Testing Results (2026-03-18)

### Issue 1: Bug in `retry.py` — `str.append()` (FIXED)

**Discovered during:** User ran `unfamiliar investigate` after phase 14.

**Symptom:** Error crashed on attempt 5 with:
```
AttributeError: 'str' object has no attribute 'append'
BriefingGenerationError: Failed to generate briefing: 'str' object has no attribute 'append' (after 5 attempt(s))
```

**Root cause:** `retry.py` lines 96-104 used `rate_msg.append(...)` to build the exhausted-retries log message. Strings don't have `.append()` — lists do. Also had a duplicate `break` statement (dead code). The bug was only triggered when the HTTP exception had a response body (all real 429 errors from GLM do).

**Why tests missed it:** The exhausted-retries test uses a synthetic exception that has no `.response.text` attribute, so `body` was always `None` and the `.append()` call was never reached.

**Fix:** Changed `rate_msg.append(...)` to `rate_msg += ...`. Removed duplicate `break`. All 888 tests still pass.

**Files changed:** `src/unfamiliar/llm/retry.py`

---

### Issue 2: GLM still returns 429 (ONGOING — needs investigation)

**Status:** Phase 14's core change IS confirmed working. The error traceback now shows `langchain_openai/chat_models/base.py` — not `langchain_community/chat_models/zhipuai.py`. JWT auth is gone. Bearer auth is in use. ✓

**But:** GLM's API still returns HTTP 429 error code `1302`:
```
{"error": {"code": "1302", "message": "您的账户已达到速率限制，请您控制请求频率"}}
```
("Your account has reached the rate limit, please control your request frequency")

The original hypothesis — that JWT auth routed to a stricter rate-limit tier — was not the full picture.

**New clue from the traceback:** The call now goes through:
```
langchain_openai/chat_models/base.py:1464
    raw_response = self.root_client.chat.completions.with_raw_response.parse(**payload)
```
This uses the OpenAI **Structured Outputs** API (`response_format` with JSON schema) rather than the older function-calling approach. This is a newer API feature. GLM's OpenAI-compatible endpoint may have separate (lower) quotas for structured output requests vs regular chat completions.

**Before phase 14:** `ChatZhipuAI` was in `_NO_TOOL_CALLING_MODELS`, forcing the JSON text fallback path (no structured output). But 429 still occurred, so this alone is not the cause.

**Key observation:** The same API key works for agentic coding but not for unfamiliar. The request type differs — agentic coding uses streaming chat completions; unfamiliar uses structured output.

**Investigation steps (for user):**

1. **Check GLM dashboard** at https://open.bigmodel.cn/ — look at usage stats and rate limit tier info during the error window. See if different API features have separate quotas.

2. **Try correct model name** — from memory, user's working GLM model is `glm-4.7-Flash`. The registry default is `glm-4-flash`. Test with:
   ```bash
   UNFAMILIAR_MODEL=glm-4.7-Flash python3 -m unfamiliar investigate /path/to/repo
   ```

3. **Try forcing JSON fallback path** — to test if structured output is the trigger, temporarily add `"ChatOpenAI"` to `_NO_TOOL_CALLING_MODELS` in `structured.py` and re-run. If 429 stops, the structured output feature is hitting a different quota. (Revert after testing.)

4. **Test with a minimal direct API call** to isolate the issue:
   ```python
   from openai import OpenAI
   client = OpenAI(api_key="YOUR_KEY", base_url="https://open.bigmodel.cn/api/paas/v4/")
   # Test 1: plain chat
   resp = client.chat.completions.create(model="glm-4-flash", messages=[{"role":"user","content":"hi"}])
   # Test 2: structured output
   from pydantic import BaseModel
   class Test(BaseModel):
       answer: str
   resp = client.beta.chat.completions.parse(model="glm-4-flash", messages=[{"role":"user","content":"hi"}], response_format=Test)
   ```
   If Test 1 works but Test 2 returns 429, structured output quotas are the issue.

## Next Steps
- Run investigation steps above
- If structured output is the trigger → add `supports_tool_calling: bool` field to `ProviderConfig` and set it `False` for GLM, then use JSON fallback path for GLM
- If plain chat also 429s → GLM account issue, may need to contact ZhipuAI support or try Anthropic
- Update `state.md` to reflect resolved/ongoing status after investigation

