# Test Pipeline Issue: GLM Read Timeouts

**Date:** 2026-03-14
**Branch:** 007-MVP
**Status:** Pipeline works with GLM (glm-4.7-Flash), but some mission categories fail with read timeouts.

## Executive Summary

The unfamiliar pipeline **works end-to-end with GLM** using `glm-4.7-Flash`. The remaining issue is **read timeouts** on some mission category generation calls — specifically the "testing" category fails most often.

**Working command:**
```bash
GLM_API_KEY=<key> UNFAMILIAR_PROVIDER=glm UNFAMILIAR_MODEL=glm-4.7-Flash python _planning/testing_script.py
```

---

## The Problem

### Error Message
```
Failed to generate missions: The read operation timed out (category 'testing', after 5 attempt(s))
```

### What This Is NOT

- **NOT a rate limit** — Rate limits return HTTP 429 or GLM error code 1302
- **NOT fixed by longer retry delays** — GLM already uses 30s base delay (7.5+ minutes of total wait between 5 attempts)

### Post-Phase 10 Test Runs (2026-03-14)

| Run | Time (UTC) | Categories | Failed | Testing |
|-----|------------|------------|--------|---------|
| 10 | 02:00 | 7 | 4/7 | ❌ |
| 11 | 02:57 | 4 | 2/4 | ❌ |
| 12 | 03:26 | 2 | 1/2 | ❌ |
| 13 | 03:41 | 1 | 1/1 | ❌ |
| 14 | 03:52 | 1 | 0/1 | ✅ |

**Key observation:** "testing" category failed **4 consecutive runs** (20 total retry attempts across ~2 hours) before succeeding on the 5th run.

---

## Differential Analysis

### Why "Short Timeout" Theory Is Insufficient

Each failed category goes through 5 retry attempts with exponential backoff:
- Wait times: 30s → 60s → 120s → 240s → 480s
- **Total wait between attempts: ~7.5 minutes**
- Plus actual read-wait time per attempt (likely 30-60s each)
- **Total time per failed category: ~12+ minutes**

If GLM just needed more time to generate, it would succeed within one of those 12+ minutes of waiting. Yet it failed 20 consecutive retry attempts across 4 separate runs before succeeding.

### Remaining Theories

#### 1. GLM Server-Side Intermittency

**Hypothesis:** GLM's API servers have intermittent outages or degraded performance that resolves on its own, unrelated to our request timing.

**Evidence:**
- 4 consecutive failures, then success 11 minutes later
- Pattern doesn't correlate with retry delays
- Same request content succeeds eventually

**Verification:** Track response times across multiple runs to identify patterns.

#### 2. Connection State Corruption

**Hypothesis:** HTTP connection pool gets into a bad state that persists across retries but resets when creating a new model instance (each output file run).

**Evidence:**
- Each output file run creates a "fresh model" via `create_chat_model()`
- Retries within a run fail, but fresh run succeeds
- Testing category tends to fail later in category sequence

**Verification:**
- Check if ChatZhipuAI shares underlying HTTP client across instances
- Test with connection pooling disabled
- Create completely fresh HTTP client for each category

#### 3. GLM Rate Limiting Via Timeout (Not 429)

**Hypothesis:** Some APIs don't return proper 429 errors when rate-limited — they simply stall the connection until timeout. GLM may do this for certain quota situations.

**Evidence:**
- Long gaps between runs (11-57 minutes) eventually lead to success
- Retry delays within a run don't help
- Pattern matches "cooldown needed" but not "longer timeout needed"

**Verification:** Test with much longer gaps between API calls (5-10 minutes) to see if timeout rate drops.

#### 4. Request Size Triggers Different Processing Path

**Hypothesis:** Testing category context hits some edge case in GLM's infrastructure (specific token count, content pattern, etc.) that triggers a slower or failing processing path.

**Evidence:**
- Testing category fails more than others
- Testing context builder produces more structured data (test files + coverage + untested files)
- Other categories with similar context sizes don't fail as consistently

**Verification:**
- Log exact context size per category
- Test with artificially reduced testing context
- Compare token counts between successful and failed categories

---

## Recommended Investigation Order

1. **Add response time logging** — Track how long each API call takes (successful and failed)
2. **Check connection pooling** — Inspect if ChatZhipuAI shares HTTP client state
3. **Log context sizes** — Compare token counts per category to identify patterns
4. **Test with longer gaps** — Try 5-10 minute cooldowns between categories

---

## GLM Model Compatibility

| Model | Status | Notes |
|-------|--------|-------|
| `glm-4.7-Flash` | ✅ | Works; read timeouts are a separate issue |
| `glm-5` | ❌ | Error 1113 — API access may require separate tier |
| Other glm-4 variants | ❌ | Various errors (model not found, insufficient balance) |

**Recommendation:** Use `UNFAMILIAR_MODEL=glm-4.7-Flash`

---

## Key Files

| File | Role |
|------|------|
| `src/unfamiliar/llm/retry.py` | Retry with exponential backoff (30s base for GLM) |
| `src/unfamiliar/llm/providers/factory.py` | Provider factory, creates ChatZhipuAI instances |
| `src/unfamiliar/llm/missions/generator.py` | Mission generation with retry wrapper |
| `src/unfamiliar/llm/missions/context.py` | Mission context builder (testing category has most data) |
| `_planning/testing_script.py` | Manual integration test script |

---

## Open Questions

1. **HTTP connection reuse** — Does ChatZhipuAI share connection state across instances?
2. **GLM timeout behavior** — Does GLM ever use connection stalling instead of 429 for rate limits?
3. **Context size correlation** — Is there a token count threshold that triggers timeouts?
