# Failure Mode Investigation: `qwen14b_online_20260226_145200`

> **Date:** 2026-02-26 | **Status:** Complete

## Inputs analyzed
- Results JSONL: `evals/results/qwen14b_online_20260226_145200.jsonl`
- Policy events JSONL: `evals/results/qwen14b_online_20260226_145200.adaptive_policy.jsonl`
- Attempt artifacts under:
  - `evals/artifacts/qwen14b_online_20260226_145200/contexts/...`

## High-level counts (adaptive condition only)
- Total adaptive attempts: 60
- `success`: 23
- `tests_still_failing`: 17
- `patch_verifier_rejected`: 12
- `syntax_preflight_failed`: 8

This matches the failure buckets called out in run discussion.

## A) `patch_verifier_rejected` (12)

### What happened
- 12/12 are adaptive attempts, exactly 4 cases x 3 attempts:
  - `cd_abc221_h_26320903_hard`
  - `cd_abc221_h_40746464_hard`
  - `cd_abc222_g_45118973_hard`
  - `cd_abc222_g_45276473_hard`
- 12/12 have:
  - `patch_verifier_score = 1`
  - reasons: `substantive_change`, `misses_crash_scope`, `non_test_edit`
  - `patch_verifier_inferred_source_scope = []`
- 12/12 are runner-forced fallback diffs (no model patch), with only marker edits:
  - `# llmdebug-force-attempt-1|2|3`

### Root-cause chain
1. Patcher call fails before producing usable edits.
2. Runner force-patch is injected to keep attempt flow alive.
3. Verifier sees a non-test file edit, but no crash/source scope match.
4. Attempt is rejected before tests run.

### Why `misses_crash_scope` is systematic here
- For these cases, snapshot crash frames point to `test_buggy.py` (assertion in test harness).
- Current source-scope inference inspects imports in test files.
- ConDefects tests use `subprocess.run([sys.executable, "buggy.py"], ...)` and typically do not import `buggy`.
- Result: inferred source scope is empty, so verifier cannot credit edits to `buggy.py`.

### Error split inside these 12
- 6 attempts: immediate HTTP 400 context-limit style error:
  - `"The number of tokens to keep from the initial prompt is greater than the context length..."`
- 6 attempts: `TimeoutError: timed out` with long propose time around 480s, plus mixed stage failures (`json_schema` timeout, `json_object` 400, `plain` timeout).

## B) `syntax_preflight_failed` (8 adaptive)

### What happened
- 8 adaptive failures across 4 cases:
  - `async_cancel_swallowed_medium` (attempts 1,2,3)
  - `async_missing_await_chain_easy` (attempts 1,2,3)
  - `async_lock_across_await_hard` (attempt 1)
  - `async_taskgroup_partial_failure_hard` (attempt 1)
- 8/8 have:
  - `syntax_preflight_last_error_type = PyCompileError`
  - `syntax_preflight_failures = 2` (max cycles exhausted)

### Syntax error pattern breakdown (adaptive 8)
- `literal_backslash_n`: 3
  - Generated patch lines contain literal escaped newlines (for example `try:\\n ...`), causing invalid Python source.
- `await_outside_async`: 3
  - Model inserts `await` inside `def`, not `async def`.
- `unterminated_string`: 1
  - Example artifact includes `await asyncio.sleep(0)",` leading to broken string literal.
- `indentation_error`: 1
  - Mis-indented `if` block in `finally`.

### Important artifact signal
- In several cases, cycle 2 patch remains structurally invalid and repeats the same error class.
- This indicates retry feedback alone is not sufficient without a sanitation or constrained fix step.

## C) Timeout patcher exceptions (6 adaptive)

### What happened
- 6 adaptive attempts have `TimeoutError: timed out`.
- Cases:
  - `cd_abc222_g_45118973_hard` (attempts 1-3)
  - `cd_abc222_g_45276473_hard` (attempts 1-3)
- All 6 became `patch_verifier_rejected` due forced fallback patch flow (above).

### Correlated signals
- Very large prompt sizes in these attempts (roughly 40k to 43k chars).
- Propose duration clusters around ~480s.
- Structured output attempts show repeated failures across modes before fallback exhaustion.

## Conclusions
1. `patch_verifier_rejected` here is mainly a transport/context failure symptom, not a bad patch-quality signal.
2. `syntax_preflight_failed` is dominated by patch-text formatting corruption and async-signature mistakes.
3. Timeout errors and context-limit 400s are concentrated in a small subset of hard ConDefects cases.

## Recommended follow-up actions
1. Improve source-scope inference for subprocess tests.
   - Parse `subprocess.run([...,"buggy.py"...])` style targets as inferred source scope.
   - Status: implemented.
2. Skip force-patch marker fallback when upstream failure is transport/context.
   - Record as explicit `patcher_request_failed` instead of generating verifier noise.
   - Status: implemented.
3. Add patch sanitation guard before syntax preflight.
   - Target literal `\\n`/quote artifacts and obvious indentation corruption.
   - Status: implemented.
4. Add adaptive prompt compaction for long ConDefects contexts.
   - Status: deferred for now.
   - Rationale: this may be partially addressed by a larger context window; revisit if context/timeout failures persist.
