# Adaptive Failed-Case Recovery Findings (Baseline vs Adaptive)

> **Date:** 2026-03-04 | **Status:** Complete
>
> Documented from completed local result files in `results/complete/`.

## Purpose & Goal

Evaluate whether the adaptive pipeline can recover cases that the earlier baseline run did not solve.

## Scientific Gap

We had strong full-benchmark baseline performance, but needed direct evidence on a harder practical question:
"When we re-run previously unsolved cases with adaptive tooling, how many can we rescue, at what cost?"

## Hypothesis / Claim

Adaptive tooling should convert a meaningful fraction of previously unsolved cases to success without increasing overall run cost on that failed-case subset.

## Data Analyzed

- `results/complete/baseline-traceback-20260301T201521.jsonl`
- `results/complete/adaptive-online-20260304T125733.jsonl`
- `results/complete/adaptive-online-20260304T125733.adaptive_policy.jsonl`

## Methodology

1. Parse attempt-level JSONL records.
2. Collapse to final per-case outcomes (last attempt per `case_id`).
3. Compare baseline vs adaptive on overlapping case IDs (`n=157`).
4. Report effectiveness and efficiency:
   - solved cases / solve rate,
   - attempts,
   - token and wall-time totals,
   - final failure-mode composition.
5. Analyze adaptive policy traces (helper invocation, action mix, reward traces).

## Key Metrics

### Matched failed-case subset (`n=157`)

| Metric | Baseline on same 157 | Adaptive run on 157 |
|---|---:|---:|
| Solved cases | 0 | 25 |
| Solve rate | 0.00% | 15.92% |
| Attempts | 308 | 255 |
| Avg attempts/case | 1.96 | 1.62 |
| Total tokens | 2,472,696 | 1,990,787 |
| Avg tokens/case | 15,749.66 | 12,680.17 |
| Total wall time | 56,021.26s | 24,590.60s |
| Avg wall time/case | 356.82s | 156.63s |

Adaptive solve-rate confidence interval (Wilson, 95%): 11.02% to 22.45%.

### Full baseline run context

- Baseline full run: 1,221 / 1,664 solved (73.38%).
- The adaptive run here is a targeted failed-case subset run, not a full benchmark rerun.

## Findings

### F1. Recovery signal is strong and clearly positive

- Adaptive recovered **25 previously unsolved cases** out of 157.
- There were **no regressions** on the matched subset (no baseline success turned into adaptive failure in this subset because baseline had zero successes on it).
- Absolute uplift on this subset: **+15.92 percentage points**.

Interpretation: the adaptation/tooling stack is already delivering real rescue value, not just noise.

### F2. Recovery came with lower total cost on this hard subset

Compared to baseline behavior on the same cases:

- Attempts decreased by 17.2% (308 -> 255).
- Total tokens decreased by 19.5% (2.47M -> 1.99M).
- Total wall time decreased by 56.1% (56.0k s -> 24.6k s).

Interpretation: the observed gain is not a simple "spend much more to solve a few more" pattern.

### F3. Remaining failures are concentrated and actionable

Adaptive unresolved cases: 132/157.

Final-note breakdown:

- `tests_still_failing`: 62
- `patcher_request_failed`: 61
- `syntax_preflight_failed`: 9

Category concentration:

- `logic_error`: 123/132 unresolved

Interpretation: next improvements should target hard logic-error reasoning and request-budget/transport resilience.

### F4. Helper usage appears directionally useful

From adaptive run telemetry:

- Helper invoked on 42/255 attempts (16.47%).
- Attempt-level success when `invoke_helper`: 15.15%.
- Attempt-level success when `skip_helper`: 6.82%.

Case-level directional signal:

- Cases with any helper use: 10/42 solved (23.81%).
- Cases with no helper use: 15/115 solved (13.04%).

Interpretation: helper invocation is not universally beneficial, but current data suggests positive utility when used.

### F5. Request-budget failures are a major bottleneck

- Adaptive final `patcher_request_failed` cases: 61/61 were `request_budget_exceeded`.
- Cases with any request-budget failure had 0 solved outcomes in this run.

Interpretation: request-budget handling is currently a hard failure boundary for this subset.

## Validity Notes / Caveats

This is a positive outcome, but not a pure single-variable A/B:

- Baseline run and adaptive run used different `llmdebug` versions (`2.18.0` vs `2.21.0`).
- Model identifier differed (`Qwen3.5-35B-A3B-UD-Q4_K_M` vs `Qwen3.5-35B-A3B-Q4_K_M`).
- Search budget differed (`k=3` vs `k=2`).
- Adaptive configuration enabled online policy + helper gating.

Therefore, the measured uplift should be interpreted as "adaptive/tooling stack effect" on failed cases, not adaptation in isolation.

## Evaluation Plan (Next Iteration)

1. Re-run the remaining 132 failures with adjusted request-budget policy to reduce `request_budget_exceeded` terminal failures.
2. Run an ablation on the same 157 cases with identical model/version and only adaptive toggles changed.
3. Prioritize hard `logic_error` unresolved cases with targeted prompt/controller changes.
4. Keep tracking the same metrics: solve rate, attempts, tokens, wall time, and failure-note mix.

## Concrete Next Steps

1. Promote these recovery metrics into the ongoing roadmap as a "recovery milestone".
2. Schedule one focused rerun for request-budget resilience.
3. Schedule one controlled adaptive-on/off ablation for cleaner attribution.
