# z0 Rescue Findings (March 1, 2026)

## Scope

This note summarizes the observed behavior for:

- Baseline traceback-only run: `results/z0/z0.jsonl`
- Adaptive rescue run: `results/z0/z0-rescue-20260301T134913.jsonl`
- Case lists generated from baseline:
  - `results/z0/failing_cases.traceback_only.txt`
  - `results/z0/rescue_cases.traceback_only.failed_or_attempt_ge2.txt`

## Run Context

- Baseline `z0`:
  - `condition=traceback_only`
  - `transport_fail_fast_streak=12`
  - Recorded `patch_meta.transport_retry_limit=2` (older retry profile in that run)
- Rescue `z0-rescue-20260301T134913`:
  - `condition=adaptive_llm_discretion`
  - `transport_fail_fast_streak=5`
  - `openai_transport_max_retries=6`
  - `openai_transport_retry_backoff_base_sec=1.0`
  - `openai_transport_retry_backoff_max_sec=8.0`

## Baseline Summary (`z0`)

- Terminal units: `1000`
- Success: `784` (`78.4%`)
- Failure: `216` (`21.6%`)
- Terminal notes:
  - `success`: `784`
  - `tests_still_failing`: `133`
  - `patcher_request_failed`: `83`
- Terminal `patcher_request_failed` details:
  - `83/83` are `patcher_request_failure_kind=transport`
  - `83/83` are `patcher_request_failure_error=request_budget_exceeded`
  - Concentrated in ConDefects, `snapshot_dependency=helpful`

## Rescue Selection and Coverage

- Rescue list (`failed-or-attempt-ge2`): `313` cases
- Rescue run reached terminal state for: `111` cases
- Rescue run stopped early due to fail-fast (`transport_fail_fast_streak=5`)

## Rescue Summary (`z0-rescue-20260301T134913`)

- Terminal units: `111`
- Success: `32` (`28.83%`)
- Failure: `79`
- Terminal notes:
  - `tests_still_failing`: `40`
  - `patcher_request_failed`: `39`
  - `success`: `32`

### Terminal failure-mode detail

- Terminal `patcher_request_failed`: `39/111`
- `39/39` are:
  - `patcher_request_failure_kind=transport`
  - `patcher_request_failure_error=request_budget_exceeded`
- All terminal `patcher_request_failed` occurred at attempt `3`.
- Category/source concentration of these 39 terminal transport failures:
  - `logic_error`: `33`
  - `type_coercion`: `6`
  - `bug_source=condefects`: `39`
  - `snapshot_dependency=helpful`: `39`

### Attempt-level failure telemetry (rescue run)

- Attempt rows with `patcher_request_failed=true`: `122`
- `122/122` are `request_budget_exceeded`
- Tokens/latency profile for those 122 attempts:
  - `tokens_total == 0`: `59` (`48.4%`)
  - `tokens_total` median: `1940`, p95: `16366`, max: `30491`
  - `llm_call_count` median: `4`, p90: `7`, max: `13`
  - `propose_duration_sec` median: `200.99`, p90: `228.97`, max: `253.55`

Interpretation: the dominant failure mode is propose-budget exhaustion on hard cases, not a connection-reset crash signature.

## Confirmed Fail-Fast Trigger Window

First 5-consecutive terminal transport failures (triggering streak `5`):

1. `cd_abc265_b_45498956_easy` (attempt 3, `patcher_request_failed`)
2. `cd_abc267_g_35931814_hard` (attempt 3, `patcher_request_failed`)
3. `cd_abc265_b_46016041_easy` (attempt 3, `patcher_request_failed`)
4. `cd_abc267_f_45545460_hard` (attempt 3, `patcher_request_failed`)
5. `cd_abc268_e_45307852_hard` (attempt 3, `patcher_request_failed`)

Each had `patcher_request_failure_error=request_budget_exceeded`, `effective_timeout_sec=0`, and `remaining_budget_sec=0` at failure.

## Paired Comparison: Rescue vs Baseline on Processed Rescue Cases

Paired set: the 111 rescue-terminal cases compared against their traceback-only outcome in baseline.

- Baseline success on paired set: `36`
- Rescue success on paired set: `32`
- Net delta on paired set: `-4`

Transition counts:

- `success -> success`: `25`
- `fail -> success`: `7`
- `success -> fail`: `11`
- `fail -> fail`: `68`

## Failed-Only View (Primary Rescue Lens)

Among baseline traceback-only failures that were processed before rescue fail-fast:

- Baseline failures in paired set: `75`
- Recovered by rescue (`fail -> success`): `7`
- Rescue gain on this slice: `7/75 = 9.33%`

Relative to all baseline traceback failures (`216` total), current recovered count is:

- `7/216 = 3.24%`

Recovered case IDs:

- `cd_abc238_a_45742740_easy`
- `cd_abc244_d_45311340_easy`
- `cd_abc250_d_45700046_medium`
- `cd_abc257_a_45802027_easy`
- `cd_abc264_c_45221303_medium`
- `cd_abc265_b_45487693_easy`
- `cd_abc266_c_45028964_medium`

## Practical Conclusion So Far

- Rescue is positive on the failed-only subset that was actually processed (`+7` recoveries out of 75 prior failures).
- Overall paired net is negative because the rescue selection also included unstable baseline successes (`attempt>=2`) and some regressed.
- The dominant blocker in rescue is concentrated `request_budget_exceeded` failures in the ConDefects tail, which interacts strongly with fail-fast streak behavior.

## Protocol Decision (Next Runs)

Adopt this evaluation protocol going forward:

1. Run a traceback-only baseline pass.
2. Build rerun list from traceback-only terminal failures only (`--mode failed`).
3. Run adaptive tooling (`adaptive_llm_discretion`) only on that failed list.
4. Keep online policy learning enabled on the adaptive pass (`--adaptive-online-policy-mode on`).

Rationale:

- This preserves the observed failed-only rescue signal.
- It avoids mixing in unstable baseline-success cases that can make paired net uplift look negative.
- It keeps adaptive policy learning focused on genuinely hard/failed trajectories.
