# Adaptive Solver-Rate Improvements (2026-03-06)

> **Date:** 2026-03-06 | **Status:** Complete

## Purpose & Goal

Identify practical improvements that can increase solver rate in adaptive mode, with emphasis on changes that improve solved-case yield rather than only reducing token or latency cost.

## Scientific Gap

The current adaptive loop already logs rich telemetry and uses online decision-making, but its decision space and reward shaping are still narrow relative to the real failure modes showing up in runs. In practice, many adaptive failures appear to come from helper unreliability, transport/tooling failures, patch-verifier rejection, and hard one-shot budget caps rather than from a simple lack of exploration.

## Hypothesis / Claim

Solver rate should improve most if we:

1. make helper invocation more reliable,
2. stop treating transport and formatting failures as meaningful policy feedback,
3. broaden the adaptive action space beyond binary helper/budget choices, and
4. use a denser notion of progress than only failure-signature change or new evidence IDs.

## Methodology Used For This Review

1. Inspect the adaptive execution loop in [evals/run_eval.py](./../evals/run_eval.py).
2. Inspect the helper and budget online-policy controllers in [evals/adaptive_online_policy.py](./../evals/adaptive_online_policy.py) and [evals/adaptive_budget_online_policy.py](./../evals/adaptive_budget_online_policy.py).
3. Inspect the Stage-A request contract in [evals/stage_a_contract.py](./../evals/stage_a_contract.py) and OpenAI helper/tool invocation paths in [evals/patchers/openai_compat_patcher.py](./../evals/patchers/openai_compat_patcher.py).
4. Inspect analysis surfaces in [evals/analyze_results.py](./../evals/analyze_results.py) and [evals/eval_summary.py](./../evals/eval_summary.py).
5. Sample stored result and policy-event JSONLs in `evals/results/` to compare adaptive behavior across runs.

## Architectural Snapshot

### Components

- `run_one_condition()` in [evals/run_eval.py](./../evals/run_eval.py) orchestrates retries, helper decisions, budget decisions, prompt building, patch proposal, verification, and reward logging.
- The helper policy uses LinUCB with actions `skip_helper` and `invoke_helper`.
- The budget policy uses LinUCB with actions `keep_budget` and `increase_budget`.
- Stage-A can request only `file`, `search`, and `frame` evidence, plus optional snapshot or compaction flags.
- Stage-B receives either monolithic context or staged evidence and proposes a patch.

### Data Flow

1. Run tests and collect traceback/snapshot.
2. Compute failure signature, stagnation state, pregate features, and policy features.
3. Decide whether to invoke helper and whether to increase propose budget.
4. If helper runs, request Stage-A evidence and optionally compact context.
5. Build Stage-B prompt and propose/apply patch.
6. Run syntax preflight, patch verification, and tests.
7. Compute local reward and update online controllers.

### Interfaces And Invariants

- Adaptive helper decisions are currently binary.
- Budget increases are one-tier-at-a-time and only affect OpenAI propose time.
- Progress is primarily inferred from failure-signature change and new evidence IDs.
- Default helper, compaction, and budget-up counts are tightly capped.

### Main Risks

- Useful helper attempts can be under-selected because helper cost is immediate while rescue benefit is delayed.
- Stage-A transport or payload failures can poison learning.
- Partial improvements can be misclassified as no progress.
- Budget policy can appear ineffective because it controls too little.

## Current Behavior Summary

### What Looks Promising

- In sampled runs where helper-materialized context occurs, those cases are often competitive with or better than no-helper cases.
- In `qwen14b_online_20260226_145200`, adaptive matched traceback-only overall, but helper-materialized adaptive cases solved at a stronger rate than adaptive no-helper cases.
- In `rl_tuned_new`, the few pairwise uplift cases all came from cases where helper context materialized.

### What Looks Limiting

- Several adaptive runs show many `helper_budget_exhausted` and `llm_invalid_response_no_helper` outcomes.
- Patch-verifier rejection remains a frequent terminal note after a patch is produced.
- Stored runs show much stronger evidence for helper-path bottlenecks than for budget-policy bottlenecks.
- Current analysis code expects newer schema versions than many archived adaptive runs, which makes historical comparison harder.

## Priority Improvement Ideas

### 1) Helper Reliability And Transport-Noise Handling (P0)

### Problem

Adaptive helper benefit is bottlenecked by reliability. In sampled runs, a meaningful share of adaptive failures come from:

- Stage-A request failures,
- invalid or missing Stage-A payloads,
- helper budget exhaustion after low-yield usage,
- transport failures such as `request_failed` and `http_error`.

These failures currently mix together model judgment, transport instability, and formatting fragility.

### Why This Matters For Solver Rate

When helper context materializes, it often looks directionally useful. The problem is not only helper selection; it is also helper execution quality. If helper invocation remains noisy, the online policy will learn against a distorted reward surface and underuse a potentially valuable action.

### Proposal

Implement a helper-reliability pass before deeper policy tuning:

1. Separate transport failures from semantic helper failures.
2. Harden Stage-A structured output handling.
3. Add a retry and fallback strategy specifically for helper requests.
4. Prevent transport failures from counting as strong negative policy evidence.
5. Add clearer telemetry so we can distinguish:
   - helper requested and succeeded,
   - helper requested but returned invalid payload,
   - helper requested but transport failed,
   - helper skipped by policy,
   - helper skipped by guardrail.

### Concrete Changes

#### A. Failure taxonomy hardening

- Add explicit helper failure classes such as:
  - `helper_transport_failed`
  - `helper_invalid_payload`
  - `helper_request_budget_exceeded`
  - `helper_unavailable_backend`
- Preserve existing low-level fields, but promote a normalized helper outcome field into `attempt_metadata` and `patch_meta`.

#### B. Structured-output resilience for Stage-A

- Prefer the most reliable structured response path for helper requests.
- On helper JSON parse failure, retry once with a stricter helper-only prompt and reduced verbosity.
- Add a conservative fallback parser path that can recover obvious valid JSON objects from wrapped text.
- If Stage-A still fails, mark it as helper-format failure instead of collapsing it into generic no-helper behavior.

#### C. Transport-aware retry policy

- Add a helper-specific retry budget for transient transport failures.
- Treat `ConnectionRefusedError`, retryable HTTP failures, and temporary backend unavailability as retryable helper faults.
- Avoid consuming the full helper budget on transient transport failures when no evidence was actually returned.

#### D. Reward/update hygiene

- Do not apply normal negative learning updates when helper failure is clearly transport-driven.
- Either skip the update or down-weight it with a dedicated `update_reason`, so the controller does not learn that `invoke_helper` is intrinsically bad when the backend simply failed.

#### E. Better helper accounting

- Track materialized evidence by type:
  - file evidence,
  - frame evidence,
  - search evidence,
  - snapshot included,
  - compacted context used.
- Track helper retries, helper transport retries, and helper effective success rate separately from raw invocation rate.

### Acceptance Criteria

- Adaptive runs can distinguish transport failure from helper semantic failure at the attempt level.
- Helper transport failures no longer consume learning signal the same way as clean semantic failures.
- Stage-A invalid-payload rate decreases.
- Helper materialization rate increases without indiscriminately increasing helper invocations.
- Solver rate on adaptive runs improves or, at minimum, helper rescue rate improves without regression in overall solve rate.

### 2) Expand The Adaptive Action Space (P1)

### Problem

The helper action space is binary, so the controller must choose between no help and a full helper invocation. That is too coarse for many cases.

### Proposal

Introduce intermediate helper actions:

- `frame_only`
- `file_only`
- `search_only`
- `snapshot_lite`
- `compact_then_patch`
- `retry_helper_strict_json`

This should let the controller buy smaller, cheaper pieces of evidence on ambiguous cases.

### 3) Reward Eventual Rescue, Not Only Local Efficiency (P1)

### Problem

Current reward is mostly local to the current attempt. `skip_helper` can look cheap and attractive because it avoids helper cost immediately, while `invoke_helper` pays token and latency cost up front even when it helps solve the case on the next attempt.

### Proposal

- Add delayed credit when helper use leads to success within the next one or two attempts.
- Increase reward weight for rescue on repeated-signature or hard cases.
- Keep local cost penalties, but reduce the bias toward cheap non-help on context-hungry cases.

### 4) Richer Progress Signals (P1)

### Problem

Progress is currently very coarse. A case can improve materially while keeping the same failure signature.

### Proposal

Add progress features and reward terms for:

- patch-verifier score improvement,
- syntax-preflight improvement,
- failing-test count change,
- crash-file or source-scope alignment improvement,
- movement from transport failure to semantic failure,
- reduction in patch-noop or verifier-rejection behavior.

### 5) Broaden What The Budget Policy Can Control (P2)

### Problem

The budget policy currently controls only propose time. That is too narrow if many failures are context-limited or helper-limited.

### Proposal

Let budget control optionally tune:

- `max_tokens`,
- helper-call allowance,
- Stage-A char and request budgets,
- retry-history budget,
- response mode selection for patching,
- maybe compaction allowance.

### 6) Patch-Verifier Recovery Loop (P2)

### Problem

Many adaptive attempts terminate at patch-verifier rejection even when the patch is substantive.

### Proposal

When the verifier rejects a substantive patch:

1. feed verifier reasons back into a short corrective prompt,
2. allow one cheap repair cycle before abandoning the attempt,
3. log whether verifier-guided repair changes the terminal outcome.

### 7) Difficulty-Aware Caps And Priors (P2)

### Problem

Hard and snapshot-dependent cases likely need more than the default one-helper, one-topup pattern.

### Proposal

Use case metadata to set adaptive priors or override defaults:

- more helper budget for `hard` cases,
- more helper budget for `snapshot_dependency=required`,
- more conservative helper use on easy and low-snapshot cases.

### 8) Analytics And Evaluation Hardening (P0 For Research Velocity)

### Problem

The improvement loop is partially blind because the current analyzer expects schema 5 while many real adaptive runs are older schema versions, and the summary surfaces do not expose enough decision-conditioned solver metrics.

### Proposal

- restore compatibility or migration tooling for schema-3 adaptive runs,
- report solver rate conditioned on selected and applied adaptive action,
- report helper rescue rate by action, difficulty, and snapshot dependency,
- add explicit rollups for helper transport failure, invalid payload, and verifier-repair opportunities.

## Recommended Execution Order

1. Helper reliability and transport-noise handling.
2. Reward and progress redesign.
3. Patch-verifier recovery loop.
4. Richer helper action space.
5. Broader budget-control action space.
6. Difficulty-aware priors and cap tuning.
7. Longer-horizon policy refinement after cleaner telemetry exists.

## Detailed Plan For (1): Helper Reliability And Transport-Noise Handling

### Goal

Increase solver rate by making helper invocation trustworthy enough that the controller can use it without being dominated by transport failures and malformed Stage-A responses.

### Scope

In scope:

- Stage-A helper request path
- helper transport retry handling
- helper outcome taxonomy
- helper-related reward/update treatment
- helper telemetry and analysis fields

Out of scope for the first pass:

- redesigning the full reward function
- adding new helper action types
- changing budget-policy action space
- implementing verifier-guided repair

### Working Hypotheses

1. A noticeable share of adaptive underperformance is caused by helper transport or payload failures rather than bad helper selection.
2. Excluding or down-weighting transport-driven helper failures from learning updates will reduce false pressure toward `skip_helper`.
3. Improving Stage-A structured output reliability will increase helper materialization rate and raise adaptive rescue rate.

### Proposed Workstreams

### Workstream A: Normalize helper outcomes

- Add a single normalized helper outcome field in attempt metadata.
- Distinguish at least:
  - `not_invoked`
  - `policy_skipped`
  - `guardrail_skipped`
  - `transport_failed`
  - `invalid_payload`
  - `invoked_no_context`
  - `materialized_context`

### Workstream B: Harden Stage-A request handling

- Add one helper-specific retry on invalid JSON or empty payload.
- Keep the retry prompt smaller and stricter than the first prompt.
- Capture raw helper response and parse failure reason in normalized form.

### Workstream C: Make transport failures policy-safe

- Detect retryable helper transport failures explicitly.
- Skip or down-weight the online-policy update for those attempts.
- Keep a separate counter for helper transport instability so it remains visible.

### Workstream D: Improve helper-budget accounting

- Do not count transient helper transport failures the same way as successful helper invocations.
- Revisit whether a helper call should consume budget only after a meaningful helper response arrives.

### Workstream E: Expose evaluation metrics

- Add rollups for:
  - helper transport failure rate,
  - helper invalid-payload rate,
  - helper materialization rate,
  - solver rate when helper materializes,
  - solver rate when helper fails in transport,
  - skipped-update count for helper transport failures.

### Sequenced Implementation Steps

1. Add helper outcome normalization and metadata fields.
2. Add helper transport and payload failure classification.
3. Add helper-specific retry/fallback handling in the patcher path.
4. Adjust adaptive update logic so transport faults do not produce normal learning updates.
5. Extend analysis rollups and summary output.
6. Run a short paired adaptive experiment to validate the signal.

### Evaluation Plan For (1)

### Metrics

Primary:

- adaptive solve rate,
- helper rescue rate,
- helper materialization case rate.

Secondary:

- helper transport failure rate,
- helper invalid-payload rate,
- share of `skip_helper` decisions after cleaning transport noise,
- patch-verifier rejection rate,
- token and wall-time deltas.

### Experimental Design

1. Use the same failed-case pool for baseline adaptive and helper-reliability branch runs.
2. Keep model, prompt mode, and retry count fixed.
3. Compare:
   - current adaptive,
   - adaptive plus helper reliability hardening.
4. Break down results by difficulty and snapshot dependency.

### Threats To Validity

- Backend health can move independently of code changes.
- Older result schemas make historical comparisons less clean.
- Better helper reliability may increase token spend even if solver rate improves.

### Concrete Next Steps

1. Implement helper outcome normalization in the adaptive attempt metadata path.
2. Harden Stage-A helper parsing and retry behavior in the OpenAI-compatible patcher.
3. Add transport-safe update behavior in the online-policy finalization path.
4. Extend analysis to report helper transport and invalid-payload rollups.
5. Run a short matched failed-case adaptive rerun to validate lift.
