# Adaptive Context Retrieval for LLM Debugging: Findings and Next Steps

Date: 2026-02-25
Status: Documented after four parallel research passes focused on adaptive retrieval/tool use, iterative debugging loops, evidence indexing, and context-length effects.

## Purpose

This note answers two questions from our recent runs:

1. Why does the model often request expensive helper context (snapshot) even when traceback-only may be enough?
2. Why can adding snapshot context fail to improve (or even reduce) solve rate?

It also proposes concrete interventions beyond the changes already implemented.

## What We Already Implemented (Baseline)

These are already in the pipeline:

- `--adaptive-max-helper-calls` default is now `1` (instead of unlimited).
- Adaptive retries stay `full` by default (not implicitly forced to delta mode).
- Stage-A prompt was rewritten to be cost-aware and to discourage unnecessary helper calls.
- Stage-A uses a *snapshot index* (exception + frame list only) for adaptive conditions to reduce bias from full locals/state.
- Stage-A can request full snapshot inclusion in Stage-B via `need_snapshot=true` with `snapshot_justification`.
- Attempt-1 full snapshot inclusion is denied when `snapshot_justification` is missing/generic.
- Progress-based adaptive stop logic was added for repeated no-progress trajectories.
- Evidence blocks use stable short IDs and an explicit Stage-A evidence index section.

This document focuses on what to do next beyond that baseline.

## Method (Research Tracks)

Four targeted literature sweeps were run in parallel:

1. Adaptive retrieval/tool-invocation policy design.
2. Iterative self-refinement debugging loops and retry-failure modes.
3. Evidence indexing/citation mechanisms (including short IDs).
4. Empirical studies on context length and when extra context hurts code tasks.

## Core Findings

## F0. Long-context position effects (“lost in the middle”) are real and actionable

Even when a model *can* accept long contexts, it may fail to use relevant facts that appear in the
middle of a long prompt. This is a strong candidate explanation for “snapshot provided but solve
rate does not improve”: the extra evidence can bury the key clue rather than clarifying it.

Practical implication for llmdebug-style snapshots:

- Prefer an index + short “best” snippets over raw dumps.
- Repeat the single most diagnostic evidence near the end of the prompt (recency advantage).
- Keep a stable evidence ID list so later prompts refer to IDs, not pasted text.

## F1. More context is not monotonic; noise can dominate utility

Repeated signal across SWE-style and repo-level benchmarks:

- Increasing retrieved/context tokens often raises retrieval recall but can lower final solve rate.
- Collapsing/pruning context to edit-adjacent or high-value evidence often improves outcomes over full context dumps.
- Several studies show strong degradation at long context windows in code tasks.

Implication for us: snapshot usefulness depends on selection/packaging quality, not raw volume.

## F2. Selective retrieval beats always-retrieve policies

Strong evidence supports explicit retrieve-or-not control:

- Learned or policy-driven selective retrieval reduces waste and can preserve/improve quality.
- A small top-k high-quality context set outperforms broad low-ranked retrieval.

Implication for us: model discretion alone is usually insufficient; explicit guardrails and utility gating are needed.

## F3. Iterative repair has diminishing returns without reset/diversification

Common failure pattern:

- Attempt 1 -> 2 often helps.
- Later attempts can plateau/loop unless evidence changes, hypotheses diversify, or controller resets context.

Implication for us: retries must be conditioned on measurable progress, not just remaining budget.

## F4. Cost-aware controllers are becoming a first-class design pattern

Recent agent work treats tool budget explicitly (tokens + tool calls + runtime).

Implication for us: helper invocation should be treated as an investment decision with expected-value checks.

## F4b. “Selective augmentation” and “compressed retrieval” are standard RAG mitigations

In retrieval-augmented generation, multiple works show that:

- retrieved documents should be compressed before inclusion, and
- the system should be allowed to add *nothing* when retrieval is not helpful.

This maps closely onto snapshots: we should be able to return “no additional evidence” (or only a
tiny index) when the helper context is unlikely to help.

## F5. Short evidence IDs are useful operationally, but not a standalone accuracy lever

Evidence supports citation-grounded workflows and compact referencing. Direct evidence that "short hash IDs alone" improve solve rate is limited.

Implication for us: short IDs are good infrastructure (token efficiency, traceability), but they need ranking/pruning/policy logic to improve outcomes.

## Why Our Observed Behavior Is Plausible

Observed behavior:

- The model requests helper context frequently.
- Even with helper context, solve rate does not reliably increase.

Research-consistent explanation:

- LLMs are biased toward asking for more evidence when uncertainty is high unless the policy penalizes low-value expansion.
- Added context can bury the true signal (localization dilution), increasing reasoning error despite more information.

## What To Do Next (Beyond Current Baseline)

## P0 (Immediate, low-risk)

1. Add a mandatory "helper justification" contract.
- Before helper call, model must emit:
  - missing fact,
  - expected decision impact,
  - max evidence budget request.
- Reject helper request if justification is generic or non-falsifiable.

2. Introduce a two-stage helper path.
- Stage H1 (cheap): request only compact index + top suspicious snippets.
- Stage H2 (expensive snapshot chunks): allowed only if H1 did not resolve blocking uncertainty.

3. Add a no-new-evidence retry block.
- If next attempt does not introduce new evidence IDs or a changed hypothesis, force stop or forced diversification.

4. Add stricter helper-call quota schedule.
- Keep default cap at 1, but also disallow helper on attempt 1 unless failure pattern is high-confidence "needs runtime/state" class.

5. Add explicit helper budget awareness in the prompt.
- Always show remaining helper-call budget and remaining context budget (“Budget Tracker”-style).
- Include a rule: “If you spend the last helper call, you must propose a final patch or stop.”

## P1 (High impact, moderate effort)

1. Train/fit a lightweight invocation policy from our run telemetry.
- Target: predict positive marginal utility of helper call.
- Features:
  - traceback entropy/signature type,
  - localization confidence,
  - prior attempt delta,
  - evidence novelty,
  - token budget remaining.

2. Implement evidence ranking by expected patch utility.
- Rank chunks by:
  - crash proximity,
  - executed-line overlap,
  - suspiciousness (SBFL-style),
  - reference reuse in successful historical fixes.
- Enforce top-k and token caps per class.

3. Add diversity-controlled retries.
- Retry N+1 must differ along one axis:
  - localization hypothesis,
  - patch strategy,
  - evidence subset.
- Prevent same-strategy loops.

4. Add a “fresh start” replan after decay signals.
- Reset conversational history.
- Keep only a structured state object (failure signature, confirmed facts, rejected hypotheses, top evidence IDs).
- Re-run Stage-A with this compact state to avoid compounding noise.

## P2 (Experimental, should be A/B tested)

1. Add a small "counterfactual controller" arm.
- On helper request, run a cheap shadow decision: "proceed without helper" vs "with helper-lite".
- Use online outcome signals to calibrate invocation policy.

2. Add context-collapsing summaries for old evidence.
- Keep raw chunks out of later attempts unless re-requested.
- Carry forward only compact state + IDs + unresolved constraints.

3. Add failure-mode classifier for routing.
- Route failures into classes (logic, off-by-one, API misuse, stateful async, data-shape, etc.).
- Use class-specific helper policies and budgets.

## Recommended Evaluation Plan (Next 1-2 Iterations)

Run an intervention matrix over the same case set and seeds:

1. Current baseline (already implemented).
2. Baseline + mandatory helper justification.
3. Baseline + two-stage helper (H1/H2).
4. Baseline + no-new-evidence retry block.
5. Baseline + (2)+(3)+(4).

Track at least:

- `solve_rate`, `solve_rate_delta_vs_traceback_only`
- `helper_invocation_case_rate`
- `helper_rescue_rate`
- `helper_success_when_invoked`
- `mean_tokens_total`, `mean_tokens_per_attempt`
- `mean_llm_calls_per_case`
- `no_progress_stop_rate`
- solve-on-attempt distribution

Success criterion for rollout:

- Non-inferior or improved solve rate vs current baseline,
- materially lower helper invocation and token cost,
- improved helper rescue precision (helper used less often, but with higher payoff when used).

## Critical Risks To Watch

1. Over-constraining helper usage may block genuinely hard cases.
- Mitigation: allow controlled override on specific failure classes.

2. Over-pruning context may remove key causal detail.
- Mitigation: preserve reversible retrieval path from ID -> full chunk.

3. Policy overfit to one model family (for example Qwen-only behavior).
- Mitigation: cross-model validation (at least one additional model family).

## References (Primary)

- Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni, Liang. "Lost in the Middle: How Language Models Use Long Contexts." TACL (2024). arXiv:2307.03172. https://arxiv.org/abs/2307.03172
- Hsieh et al. "Found in the middle: Calibrating Positional Attention Bias Improves Long Context Utilization." Findings of ACL (2024). https://aclanthology.org/2024.findings-acl.890/
- Xu, Shi, Choi. "RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation." arXiv:2310.04408 (2023). https://arxiv.org/abs/2310.04408
- Asai et al. "Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection." arXiv:2310.11511 (2023). https://arxiv.org/abs/2310.11511
- Schick et al. "Toolformer: Language Models Can Teach Themselves to Use Tools." arXiv:2302.04761 (2023). https://arxiv.org/abs/2302.04761
- Yao et al. "ReAct: Synergizing Reasoning and Acting in Language Models." arXiv:2210.03629 (2022). https://arxiv.org/abs/2210.03629
- Shinn et al. "Reflexion: Language Agents with Verbal Reinforcement Learning." arXiv:2303.11366 (2023). https://arxiv.org/abs/2303.11366
- Madaan et al. "Self-Refine: Iterative Refinement with Self-Feedback." arXiv:2303.17651 (2023). https://arxiv.org/abs/2303.17651
- Yang et al. "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering." arXiv:2405.15793 (2024). https://arxiv.org/abs/2405.15793
- Chen et al. "SWE-Exp: Experience-Driven Software Issue Resolution." arXiv:2507.23361 (2025). https://arxiv.org/abs/2507.23361

- Repoformer (ICML 2024): https://proceedings.mlr.press/v235/wu24a.html
- RepoCoder (EMNLP 2023): https://aclanthology.org/2023.emnlp-main.151/
- SWE-bench (ICLR 2024 / arXiv): https://arxiv.org/abs/2310.06770
- CodeRAG-Bench (NAACL Findings 2025): https://aclanthology.org/2025.findings-naacl.176/
- Lost in the Middle (TACL 2024): https://aclanthology.org/2024.tacl-1.9/
- LongCodeBench (arXiv 2025): https://arxiv.org/abs/2505.07897
- SWE-Pruner (arXiv 2026): https://arxiv.org/abs/2601.16746
- Impact-driven Context Filtering / CODEFILTER (COLM 2025): https://openreview.net/forum?id=0Y2zXLFBji
- Self-RAG (arXiv 2023): https://arxiv.org/abs/2310.11511
- RECOMP (arXiv 2023): https://arxiv.org/abs/2310.04408
- Budget-Aware Tool-Use (arXiv 2025): https://arxiv.org/abs/2511.17006
- Revisit Self-Debugging with Self-Generated Tests (ACL 2025): https://aclanthology.org/2025.acl-long.881/
- Rethinking Repetition Problems in Code Generation (ACL 2025): https://aclanthology.org/2025.acl-long.48/
- Evaluating AGENTS.md (arXiv 2026): https://arxiv.org/abs/2602.11988
- Debugging Effectiveness Decay (Scientific Reports 2025): https://www.nature.com/articles/s41598-025-27846-5

## Notes

Some 2025-2026 items are preprints; treat them as directional evidence until broader replication is available.
