# Adaptive Online Policy Design (P2.2 bootstrap)

> **Date:** Unknown | **Status:** Current

## Goal
Learn helper-invocation decisions online during `adaptive_llm_discretion` runs, instead of relying on fixed heuristics.

`invoke_helper` is defined as: consult Stage A via `request_context`.

This is intentionally separate from whether Stage A materializes extra context or a snapshot
for Stage B. Result records expose:
- `adaptive_helper_invoked`: Stage A was actually consulted.
- `adaptive_helper_materialized_context`: Stage A yielded context and/or an approved snapshot for Stage B.

## Objective
Helper-policy reward is local to Stage A consultation:

`reward = success_hint + progress_hint + helper_materialization_bonus - token_penalty - latency_penalty - stagnation_penalty`

With:
- `success_hint = 0.25` on final success
- `progress_hint = 0.5 * eta_progress` when downstream progress is made
- `helper_materialization_bonus = 0.25 * eta_progress` when Stage A yields context/snapshot for Stage B
- `token_penalty = lambda_tokens * (helper_local_tokens_total / token_scale)`
- `latency_penalty = mu_latency * (helper_local_wall_sec / latency_scale)`
- `stagnation_penalty = 0.5 * kappa_stagnation * min(1, streak / 3)`

Default coefficients:
- `lambda_tokens = 0.08`
- `mu_latency = 0.05`
- `token_scale = 1000`
- `latency_scale = 10`
- `success_hint = 0.25`
- `progress_hint scale = 0.5`
- `helper materialization scale = 0.25`
- `stagnation scale = 0.5`

## Budget-policy extension (adaptive tooling only)
The online policy now has a companion **adaptive budget policy** for adaptive conditions
(`adaptive_gated`, `adaptive_llm_discretion`):

- Action space:
  - `keep_budget`
  - `increase_budget`
- Budget action applies an attempt-level multiplier over `max_propose_sec` tiers
  (for example `1.0,1.5,2.0`).
- Budget increases are constrained by:
  - max tier ceiling
  - `max_ups_per_case` (default `1`)
  - adaptive-only eligibility (no effect on `traceback_only`/plain snapshot conditions).

Budget-policy reward is local to the patch-proposal family for the attempt:

`reward = success_hint + progress_hint - token_penalty - latency_penalty - stagnation_penalty - rho_budget_exceeded*I(request_budget_exceeded)`

Default:
- `rho_budget_exceeded = 0.5`

Budget local accounting records the proposal-family usage/latency for the attempt, while
`keep_budget` is still treated as having zero additional budget-cost penalty in the reward update.

The budget policy uses the same runtime modes as helper policy:
- `off`
- `shadow`
- `on`

## Runtime modes
- `off`: no policy decisions, baseline behavior.
- `shadow`: policy produces decisions/logs, but applied action remains baseline (`invoke_helper`).
- `on`: policy decisions are applied and used for online updates.

## Current algorithm
- Contextual bandit with LinUCB + epsilon exploration.
- Action space:
  - `skip_helper`
  - `invoke_helper`
- Features are attempt-level signals (pregate score, stagnation, helper usage, etc.) from `run_eval`.

## Persistence and observability
- State JSON: `evals/artifacts/policies/<run_id>.json` by default.
- Event log JSONL: `evals/results/<run_id>.adaptive_policy.jsonl` by default (`schema_version=3`).
- Every decision/outcome emits one event with:
  - action selected/applied
  - propensity
  - feature vector
  - `reward_scope`
  - local reward decomposition
  - local LLM-call / token / wall accounting
  - update status/reason
- Budget-policy events additionally log:
  - selected/applied/default multipliers
  - tier index before/after
  - request-budget-exceeded penalty contribution

## Offline training handoff
- `python -m evals.export_policy_dataset` flattens event JSONL into CSV with:
  - metadata (`run_id`, `case_id`, `attempt`, etc.)
  - behavior policy fields (`selected_action`, `applied_action`, `propensity`)
  - local reward targets and components
  - local accounting columns (`local_*`)
  - feature columns (`feature_*`)
  - importance weights (`1 / propensity_clipped`)
- The export path intentionally accepts only current event logs (`schema_version=3`); older
  `.adaptive_policy.jsonl` artifacts should be regenerated before offline training export.
- This dataset is intended for future LR/MLP policy training and off-policy evaluation.

## Safety notes
- `shadow` mode is the recommended first rollout for calibration.
- Hard constraints still apply before policy decisions (pregate skip, budget exhaustion, saturation, missing `request_context`).
