# Adaptive Online Policy Design (P2.2 bootstrap)

## Goal
Learn helper-invocation decisions online during `adaptive_llm_discretion` runs, instead of relying on fixed heuristics.

## Objective
Per-attempt reward is:

`reward = success - lambda_tokens * (tokens_total / token_scale) - mu_latency * (attempt_wall_sec / latency_scale)`

Default coefficients:
- `lambda_tokens = 0.08`
- `mu_latency = 0.05`
- `token_scale = 1000`
- `latency_scale = 10`

## Runtime modes
- `off`: no policy decisions, baseline behavior.
- `shadow`: policy produces decisions/logs, but applied action remains baseline (`invoke_helper`).
- `on`: policy decisions are applied and used for online updates.

## Current algorithm
- Contextual bandit with LinUCB + epsilon exploration.
- Action space:
  - `skip_helper`
  - `invoke_helper`
- Features are attempt-level signals (pregate score, stagnation, helper usage, etc.) from `run_eval`.

## Persistence and observability
- State JSON: `evals/artifacts/policies/<run_id>.json` by default.
- Event log JSONL: `evals/results/<run_id>.adaptive_policy.jsonl` by default.
- Every decision/outcome emits one event with:
  - action selected/applied
  - propensity
  - feature vector
  - reward decomposition
  - update status/reason

## Offline training handoff
- `python -m evals.export_policy_dataset` flattens event JSONL into CSV with:
  - metadata (`run_id`, `case_id`, `attempt`, etc.)
  - behavior policy fields (`selected_action`, `applied_action`, `propensity`)
  - reward targets and components
  - feature columns (`feature_*`)
  - importance weights (`1 / propensity_clipped`)
- This dataset is intended for future LR/MLP policy training and off-policy evaluation.

## Safety notes
- `shadow` mode is the recommended first rollout for calibration.
- Hard constraints still apply before policy decisions (pregate skip, budget exhaustion, saturation, missing `request_context`).
