# Adaptive Online Learning Policy - Quick Reference

## 🎯 Core Concept

A **contextual-bandit (LinUCB)** controller that learns which helper-invocation **action** (out of 8) to take given the current **context** (50+ features), by observing **rewards** from past attempts.

## 📊 The Learning Loop

```
┌─────────────────────────────────────────────────────────────────┐
│                      DECISION-OUTCOME CYCLE                      │
└─────────────────────────────────────────────────────────────────┘

1. EXTRACT FEATURES
   • Attempt progress, helper budget, failure signals
   • Context request state, difficulty metadata
   • 50+ raw features from eval state
   ▼
2. SCALE FEATURES (profile "v1")
   • Count features: log1p normalization
   • Clip all to [-4, 4]
   ▼
3. DECIDE ACTION
   • Warm-up phase (first 200): Uniform random over 8 actions
   • Then: LinUCB greedy (95%) + ε-greedy (5%)
   • Score = exploit + α × explore
   ▼
4. EXECUTE ATTEMPT
   • Use selected action to route helper
   • Track tokens, latency, success
   ▼
5. OBSERVE REWARD
   • Success hint + progress bonus
   - Token penalty - latency penalty
   - Stagnation penalty + delayed rescue bonus
   ▼
6. UPDATE MODEL (if mode="on")
   • A_k ← A_k + x·x^T
   • b_k ← b_k + reward·x
   • Every 50 updates: save to disk
```

## 🎮 8 Actions the Policy Can Choose

| Action | Meaning |
|--------|---------|
| `skip_helper` | Don't invoke helper at all |
| `invoke_helper` | Standard invocation (baseline) |
| `frame_only` | Stack frames only (no code/search) |
| `file_only` | Only requested files |
| `search_only` | Only search results |
| `snapshot_lite` | Minimal context |
| `compact_then_patch` | Compress context first |
| `retry_helper_strict_json` | Stricter JSON validation |

## 🧠 What Gets Learned

**Per-action matrices** A_k and b_k maintain:
- **A_k**: Accumulated feature covariance (outer products of contexts)
- **b_k**: Reward-weighted feature accumulation
- **θ_k = A_k^{-1} b_k**: Estimated reward coefficients for action k

**Scoring formula**:
```
score_k = θ_k^T · x  +  α × √(x^T · A_k^{-1} · x)
          └─────┬─────┘     └──────────┬──────────┘
          exploit           exploration bonus
```

## 📈 50+ Input Features

### Attempt Progress (4)
- `attempt_frac`: current / max attempts
- `attempts_remaining_frac`: (max - current) / max
- `timed_out`: 1 if timeout exceeded
- (others)

### Helper Budget (8)
- `helper_calls_used`: absolute count
- `helper_budget_remaining_frac`: fraction left
- `helper_topup_used_frac`, `...remaining_frac`
- `helper_budget_exhausted`: binary
- `helper_saturation_guarded`: binary
- (others)

### Failure Signals (4)
- `failure_signature_repeat`: same test failure?
- `stagnation_streak`: attempts with no progress
- `adaptive_no_progress_streak`: no new evidence
- (others)

### Context Requests (10)
- `last_helper_need_more_context`: binary
- `last_helper_invalid_response`: binary
- `last_helper_declined`: binary
- `supports_context_request`: patcher capability
- `last_helper_action_executed`: encoded flags
- (others)

### Context Compaction (8)
- `context_compaction_enabled`: binary
- `supports_context_compaction`: binary
- `context_compaction_calls_used_frac`
- `context_compaction_calls_remaining_frac`
- `last_context_compaction_success`: binary
- `last_context_compaction_ratio`: 0-1 compression
- `last_context_compaction_trigger_*`: categorical
- (others)

### Case Metadata (4)
- `case_difficulty_ord`: 0 (easy) / 0.5 / 1 (hard)
- `case_snapshot_dependency_ord`: 0 / 0.5 / 1
- `case_bug_source_imported`: binary
- (others)

### Pre-gate Signals (3)
- `pregate_score`: 0-1 confidence
- `pregate_fired`: should skip?
- `pregate_type_specificity`: exception type precision

## 💰 Reward Formula

```
reward = success_hint (if success)
       + η_progress × progress_score × SCALE
       + materialization_bonus (if context used)
       - λ_tokens × (tokens_total / 1000)
       - μ_latency × (wall_sec / 10)
       - κ_stagnation × min(1, no_progress_streak/3) × SCALE
       + delayed_rescue_bonus (if resolved later)
```

**Default parameters** (eval_types.py):
- λ_tokens = 0.08
- μ_latency = 0.05
- η_progress = 0.15
- κ_stagnation = 0.10
- token_scale = 1000, latency_scale = 10

## 🔧 Configuration

```python
OnlinePolicyConfig(
    mode: "off" | "shadow" | "on"        # off=no policy, shadow=eval only, on=active
    algorithm: "linucb"                  # (only algorithm available)
    alpha: 0.8                           # Exploration bonus strength
    epsilon: 0.05                        # ε-greedy probability
    l2: 1.0                              # Regularization for matrix inversion
    warmup_decisions: 200                # Uniform sampling for first N
    save_every: 50                       # Save state after N updates
    feature_scaling_profile: "v1"        # "v1" (default) or "none"
    state_path: Path | None              # Persist model state to disk
    events_path: Path | None             # Log all decisions+outcomes as JSONL
    random_seed: int | None              # RNG seed
)
```

## 🚀 Modes

| Mode | Behavior | Use Case |
|------|----------|----------|
| `off` | Always use default action | Baseline condition |
| `shadow` | Policy selects action but default is executed; rewards observed under default | Offline evaluation without risk |
| `on` | Policy selects and executes action; actual rewards observed | Active learning (production) |

## 📊 LinUCB Algorithm Overview

1. **Maintain** A_k (d×d matrix) and b_k (d vector) for each of K=8 actions
2. **On decision** with context x:
   - Compute θ̂_k = A_k^{-1} b_k for each action
   - Score each action: UCB_k = θ̂_k^T x + α √(x^T A_k^{-1} x)
   - Select argmax_k UCB_k (greedy) or random (ε-greedy)
3. **On reward** r for action a with context x:
   - A_a ← A_a + x x^T
   - b_a ← b_a + r·x
4. **Save state** every 50 updates to enable resuming

## 📝 Key Data Structures

```python
# Decision made by policy
PolicyDecision:
  - selected_action: action chosen by LinUCB
  - applied_action: action actually executed (selected if mode="on", else default)
  - propensity: P(selected_action) — for off-policy corrections
  - scores: dict of UCB scores for all 8 actions
  - feature_values: the scaled features used
  - decision_id: UUID to link to outcome

# Outcome recorded after attempt
PolicyEvent:
  - paired with PolicyDecision (via decision_id)
  - reward: observed reward signal
  - behavior_action: what actually happened (for off-policy eval)
  - behavior_propensity: P(behavior_action) — for weighting
  - model_updated: whether A/b matrices were updated
  - [all reward components broken down]
```

## 🎯 Budget Policy (Secondary)

A **separate LinUCB instance** with just 2 actions:
- `keep_budget` — use current request budget cap
- `increase_budget` — escalate to next tier

**Configuration**:
```python
BudgetPolicyConfig(
    alpha: 0.6        # Lower than online (0.8)
    epsilon: 0.03     # Lower (0.05)
    warmup: 0         # No warm-up (vs 200)
)
```

## 📂 Key Files

| File | Purpose |
|------|---------|
| `adaptive_online_policy.py` | Core LinUCB controller + decision/outcome classes |
| `adaptive_budget_online_policy.py` | Budget escalation policy (secondary) |
| `policy_feature_scaling.py` | Feature normalization (log1p for counts, clipping) |
| `run_eval.py` | Orchestration: feature extraction, decision calling, reward calculation |
| `stage_a_contract.py` | JSON schema for helper requests (file/search/frame) |
| `analyze_results.py` | Off-policy evaluation (OPE) of target policies |

## 🔄 Integration in run_eval.py

```python
# Initialize before condition loop
online_policy_controller = AdaptiveOnlinePolicyController(config)

# Per attempt
decision = online_policy_controller.decide(
    feature_values=scaled_features,
    default_action=ACTION_INVOKE_HELPER
)
helper_action = decision.applied_action  # Use this to route helper

# After attempt, observe reward
online_policy_controller.record_outcome(
    decision=decision,
    reward=reward_value,
    success=success,
    # ... + many other parameters
)

# Before exiting
online_policy_controller.close()  # Flush state to disk
```

## 🎓 Learning Dynamics

- **Warm-up (decisions 1-200)**: Uniform random → explores all actions equally
- **Early phase**: ε-greedy with high uncertainty → balanced exploration/exploitation
- **Steady state**: Greedy arm selection (high confidence actions) with 5% random exploration
- **Uncertainty bonus**: √(x^T A^{-1} x) is **larger** when context x is new → encourages exploration of unfamiliar situations

## 💡 Key Insights

1. **Feature scaling is critical**: Prevents large features from dominating the regression; log1p handles skewed distributions (streaks, budgets)

2. **Reward decomposition helps interpretability**: Each component (success, progress, token cost, etc.) can be analyzed independently

3. **Deferred credit**: If helper materializes context but attempt fails, credit is deferred to later attempt that resolves case → encourages long-term thinking

4. **Off-policy evaluation**: Shadow mode enables safe experimentation; OPE in `analyze_results.py` estimates expected reward of alternative policies

5. **State persistence**: Models persist to disk every 50 updates → enables training across multiple eval runs

