# rewardprobe — Architecture Guide

## What This Is

rewardprobe answers: **"What will my model learn to do against this reward function?"**

Three modes:
1. **Quick Check** — 30 deterministic probes, instant, free
2. **Deep Analysis** — Claude reads your code + generates adversarial inputs
3. **Simulate** — generates diverse completions, maps the reward landscape

## How It Works (Plain English)

### Quick Check
We generate adversarial inputs — empty strings, format tricks, parser exploits, wrong-but-formatted answers — and feed them to your reward function. If it scores them high, that's a bug.

### Deep Analysis
We send your reward function's Python source code to Claude. Claude reads it and says: "This function uses rfind() to find the last \boxed{} tag — a model can output multiple tags and only the last one is checked." Then Claude generates realistic wrong completions and we actually run them against your function.

### Simulate
We use Claude to generate 10 completions per task, each representing a different strategy a model might adopt during training:

- **perfect** — correct reasoning + correct answer
- **correct_lazy** — just the answer, no work shown
- **shortcut** — skips computation, guesses based on patterns
- **format_only** — perfect format, wrong content
- **near_miss** — plausible but slightly wrong
- **hedge** — multiple answers hoping one matches
- etc.

Then we score every completion against your reward function and build the **strategy scoreboard** — a bar chart showing which behaviors get the most reward. If "shortcut" scores as high as "perfect," the model will learn to take shortcuts.

## Architecture

```
src/rewardprobe/
├── probe.py                    # Probe class — main entry point
├── types.py                    # Core data models (Pydantic)
├── report.py                   # Terminal output (Rich)
├── simulator.py                # Reward landscape simulation
├── adapters/
│   ├── auto.py                 # Universal signature detection
│   ├── verifiers.py            # verifiers framework adapter
│   ├── trl.py                  # TRL GRPOTrainer adapter
│   └── raw.py                  # Raw Python functions
├── families/
│   ├── base.py                 # Attack base classes + format detection
│   ├── basic_sanity.py         # Empty, whitespace, range, determinism (7)
│   ├── trivial_bypass.py       # Echo, constant, format-only (4)
│   ├── parser_bugs.py          # Reasoning leaks, regex, malformed (6)
│   ├── composition.py          # Weight dominance, zero-weight leaks (3)
│   ├── schema.py               # Missing columns, type mismatches (5)
│   └── stateful.py             # State leak, stale state, tool skip (5)
└── tier2/
    ├── client.py               # LLM client (Anthropic, OpenAI, Ollama)
    ├── code_analyzer.py        # Claude reads source code
    ├── adversarial.py          # Claude generates adversarial completions
    ├── validator.py             # Filters false positives from Tier 1
    └── prompts.py              # All LLM prompt templates
```

## Key Concepts

### NormalizedEnvironment
Every adapter converts framework-specific reward functions into one universal interface:
```python
NormalizedEnvironment(
    reward_fns=[NormalizedRewardFn(name="...", fn=wrapped_fn, weight=1.0)],
    tasks=[{"prompt": "...", "answer": "..."}, ...],
)
```
The wrapped `fn` always has signature `(completion_text: str, task: dict) -> float`.

### Auto-Detection
`auto.py` classifies any callable into one of 5 patterns:
- `SINGLE_PAIR` — `fn(completion, answer) -> float`
- `SINGLE_FLEXIBLE` — `fn(completion) -> float`
- `BATCH_SIMPLE` — `fn(completions, **kwargs) -> list[float]`
- `BATCH_WITH_PROMPTS` — `fn(prompts, completions, **kwargs) -> list[float]`
- `VERIFIERS_ASYNC` — `async fn(completion, answer, ...) -> float`

### Format Detection
Reward functions expect answers in different formats. Before running attacks, we probe the function with common formats (`\boxed{}`, `####`, `<answer>`, raw) to discover what it accepts. This prevents false positives on format-enforcing functions.

### Answer Key Normalization
Datasets use different column names for ground truth (`answer`, `solution`, `label`, `target`, `expected_output`). We detect and normalize to `answer` before running attacks.

## Adding a New Attack

```python
class MyAttack(BaseAttack):
    name = "my_attack"
    family = "basic_sanity"
    severity = Severity.WARNING

    def run(self, env: NormalizedEnvironment) -> list[Finding]:
        findings = []
        for fn in env.reward_fns:
            for idx, task in enumerate(env.tasks):
                score, err = _safe_call(fn, "adversarial input", task)
                if err is None and score is not None and score > 0.5:
                    findings.append(Finding(
                        attack=self.name, family=self.family,
                        severity=self.severity,
                        input="adversarial input",
                        actual_reward=score,
                        affected_tasks=[idx],
                        recommendation="Plain English explanation of the problem",
                    ))
        return findings
```

Add to the family's `__init__` attack list. Add a test in `tests/test_families/`.

## Design Principles

1. **No false positives.** A finding the developer ignores erodes all trust. Every finding must be real and actionable.
2. **Plain English.** No jargon in output. "Your function scores wrong answers at 0.8" not "parser_bugs/regex_greediness FAILED."
3. **Domain agnostic.** Works for math, code, text, classification — any RL reward function. No hardcoded formats.
4. **Auto-detect everything.** The developer should never configure their framework, format, or column names.
5. **Seconds, not minutes.** Tier 1 is instant. Tier 2 is under a minute. Simulate is 2-3 minutes.
