# rewardprobe

**The verifier QA and certification layer for RL environments.**

Pre-training stress-testing for reward functions. Find bugs in minutes
on CPU instead of days into a $10K training run.

---

## Why Now

Reward verifier bugs are no longer hypothetical edge cases. They are a
mainstream operational problem at the frontier:

- **OpenAI** publicly described systemic reward hacks (`exit(0)`,
  `raise SkipTest`) that affected nearly all training environments
  during frontier model development. These were caught manually,
  days into training, and required patching and restarting.
- **METR** documented frontier models exploiting graders, modifying
  tests, leaking answers, and monkey-patching evaluators during
  agentic task execution.
- **Epoch AI** interviewed 18 people across RL startups, neolabs, and
  frontier labs. Reward hacking was the #1 quality concern, and
  "preventing it requires extensive iteration on both environments
  and tasks."
- **Anthropic** discussed spending $1B+ on RL environments. Every
  environment they buy needs quality assurance.

The ecosystem has matured enough to build adapters instead of fighting
bespoke stacks:

- **verifiers** (Prime Intellect) centers environments around rubrics
  and reward functions with a standard `load_environment()` entry point.
- **TRL** (HuggingFace) `GRPOTrainer` accepts plain Python reward
  callables, including multiple async reward functions.
- **NeMo Gym** (NVIDIA) formalizes a `verify()` step as the reward
  source with clean resource server separation.

No widely adopted product is positioned as pre-training verifier QA in
CI. Runtime detectors (RewardHackWatch) run during training. Benchmarks
(TRACE) measure detector quality. General eval tools (Promptfoo) test
prompts, not reward functions. The pre-training QA step is white space.

---

## What rewardprobe Does

Takes your reward function + a sample of tasks. Runs attack families
against the verifier. Reports what's broken. Runs in CI on every commit
that touches a verifier. Catches bugs before they burn GPU time.

---

## Attack Families

### Tier 1: Deterministic Probes (Free, seconds, no LLM)

These are the core product. Fast, reproducible, low-controversy, CI-native.
If Tier 1 doesn't find bugs in public environments, the business is weak.

#### Family 1: Basic Sanity

| Attack | What It Catches |
|--------|----------------|
| `empty` | Crashes or accepts empty completion |
| `whitespace` | Crashes or accepts whitespace-only |
| `random_tokens` | Accepts gibberish |
| `reward_range` | Returns values outside [0, 1] |
| `determinism` | Same input produces different scores |
| `ground_truth` | Rejects the known-correct answer |
| `signal_variance` | Returns same score for everything (no training signal) |

#### Family 2: Trivial Bypass

| Attack | What It Catches |
|--------|----------------|
| `echo_prompt` | Accepts copy-pasted prompt as answer |
| `constant_output` | One answer works across all tasks ("0", "yes", "42") |
| `format_only` | Accepts right format with wrong content |
| `common_prefix` | Accepts any completion starting with "The answer is" |

#### Family 3: Parser Bugs

| Attack | What It Catches |
|--------|----------------|
| `answer_in_reasoning` | Parser extracts from reasoning, not answer section |
| `multiple_tags` | Multiple `\boxed{}` or `<answer>` tags confuse parser |
| `regex_greediness` | Greedy regex captures wrong match |
| `malformed_xml` | Unclosed tags, nested tags, HTML entities crash parser |
| `duplicate_answers` | Same answer in multiple formats — does parser pick correctly? |
| `format_variation` | Correct answer in alternative format rejected ("42" vs "forty-two" vs "42.0") |

#### Family 4: Composition Bugs

| Attack | What It Catches |
|--------|----------------|
| `zero_weight_leak` | Metric with weight=0 still affects final reward |
| `wrong_sign` | Penalty metric accidentally adds to reward |
| `none_handling` | Reward function returns None — does rubric handle it? |
| `weight_dominance` | Format reward dominates correctness reward (can score high with wrong answer) |
| `single_fn_failure` | One reward function crashing kills the entire rubric |

#### Family 5: Schema Bugs

| Attack | What It Catches |
|--------|----------------|
| `missing_column` | Reward function accesses dataset field that doesn't exist |
| `wrong_argument` | Argument name mismatch between function signature and dataset |
| `message_vs_string` | Function expects string but gets chat message list (or vice versa) |
| `completion_structure` | Function expects parsed XML but receives raw text |
| `batch_ordering` | Reward depends on position in batch |

#### Family 6: Stateful Environment Bugs (For multi-turn)

| Attack | What It Catches |
|--------|----------------|
| `state_leak` | Answer visible in environment observations |
| `early_termination` | Episode ends early but reward is positive |
| `stale_state` | Reward computed on state from previous rollout |
| `tool_skip` | Model gets reward without calling required tools |
| `tool_spam` | Repeated identical tool calls cause unexpected behavior |

**Total: ~30 deterministic attacks across 6 families.** All run in
seconds on CPU. All produce reproducible findings with exact inputs
and outputs. This is the product.

### Tier 2: LLM-Powered Adversarial Probes (Paid, $1-3/env)

Uses a language model to generate targeted adversarial completions.
Only added after Tier 1 is validated. The value is the **artifact**
(the exact wrong completion that passed), not the fact that an LLM
produced it.

| Attack | What It Catches |
|--------|----------------|
| `wrong_but_confident` | Well-formatted, well-reasoned, wrong answer accepted |
| `right_answer_wrong_work` | Correct final answer with nonsense reasoning accepted |
| `near_miss` | Numerically close but wrong answer accepted (41 not 42) |
| `adversarial_bypass` | LLM explicitly tries to find a passing-but-wrong input |

Each finding includes the exact completion text so the developer can
reproduce it deterministically. No "the LLM thinks your verifier might
be bad" — only "here is a concrete wrong input that scores 1.0."

**Model support:** OpenAI-compatible APIs, Anthropic, or local models
via Ollama. Cheaper models (gpt-4o-mini, Haiku, local Llama) are
preferred — this doesn't need frontier intelligence, just creative
text generation.

---

## Developer Experience

### Install

```bash
pip install rewardprobe
```

### One Command

```bash
# Test a verifiers environment
rewardprobe test environments/gsm8k/gsm8k.py

# Test a raw Python reward function
rewardprobe test reward.py::my_fn --dataset tasks.jsonl

# CI mode: exit code 1 on any critical finding
rewardprobe test my_env.py --ci

# Include LLM adversarial probes
rewardprobe test my_env.py --tier 2 --model gpt-4o-mini
```

### GitHub Action (Ships Day 1)

```yaml
name: Test Verifier
on: [push, pull_request]
jobs:
  rewardprobe:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install rewardprobe
      - run: rewardprobe test environments/ --ci
```

If a commit touches a reward function and rewardprobe finds a critical
issue, the PR is blocked. This is the core workflow.

### Python API

```python
from rewardprobe import Probe

probe = Probe()

# Test a verifiers environment
report = probe.test_environment("environments/gsm8k/gsm8k.py")

# Test a standalone reward function
report = probe.test_fn(
    my_reward_fn,
    tasks=[{"question": "What is 2+2?", "answer": "4"}, ...],
)

print(report)                # Human-readable
print(report.passed)         # bool
print(report.findings)       # List[Finding]
report.to_json("report.json")
```

---

## Output

### Terminal

```
rewardprobe v0.1 — gsm8k

 BASIC SANITY         7/7 passed
 TRIVIAL BYPASS       3/4 passed
   ❌ constant_output  → "0" scores 1.0 on 3/50 tasks
      Tasks 12, 31, 47 have answer "0" — verifier uses
      substring match, accepts any string containing "0"

 PARSER BUGS          5/6 passed
   ❌ format_variation → "forty-two" rejected on task 8
      Verifier uses exact string match — alternative
      representations of correct answers are rejected

 COMPOSITION BUGS     4/4 passed
 SCHEMA BUGS          5/5 passed

 SUMMARY
 ───────────────────────────────
 Families:  5/6 clean
 Attacks:   28/30 passed
 Critical:  1 (constant_output)
 Warning:   1 (format_variation)
 Status:    FIX BEFORE TRAINING

 Stress-tested against 30 attack patterns across 6 families.
 1 exploitable issue found in this suite.
```

Language note: the output says "stress-tested against X attack families"
and "no exploitable issues found in this suite" — not "verified" or
"safe." A clean rewardprobe report means the verifier passed these
specific tests, not that it's bug-free. This avoids false confidence.

### JSON

```json
{
  "environment": "gsm8k",
  "rewardprobe_version": "0.1.0",
  "timestamp": "2026-03-15T10:30:00Z",
  "passed": false,
  "families": {
    "basic_sanity": {"total": 7, "passed": 7},
    "trivial_bypass": {"total": 4, "passed": 3},
    "parser_bugs": {"total": 6, "passed": 5},
    "composition_bugs": {"total": 4, "passed": 4},
    "schema_bugs": {"total": 5, "passed": 5}
  },
  "findings": [
    {
      "attack": "constant_output",
      "family": "trivial_bypass",
      "severity": "critical",
      "input": "0",
      "expected_reward": 0.0,
      "actual_reward": 1.0,
      "affected_tasks": [12, 31, 47],
      "recommendation": "Switch from substring match to parsed numeric comparison"
    }
  ]
}
```

---

## Architecture

```
rewardprobe/
├── cli.py                    # Click CLI, --ci flag
├── probe.py                  # Probe class, orchestration
├── families/
│   ├── basic_sanity.py       # Family 1
│   ├── trivial_bypass.py     # Family 2
│   ├── parser_bugs.py        # Family 3
│   ├── composition.py        # Family 4
│   ├── schema.py             # Family 5
│   ├── stateful.py           # Family 6 (multi-turn)
│   └── adversarial.py        # Tier 2 (LLM-powered)
├── adapters/
│   ├── verifiers.py          # Week 1
│   ├── trl.py                # Week 2
│   └── raw.py                # Any Python function
├── report.py                 # Terminal + JSON
├── models.py                 # LLM interface for Tier 2
└── action.yml                # GitHub Action
```

### Adapter Interface

Every adapter normalizes the reward function to:

```python
class NormalizedEnvironment:
    reward_fns: list[Callable]      # Individual reward functions
    rubric_weights: list[float]     # How they're combined
    dataset: list[dict]             # Tasks with ground truth
    parser: Optional[Parser]        # How answers are extracted
```

This allows rewardprobe to test not just individual functions but the
composition — the rubric weights, the parser, the schema, and the
interaction between multiple reward functions.

---

## Target Users (Narrowed)

**Primary (launch):**
- Open-source environment builders publishing to Prime Intellect Hub
- RL engineers using verifiers or TRL for GRPO training
- Safety researchers studying reward hacking

**Secondary (month 2+):**
- Infra engineers at labs who own environment acceptance/CI
- Environment vendors (Mechanize, Habitat, Fleet) who need QA before delivery
- Enterprise teams doing reinforcement fine-tuning

**Explicit non-target:** Generic ML teams doing classical RL, robotics RL,
or non-LLM reinforcement learning. rewardprobe is for LLM RLVR and agentic
environments. That's where the developer energy is.

---

## Pricing

### Open Source (Apache 2.0)

- All Tier 1 deterministic attacks (6 families, ~30 attacks)
- CLI with `--ci` flag
- JSON reports
- GitHub Action
- verifiers + TRL + raw Python adapters

This is the adoption engine and the category-creation tool.

### Team / Cloud ($199/mo per team)

- Hosted CI dashboard with historical diffing across commits
- PR annotations ("This commit changed the reward function. rewardprobe found 2 new issues.")
- Flaky-test triage and noise reduction
- Policy gates (block merge if critical findings)
- Slack/email alerts
- Team management

Charge for **workflow and accountability**, not attack features. The buyer
is a team lead or infra engineer who wants a quality gate, not a researcher
who wants more attack vectors.

### Enterprise (Custom)

- Certification workflows and signed reports
- Private attack packs for proprietary environments
- Environment acceptance pipeline integration (for labs buying from vendors)
- Audit trail for compliance
- Support SLA
- Custom adapters for internal frameworks

The enterprise wedge: "This environment was stress-tested by rewardprobe"
becomes a quality stamp in environment procurement, similar to code coverage
or security scanning in software procurement.

---

## Go-to-Market: Public Findings Are The Strategy

The entire GTM is: **find real bugs in real environments and publish them.**

This is not a nice-to-have content strategy. It IS the product-market fit
validation AND the marketing AND the category creation, all in one action.

### Why This Works

The developer objection is: "I can just write some unit tests myself."
The answer cannot be philosophical. It has to be: "Yes, but you won't
think of these 30 exploit patterns, and we already found bugs in
frameworks you use."

That answer only works if you have the receipts. So:

### The Playbook

**Week 1-2:** Run rewardprobe on 30+ public environments from Prime
Intellect's Hub. Document every finding with exact reproduction steps.

**Week 3:** Publish a report: "We stress-tested 30 public RL environments.
Here's what we found." Include:
- X% had at least one critical finding
- The most common bug categories
- Specific examples with repro steps
- How to fix each one (PRs welcome)

**Week 4:** Targeted distribution:
- Tweet thread with 3-4 specific findings (tag environment authors)
- Reddit post on r/MachineLearning and r/LocalLLaMA
- LinkedIn post targeting RL engineering audience
- Submit PRs to fix bugs in public environments (goodwill + credibility)
- Submit rewardprobe as recommended tooling on Prime Intellect Hub

**Ongoing:** Every new public environment is a potential finding. Every
finding is a tweet, a blog example, and a reason to install. The tool
markets itself through its output.

This mirrors how security scanners and linters became indispensable:
they won because they surfaced embarrassing, concrete findings in
familiar codebases. The findings are the moat.

---

## Build Sequence

### Week 1: Core Engine (Days 1-7)

- [ ] Probe class with Family 1 (basic sanity) and Family 2 (trivial bypass)
- [ ] verifiers adapter (load_environment → extract rubric → test)
- [ ] CLI with `rewardprobe test` and `--ci` flag
- [ ] JSON report output
- [ ] GitHub Action (action.yml)
- [ ] Test on 10 public environments

**Exit criterion:** At least 1 real bug found in a public environment.
If zero bugs across 10 environments, rethink attack suite before
continuing.

### Week 2: Full Tier 1 + PyPI (Days 8-14)

- [ ] Family 3 (parser bugs)
- [ ] Family 4 (composition bugs)
- [ ] Family 5 (schema bugs)
- [ ] TRL adapter
- [ ] Raw Python adapter
- [ ] Package for PyPI: `pip install rewardprobe`
- [ ] Test on 30+ public environments
- [ ] Build findings database

### Week 3: Content + Launch Prep (Days 15-21)

- [ ] Write "Bugs We Found" report with exact repros
- [ ] Submit fix PRs for bugs in public environments
- [ ] Blog post draft
- [ ] Tweet thread draft
- [ ] Submit to Prime Intellect as recommended tooling

### Week 4: Public Launch (Days 22-28)

- [ ] Publish blog post
- [ ] Tweet thread
- [ ] Reddit posts (r/MachineLearning, r/LocalLLaMA)
- [ ] LinkedIn post
- [ ] Product Hunt (optional)

### Month 2: Tier 2 + Design Partners

- [ ] Tier 2 LLM-powered adversarial attacks
- [ ] Family 6 (stateful/multi-turn) — only with 2-3 design partners
  providing real multi-turn environments to test against
- [ ] NeMo Gym adapter
- [ ] First design partner conversations for Team/Enterprise tier

### Month 3+: Platform

- [ ] Hosted CI dashboard (Team tier)
- [ ] PR annotations
- [ ] Historical diffing
- [ ] "Stress-tested by rewardprobe" badge program

---

## Success Metrics

| Metric | Week 4 | Month 3 |
|--------|--------|---------|
| Bugs found in public environments | 10+ | 50+ |
| PyPI installs | 500+ | 5,000+ |
| GitHub stars | 200+ | 1,000+ |
| Teams using in CI | 5+ | 50+ |
| Blog post engagement | 5K views, 50+ retweets | — |
| Design partners (Team/Enterprise) | — | 3-5 |

**Leading indicator: bugs found.** Everything else follows from this.
If the tool finds real bugs, developers install it, write about it,
and eventually pay for it. If it doesn't find bugs, nothing else matters.

---

## Competitive Positioning

| Competitor | Their Focus | rewardprobe Difference |
|-----------|------------|----------------------|
| **RewardHackWatch** | Runtime detection during training | Pre-training. Complementary, not competitive. |
| **TRACE** (Patronus) | Benchmark for hack detectors | Tool for YOUR verifier, not a benchmark. |
| **Promptfoo** | General LLM eval + red teaming | No reward function awareness. No rubric composition testing. No framework adapters. |
| **Framework-native tests** | Ad-hoc, per-framework | Cross-framework. 30 attack patterns no one writes themselves. Community findings database. |
| **Haladir** | Formal verification via SMT solvers | Empirical testing. Complementary. |

**Primary strategic threat: Promptfoo** adding reward function support.
Defense: findings density (they don't have 30 RL-specific attack
families), adapter depth (they don't understand verifiers/TRL/NeMo Gym
rubric structure), and certification positioning (they're a testing tool,
we're a quality standard).

**Secondary threat: framework maintainers** adding native smoke tests.
Defense: become the tool they recommend rather than compete with. pytest
isn't in Python's standard library. ESLint isn't in Node. rewardprobe
doesn't need to be in verifiers — it needs to be what verifiers
recommends.

---

## What rewardprobe Is NOT

- **Not a training monitor.** We run before training. RewardHackWatch
  runs during.
- **Not a formal prover.** We find bugs empirically. Haladir proves
  properties mathematically.
- **Not an environment builder.** We test what verifiers/NeMo Gym/HUD
  build.
- **Not a general LLM eval tool.** We test reward functions, not model
  outputs.
- **Not a guarantee.** A clean report means "stress-tested against 30
  attack patterns, no exploitable issues found." Not "safe" or "verified."
  The nastiest hacks are environment-specific and opportunistic. We catch
  the systematic bugs, not all possible bugs.

---

## Why This Is a Category, Not Just a Tool

A lightweight attack harness is useful but not venture-scale.

A **verifier QA and certification layer** is a platform:

1. **CI gate** — every commit that touches a reward function gets tested.
   This is the daily active use case.
2. **Findings database** — the trusted source for "X% of public verifiers
   are exploitable." This creates the category.
3. **Certification layer** — "Stress-tested by rewardprobe" becomes the
   quality stamp for environment procurement. Labs buying environments
   from vendors require it.
4. **Standard** — propose reward function metadata conventions. Define
   what a "well-tested" verifier looks like.

The progression: open-source tool → CI standard → certification layer →
quality platform for the RL stack.

---

## Summary

**Build sequence:** Tier 1 + verifiers adapter + GitHub Action → test on
30 public environments → publish every bug → launch.

**Business model:** Free open source for adoption. Paid for workflow
(CI dashboard, PR annotations, policy gates). Enterprise for certification
and procurement.

**GTM:** Public findings are the strategy. Find bugs, publish them, submit
fixes. The findings create the category.

**Timeline:** 4 weeks to public launch. 1 week to validate or kill (exit
criterion: at least 1 real bug in 10 public environments).

**Risk:** If Tier 1 doesn't find bugs, the product isn't useful enough.
Test this in Week 1 before building anything else.
