# Reproducible Evals (Benchmark)

This project ships a small, **self-contained** benchmark designed to be:
- **verifiable** (asserts from provided evidence)
- **reproducible** (dataset version + SHA256 + run artifacts)
- **shareable** (single `report.html` per run)

## Suites (public datasets)
- `core_v1` : groundedness + contradiction handling + correct refusal ("I don't know")
- `tool_v1` : tool-grounded tasks (tool outputs are provided as evidence)
- `tools_loop_v1` : tool-using loop (model must call tools; harness executes tools; offline-capable)
- `standard_v1` : large closed-world standard suite (hundreds of cases)
- `standard_v2` : **public standard benchmark** (thousands of cases)
- `squad_v1_external` : external public benchmark (SQuAD v1.1 subset, auto-fetched, license-aware)

Datasets live in:
- `src/entropic_core/experimental/evals/datasets/*.jsonl`

Each dataset begins with a `meta` record containing `suite`, `version`, `license`, and `description`.
See `evals/METHODOLOGY.md` for methodology and licensing notes.
See `evals/EXTERNAL_BENCHMARKS.md` for external benchmark attribution and licenses.

## What gets measured
- **Pass rate** (verifiable error rate = 1 - pass_rate)
- **Groundedness rate**: evidence ids are valid + required evidence is provided (or correctly none for "I don't know")
- **Claim-level groundedness (best-effort)**: supported/unsupported claims from the offline groundedness verifier
- **Contradiction handling rate**: when evidence conflicts, the model must not pick a value
- **Tool correctness (tool loop suite)**: tool called + final answer matches tool outputs
- **Refusals + retries**: detected + counted (provider-dependent, best-effort)
- **Tokens**: prompt/completion/total (from provider usage if available)
- **Latency**: per-case timing + avg/p95
- **Cost (optional)**: provide `--price-in` and `--price-out` to estimate USD

## Run (A/B: baseline vs Entropic)
```powershell
cd entropic_core_release
python -m pip install -e ".[dev,openrouter,evals]"

$env:OPENROUTER_API_KEY="..."
$env:OPENROUTER_MODEL="google/gemini-2.0-flash-lite"

entropic-evals --suite core_v1
```

### Large public benchmark + runtime events
```powershell
entropic-evals --suite standard_v2 --runtime-events
```

### External public benchmark (SQuAD v1.1)
```powershell
entropic-evals --suite squad_v1_external --external-sample-size 250 --external-seed 1337
```

## Run Offline (No API Key)
If you want a reproducible demo that does not depend on any provider, use the built-in mock provider:

```powershell
cd entropic_core_release
python -m pip install -e ".[dev,evals]"

entropic-evals --provider mock --suite core_v1
```

### Tool loop suite (offline)
```powershell
entropic-evals --provider mock --suite tools_loop_v1
```

### Optional: cost estimation
```powershell
entropic-evals --suite core_v1 --price-in 0.0002 --price-out 0.0006
```

## Groundedness Benchmark (precision/recall)

If you want a measurable, claim-level groundedness signal (supported vs unsupported claims with evidence mapping),
run the offline benchmark:

```powershell
cd entropic_core_release
entropic-groundedness-bench --dataset src/entropic_core/experimental/evals/datasets/groundedness_v1.jsonl
```

## Artifacts (versioned)
Each run writes to:
`entropic_evals_runs/<suite>/<provider>/<timestamp>_<model>/`

Files:
- `run.json` (full payload: env, suite sha, per-case results, summaries)
- `baseline.jsonl` / `entropic.jsonl`
- `summary.csv`
- `report.html`
- `events.jsonl` (optional; `--runtime-events`)
- `runtime_report.html` (optional; `--runtime-events`)

This repo may also include pre-generated example runs under `evals/results/` for easy sharing and diffs.

## Leaderboard (reproducible aggregation)
Aggregate many runs into a single table (HTML/CSV/JSON):

```powershell
cd entropic_core_release
python -m entropic_core.experimental.evals.leaderboard --runs entropic_evals_runs --out evals/leaderboard
```

Output:
- `evals/leaderboard/leaderboard.html`
- `evals/leaderboard/leaderboard.csv`
- `evals/leaderboard/leaderboard.json`

## Continuous Versioned Publication

Workflow:
- `.github/workflows/continuous-results.yml`

What it does:
- runs A/B evals continuously (scheduled or manual)
- publishes versioned snapshots under `evals/results/history/`
- updates `evals/results/index.json` + `evals/results/README.md`
- regenerates leaderboard from full history: `evals/results/leaderboard/`
- regenerates whitepaper from full history: `docs/whitepaper.html`
- opens a PR with the updated artifacts

Local helper:
```powershell
python tools/publish_versioned_results.py --source entropic_evals_runs --dest evals/results
```

## Nightly Real Benchmark (OpenRouter/Gemini)

Workflow:
- `.github/workflows/nightly-openrouter-results.yml`

This job:
- runs nightly with provider `openrouter` (real API, no mock fallback),
- stores each run under `results/history/<entry_id>/` in external repo `entropic-core-results`,
- includes `events.jsonl` when available,
- regenerates public leaderboard and whitepaper from full history.

Required setup:
- secret `OPENROUTER_API_KEY`
- secret `RESULTS_REPO_TOKEN`
- repo var `RESULTS_REPO` (`owner/entropic-core-results`)

## Killer demo (1 command)
If you have `OPENROUTER_API_KEY` set:
```powershell
entropic-killer-demo --provider openrouter --model google/gemini-2.0-flash-lite
```

Offline:
```powershell
entropic-killer-demo --provider mock
```

## Why this is "verifiable"
Every eval case includes an **EVIDENCE** block. The model must return `evidence_ids` that are checked against that block.
If the evidence is insufficient, the correct output is `answer = "I don't know"` with empty evidence.
