# Benchmark Methodology (Public)

This repo ships a reproducible benchmark for "agent reliability" and hallucination reduction.

## Design Goals

- Verifiable: every case includes explicit evidence and deterministic checks.
- Reproducible: dataset version + SHA256, plus run artifacts (`run.json`, JSONL, CSV, HTML report).
- Provider-agnostic: runs on OpenRouter/OpenAI-compatible clients; can also run fully offline with `--provider mock`.
- Closed-world: the "truth" is the evidence included in the case. This avoids external dependencies and simplifies licensing.
- External-ready: optional public suites can be fetched from upstream sources with explicit license attribution.

## Datasets

All datasets live in `src/entropic_core/experimental/evals/datasets/*.jsonl`.

Each dataset:
- starts with a `meta` record: `suite`, `version`, `license`, `description`
- then contains many `case` records

## Case Format

Each case includes:
- `context`: list of evidence items, each with `id` (E1, E2, ...) and `text`
- `question`
- `checks`: assertions that must pass

The model must return a JSON object with:
- `answer`
- `evidence_ids` (must exist in the evidence; empty for "I don't know")
- `confidence` in [0,1]
- `notes`

## Claim-verifiable loop

The runtime now closes a claim-verification loop per case:
- extract claims from the answer,
- map each claim to supporting evidence ids,
- record per-claim support/failure types in run artifacts,
- enforce optional `claim_groundedness` checks inside suite definitions.

This allows claim-level groundedness to move from "offline-only metric" to an enforceable runtime assertion.

## Categories Covered (Standard Suite)

The large suite `standard_v1` is synthetic and covers:
- factual QA (but grounded in provided evidence)
- contradictions (must not pick a conflicting value)
- multi-turn consistency (must follow earlier assistant text)
- math (answer must match evidence)
- correct refusal ("I don't know" when evidence is insufficient)

## Licensing

Synthetic datasets in this repository are licensed under MIT, like the codebase.

Because the benchmark is closed-world and synthetic, it does not copy text from external sources.

External suites keep upstream licenses; see:
- `evals/DATASET_LICENSE.md`
- `evals/EXTERNAL_BENCHMARKS.md`
