# External Benchmarks

This project supports **external public benchmarks** in addition to synthetic in-repo suites.

## `squad_v1_external`

- Source: SQuAD v1.1 `dev-v1.1.json`
- Homepage: <https://rajpurkar.github.io/SQuAD-explorer/>
- License: **CC BY-SA 4.0**
- License URL: <https://creativecommons.org/licenses/by-sa/4.0/>
- Attribution: Rajpurkar et al., Stanford Question Answering Dataset (SQuAD)

### How it is used

- The dataset is **not vendored** in this repository.
- It is fetched at runtime from public source URLs and converted to Entropic JSONL format.
- Conversion is deterministic for a fixed `sample_size` + `seed`.

Command example:

```powershell
entropic-evals --suite squad_v1_external --external-sample-size 250 --external-seed 1337
```

Force refresh of local cached JSONL:

```powershell
entropic-evals --suite squad_v1_external --refresh-external
```

## Notes for commercial usage

- Synthetic suites in this repo are MIT-licensed.
- `squad_v1_external` follows SQuAD's upstream **CC BY-SA 4.0** terms.
- Always keep attribution and source/license metadata in shared reports.
