# Evals

NoiseGate uses [promptfoo](https://promptfoo.dev) to evaluate triage quality against real HackerOne disclosed reports. The eval dataset includes both accepted (`resolved`) and rejected (`not-applicable`, `informative`) reports, testing for both false negatives and false positives.

## Prerequisites

- Node.js (for promptfoo)
- NoiseGate installed with eval dependencies: `pip install -e ".[evals]"`
- An API key configured: `noisegate llm`
- Playwright browsers installed: `playwright install chromium`

## Generate the dataset

```bash
python evals/generate_dataset.py
```

This downloads the [HackerOne disclosed reports dataset](https://huggingface.co/datasets/Hacker0x01/hackerone_disclosed_reports) from HuggingFace, samples test cases, and imports real program definitions from HackerOne for each program in the sample. Program definitions are saved to `evals/programs/` and cached across runs.

Options:

- `--count N` — number of test cases (default: 10)
- `--seed N` — random seed for reproducible sampling (default: 42)

## Run evals

```bash
cd evals
npx promptfoo@latest eval
```

## View results

```bash
npx promptfoo@latest view
```

## What the evals measure

Each test case runs the full triage pipeline against a real program definition and asserts:

- **Schema validity** — output is well-formed JSON with non-empty check reasons
- **Verdict alignment** — `resolved` reports should not get FAIL; `not-applicable`/`informative` reports should not get PASS
