# Prompt Evals

Prompt evals validate prompt behaviour offline and deterministically using
a mock provider. No real model calls are made. Every invocation produces
identical results for the same inputs.

## What Are Prompt Evals?

Prompt evals run a set of test cases against a prompt template. Each case
provides structured inputs, expected output properties, and optional budget
constraints. The mock provider returns deterministic, hash-based outputs
so that suites can be executed in CI without network access or API keys.

## Suite Format (prompt-eval-suite/1.0)

```json
{
  "schema_version": "prompt-eval-suite/1.0",
  "suite_id": "unique-suite-name",
  "prompt_id": "builder-implement-feature",
  "description": "Human-readable description",
  "cases": [
    {
      "case_id": "case-unique-id",
      "description": "What this case tests",
      "input": { "task_description": "...", "context": {} },
      "expected": {
        "contains_patterns": ["def ", "return"],
        "required_keys": ["implementation", "explanation"]
      },
      "comparison": "prompt_structural",
      "budget_expectation": { "max_tokens": 2000 }
    }
  ]
}
```

### Fields

| Field | Required | Description |
|-------|----------|-------------|
| `schema_version` | yes | Must be `prompt-eval-suite/1.0` |
| `suite_id` | yes | Unique identifier for the suite |
| `prompt_id` | yes | Which prompt template is under test |
| `cases` | yes | Array of eval cases (min 1) |
| `cases[].case_id` | yes | Unique case identifier |
| `cases[].input` | yes | Structured input for the prompt |
| `cases[].expected` | no | Expected output constraints |
| `cases[].comparison` | no | Comparison strategy (default: `prompt_structural`) |
| `cases[].budget_expectation` | no | Token budget constraints |

### Checks

- **required_keys**: output JSON must contain these top-level keys.
- **contains_patterns**: output text must contain these substrings (case-insensitive).
- **budget_expectation.max_tokens**: estimated tokens must not exceed this limit.

## Golden Workflow

1. **Create suite** -- write a `prompt-eval-suite/1.0` JSON file in
   `evals/prompt_suites/`.
2. **Run** -- execute the suite offline:
   ```bash
   python -m abi_evals.cli prompt-eval --suite evals/prompt_suites/my_suite.json
   ```
3. **Verify** -- inspect results, ensure all cases pass.
4. **Promote** -- commit the suite to the repo; CI runs it on every push.

## CLI Usage

```bash
# Run and print to stdout
python -m abi_evals.cli prompt-eval --suite path/to/suite.json

# Run and write results to file
python -m abi_evals.cli prompt-eval --suite path/to/suite.json --out results.json

# Via bash shim
script/eval-prompt-suite --suite path/to/suite.json
```

## Safety Properties

- **Offline**: the mock provider makes zero network calls.
- **Deterministic**: same inputs always produce the same outputs and run_id.
- **Stdlib-only**: no third-party dependencies beyond pytest (dev-only).
- **No real model calls**: outputs are hash-generated stubs, not LLM responses.
- **Sorted execution**: cases are always sorted by `case_id` for reproducibility.
