---
title: Parametrize
description: Generate multiple test cases from a single function
---

The `@parametrize` decorator lets you generate multiple evaluations from one function, similar to pytest's parametrize.

## Basic Usage

```python
from ezvals import eval, parametrize, EvalContext

@eval(dataset="math")
@parametrize("a,b,expected", [
    (2, 3, 5),
    (10, 20, 30),
    (0, 0, 0),
])
def test_addition(ctx: EvalContext, a, b, expected):
    result = a + b
    ctx.input = f"{a} + {b}"
    ctx.output = result
    assert result == expected, f"Expected {expected}, got {result}"
```

This generates three evaluations with numeric IDs:
- `test_addition[0]`
- `test_addition[1]`
- `test_addition[2]`

<Note>
Without custom `ids`, test variants are numbered sequentially. Use the `ids` parameter for readable names.
</Note>

## Parameter Formats

### Tuple List

```python
@parametrize("name,age", [
    ("Alice", 30),
    ("Bob", 25),
])
```

### Dictionary List

More readable for complex cases:

```python
@parametrize("operation,a,b,expected", [
    {"operation": "add", "a": 2, "b": 3, "expected": 5},
    {"operation": "multiply", "a": 4, "b": 5, "expected": 20},
    {"operation": "subtract", "a": 10, "b": 3, "expected": 7},
])
```

## Custom IDs

Name your test cases:

```python
@parametrize("threshold", [0.2, 0.5, 0.8], ids=["low", "mid", "high"])
def test_thresholds(ctx: EvalContext, threshold):
    ...
```

Generates:
- `test_thresholds[low]`
- `test_thresholds[mid]`
- `test_thresholds[high]`

## Stacking Decorators (Cartesian Product)

Stack multiple `@parametrize` decorators for all combinations:

```python
@eval(dataset="model_comparison")
@parametrize("model", ["gpt-4", "gpt-3.5", "claude"])
@parametrize("temperature", [0.0, 0.5, 1.0])
def test_models(ctx: EvalContext, model, temperature):
    ...
```

This generates 9 evaluations (3 models × 3 temperatures) with numeric IDs:
- `test_models[0]`
- `test_models[1]`
- ... through `test_models[8]`

For readable names, provide `ids` on each decorator.

## Auto-Mapping to Context

Only parameters with these special names automatically populate the context:

- `input` → `ctx.input`
- `reference` → `ctx.reference`
- `metadata` → merged into `ctx.metadata`
- `trace_data` → `ctx.trace_data`
- `latency` → `ctx.latency`
- `dataset` → overrides the function's dataset
- `labels` → merged with the function's labels

```python
@eval(dataset="qa")
@parametrize("input,reference", [
    ("What is 2+2?", "4"),
    ("What is the capital of France?", "Paris"),
])
async def test_qa(ctx: EvalContext):
    # ctx.input and ctx.reference are auto-populated
    ctx.output = await my_agent(ctx.input)
    assert ctx.output == ctx.reference
```

### Per-Case Dataset and Labels

You can override dataset or add labels on a per-case basis:

```python
@eval(dataset="default", labels=["base"])
@parametrize("input,dataset,labels", [
    ("case A", "custom_dataset", ["extra"]),  # dataset="custom_dataset", labels=["base", "extra"]
    ("case B", None, None),                    # dataset="default", labels=["base"]
])
def test_per_case(ctx: EvalContext):
    ctx.output = process(ctx.input)
```

- **Dataset**: Per-case value overrides the function's dataset
- **Labels**: Per-case labels are merged with function labels (duplicates removed)

## Accessing Non-Special Parameters

For parameters that aren't special context fields, you **must** include them in your function signature:

```python
@parametrize("prompt,expected", [...])
def test_example(ctx: EvalContext, prompt, expected):
    # Parameters are passed as function arguments
    ctx.input = prompt
    ctx.output = process(prompt)
    assert ctx.output == expected
```

<Warning>
Arbitrary parameters are **not** injected onto `ctx`. If you omit parameters from the function signature, you'll get "unexpected keyword argument" errors.
</Warning>

## Running Specific Variants

Run a specific parametrized test:

```bash
# Run all variants
ezvals run evals.py::test_math

# Run specific variant
ezvals run evals.py::test_math[2][3][5]
```

## Combining with Other Options

```python
@eval(
    dataset="sentiment",
    default_score_key="accuracy",
    timeout=5.0
)
@parametrize("model", ["gpt-4", "gpt-3.5"])
@parametrize("text,expected", [
    ("I love this!", "positive"),
    ("This is terrible", "negative"),
])
async def test_sentiment(ctx: EvalContext, model, text, expected):
    ctx.input = text
    ctx.metadata["model"] = model  # Track which model was used
    ctx.metadata["text_length"] = len(text)
    ctx.output = await analyze_sentiment(text, model=model)
    assert ctx.output == expected
```

## Data-Driven Testing

Load test cases from external sources:

```python
import json

# Load from file
with open("test_cases.json") as f:
    test_cases = json.load(f)

@eval(dataset="qa")
@parametrize("question,answer", test_cases)
async def test_qa(ctx: EvalContext, question, answer):
    ctx.input = question
    ctx.reference = answer
    ctx.output = await my_agent(question)
    assert ctx.output == answer
```

## Example: Comprehensive Test Suite

```python
from ezvals import eval, parametrize, EvalContext

PROMPTS = [
    {"input": "Hello", "expected_intent": "greeting", "expected_sentiment": "positive"},
    {"input": "I need help", "expected_intent": "support", "expected_sentiment": "neutral"},
    {"input": "This is broken!", "expected_intent": "complaint", "expected_sentiment": "negative"},
]

@eval(dataset="intent_detection", default_score_key="accuracy")
@parametrize("input,expected_intent,expected_sentiment", PROMPTS)
async def test_intent(ctx: EvalContext, expected_intent, expected_sentiment):
    # ctx.input is auto-populated (special name)
    result = await detect_intent(ctx.input)
    ctx.output = result

    assert result["intent"] == expected_intent
    assert result["sentiment"] == expected_sentiment
```
