---
title: "Writing Merits"
---

Merit is intentionally “pytest-shaped”: you write plain Python, Merit discovers `merit_*` cases, injects dependencies by parameter name (like pytest fixtures), runs them, and reports results.

This page focuses on **how to write merits** and (importantly) **where the behavior lives in the codebase**, so you can trust what’s happening.

## TLDR - If you know pytest, you already know 80%

| pytest | merit |
| --- | --- |
| `def test_*(): ...` | `def merit_*(): ...` |
| `@pytest.fixture` | `@merit.resource` |
| `@pytest.mark.parametrize(...)` | `@merit.parametrize(...)` |
| `@pytest.mark.skip(...)` | `@merit.tag.skip(...)` |
| `@pytest.mark.xfail(...)` | `@merit.tag.xfail(...)` |
| `@pytest.mark.repeat(3)` | `@merit.repeat(3)` |

## Merits

Merit follows pytest-style discovery patterns to find merit functions in your codebase:

<Info>
  **Related docs**: [Merit (Concept)](/concepts/merit), [Testing (API)](/apis/testing)
</Info>

**Files**: Merit discovers Python files starting with `merit_`:
- `merit_chatbot.py` ✓
- `merit_agent.py` ✓
- `helpers.py` ✗

**Functions**: Inside discovered files, Merit collects functions starting with `merit_`:

```python
def merit_weather_agent():  # ✓ Discovered
    pass

def helper_function():      # ✗ Not discovered
    pass
```

**Classes**: Classes starting with `Merit` are discovered, and their `merit_*` methods become merit cases:

```python
class MeritCustomerSupport:     # ✓ Discovered
    def merit_greeting(self):    # ✓ Collected
        pass

    def helper(self):            # ✗ Not collected
        pass
```

## Modifiers

Modifiers are **decorators** that change how a merit is *collected* and/or *executed*. Some modifiers expand a single merit into many cases (like parametrization or dataset iteration), while others change execution semantics (like repeating a case, or marking it as skipped/xfail). Apply them to `merit_*` functions or `Merit*` classes.

<Info>
  **Related docs**: [Testing (API)](/apis/testing), [Case (Concept)](/concepts/case)
</Info>

**`@merit.parametrize(names, values)`** - Run the same merit with different inputs

```python
@merit.parametrize("model,temp", [("gpt-4", 0.7), ("claude-3", 0.5)])
def merit_model_response(model: str, temp: float, chatbot):
    response = chatbot.generate(model=model, temperature=temp)
    assert response
```

**`@merit.iter_cases(*cases)`** - Iterate over `Case` objects from external sources

```python
import json
from merit import Case

cases = [Case(**item) for item in json.load(open("cases.json"))]

@merit.iter_cases(*cases)
def merit_from_dataset(case: Case, classifier):
    result = classifier(**case.sut_input_values)
    if case.references:
        assert result == case.references["some_ref_value"]
```

**`@merit.tag(*tags)`** - Organize and filter merits by tags

```python
@merit.tag("smoke", "fast")
def merit_health_check(api):
    assert api.health_check()

# Run: merit test --tag smoke
```

**`@merit.tag.skip(reason=...)`** - Skip merits unconditionally

```python
@merit.tag.skip(reason="Feature not implemented")
def merit_upcoming():
    pass

@merit.tag.skip(reason="Requires API key")
def merit_external_api():
    pass
```

**`@merit.tag.xfail(reason=..., strict=False)`** - Mark merits expected to fail

```python
@merit.tag.xfail(reason="Known bug #123")
def merit_known_issue():
    assert False  # Won't fail the suite

@merit.tag.xfail(reason="Should still fail", strict=True)
def merit_strict():
    pass  # If this passes, suite FAILS
```

**`@merit.repeat(n, min_passes=n)`** - Run merits multiple times to see if AI behavior is consistent

```python
@merit.repeat(10)  # All 10 must pass
def merit_consistent(llm):
    assert "hello" in llm.generate("Say hello")

@merit.repeat(10, min_passes=8)  # 8 out of 10
def merit_mostly_correct(llm):
    assert "hola" in llm.generate("Say hello in Spanish")
```

## Resources

Resources are the Merit equivalent of pytest fixtures: named, injectable dependencies that Merit resolves by parameter name.

<Info>
  **Related docs**: [Resource (Concept)](/concepts/resource), [Metric (Concept)](/concepts/metric), [SUT (Concept)](/concepts/sut), [Tracing (API)](/apis/tracing)
</Info>

**`@merit.resource(scope="case")`** - Define injectable dependencies with lifecycle management

```python
@merit.resource
def database():
    conn = connect_db()
    yield conn  # Injected into merits
    conn.close()  # Automatic cleanup

@merit.resource(scope="session")
def ml_model():
    return load_model()  # Shared across entire run

def merit_query(database, ml_model):
    # Both injected automatically
    result = database.query("SELECT 1")
    prediction = ml_model.predict(result)
    assert prediction
```

Scopes: `"case"` (default), `"suite"`, `"session"`.

**`@merit.metric(scope="session")`** - Define a metric as a scoped, injectable measurement object

Metrics behave like resources, but they’re intended to accumulate measurements across many cases and then assert on aggregates at the end of their scope.

The most common pattern is to inject a `Metric` into your merits and use the `metrics(...)` context manager to record assertion outcomes into that metric.

```python
from merit import Metric, metrics

@merit.metric
def accuracy():
    metric = Metric()
    yield metric
    assert metric.mean > 0.8  # Check after all data collected

@merit.parametrize("input,expected", [("a", 1), ("b", 2)])
def merit_classifier(input, expected, classifier, accuracy: Metric):
    result = classifier(input)

    # Assertions inside metrics() are recorded as True/False
    with metrics(accuracy):
        assert result == expected
```

Scopes: `"session"` (default), `"suite"`, `"case"`.

**`@merit.sut`** - Register a System Under Test (SUT) as an injectable callable

A SUT is the thing you’re actually evaluating (an agent function, a pipeline, a classifier, a client wrapper, etc.). Declaring it with `@merit.sut` makes it injectable and traceable, so you can assert not only on the output, but also on how it behaved internally (for example: tool calls).

<Note>
  A SUT must be **callable**. Merit will inject it into your merit function and you will call it like a normal function (or callable object).
</Note>

```python
from demo_app import agent

@merit.sut
def weather_agent(prompt: str):
    return agent(prompt, tools=["get_weather"])

def merit_agent_uses_tools(weather_agent, trace_context):
    result = weather_agent("What's the weather?")

    # Access trace spans for assertions
    sut_spans = trace_context.get_sut_spans(name="weather_agent")
    assert sut_spans

    # If you want to assert on tool calls, query LLM spans explicitly.
    # Note: attribute keys come from OpenTelemetry LLM instrumentations.
    tool_names = [
        s.attributes.get("llm.request.functions.0.name")
        for s in trace_context.get_llm_calls()
        if s.attributes
    ]
    assert "get_weather" in tool_names
```

## Custom Assert

Merit transforms Python's `assert` keyword to provide richer testing capabilities for AI systems. When you run merit files through Merit's runner, assertions behave differently than standard Python.

### Continue on failure (default behavior)

By default, Merit **continues running remaining assertions** even after one fails. This is different from standard Python, where the first failed assertion stops execution immediately.

```python
def merit_multiple_checks(classifier):
    result = classifier("test input")

    assert result.confidence > 0.8  # Fails
    assert result.label != ""        # Still runs
    assert result.valid              # Still runs
```

All three assertions will be evaluated and reported, even if the first one fails. This behavior lets you see all test failures in a single run rather than fixing them one at a time.

To stop on the first failure, use the `--fail-fast` CLI flag:

```bash
merit test --fail-fast
```

### Integration with metrics

When assertions are evaluated inside a `metrics()` context manager, Merit automatically records whether each assertion passed or failed to the specified metrics:

```python
from merit import Metric, metrics

@merit.metric
def accuracy():
    metric = Metric()
    yield metric
    assert metric.mean > 0.8

@merit.parametrize("input,expected", [("a", 1), ("b", 2), ("c", 3)])
def merit_classifier(input, expected, classifier, accuracy: Metric):
    result = classifier(input)

    # Assertions inside metrics() are recorded as 1 (pass) or 0 (fail)
    with metrics(accuracy):
        assert result == expected
```

### Only works through Merit's runner

**Important:** Merit's assertion transformation only applies when you run files through Merit's test runner:

```bash
merit test merit_my_tests.py    # ✓ Transformed assertions
python merit_my_tests.py         # ✗ Standard Python behavior
uv run merit_my_tests.py         # ✗ Standard Python behavior
```

### Assert messages

Assert messages work as expected and are captured in the `AssertionResult`:

```python
def merit_validation(response):
    assert response.status == 200, f"Expected 200, got {response.status}"
    assert "error" not in response.body, "Response contains error"
```


<CardGroup cols={2}>
  <Card title="Merit Functions (Concept)" icon="code" href="/concepts/merit">
    Discovery rules, organization, and patterns
  </Card>
  <Card title="Quick Start" icon="rocket" href="/get-started/quick-start">
    End-to-end example with SUTs, predicates, metrics, and traces
  </Card>
</CardGroup>
