---
title: "Merit"
---

**Merits** are the core building blocks of your AI system evaluations. Like pytest discovers `test_*` functions, Merit discovers and runs `merit_*` functions - each one checking how well your AI system performs using Merit's APIs and components.

## Basic Usage

The simplest merit function is a function whose name starts with `merit_`. Merit automatically discovers and executes these functions.

```python
import merit
from merit.predicates import has_unsupported_facts

# Define your AI system (or import from your codebase)
def chatbot(prompt: str) -> str:
    return call_llm(prompt)

# Merit function: discovered and run automatically
async def merit_chatbot_no_hallucinations():
    context = "Our store hours are 9 AM to 6 PM Monday-Saturday."
    response = chatbot("When are you open?")

    # Use semantic predicates to check output quality
    assert not await has_unsupported_facts(response, context)
```

Run all merit functions in your project:

```bash
merit test
```

Merit discovers all `merit_*` functions, executes them, and generates a report - just like pytest but for AI system evaluation.

## Merit Discovery

Merit follows pytest's discovery patterns, finding merits in files, functions, and classes that follow naming conventions.

### Files

Merit discovers Python files starting with `merit_`:

<Tree>
  <Tree.Folder name="project" defaultOpen>
    <Tree.File name="merit_chatbot.py ✓ Discovered" />
    <Tree.File name="merit_agent.py ✓ Discovered" />
    <Tree.Folder name="tests" defaultOpen>
      <Tree.File name="merit_rag.py ✓ Discovered" />
      <Tree.File name="helpers.py ✗ Not discovered" />
    </Tree.Folder>
    <Tree.Folder name="src" defaultOpen>
      <Tree.File name="agent.py ✗ Not discovered" />
    </Tree.Folder>
  </Tree.Folder>
</Tree>

### Functions

Within discovered files, Merit collects functions starting with `merit_`:

```python
# merit_agents.py

def merit_weather_agent():     # ✓ Collected
    pass

def merit_calculator():        # ✓ Collected
    pass

def helper_function():         # ✗ Not collected (no merit_ prefix)
    pass

def test_something():          # ✗ Not collected (pytest convention)
    pass
```

### Classes

Classes starting with `Merit` are discovered, and their `merit_*` methods become merit cases:

```python
# merit_agents.py

class MeritCustomerSupport:    # ✓ Class discovered

    def merit_greeting(self):   # ✓ Method collected
        pass

    def merit_farewell(self):   # ✓ Method collected
        pass

    def helper(self):           # ✗ Not collected (no merit_ prefix)
        pass

class TestSomething:           # ✗ Not discovered (pytest convention)
    pass
```

## Dependency Injection

Merit automatically injects dependencies by matching parameter names to registered resources, SUTs, and metrics.

```python
import merit
from merit import Metric, metrics

# Define resources (dependencies)
@merit.resource
def chatbot():
    return ChatBot(model="gpt-4")

@merit.metric
def accuracy():
    metric = Metric()
    yield metric
    assert metric.mean > 0.8

# Merit function with injected dependencies
def merit_chatbot_accuracy(chatbot, accuracy: Metric):
    # chatbot and accuracy automatically injected by name

    test_cases = [
        ("What's 2+2?", "4"),
        ("Capital of France?", "Paris"),
    ]

    for question, expected in test_cases:
        answer = chatbot.ask(question)
        with metrics(accuracy):
            assert expected.lower() in answer.lower()
```

## Async Support

Merit automatically detects and runs async functions:

```python
from merit.predicates import follows_policy

# Sync merit function
def merit_sync_test(calculator):
    result = calculator.add(2, 2)
    assert result == 4

# Async merit function - automatically detected
async def merit_async_test(chatbot):
    response = await chatbot.ask_async("Hello")

    # Many predicates are async
    policy = "Agent is friendly and professional"
    assert await follows_policy(response, policy)
```

Resources can be async too:

```python
@merit.resource
async def async_database():
    conn = await connect_async()
    yield conn
    await conn.close()

async def merit_query(async_database):
    result = await async_database.query("SELECT 1")
    assert result
```

Merit handles the async execution automatically - no `asyncio.run()` needed.

## Iterate Merits

AI systems are inherently non-deterministic, making it essential to test them across multiple scenarios and runs. Merit provides three distinct approaches to iterate the same merit definition, each optimized for different use cases: quick parametrization for a few variations, structured cases for large datasets, and repeated execution to assess consistency.

### Iterate with different parameters

When you need to run the same merit with a small set of different inputs, `@merit.parametrize` offers the most concise syntax. It's ideal for testing a handful of variations without the overhead of defining structured case objects.

```python
import merit

@merit.parametrize("city,state", [
    ("Boston", "Massachusetts"),
    ("Austin", "Texas"),
])
def merit_geography_bot(city: str, state: str, geography_bot):
    result = geography_bot.ask(f"What state is {city} in?")
    assert state in result
```

This creates 2 merit cases:
- `merit_geography_bot(city='Boston', state='Massachusetts')`
- `merit_geography_bot(city='Austin', state='Texas')`

Multiple parameters can be stacked:

```python
@merit.parametrize("model", ["gpt-4", "claude-3"])
@merit.parametrize("temperature", [0.0, 0.7, 1.0])
def merit_model_combinations(model: str, temperature: float):
    # Runs 6 times: 2 models × 3 temperatures
    pass
```

<Tip>
Parametrization works best when you have a small number of input variations (typically fewer than 10) and don't require strict type definitions.
</Tip>

### Iterate with different cases

When evaluating against tens or hundreds of examples, or when you need consistent typing and structure, use `@merit.iter_cases` with `Case` objects. This approach provides type safety through Pydantic validation and enables loading test cases from external sources like JSON files or databases.

```python
from merit import Case
import json

# Load merit cases from file
with open("merit_cases.json") as f:
    cases = [Case(**item) for item in json.load(f)]

@merit.iter_cases(*cases)
def merit_from_dataset(case: Case, classifier):
    result = classifier(**case.sut_input_values)

    expected = case.references["expected_label"]
    assert result == expected
```

### Repeat with same data

AI systems can produce different outputs for identical inputs due to their non-deterministic nature. Use `@merit.repeat` to run the same merit multiple times with the same data, measuring consistency and reliability of your AI component.

```python
import merit

@merit.repeat(count=5)
def merit_chatbot_consistent_greeting(chatbot):
    """Run 5 times - all must pass."""
    response = chatbot.ask("Hello")
    assert "hi" in response.lower() or "hello" in response.lower()

@merit.repeat(count=10, min_passes=8)
def merit_sentiment_mostly_accurate(classifier):
    """Run 10 times - at least 8 must pass."""
    result = classifier("This product is amazing!")
    assert result.sentiment == "positive"
```

<Tip>
The `min_passes` parameter is sometimes referred to as "pass@k" in the AI evaluation community. For example, `@merit.repeat(count=10, min_passes=8)` checks if your system achieves the desired behavior in at least 8 out of 10 attempts (pass@8/10).
</Tip>

## Organizing Merits with Tags

### Running only specific Merits

Use `@merit.tag` to organize and filter merits:

```python
import merit

@merit.tag("smoke", "fast")
def merit_health_check(api_client):
    response = api_client.get("/health")
    assert response.status_code == 200

@merit.tag("integration", "slow")
def merit_end_to_end_workflow(system):
    # Long-running integration merit
    pass

# Tag entire classes
@merit.tag("customer-support")
class MeritSupportBot:

    @merit.tag("greeting")
    def merit_hello(self, support_bot):
        pass

    @merit.tag("farewell")
    def merit_goodbye(self, support_bot):
        pass
```

Run specific tags from CLI:

```bash
merit test --tag smoke       # Only smoke merits
merit test --tag slow        # Only slow merits
```

### Skipping Merits unconditionally

Skip merits with `@merit.tag.skip`:

```python
import merit

@merit.tag.skip(reason="Feature not implemented yet")
def merit_upcoming_feature():
    pass

@merit.tag.skip(reason="Requires API key")
def merit_external_api():
    pass
```

### Skipping Merits conditionally

```python
def merit_conditional_skip():
    if not os.getenv("API_KEY"):
        merit.skip("API_KEY not configured")
    # Test continues if condition not met
    assert True
```

<Tip>
You can also use `merit.skip()` inside resources to conditionally skip merits when dependencies aren't available. This centralizes skip logic where the resource is defined rather than in every merit that uses it.
</Tip>

### Expected Failures

Mark merits expected to fail with `@merit.tag.xfail`:

```python
@merit.tag.xfail(reason="Known bug #123")
def merit_known_issue():
    # This failure won't fail the merit suite
    assert False

@merit.tag.xfail(reason="Model not accurate yet", strict=True)
def merit_strict_xfail():
    # If this passes, the merit suite FAILS (unexpected pass)
    pass
```

Use `strict=True` when the merit passing would be surprising and worth investigating.

## Recommendations

### 1. Name functions descriptively

Merit function names become merit case identifiers in reports. Use descriptive names that explain what's being evaluated.

**Don't do this:**
```python
def merit_test1():
    pass

def merit_test2():
    pass

def merit_chatbot():  # Too vague
    pass
```

**Do this:**
```python
def merit_chatbot_responds_to_greetings():
    """Check that chatbot handles basic greetings appropriately."""
    pass

def merit_chatbot_no_hallucinations_in_faq():
    """Verify chatbot doesn't invent facts when answering FAQ questions."""
    pass

def merit_chatbot_follows_brand_voice():
    """Ensure chatbot responses match company's tone and style guidelines."""
    pass
```

Descriptive names make reports self-documenting and help team members understand merit failures.

### 2. Use dependency injection over global imports

Merit's dependency injection system enables better resource management and merit isolation. Inject dependencies as parameters instead of importing globally.

**Don't do this:**
```python
# merit_agent.py
from app import agent  # Global import

def merit_weather_queries():
    # Using global - can't control lifecycle or swap implementations
    response = agent("What's the weather?")
    assert response
```

**Do this:**
```python
# merit_agent.py
import merit
from app import agent as production_agent

@merit.resource
def agent():
    """Evaluation instance of agent with controlled lifecycle."""
    instance = production_agent.create(env="test")
    yield instance
    instance.cleanup()

def merit_weather_queries(agent):
    # Injected - Merit manages lifecycle and can track usage
    response = agent("What's the weather?")
    assert response
```

This pattern enables:
- Automatic setup and teardown
- Resource scoping and reuse
- Merit isolation
- Better reporting and analytics

### 3. Organize related merits in Merit classes

Group related merits in classes for better organization and shared tags/setup:

**Don't do this:**
```python
# merit_support.py - flat functions with repeated tags

@merit.tag("customer-support", "greeting")
def merit_support_greeting_casual():
    pass

@merit.tag("customer-support", "greeting")
def merit_support_greeting_formal():
    pass

@merit.tag("customer-support", "farewell")
def merit_support_farewell_casual():
    pass

@merit.tag("customer-support", "farewell")
def merit_support_farewell_formal():
    pass
```

**Do this:**
```python
# merit_support.py - organized in classes

@merit.tag("customer-support")
class MeritSupportGreetings:
    """Evaluate support bot greeting scenarios."""

    @merit.tag("casual")
    def merit_greeting_casual(self, support_bot):
        pass

    @merit.tag("formal")
    def merit_greeting_formal(self, support_bot):
        pass

@merit.tag("customer-support")
class MeritSupportFarewells:
    """Evaluate support bot farewell scenarios."""

    @merit.tag("casual")
    def merit_farewell_casual(self, support_bot):
        pass

    @merit.tag("formal")
    def merit_farewell_formal(self, support_bot):
        pass
```

Classes provide:
- Logical grouping in reports
- Shared tags that cascade to methods
- Better code organization
- Easier navigation in IDEs
