[ Skip to content ](https://ai.pydantic.dev/evals/evaluators/built-in/#built-in-evaluators)
**Join us at the inaugural PyAI Conf in San Francisco on March 10th![Learn More](https://pyai.events/?utm_source=pydantic-ai) **
[ ![logo](https://ai.pydantic.dev/img/logo-white.svg) ](https://ai.pydantic.dev/ "Pydantic AI")
Pydantic AI
Built-in Evaluators
Type to start searching
[ pydantic/pydantic-ai
  * v1.63.0
  * 15.1k
  * 1.7k

](https://github.com/pydantic/pydantic-ai "Go to repository")
[ ![logo](https://ai.pydantic.dev/img/logo-white.svg) ](https://ai.pydantic.dev/ "Pydantic AI") Pydantic AI
[ pydantic/pydantic-ai
  * v1.63.0
  * 15.1k
  * 1.7k

](https://github.com/pydantic/pydantic-ai "Go to repository")
  * [ Pydantic AI  ](https://ai.pydantic.dev/)
  * [ Installation  ](https://ai.pydantic.dev/install/)
  * [ Getting Help  ](https://ai.pydantic.dev/help/)
  * [ Troubleshooting  ](https://ai.pydantic.dev/troubleshooting/)
  * [ Pydantic AI Gateway  ](https://ai.pydantic.dev/gateway/)
  * Documentation
    * Core Concepts
      * [ Agents  ](https://ai.pydantic.dev/agent/)
      * [ Dependencies  ](https://ai.pydantic.dev/dependencies/)
      * [ Function Tools  ](https://ai.pydantic.dev/tools/)
      * [ Output  ](https://ai.pydantic.dev/output/)
      * [ Messages and chat history  ](https://ai.pydantic.dev/message-history/)
      * [ Direct Model Requests  ](https://ai.pydantic.dev/direct/)
    * Models & Providers
      * [ Overview  ](https://ai.pydantic.dev/models/overview/)
      * [ OpenAI  ](https://ai.pydantic.dev/models/openai/)
      * [ Anthropic  ](https://ai.pydantic.dev/models/anthropic/)
      * [ Google  ](https://ai.pydantic.dev/models/google/)
      * [ xAI  ](https://ai.pydantic.dev/models/xai/)
      * [ Bedrock  ](https://ai.pydantic.dev/models/bedrock/)
      * [ Cerebras  ](https://ai.pydantic.dev/models/cerebras/)
      * [ Cohere  ](https://ai.pydantic.dev/models/cohere/)
      * [ Groq  ](https://ai.pydantic.dev/models/groq/)
      * [ Hugging Face  ](https://ai.pydantic.dev/models/huggingface/)
      * [ Mistral  ](https://ai.pydantic.dev/models/mistral/)
      * [ OpenRouter  ](https://ai.pydantic.dev/models/openrouter/)
      * [ Outlines  ](https://ai.pydantic.dev/models/outlines/)
    * Tools & Toolsets
      * [ Function Tools  ](https://ai.pydantic.dev/tools/)
      * [ Advanced Tool Features  ](https://ai.pydantic.dev/tools-advanced/)
      * [ Toolsets  ](https://ai.pydantic.dev/toolsets/)
      * [ Deferred Tools  ](https://ai.pydantic.dev/deferred-tools/)
      * [ Built-in Tools  ](https://ai.pydantic.dev/builtin-tools/)
      * [ Common Tools  ](https://ai.pydantic.dev/common-tools/)
      * [ Third-Party Tools  ](https://ai.pydantic.dev/third-party-tools/)
    * Advanced Features
      * [ Image, Audio, Video & Document Input  ](https://ai.pydantic.dev/input/)
      * [ Thinking  ](https://ai.pydantic.dev/thinking/)
      * [ HTTP Request Retries  ](https://ai.pydantic.dev/retries/)
    * MCP
      * [ Overview  ](https://ai.pydantic.dev/mcp/overview/)
      * [ Client  ](https://ai.pydantic.dev/mcp/client/)
      * [ FastMCP Client  ](https://ai.pydantic.dev/mcp/fastmcp-client/)
      * [ Server  ](https://ai.pydantic.dev/mcp/server/)
    * [ Multi-Agent Patterns  ](https://ai.pydantic.dev/multi-agent-applications/)
    * [ Web Chat UI  ](https://ai.pydantic.dev/web/)
    * [ Embeddings  ](https://ai.pydantic.dev/embeddings/)
    * [ Testing  ](https://ai.pydantic.dev/testing/)
  * Pydantic Evals
    * [ Overview  ](https://ai.pydantic.dev/evals/)
    * Getting Started
      * [ Quick Start  ](https://ai.pydantic.dev/evals/quick-start/)
      * [ Core Concepts  ](https://ai.pydantic.dev/evals/core-concepts/)
    * Evaluators
      * [ Overview  ](https://ai.pydantic.dev/evals/evaluators/overview/)
      * Built-in Evaluators  [ Built-in Evaluators  ](https://ai.pydantic.dev/evals/evaluators/built-in/)
        * [ Comparison Evaluators  ](https://ai.pydantic.dev/evals/evaluators/built-in/#comparison-evaluators)
          * [ EqualsExpected  ](https://ai.pydantic.dev/evals/evaluators/built-in/#equalsexpected)
          * [ Equals  ](https://ai.pydantic.dev/evals/evaluators/built-in/#equals)
          * [ Contains  ](https://ai.pydantic.dev/evals/evaluators/built-in/#contains)
        * [ Type Validation  ](https://ai.pydantic.dev/evals/evaluators/built-in/#type-validation)
          * [ IsInstance  ](https://ai.pydantic.dev/evals/evaluators/built-in/#isinstance)
        * [ Performance Evaluation  ](https://ai.pydantic.dev/evals/evaluators/built-in/#performance-evaluation)
          * [ MaxDuration  ](https://ai.pydantic.dev/evals/evaluators/built-in/#maxduration)
        * [ LLM-as-a-Judge  ](https://ai.pydantic.dev/evals/evaluators/built-in/#llm-as-a-judge)
          * [ LLMJudge  ](https://ai.pydantic.dev/evals/evaluators/built-in/#llmjudge)
        * [ Span-Based Evaluation  ](https://ai.pydantic.dev/evals/evaluators/built-in/#span-based-evaluation)
          * [ HasMatchingSpan  ](https://ai.pydantic.dev/evals/evaluators/built-in/#hasmatchingspan)
        * [ Built-in Report Evaluators  ](https://ai.pydantic.dev/evals/evaluators/built-in/#built-in-report-evaluators)
        * [ Quick Reference Table  ](https://ai.pydantic.dev/evals/evaluators/built-in/#quick-reference-table)
          * [ Case-Level Evaluators  ](https://ai.pydantic.dev/evals/evaluators/built-in/#case-level-evaluators)
          * [ Report-Level Evaluators  ](https://ai.pydantic.dev/evals/evaluators/built-in/#report-level-evaluators)
        * [ Combining Evaluators  ](https://ai.pydantic.dev/evals/evaluators/built-in/#combining-evaluators)
        * [ Next Steps  ](https://ai.pydantic.dev/evals/evaluators/built-in/#next-steps)
      * [ LLM Judge  ](https://ai.pydantic.dev/evals/evaluators/llm-judge/)
      * [ Custom Evaluators  ](https://ai.pydantic.dev/evals/evaluators/custom/)
      * [ Report Evaluators  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/)
      * [ Span-Based  ](https://ai.pydantic.dev/evals/evaluators/span-based/)
    * How-To Guides
      * [ Logfire Integration  ](https://ai.pydantic.dev/evals/how-to/logfire-integration/)
      * [ Dataset Management  ](https://ai.pydantic.dev/evals/how-to/dataset-management/)
      * [ Dataset Serialization  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/)
      * [ Concurrency & Performance  ](https://ai.pydantic.dev/evals/how-to/concurrency/)
      * [ Multi-Run Evaluation  ](https://ai.pydantic.dev/evals/how-to/multi-run/)
      * [ Retry Strategies  ](https://ai.pydantic.dev/evals/how-to/retry-strategies/)
      * [ Metrics & Attributes  ](https://ai.pydantic.dev/evals/how-to/metrics-attributes/)
    * Examples
      * [ Simple Validation  ](https://ai.pydantic.dev/evals/examples/simple-validation/)
  * Pydantic Graph
    * [ Overview  ](https://ai.pydantic.dev/graph/)
    * [ Beta API  ](https://ai.pydantic.dev/graph/beta/)
      * [ Steps  ](https://ai.pydantic.dev/graph/beta/steps/)
      * [ Joins & Reducers  ](https://ai.pydantic.dev/graph/beta/joins/)
      * [ Decisions  ](https://ai.pydantic.dev/graph/beta/decisions/)
      * [ Parallel Execution  ](https://ai.pydantic.dev/graph/beta/parallel/)
  * Integrations
    * [ Debugging & Monitoring with Pydantic Logfire  ](https://ai.pydantic.dev/logfire/)
    * Durable Execution
      * [ Overview  ](https://ai.pydantic.dev/durable_execution/overview/)
      * [ Temporal  ](https://ai.pydantic.dev/durable_execution/temporal/)
      * [ DBOS  ](https://ai.pydantic.dev/durable_execution/dbos/)
      * [ Prefect  ](https://ai.pydantic.dev/durable_execution/prefect/)
    * UI Event Streams
      * [ Overview  ](https://ai.pydantic.dev/ui/overview/)
      * [ AG-UI  ](https://ai.pydantic.dev/ui/ag-ui/)
      * [ Vercel AI  ](https://ai.pydantic.dev/ui/vercel-ai/)
    * [ Agent2Agent (A2A)  ](https://ai.pydantic.dev/a2a/)
  * Related Packages
    * [ clai  ](https://ai.pydantic.dev/cli/)
  * Examples
    * [ Setup  ](https://ai.pydantic.dev/examples/setup/)
    * Getting Started
      * [ Pydantic Model  ](https://ai.pydantic.dev/examples/pydantic-model/)
      * [ Weather agent  ](https://ai.pydantic.dev/examples/weather-agent/)
    * Conversational Agents
      * [ Chat App with FastAPI  ](https://ai.pydantic.dev/examples/chat-app/)
      * [ Bank support  ](https://ai.pydantic.dev/examples/bank-support/)
    * Data & Analytics
      * [ SQL Generation  ](https://ai.pydantic.dev/examples/sql-gen/)
      * [ Data Analyst  ](https://ai.pydantic.dev/examples/data-analyst/)
      * [ RAG  ](https://ai.pydantic.dev/examples/rag/)
    * Streaming
      * [ Stream markdown  ](https://ai.pydantic.dev/examples/stream-markdown/)
      * [ Stream whales  ](https://ai.pydantic.dev/examples/stream-whales/)
    * Complex Workflows
      * [ Flight booking  ](https://ai.pydantic.dev/examples/flight-booking/)
      * [ Question Graph  ](https://ai.pydantic.dev/examples/question-graph/)
    * Business Applications
      * [ Slack Lead Qualifier with Modal  ](https://ai.pydantic.dev/examples/slack-lead-qualifier/)
    * UI Examples
      * [ Agent User Interaction (AG-UI)  ](https://ai.pydantic.dev/examples/ag-ui/)
  * API Reference
    * pydantic_ai
      * [ pydantic_ai.ag_ui  ](https://ai.pydantic.dev/api/ag_ui/)
      * [ pydantic_ai.agent  ](https://ai.pydantic.dev/api/agent/)
      * [ pydantic_ai.builtin_tools  ](https://ai.pydantic.dev/api/builtin_tools/)
      * [ pydantic_ai.common_tools  ](https://ai.pydantic.dev/api/common_tools/)
      * [ pydantic_ai — Concurrency  ](https://ai.pydantic.dev/api/concurrency/)
      * [ pydantic_ai.direct  ](https://ai.pydantic.dev/api/direct/)
      * [ pydantic_ai.durable_exec  ](https://ai.pydantic.dev/api/durable_exec/)
      * [ pydantic_ai.embeddings  ](https://ai.pydantic.dev/api/embeddings/)
      * [ pydantic_ai.exceptions  ](https://ai.pydantic.dev/api/exceptions/)
      * [ pydantic_ai.ext  ](https://ai.pydantic.dev/api/ext/)
      * [ pydantic_ai.format_prompt  ](https://ai.pydantic.dev/api/format_prompt/)
      * [ pydantic_ai.mcp  ](https://ai.pydantic.dev/api/mcp/)
      * [ pydantic_ai.messages  ](https://ai.pydantic.dev/api/messages/)
      * [ pydantic_ai.models.anthropic  ](https://ai.pydantic.dev/api/models/anthropic/)
      * [ pydantic_ai.models  ](https://ai.pydantic.dev/api/models/base/)
      * [ pydantic_ai.models.bedrock  ](https://ai.pydantic.dev/api/models/bedrock/)
      * [ pydantic_ai.models.cerebras  ](https://ai.pydantic.dev/api/models/cerebras/)
      * [ pydantic_ai.models.cohere  ](https://ai.pydantic.dev/api/models/cohere/)
      * [ pydantic_ai.models.fallback  ](https://ai.pydantic.dev/api/models/fallback/)
      * [ pydantic_ai.models.function  ](https://ai.pydantic.dev/api/models/function/)
      * [ pydantic_ai.models.google  ](https://ai.pydantic.dev/api/models/google/)
      * [ pydantic_ai.models.xai  ](https://ai.pydantic.dev/api/models/xai/)
      * [ pydantic_ai.models.groq  ](https://ai.pydantic.dev/api/models/groq/)
      * [ pydantic_ai.models.huggingface  ](https://ai.pydantic.dev/api/models/huggingface/)
      * [ pydantic_ai.models.instrumented  ](https://ai.pydantic.dev/api/models/instrumented/)
      * [ pydantic_ai.models.mcp_sampling  ](https://ai.pydantic.dev/api/models/mcp-sampling/)
      * [ pydantic_ai.models.mistral  ](https://ai.pydantic.dev/api/models/mistral/)
      * [ pydantic_ai.models.openai  ](https://ai.pydantic.dev/api/models/openai/)
      * [ pydantic_ai.models.openrouter  ](https://ai.pydantic.dev/api/models/openrouter/)
      * [ pydantic_ai.models.outlines  ](https://ai.pydantic.dev/api/models/outlines/)
      * [ pydantic_ai.models.test  ](https://ai.pydantic.dev/api/models/test/)
      * [ pydantic_ai.models.wrapper  ](https://ai.pydantic.dev/api/models/wrapper/)
      * [ pydantic_ai.output  ](https://ai.pydantic.dev/api/output/)
      * [ pydantic_ai.profiles  ](https://ai.pydantic.dev/api/profiles/)
      * [ pydantic_ai.providers  ](https://ai.pydantic.dev/api/providers/)
      * [ pydantic_ai.result  ](https://ai.pydantic.dev/api/result/)
      * [ pydantic_ai.retries  ](https://ai.pydantic.dev/api/retries/)
      * [ pydantic_ai.run  ](https://ai.pydantic.dev/api/run/)
      * [ pydantic_ai.settings  ](https://ai.pydantic.dev/api/settings/)
      * [ pydantic_ai.tools  ](https://ai.pydantic.dev/api/tools/)
      * [ pydantic_ai.toolsets  ](https://ai.pydantic.dev/api/toolsets/)
      * [ pydantic_ai.ui.ag_ui  ](https://ai.pydantic.dev/api/ui/ag_ui/)
      * [ pydantic_ai.ui  ](https://ai.pydantic.dev/api/ui/base/)
      * [ pydantic_ai.ui.vercel_ai  ](https://ai.pydantic.dev/api/ui/vercel_ai/)
      * [ pydantic_ai.usage  ](https://ai.pydantic.dev/api/usage/)
    * pydantic_evals
      * [ pydantic_evals.dataset  ](https://ai.pydantic.dev/api/pydantic_evals/dataset/)
      * [ pydantic_evals.evaluators  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/)
      * [ pydantic_evals.reporting  ](https://ai.pydantic.dev/api/pydantic_evals/reporting/)
      * [ pydantic_evals.otel  ](https://ai.pydantic.dev/api/pydantic_evals/otel/)
      * [ pydantic_evals.generation  ](https://ai.pydantic.dev/api/pydantic_evals/generation/)
    * pydantic_graph
      * [ pydantic_graph  ](https://ai.pydantic.dev/api/pydantic_graph/graph/)
      * [ pydantic_graph.nodes  ](https://ai.pydantic.dev/api/pydantic_graph/nodes/)
      * [ pydantic_graph.persistence  ](https://ai.pydantic.dev/api/pydantic_graph/persistence/)
      * [ pydantic_graph.mermaid  ](https://ai.pydantic.dev/api/pydantic_graph/mermaid/)
      * [ pydantic_graph.exceptions  ](https://ai.pydantic.dev/api/pydantic_graph/exceptions/)
      * Beta API
        * [ pydantic_graph.beta  ](https://ai.pydantic.dev/api/pydantic_graph/beta/)
        * [ pydantic_graph.beta.graph  ](https://ai.pydantic.dev/api/pydantic_graph/beta_graph/)
        * [ pydantic_graph.beta.graph_builder  ](https://ai.pydantic.dev/api/pydantic_graph/beta_graph_builder/)
        * [ pydantic_graph.beta.step  ](https://ai.pydantic.dev/api/pydantic_graph/beta_step/)
        * [ pydantic_graph.beta.join  ](https://ai.pydantic.dev/api/pydantic_graph/beta_join/)
        * [ pydantic_graph.beta.decision  ](https://ai.pydantic.dev/api/pydantic_graph/beta_decision/)
        * [ pydantic_graph.beta.node  ](https://ai.pydantic.dev/api/pydantic_graph/beta_node/)
    * fasta2a
      * [ fasta2a  ](https://ai.pydantic.dev/api/fasta2a/)
  * Project
    * [ Contributing  ](https://ai.pydantic.dev/contributing/)
    * [ Upgrade Guide  ](https://ai.pydantic.dev/changelog/)
    * [ Version policy  ](https://ai.pydantic.dev/version-policy/)


  * [ Comparison Evaluators  ](https://ai.pydantic.dev/evals/evaluators/built-in/#comparison-evaluators)
    * [ EqualsExpected  ](https://ai.pydantic.dev/evals/evaluators/built-in/#equalsexpected)
    * [ Equals  ](https://ai.pydantic.dev/evals/evaluators/built-in/#equals)
    * [ Contains  ](https://ai.pydantic.dev/evals/evaluators/built-in/#contains)
  * [ Type Validation  ](https://ai.pydantic.dev/evals/evaluators/built-in/#type-validation)
    * [ IsInstance  ](https://ai.pydantic.dev/evals/evaluators/built-in/#isinstance)
  * [ Performance Evaluation  ](https://ai.pydantic.dev/evals/evaluators/built-in/#performance-evaluation)
    * [ MaxDuration  ](https://ai.pydantic.dev/evals/evaluators/built-in/#maxduration)
  * [ LLM-as-a-Judge  ](https://ai.pydantic.dev/evals/evaluators/built-in/#llm-as-a-judge)
    * [ LLMJudge  ](https://ai.pydantic.dev/evals/evaluators/built-in/#llmjudge)
  * [ Span-Based Evaluation  ](https://ai.pydantic.dev/evals/evaluators/built-in/#span-based-evaluation)
    * [ HasMatchingSpan  ](https://ai.pydantic.dev/evals/evaluators/built-in/#hasmatchingspan)
  * [ Built-in Report Evaluators  ](https://ai.pydantic.dev/evals/evaluators/built-in/#built-in-report-evaluators)
  * [ Quick Reference Table  ](https://ai.pydantic.dev/evals/evaluators/built-in/#quick-reference-table)
    * [ Case-Level Evaluators  ](https://ai.pydantic.dev/evals/evaluators/built-in/#case-level-evaluators)
    * [ Report-Level Evaluators  ](https://ai.pydantic.dev/evals/evaluators/built-in/#report-level-evaluators)
  * [ Combining Evaluators  ](https://ai.pydantic.dev/evals/evaluators/built-in/#combining-evaluators)
  * [ Next Steps  ](https://ai.pydantic.dev/evals/evaluators/built-in/#next-steps)


  1. [ Pydantic AI  ](https://ai.pydantic.dev/)
  2. [ Pydantic Evals  ](https://ai.pydantic.dev/evals/)
  3. [ Evaluators  ](https://ai.pydantic.dev/evals/evaluators/overview/)


# Built-in Evaluators
Pydantic Evals provides several built-in evaluators for common evaluation tasks.
## Comparison Evaluators
### EqualsExpected
Check if the output exactly equals the expected output from the case.
```
from pydantic_evals.evaluators import EqualsExpected

EqualsExpected()

```

**Parameters:** None
**Returns:** `bool` - `True` if `ctx.output == ctx.expected_output`
**Example:**
```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EqualsExpected

dataset = Dataset(
    cases=[
        Case(
            name='addition',
            inputs='2 + 2',
            expected_output='4',
        ),
    ],
    evaluators=[EqualsExpected()],
)

```

**Notes:**
  * Skips evaluation if `expected_output` is `None` (returns empty dict `{}`)
  * Uses Python's `==` operator, so works with any comparable types
  * For structured data, considers nested equality


* * *
### Equals
Check if the output equals a specific value.
```
from pydantic_evals.evaluators import Equals

Equals(value='expected_result')

```

**Parameters:**
  * `value` (Any): The value to compare against
  * `evaluation_name` (str | None): Custom name for this evaluation in reports


**Returns:** `bool` - `True` if `ctx.output == value`
**Example:**
```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import Equals

# Check output is always "success"
dataset = Dataset(
    cases=[Case(inputs='test')],
    evaluators=[
        Equals(value='success', evaluation_name='is_success'),
    ],
)

```

**Use Cases:**
  * Checking for sentinel values
  * Validating consistent outputs
  * Testing classification into specific categories


* * *
### Contains
Check if the output contains a specific value or substring.
```
from pydantic_evals.evaluators import Contains

Contains(
    value='substring',
    case_sensitive=True,
    as_strings=False,
)

```

**Parameters:**
  * `value` (Any): The value to search for
  * `case_sensitive` (bool): Case-sensitive comparison for strings (default: `True`)
  * `as_strings` (bool): Convert both values to strings before checking (default: `False`)
  * `evaluation_name` (str | None): Custom name for this evaluation in reports


**Returns:** [`EvaluationReason`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluationReason "EvaluationReason



      dataclass
  ") - Pass/fail with explanation
**Behavior:**
For **strings** : checks substring containment
  * `Contains(value='hello', case_sensitive=False)`
  * Matches: "Hello World", "say hello", "HELLO"
  * Doesn't match: "hi there"


For **lists/tuples** : checks membership
  * `Contains(value='apple')`
  * Matches: `['apple', 'banana']`, `('apple',)`
  * Doesn't match: `['apples', 'orange']`


For **dicts** : checks key-value pairs
  * `Contains(value={'name': 'Alice'})`
  * Matches: `{'name': 'Alice', 'age': 30}`
  * Doesn't match: `{'name': 'Bob'}`


**Example:**
```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import Contains

dataset = Dataset(
    cases=[Case(inputs='test')],
    evaluators=[
        # Check for required keywords
        Contains(value='terms and conditions', case_sensitive=False),
        # Check for PII (fail if found)
        # Note: Use a custom evaluator that returns False when PII found
    ],
)

```

**Use Cases:**
  * Required content verification
  * Keyword detection
  * PII/sensitive data detection
  * Multi-value validation


* * *
## Type Validation
### IsInstance
Check if the output is an instance of a type with the given name.
```
from pydantic_evals.evaluators import IsInstance

IsInstance(type_name='str')

```

**Parameters:**
  * `type_name` (str): The type name to check (uses `__name__` or `__qualname__`)
  * `evaluation_name` (str | None): Custom name for this evaluation in reports


**Returns:** [`EvaluationReason`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluationReason "EvaluationReason



      dataclass
  ") - Pass/fail with type information
**Example:**
```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import IsInstance

dataset = Dataset(
    cases=[Case(inputs='test')],
    evaluators=[
        # Check output is always a string
        IsInstance(type_name='str'),
        # Check for Pydantic model
        IsInstance(type_name='MyModel'),
        # Check for dict
        IsInstance(type_name='dict'),
    ],
)

```

**Notes:**
  * Matches against both `__name__` and `__qualname__` of the type
  * Works with built-in types (`str`, `int`, `dict`, `list`, etc.)
  * Works with custom classes and Pydantic models
  * Checks the entire MRO (Method Resolution Order) for inheritance


**Use Cases:**
  * Format validation
  * Structured output verification
  * Type consistency checks


* * *
## Performance Evaluation
### MaxDuration
Check if task execution time is under a maximum threshold.
```
from datetime import timedelta

from pydantic_evals.evaluators import MaxDuration

MaxDuration(seconds=2.0)
# or
MaxDuration(seconds=timedelta(seconds=2))

```

**Parameters:**
  * `seconds` (float | timedelta): Maximum allowed duration


**Returns:** `bool` - `True` if `ctx.duration <= seconds`
**Example:**
```
from datetime import timedelta

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import MaxDuration

dataset = Dataset(
    cases=[Case(inputs='test')],
    evaluators=[
        # SLA: must respond in under 2 seconds
        MaxDuration(seconds=2.0),
        # Or using timedelta
        MaxDuration(seconds=timedelta(milliseconds=500)),
    ],
)

```

**Use Cases:**
  * SLA compliance
  * Performance regression testing
  * Latency requirements
  * Timeout validation


**See Also:** [Concurrency & Performance](https://ai.pydantic.dev/evals/how-to/concurrency/)
* * *
## LLM-as-a-Judge
### LLMJudge
Use an LLM to evaluate subjective qualities based on a rubric.
```
from pydantic_evals.evaluators import LLMJudge

LLMJudge(
    rubric='Response is accurate and helpful',
    model='openai:gpt-5.2',
    include_input=False,
    include_expected_output=False,
    model_settings=None,
    score=False,
    assertion={'include_reason': True},
)

```

**Parameters:**
  * `rubric` (str): The evaluation criteria (required)
  * `model` (Model | KnownModelName | None): Model to use (default: `'openai:gpt-5.2'`)
  * `include_input` (bool): Include task inputs in the prompt (default: `False`)
  * `include_expected_output` (bool): Include expected output in the prompt (default: `False`)
  * `model_settings` (ModelSettings | None): Custom model settings
  * `score` (OutputConfig | False): Configure score output (default: `False`)
  * `assertion` (OutputConfig | False): Configure assertion output (default: includes reason)


**Returns:** Depends on `score` and `assertion` parameters (see below)
**Output Modes:**
By default, returns a **boolean assertion** with reason:
  * `LLMJudge(rubric='Response is polite')`
  * Returns: `{'LLMJudge_pass': EvaluationReason(value=True, reason='...')}`


Return a **score** (0.0 to 1.0) instead:
  * `LLMJudge(rubric='Response quality', score={'include_reason': True}, assertion=False)`
  * Returns: `{'LLMJudge_score': EvaluationReason(value=0.85, reason='...')}`


Return **both** score and assertion:
  * `LLMJudge(rubric='Response quality', score={'include_reason': True}, assertion={'include_reason': True})`
  * Returns: `{'LLMJudge_score': EvaluationReason(value=0.85, reason='...'), 'LLMJudge_pass': EvaluationReason(value=True, reason='...')}`


**Customize evaluation names:**
  * `LLMJudge(rubric='Response is factually accurate', assertion={'evaluation_name': 'accuracy', 'include_reason': True})`
  * Returns: `{'accuracy': EvaluationReason(value=True, reason='...')}`


**Example:**
```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import LLMJudge

dataset = Dataset(
    cases=[Case(inputs='test', expected_output='result')],
    evaluators=[
        # Basic accuracy check
        LLMJudge(
            rubric='Response is factually accurate',
            include_input=True,
        ),
        # Quality score with different model
        LLMJudge(
            rubric='Overall response quality',
            model='anthropic:claude-sonnet-4-6',
            score={'evaluation_name': 'quality', 'include_reason': False},
            assertion=False,
        ),
        # Check against expected output
        LLMJudge(
            rubric='Response matches the expected answer semantically',
            include_input=True,
            include_expected_output=True,
        ),
    ],
)

```

**See Also:** [LLM Judge Deep Dive](https://ai.pydantic.dev/evals/evaluators/llm-judge/)
* * *
## Span-Based Evaluation
### HasMatchingSpan
Check if OpenTelemetry spans match a query (requires Logfire configuration).
```
from pydantic_evals.evaluators import HasMatchingSpan

HasMatchingSpan(
    query={'name_contains': 'tool_call'},
    evaluation_name='called_tool',
)

```

**Parameters:**
  * `query` ([`SpanQuery`](https://ai.pydantic.dev/api/pydantic_evals/otel/#pydantic_evals.otel.SpanQuery "SpanQuery")): Query to match against spans
  * `evaluation_name` (str | None): Custom name for this evaluation in reports


**Returns:** `bool` - `True` if any span matches the query
**Example:**
```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import HasMatchingSpan

dataset = Dataset(
    cases=[Case(inputs='test')],
    evaluators=[
        # Check that a specific tool was called
        HasMatchingSpan(
            query={'name_contains': 'search_database'},
            evaluation_name='used_database',
        ),
        # Check for errors
        HasMatchingSpan(
            query={'has_attributes': {'error': True}},
            evaluation_name='had_errors',
        ),
        # Check duration constraints
        HasMatchingSpan(
            query={
                'name_equals': 'llm_call',
                'max_duration': 2.0,  # seconds
            },
            evaluation_name='llm_fast_enough',
        ),
    ],
)

```

**See Also:** [Span-Based Evaluation](https://ai.pydantic.dev/evals/evaluators/span-based/)
* * *
## Built-in Report Evaluators
In addition to the case-level evaluators above, Pydantic Evals provides report evaluators that analyze entire experiment results. These are passed via the `report_evaluators` parameter on `Dataset`.
Report Evaluator | Purpose | Output
---|---|---
[`ConfusionMatrixEvaluator`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ConfusionMatrixEvaluator "ConfusionMatrixEvaluator



      dataclass
  ") | Classification confusion matrix | `ConfusionMatrix`
[`PrecisionRecallEvaluator`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.PrecisionRecallEvaluator "PrecisionRecallEvaluator



      dataclass
  ") | PR curve with AUC | `PrecisionRecall`
**See:** [Report Evaluators](https://ai.pydantic.dev/evals/evaluators/report-evaluators/) for full documentation, parameters, and examples, including how to write custom report evaluators that produce `ScalarResult` and `TableResult` analyses.
* * *
## Quick Reference Table
### Case-Level Evaluators
Evaluator | Purpose | Return Type | Cost | Speed
---|---|---|---|---
[`EqualsExpected`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EqualsExpected "EqualsExpected



      dataclass
  ") | Exact match with expected | `bool` | Free | Instant
[`Equals`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Equals "Equals



      dataclass
  ") | Equals specific value | `bool` | Free | Instant
[`Contains`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Contains "Contains



      dataclass
  ") | Contains value/substring |  `bool` + reason | Free | Instant
[`IsInstance`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.IsInstance "IsInstance



      dataclass
  ") | Type validation |  `bool` + reason | Free | Instant
[`MaxDuration`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.MaxDuration "MaxDuration



      dataclass
  ") | Performance threshold | `bool` | Free | Instant
[`LLMJudge`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.LLMJudge "LLMJudge



      dataclass
  ") | Subjective quality |  `bool` and/or `float` | $$ | Slow
[`HasMatchingSpan`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.HasMatchingSpan "HasMatchingSpan



      dataclass
  ") | Behavioral check | `bool` | Free | Fast
### Report-Level Evaluators
Evaluator | Purpose | Output Type | Cost | Speed
---|---|---|---|---
[`ConfusionMatrixEvaluator`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ConfusionMatrixEvaluator "ConfusionMatrixEvaluator



      dataclass
  ") | Classification matrix | `ConfusionMatrix` | Free | Instant
[`PrecisionRecallEvaluator`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.PrecisionRecallEvaluator "PrecisionRecallEvaluator



      dataclass
  ") | PR curve with AUC | `PrecisionRecall` | Free | Instant
## Combining Evaluators
Best practice is to combine fast deterministic checks with slower LLM evaluations:
```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import (
    Contains,
    IsInstance,
    LLMJudge,
    MaxDuration,
)

dataset = Dataset(
    cases=[Case(inputs='test')],
    evaluators=[
        # Fast checks first (fail fast)
        IsInstance(type_name='str'),
        Contains(value='required_field'),
        MaxDuration(seconds=2.0),
        # Expensive LLM checks last
        LLMJudge(rubric='Response is helpful and accurate'),
    ],
)

```

This approach:
  1. Catches format/structure issues immediately
  2. Validates required content quickly
  3. Only runs expensive LLM evaluation if basic checks pass
  4. Provides comprehensive quality assessment


## Next Steps
  * **[LLM Judge](https://ai.pydantic.dev/evals/evaluators/llm-judge/)** - Deep dive on LLM-as-a-Judge evaluation
  * **[Custom Evaluators](https://ai.pydantic.dev/evals/evaluators/custom/)** - Write your own evaluation logic
  * **[Report Evaluators](https://ai.pydantic.dev/evals/evaluators/report-evaluators/)** - Experiment-wide analyses (confusion matrices, PR curves, etc.)
  * **[Span-Based Evaluation](https://ai.pydantic.dev/evals/evaluators/span-based/)** - Using OpenTelemetry spans for behavioral checks


© Pydantic Services Inc. 2024 to present
