[ Skip to content ](https://ai.pydantic.dev/evals/evaluators/overview/#evaluators-overview)
**Join us at the inaugural PyAI Conf in San Francisco on March 10th![Learn More](https://pyai.events/?utm_source=pydantic-ai) **
[ ![logo](https://ai.pydantic.dev/img/logo-white.svg) ](https://ai.pydantic.dev/ "Pydantic AI")
Pydantic AI
Overview
Type to start searching
[ pydantic/pydantic-ai
  * v1.63.0
  * 15.1k
  * 1.7k

](https://github.com/pydantic/pydantic-ai "Go to repository")
[ ![logo](https://ai.pydantic.dev/img/logo-white.svg) ](https://ai.pydantic.dev/ "Pydantic AI") Pydantic AI
[ pydantic/pydantic-ai
  * v1.63.0
  * 15.1k
  * 1.7k

](https://github.com/pydantic/pydantic-ai "Go to repository")
  * [ Pydantic AI  ](https://ai.pydantic.dev/)
  * [ Installation  ](https://ai.pydantic.dev/install/)
  * [ Getting Help  ](https://ai.pydantic.dev/help/)
  * [ Troubleshooting  ](https://ai.pydantic.dev/troubleshooting/)
  * [ Pydantic AI Gateway  ](https://ai.pydantic.dev/gateway/)
  * Documentation
    * Core Concepts
      * [ Agents  ](https://ai.pydantic.dev/agent/)
      * [ Dependencies  ](https://ai.pydantic.dev/dependencies/)
      * [ Function Tools  ](https://ai.pydantic.dev/tools/)
      * [ Output  ](https://ai.pydantic.dev/output/)
      * [ Messages and chat history  ](https://ai.pydantic.dev/message-history/)
      * [ Direct Model Requests  ](https://ai.pydantic.dev/direct/)
    * Models & Providers
      * [ Overview  ](https://ai.pydantic.dev/models/overview/)
      * [ OpenAI  ](https://ai.pydantic.dev/models/openai/)
      * [ Anthropic  ](https://ai.pydantic.dev/models/anthropic/)
      * [ Google  ](https://ai.pydantic.dev/models/google/)
      * [ xAI  ](https://ai.pydantic.dev/models/xai/)
      * [ Bedrock  ](https://ai.pydantic.dev/models/bedrock/)
      * [ Cerebras  ](https://ai.pydantic.dev/models/cerebras/)
      * [ Cohere  ](https://ai.pydantic.dev/models/cohere/)
      * [ Groq  ](https://ai.pydantic.dev/models/groq/)
      * [ Hugging Face  ](https://ai.pydantic.dev/models/huggingface/)
      * [ Mistral  ](https://ai.pydantic.dev/models/mistral/)
      * [ OpenRouter  ](https://ai.pydantic.dev/models/openrouter/)
      * [ Outlines  ](https://ai.pydantic.dev/models/outlines/)
    * Tools & Toolsets
      * [ Function Tools  ](https://ai.pydantic.dev/tools/)
      * [ Advanced Tool Features  ](https://ai.pydantic.dev/tools-advanced/)
      * [ Toolsets  ](https://ai.pydantic.dev/toolsets/)
      * [ Deferred Tools  ](https://ai.pydantic.dev/deferred-tools/)
      * [ Built-in Tools  ](https://ai.pydantic.dev/builtin-tools/)
      * [ Common Tools  ](https://ai.pydantic.dev/common-tools/)
      * [ Third-Party Tools  ](https://ai.pydantic.dev/third-party-tools/)
    * Advanced Features
      * [ Image, Audio, Video & Document Input  ](https://ai.pydantic.dev/input/)
      * [ Thinking  ](https://ai.pydantic.dev/thinking/)
      * [ HTTP Request Retries  ](https://ai.pydantic.dev/retries/)
    * MCP
      * [ Overview  ](https://ai.pydantic.dev/mcp/overview/)
      * [ Client  ](https://ai.pydantic.dev/mcp/client/)
      * [ FastMCP Client  ](https://ai.pydantic.dev/mcp/fastmcp-client/)
      * [ Server  ](https://ai.pydantic.dev/mcp/server/)
    * [ Multi-Agent Patterns  ](https://ai.pydantic.dev/multi-agent-applications/)
    * [ Web Chat UI  ](https://ai.pydantic.dev/web/)
    * [ Embeddings  ](https://ai.pydantic.dev/embeddings/)
    * [ Testing  ](https://ai.pydantic.dev/testing/)
  * Pydantic Evals
    * [ Overview  ](https://ai.pydantic.dev/evals/)
    * Getting Started
      * [ Quick Start  ](https://ai.pydantic.dev/evals/quick-start/)
      * [ Core Concepts  ](https://ai.pydantic.dev/evals/core-concepts/)
    * Evaluators
      * Overview  [ Overview  ](https://ai.pydantic.dev/evals/evaluators/overview/)
        * [ When to Use Different Evaluators  ](https://ai.pydantic.dev/evals/evaluators/overview/#when-to-use-different-evaluators)
          * [ Deterministic Checks (Fast & Reliable)  ](https://ai.pydantic.dev/evals/evaluators/overview/#deterministic-checks-fast-reliable)
          * [ LLM-as-a-Judge (Flexible & Nuanced)  ](https://ai.pydantic.dev/evals/evaluators/overview/#llm-as-a-judge-flexible-nuanced)
          * [ Custom Evaluators  ](https://ai.pydantic.dev/evals/evaluators/overview/#custom-evaluators)
        * [ Evaluation Types  ](https://ai.pydantic.dev/evals/evaluators/overview/#evaluation-types)
          * [ 1. Assertions (bool)  ](https://ai.pydantic.dev/evals/evaluators/overview/#1-assertions-bool)
          * [ 2. Scores (int or float)  ](https://ai.pydantic.dev/evals/evaluators/overview/#2-scores-int-or-float)
          * [ 3. Labels (str)  ](https://ai.pydantic.dev/evals/evaluators/overview/#3-labels-str)
          * [ Multiple Results  ](https://ai.pydantic.dev/evals/evaluators/overview/#multiple-results)
        * [ Combining Evaluators  ](https://ai.pydantic.dev/evals/evaluators/overview/#combining-evaluators)
        * [ Case-specific evaluators  ](https://ai.pydantic.dev/evals/evaluators/overview/#case-specific-evaluators)
          * [ Why Case-Specific Evaluators Matter  ](https://ai.pydantic.dev/evals/evaluators/overview/#why-case-specific-evaluators-matter)
          * [ Building Golden Datasets with Case-Specific LLMJudge  ](https://ai.pydantic.dev/evals/evaluators/overview/#building-golden-datasets-with-case-specific-llmjudge)
        * [ Async vs Sync  ](https://ai.pydantic.dev/evals/evaluators/overview/#async-vs-sync)
        * [ Evaluation Context  ](https://ai.pydantic.dev/evals/evaluators/overview/#evaluation-context)
        * [ Error Handling  ](https://ai.pydantic.dev/evals/evaluators/overview/#error-handling)
        * [ Report Evaluators (Experiment-Wide)  ](https://ai.pydantic.dev/evals/evaluators/overview/#report-evaluators-experiment-wide)
        * [ Next Steps  ](https://ai.pydantic.dev/evals/evaluators/overview/#next-steps)
      * [ Built-in Evaluators  ](https://ai.pydantic.dev/evals/evaluators/built-in/)
      * [ LLM Judge  ](https://ai.pydantic.dev/evals/evaluators/llm-judge/)
      * [ Custom Evaluators  ](https://ai.pydantic.dev/evals/evaluators/custom/)
      * [ Report Evaluators  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/)
      * [ Span-Based  ](https://ai.pydantic.dev/evals/evaluators/span-based/)
    * How-To Guides
      * [ Logfire Integration  ](https://ai.pydantic.dev/evals/how-to/logfire-integration/)
      * [ Dataset Management  ](https://ai.pydantic.dev/evals/how-to/dataset-management/)
      * [ Dataset Serialization  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/)
      * [ Concurrency & Performance  ](https://ai.pydantic.dev/evals/how-to/concurrency/)
      * [ Multi-Run Evaluation  ](https://ai.pydantic.dev/evals/how-to/multi-run/)
      * [ Retry Strategies  ](https://ai.pydantic.dev/evals/how-to/retry-strategies/)
      * [ Metrics & Attributes  ](https://ai.pydantic.dev/evals/how-to/metrics-attributes/)
    * Examples
      * [ Simple Validation  ](https://ai.pydantic.dev/evals/examples/simple-validation/)
  * Pydantic Graph
    * [ Overview  ](https://ai.pydantic.dev/graph/)
    * [ Beta API  ](https://ai.pydantic.dev/graph/beta/)
      * [ Steps  ](https://ai.pydantic.dev/graph/beta/steps/)
      * [ Joins & Reducers  ](https://ai.pydantic.dev/graph/beta/joins/)
      * [ Decisions  ](https://ai.pydantic.dev/graph/beta/decisions/)
      * [ Parallel Execution  ](https://ai.pydantic.dev/graph/beta/parallel/)
  * Integrations
    * [ Debugging & Monitoring with Pydantic Logfire  ](https://ai.pydantic.dev/logfire/)
    * Durable Execution
      * [ Overview  ](https://ai.pydantic.dev/durable_execution/overview/)
      * [ Temporal  ](https://ai.pydantic.dev/durable_execution/temporal/)
      * [ DBOS  ](https://ai.pydantic.dev/durable_execution/dbos/)
      * [ Prefect  ](https://ai.pydantic.dev/durable_execution/prefect/)
    * UI Event Streams
      * [ Overview  ](https://ai.pydantic.dev/ui/overview/)
      * [ AG-UI  ](https://ai.pydantic.dev/ui/ag-ui/)
      * [ Vercel AI  ](https://ai.pydantic.dev/ui/vercel-ai/)
    * [ Agent2Agent (A2A)  ](https://ai.pydantic.dev/a2a/)
  * Related Packages
    * [ clai  ](https://ai.pydantic.dev/cli/)
  * Examples
    * [ Setup  ](https://ai.pydantic.dev/examples/setup/)
    * Getting Started
      * [ Pydantic Model  ](https://ai.pydantic.dev/examples/pydantic-model/)
      * [ Weather agent  ](https://ai.pydantic.dev/examples/weather-agent/)
    * Conversational Agents
      * [ Chat App with FastAPI  ](https://ai.pydantic.dev/examples/chat-app/)
      * [ Bank support  ](https://ai.pydantic.dev/examples/bank-support/)
    * Data & Analytics
      * [ SQL Generation  ](https://ai.pydantic.dev/examples/sql-gen/)
      * [ Data Analyst  ](https://ai.pydantic.dev/examples/data-analyst/)
      * [ RAG  ](https://ai.pydantic.dev/examples/rag/)
    * Streaming
      * [ Stream markdown  ](https://ai.pydantic.dev/examples/stream-markdown/)
      * [ Stream whales  ](https://ai.pydantic.dev/examples/stream-whales/)
    * Complex Workflows
      * [ Flight booking  ](https://ai.pydantic.dev/examples/flight-booking/)
      * [ Question Graph  ](https://ai.pydantic.dev/examples/question-graph/)
    * Business Applications
      * [ Slack Lead Qualifier with Modal  ](https://ai.pydantic.dev/examples/slack-lead-qualifier/)
    * UI Examples
      * [ Agent User Interaction (AG-UI)  ](https://ai.pydantic.dev/examples/ag-ui/)
  * API Reference
    * pydantic_ai
      * [ pydantic_ai.ag_ui  ](https://ai.pydantic.dev/api/ag_ui/)
      * [ pydantic_ai.agent  ](https://ai.pydantic.dev/api/agent/)
      * [ pydantic_ai.builtin_tools  ](https://ai.pydantic.dev/api/builtin_tools/)
      * [ pydantic_ai.common_tools  ](https://ai.pydantic.dev/api/common_tools/)
      * [ pydantic_ai — Concurrency  ](https://ai.pydantic.dev/api/concurrency/)
      * [ pydantic_ai.direct  ](https://ai.pydantic.dev/api/direct/)
      * [ pydantic_ai.durable_exec  ](https://ai.pydantic.dev/api/durable_exec/)
      * [ pydantic_ai.embeddings  ](https://ai.pydantic.dev/api/embeddings/)
      * [ pydantic_ai.exceptions  ](https://ai.pydantic.dev/api/exceptions/)
      * [ pydantic_ai.ext  ](https://ai.pydantic.dev/api/ext/)
      * [ pydantic_ai.format_prompt  ](https://ai.pydantic.dev/api/format_prompt/)
      * [ pydantic_ai.mcp  ](https://ai.pydantic.dev/api/mcp/)
      * [ pydantic_ai.messages  ](https://ai.pydantic.dev/api/messages/)
      * [ pydantic_ai.models.anthropic  ](https://ai.pydantic.dev/api/models/anthropic/)
      * [ pydantic_ai.models  ](https://ai.pydantic.dev/api/models/base/)
      * [ pydantic_ai.models.bedrock  ](https://ai.pydantic.dev/api/models/bedrock/)
      * [ pydantic_ai.models.cerebras  ](https://ai.pydantic.dev/api/models/cerebras/)
      * [ pydantic_ai.models.cohere  ](https://ai.pydantic.dev/api/models/cohere/)
      * [ pydantic_ai.models.fallback  ](https://ai.pydantic.dev/api/models/fallback/)
      * [ pydantic_ai.models.function  ](https://ai.pydantic.dev/api/models/function/)
      * [ pydantic_ai.models.google  ](https://ai.pydantic.dev/api/models/google/)
      * [ pydantic_ai.models.xai  ](https://ai.pydantic.dev/api/models/xai/)
      * [ pydantic_ai.models.groq  ](https://ai.pydantic.dev/api/models/groq/)
      * [ pydantic_ai.models.huggingface  ](https://ai.pydantic.dev/api/models/huggingface/)
      * [ pydantic_ai.models.instrumented  ](https://ai.pydantic.dev/api/models/instrumented/)
      * [ pydantic_ai.models.mcp_sampling  ](https://ai.pydantic.dev/api/models/mcp-sampling/)
      * [ pydantic_ai.models.mistral  ](https://ai.pydantic.dev/api/models/mistral/)
      * [ pydantic_ai.models.openai  ](https://ai.pydantic.dev/api/models/openai/)
      * [ pydantic_ai.models.openrouter  ](https://ai.pydantic.dev/api/models/openrouter/)
      * [ pydantic_ai.models.outlines  ](https://ai.pydantic.dev/api/models/outlines/)
      * [ pydantic_ai.models.test  ](https://ai.pydantic.dev/api/models/test/)
      * [ pydantic_ai.models.wrapper  ](https://ai.pydantic.dev/api/models/wrapper/)
      * [ pydantic_ai.output  ](https://ai.pydantic.dev/api/output/)
      * [ pydantic_ai.profiles  ](https://ai.pydantic.dev/api/profiles/)
      * [ pydantic_ai.providers  ](https://ai.pydantic.dev/api/providers/)
      * [ pydantic_ai.result  ](https://ai.pydantic.dev/api/result/)
      * [ pydantic_ai.retries  ](https://ai.pydantic.dev/api/retries/)
      * [ pydantic_ai.run  ](https://ai.pydantic.dev/api/run/)
      * [ pydantic_ai.settings  ](https://ai.pydantic.dev/api/settings/)
      * [ pydantic_ai.tools  ](https://ai.pydantic.dev/api/tools/)
      * [ pydantic_ai.toolsets  ](https://ai.pydantic.dev/api/toolsets/)
      * [ pydantic_ai.ui.ag_ui  ](https://ai.pydantic.dev/api/ui/ag_ui/)
      * [ pydantic_ai.ui  ](https://ai.pydantic.dev/api/ui/base/)
      * [ pydantic_ai.ui.vercel_ai  ](https://ai.pydantic.dev/api/ui/vercel_ai/)
      * [ pydantic_ai.usage  ](https://ai.pydantic.dev/api/usage/)
    * pydantic_evals
      * [ pydantic_evals.dataset  ](https://ai.pydantic.dev/api/pydantic_evals/dataset/)
      * [ pydantic_evals.evaluators  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/)
      * [ pydantic_evals.reporting  ](https://ai.pydantic.dev/api/pydantic_evals/reporting/)
      * [ pydantic_evals.otel  ](https://ai.pydantic.dev/api/pydantic_evals/otel/)
      * [ pydantic_evals.generation  ](https://ai.pydantic.dev/api/pydantic_evals/generation/)
    * pydantic_graph
      * [ pydantic_graph  ](https://ai.pydantic.dev/api/pydantic_graph/graph/)
      * [ pydantic_graph.nodes  ](https://ai.pydantic.dev/api/pydantic_graph/nodes/)
      * [ pydantic_graph.persistence  ](https://ai.pydantic.dev/api/pydantic_graph/persistence/)
      * [ pydantic_graph.mermaid  ](https://ai.pydantic.dev/api/pydantic_graph/mermaid/)
      * [ pydantic_graph.exceptions  ](https://ai.pydantic.dev/api/pydantic_graph/exceptions/)
      * Beta API
        * [ pydantic_graph.beta  ](https://ai.pydantic.dev/api/pydantic_graph/beta/)
        * [ pydantic_graph.beta.graph  ](https://ai.pydantic.dev/api/pydantic_graph/beta_graph/)
        * [ pydantic_graph.beta.graph_builder  ](https://ai.pydantic.dev/api/pydantic_graph/beta_graph_builder/)
        * [ pydantic_graph.beta.step  ](https://ai.pydantic.dev/api/pydantic_graph/beta_step/)
        * [ pydantic_graph.beta.join  ](https://ai.pydantic.dev/api/pydantic_graph/beta_join/)
        * [ pydantic_graph.beta.decision  ](https://ai.pydantic.dev/api/pydantic_graph/beta_decision/)
        * [ pydantic_graph.beta.node  ](https://ai.pydantic.dev/api/pydantic_graph/beta_node/)
    * fasta2a
      * [ fasta2a  ](https://ai.pydantic.dev/api/fasta2a/)
  * Project
    * [ Contributing  ](https://ai.pydantic.dev/contributing/)
    * [ Upgrade Guide  ](https://ai.pydantic.dev/changelog/)
    * [ Version policy  ](https://ai.pydantic.dev/version-policy/)


  * [ When to Use Different Evaluators  ](https://ai.pydantic.dev/evals/evaluators/overview/#when-to-use-different-evaluators)
    * [ Deterministic Checks (Fast & Reliable)  ](https://ai.pydantic.dev/evals/evaluators/overview/#deterministic-checks-fast-reliable)
    * [ LLM-as-a-Judge (Flexible & Nuanced)  ](https://ai.pydantic.dev/evals/evaluators/overview/#llm-as-a-judge-flexible-nuanced)
    * [ Custom Evaluators  ](https://ai.pydantic.dev/evals/evaluators/overview/#custom-evaluators)
  * [ Evaluation Types  ](https://ai.pydantic.dev/evals/evaluators/overview/#evaluation-types)
    * [ 1. Assertions (bool)  ](https://ai.pydantic.dev/evals/evaluators/overview/#1-assertions-bool)
    * [ 2. Scores (int or float)  ](https://ai.pydantic.dev/evals/evaluators/overview/#2-scores-int-or-float)
    * [ 3. Labels (str)  ](https://ai.pydantic.dev/evals/evaluators/overview/#3-labels-str)
    * [ Multiple Results  ](https://ai.pydantic.dev/evals/evaluators/overview/#multiple-results)
  * [ Combining Evaluators  ](https://ai.pydantic.dev/evals/evaluators/overview/#combining-evaluators)
  * [ Case-specific evaluators  ](https://ai.pydantic.dev/evals/evaluators/overview/#case-specific-evaluators)
    * [ Why Case-Specific Evaluators Matter  ](https://ai.pydantic.dev/evals/evaluators/overview/#why-case-specific-evaluators-matter)
    * [ Building Golden Datasets with Case-Specific LLMJudge  ](https://ai.pydantic.dev/evals/evaluators/overview/#building-golden-datasets-with-case-specific-llmjudge)
  * [ Async vs Sync  ](https://ai.pydantic.dev/evals/evaluators/overview/#async-vs-sync)
  * [ Evaluation Context  ](https://ai.pydantic.dev/evals/evaluators/overview/#evaluation-context)
  * [ Error Handling  ](https://ai.pydantic.dev/evals/evaluators/overview/#error-handling)
  * [ Report Evaluators (Experiment-Wide)  ](https://ai.pydantic.dev/evals/evaluators/overview/#report-evaluators-experiment-wide)
  * [ Next Steps  ](https://ai.pydantic.dev/evals/evaluators/overview/#next-steps)


  1. [ Pydantic AI  ](https://ai.pydantic.dev/)
  2. [ Pydantic Evals  ](https://ai.pydantic.dev/evals/)
  3. [ Evaluators  ](https://ai.pydantic.dev/evals/evaluators/overview/)


# Evaluators Overview
Evaluators are the core of Pydantic Evals. They analyze task outputs and provide scores, labels, or pass/fail assertions.
## When to Use Different Evaluators
### Deterministic Checks (Fast & Reliable)
Use deterministic evaluators when you can define exact rules:
Evaluator | Use Case | Example
---|---|---
[`EqualsExpected`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EqualsExpected "EqualsExpected



      dataclass
  ") | Exact output match | Structured data, classification
[`Equals`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Equals "Equals



      dataclass
  ") | Equals specific value | Checking for sentinel values
[`Contains`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Contains "Contains



      dataclass
  ") | Substring/element check | Required keywords, PII detection
[`IsInstance`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.IsInstance "IsInstance



      dataclass
  ") | Type validation | Format validation
[`MaxDuration`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.MaxDuration "MaxDuration



      dataclass
  ") | Performance threshold | SLA compliance
[`HasMatchingSpan`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.HasMatchingSpan "HasMatchingSpan



      dataclass
  ") | Behavior verification | Tool calls, code paths
**Advantages:**
  * Fast execution (microseconds to milliseconds)
  * Deterministic results
  * No cost
  * Easy to debug


**When to use:**
  * Format validation (JSON structure, type checking)
  * Required content checks (must contain X, must not contain Y)
  * Performance requirements (latency, token counts)
  * Behavioral checks (which tools were called, which code paths executed)


### LLM-as-a-Judge (Flexible & Nuanced)
Use [`LLMJudge`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.LLMJudge "LLMJudge



      dataclass
  ") when evaluation requires understanding or judgment:
```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import LLMJudge

dataset = Dataset(
    cases=[Case(inputs='What is 2+2?', expected_output='4')],
    evaluators=[
        LLMJudge(
            rubric='Response is factually accurate based on the input',
            include_input=True,
        )
    ],
)

```

**Advantages:**
  * Can evaluate subjective qualities (helpfulness, tone, creativity)
  * Understands natural language
  * Can follow complex rubrics
  * Flexible across domains


**Disadvantages:**
  * Slower (seconds per evaluation)
  * Costs money
  * Non-deterministic
  * Can have biases


**When to use:**
  * Factual accuracy
  * Relevance and helpfulness
  * Tone and style
  * Completeness
  * Following instructions
  * RAG quality (groundedness, citation accuracy)


### Custom Evaluators
Custom evaluators can be useful if you want to make use of any evaluation logic we don't provide with the framework. They are frequently useful for domain-specific logic:
```
from dataclasses import dataclass

from pydantic_evals.evaluators import Evaluator, EvaluatorContext


@dataclass
class ValidSQL(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> bool:
        try:
            import sqlparse
            sqlparse.parse(ctx.output)
            return True
        except Exception:
            return False

```

**When to use:**
  * Domain-specific validation (SQL syntax, regex patterns, business rules)
  * External API calls (running generated code, checking databases)
  * Complex calculations (precision/recall, BLEU scores)
  * Integration checks (does API call succeed?)


## Evaluation Types
Detailed Return Types Guide
For full detail about precisely what custom Evaluators may return, see [Custom Evaluator Return Types](https://ai.pydantic.dev/evals/evaluators/custom/#return-types).
Evaluators essentially return three types of results:
### 1. Assertions (bool)
Pass/fail checks that appear as ✔ or ✗ in reports:
```
from dataclasses import dataclass

from pydantic_evals.evaluators import Evaluator, EvaluatorContext


@dataclass
class HasKeyword(Evaluator):
    keyword: str

    def evaluate(self, ctx: EvaluatorContext) -> bool:
        return self.keyword in ctx.output

```

**Use for:** Binary checks, quality gates, compliance requirements
### 2. Scores (int or float)
Numeric metrics:
```
from dataclasses import dataclass

from pydantic_evals.evaluators import Evaluator, EvaluatorContext


@dataclass
class ConfidenceScore(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> float:
        # Analyze and return score
        return 0.87  # 87% confidence

```

**Use for:** Quality metrics, ranking, A/B testing, regression tracking
### 3. Labels (str)
Categorical classifications:
```
from dataclasses import dataclass

from pydantic_evals.evaluators import Evaluator, EvaluatorContext


@dataclass
class SentimentClassifier(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> str:
        if 'error' in ctx.output.lower():
            return 'error'
        elif 'success' in ctx.output.lower():
            return 'success'
        return 'neutral'

```

**Use for:** Classification, error categorization, quality buckets
### Multiple Results
You can return multiple evaluations from a single evaluator:
```
from dataclasses import dataclass

from pydantic_evals.evaluators import Evaluator, EvaluatorContext


@dataclass
class ComprehensiveCheck(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> dict[str, bool | float | str]:
        return {
            'valid_format': self._check_format(ctx.output),  # bool
            'quality_score': self._score_quality(ctx.output),  # float
            'category': self._classify(ctx.output),  # str
        }

    def _check_format(self, output: str) -> bool:
        return True

    def _score_quality(self, output: str) -> float:
        return 0.85

    def _classify(self, output: str) -> str:
        return 'good'

```

## Combining Evaluators
Mix and match evaluators to create comprehensive evaluation suites:
```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import (
    Contains,
    IsInstance,
    LLMJudge,
    MaxDuration,
)

dataset = Dataset(
    cases=[Case(inputs='test', expected_output='result')],
    evaluators=[
        # Fast deterministic checks first
        IsInstance(type_name='str'),
        Contains(value='required_field'),
        MaxDuration(seconds=2.0),
        # Slower LLM checks after
        LLMJudge(
            rubric='Response is accurate and helpful',
            include_input=True,
        ),
    ],
)

```

## Case-specific evaluators
Case-specific evaluators are one of the most powerful features for building comprehensive evaluation suites. You can attach evaluators to individual [`Case`](https://ai.pydantic.dev/api/pydantic_evals/dataset/#pydantic_evals.dataset.Case "Case



      dataclass
  ") objects that only run for those specific cases:
```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import IsInstance, LLMJudge

dataset = Dataset(
    cases=[
        Case(
            name='greeting_response',
            inputs='Say hello',
            evaluators=[
                # This evaluator only runs for this case
                LLMJudge(
                    rubric='Response is warm and friendly, uses casual tone',
                    include_input=True,
                ),
            ],
        ),
        Case(
            name='formal_response',
            inputs='Write a business email',
            evaluators=[
                # Different requirements for this case
                LLMJudge(
                    rubric='Response is professional and formal, uses business language',
                    include_input=True,
                ),
            ],
        ),
    ],
    evaluators=[
        # This runs for ALL cases
        IsInstance(type_name='str'),
    ],
)

```

### Why Case-Specific Evaluators Matter
Case-specific evaluators solve a fundamental problem with one-size-fits-all evaluation: **if you could write a single evaluator rubric that perfectly captured your requirements across all cases, you'd just incorporate that rubric into your agent's instructions**. (Note: this is less relevant in cases where you want to use a cheaper model in production and assess it using a more expensive model, but in many cases it makes sense to use the best model you can in production.)
The power of case-specific evaluation comes from the nuance:
  * **Different cases have different requirements** : A customer support response needs empathy; a technical API response needs precision
  * **Avoid "inmates running the asylum"** : If your LLMJudge rubric is generic enough to work everywhere, your agent should already be following it
  * **Capture nuanced golden behavior** : Each case can specify exactly what "good" looks like for that scenario


### Building Golden Datasets with Case-Specific LLMJudge
A particularly powerful pattern is using case-specific [`LLMJudge`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.LLMJudge "LLMJudge



      dataclass
  ") evaluators to quickly build comprehensive, maintainable evaluation suites. Instead of needing exact `expected_output` values, you can describe what you care about:
```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import LLMJudge

dataset = Dataset(
    cases=[
        Case(
            name='handle_refund_request',
            inputs={'query': 'I want my money back', 'order_id': '12345'},
            evaluators=[
                LLMJudge(
                    rubric="""
                    Response should:
                    1. Acknowledge the refund request empathetically
                    2. Ask for the reason for the refund
                    3. Mention our 30-day refund policy
                    4. NOT process the refund immediately (needs manager approval)
                    """,
                    include_input=True,
                ),
            ],
        ),
        Case(
            name='handle_shipping_question',
            inputs={'query': 'Where is my order?', 'order_id': '12345'},
            evaluators=[
                LLMJudge(
                    rubric="""
                    Response should:
                    1. Confirm the order number
                    2. Provide tracking information
                    3. Give estimated delivery date
                    4. Be brief and factual (not overly apologetic)
                    """,
                    include_input=True,
                ),
            ],
        ),
        Case(
            name='handle_angry_customer',
            inputs={'query': 'This is completely unacceptable!', 'order_id': '12345'},
            evaluators=[
                LLMJudge(
                    rubric="""
                    Response should:
                    1. Prioritize de-escalation with empathy
                    2. Avoid being defensive
                    3. Offer concrete next steps
                    4. Use phrases like "I understand" and "Let me help"
                    """,
                    include_input=True,
                ),
            ],
        ),
    ],
)

```

This approach lets you:
  * **Build comprehensive test suites quickly** : Just describe what you want per case
  * **Maintain easily** : Update rubrics as requirements change, without regenerating outputs
  * **Cover edge cases naturally** : Add new cases with specific requirements as you discover them
  * **Capture domain knowledge** : Each rubric documents what "good" means for that scenario


The LLM evaluator excels at understanding nuanced requirements and assessing compliance, making this a practical way to create thorough evaluation coverage without brittleness.
## Async vs Sync
Evaluators can be sync or async:
```
from dataclasses import dataclass

from pydantic_evals.evaluators import Evaluator, EvaluatorContext


@dataclass
class SyncEvaluator(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> bool:
        return True


async def some_async_operation() -> bool:
    return True


@dataclass
class AsyncEvaluator(Evaluator):
    async def evaluate(self, ctx: EvaluatorContext) -> bool:
        result = await some_async_operation()
        return result

```

Pydantic Evals handles both automatically. Use async when: - Making API calls - Running database queries - Performing I/O operations - Calling LLMs (like [`LLMJudge`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.LLMJudge "LLMJudge



      dataclass
  "))
## Evaluation Context
All evaluators receive an [`EvaluatorContext`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext "EvaluatorContext



      dataclass
  "):
  * `ctx.inputs` - Task inputs
  * `ctx.output` - Task output (to evaluate)
  * `ctx.expected_output` - Expected output (if provided)
  * `ctx.metadata` - Case metadata (if provided)
  * `ctx.duration` - Task execution time (seconds)
  * `ctx.span_tree` - OpenTelemetry spans (if logfire configured)
  * `ctx.metrics` - Custom metrics dict
  * `ctx.attributes` - Custom attributes dict


This gives evaluators full context to make informed assessments.
## Error Handling
If an evaluator raises an exception, it's captured as an [`EvaluatorFailure`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorFailure "EvaluatorFailure



      dataclass
  "):
```
from dataclasses import dataclass

from pydantic_evals.evaluators import Evaluator, EvaluatorContext


def risky_operation(output: str) -> bool:
    # This might raise an exception
    if 'error' in output:
        raise ValueError('Found error in output')
    return True


@dataclass
class RiskyEvaluator(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> bool:
        # If this raises an exception, it will be captured
        result = risky_operation(ctx.output)
        return result

```

Failures appear in `report.cases[i].evaluator_failures` with:
  * Evaluator name
  * Error message
  * Full stacktrace


Use retry configuration to handle transient failures (see [Retry Strategies](https://ai.pydantic.dev/evals/how-to/retry-strategies/)).
## Report Evaluators (Experiment-Wide)
All the evaluators above run once per case. **Report evaluators** are different: they run once per experiment after all cases have been evaluated, and analyze the full set of results together.
Use report evaluators for experiment-wide statistics like:
  * **Confusion matrices** — visualize classification accuracy across classes
  * **Precision-recall curves** — assess ranking quality with AUC scores
  * **Scalar metrics** — overall accuracy, F1, BLEU, or any single number
  * **Summary tables** — per-class breakdowns, error category summaries


```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import ConfusionMatrixEvaluator

dataset = Dataset(
    cases=[
        Case(inputs='meow', expected_output='cat'),
        Case(inputs='woof', expected_output='dog'),
    ],
    report_evaluators=[
        ConfusionMatrixEvaluator(
            predicted_from='output',
            expected_from='expected_output',
        ),
    ],
)

```

**See:** [Report Evaluators](https://ai.pydantic.dev/evals/evaluators/report-evaluators/) for the full guide, including built-in report evaluators and how to write custom ones.
## Next Steps
  * **[Built-in Evaluators](https://ai.pydantic.dev/evals/evaluators/built-in/)** - Complete reference of all provided evaluators
  * **[LLM Judge](https://ai.pydantic.dev/evals/evaluators/llm-judge/)** - Deep dive on LLM-as-a-Judge evaluation
  * **[Custom Evaluators](https://ai.pydantic.dev/evals/evaluators/custom/)** - Write your own evaluation logic
  * **[Report Evaluators](https://ai.pydantic.dev/evals/evaluators/report-evaluators/)** - Experiment-wide analyses
  * **[Span-Based Evaluation](https://ai.pydantic.dev/evals/evaluators/span-based/)** - Evaluate using OpenTelemetry spans


© Pydantic Services Inc. 2024 to present
