[ Skip to content ](https://ai.pydantic.dev/evals/core-concepts/#core-concepts)
**Join us at the inaugural PyAI Conf in San Francisco on March 10th![Learn More](https://pyai.events/?utm_source=pydantic-ai) **
[ ![logo](https://ai.pydantic.dev/img/logo-white.svg) ](https://ai.pydantic.dev/ "Pydantic AI")
Pydantic AI
Core Concepts
Type to start searching
[ pydantic/pydantic-ai
  * v1.63.0
  * 15.1k
  * 1.7k

](https://github.com/pydantic/pydantic-ai "Go to repository")
[ ![logo](https://ai.pydantic.dev/img/logo-white.svg) ](https://ai.pydantic.dev/ "Pydantic AI") Pydantic AI
[ pydantic/pydantic-ai
  * v1.63.0
  * 15.1k
  * 1.7k

](https://github.com/pydantic/pydantic-ai "Go to repository")
  * [ Pydantic AI  ](https://ai.pydantic.dev/)
  * [ Installation  ](https://ai.pydantic.dev/install/)
  * [ Getting Help  ](https://ai.pydantic.dev/help/)
  * [ Troubleshooting  ](https://ai.pydantic.dev/troubleshooting/)
  * [ Pydantic AI Gateway  ](https://ai.pydantic.dev/gateway/)
  * Documentation
    * Core Concepts
      * [ Agents  ](https://ai.pydantic.dev/agent/)
      * [ Dependencies  ](https://ai.pydantic.dev/dependencies/)
      * [ Function Tools  ](https://ai.pydantic.dev/tools/)
      * [ Output  ](https://ai.pydantic.dev/output/)
      * [ Messages and chat history  ](https://ai.pydantic.dev/message-history/)
      * [ Direct Model Requests  ](https://ai.pydantic.dev/direct/)
    * Models & Providers
      * [ Overview  ](https://ai.pydantic.dev/models/overview/)
      * [ OpenAI  ](https://ai.pydantic.dev/models/openai/)
      * [ Anthropic  ](https://ai.pydantic.dev/models/anthropic/)
      * [ Google  ](https://ai.pydantic.dev/models/google/)
      * [ xAI  ](https://ai.pydantic.dev/models/xai/)
      * [ Bedrock  ](https://ai.pydantic.dev/models/bedrock/)
      * [ Cerebras  ](https://ai.pydantic.dev/models/cerebras/)
      * [ Cohere  ](https://ai.pydantic.dev/models/cohere/)
      * [ Groq  ](https://ai.pydantic.dev/models/groq/)
      * [ Hugging Face  ](https://ai.pydantic.dev/models/huggingface/)
      * [ Mistral  ](https://ai.pydantic.dev/models/mistral/)
      * [ OpenRouter  ](https://ai.pydantic.dev/models/openrouter/)
      * [ Outlines  ](https://ai.pydantic.dev/models/outlines/)
    * Tools & Toolsets
      * [ Function Tools  ](https://ai.pydantic.dev/tools/)
      * [ Advanced Tool Features  ](https://ai.pydantic.dev/tools-advanced/)
      * [ Toolsets  ](https://ai.pydantic.dev/toolsets/)
      * [ Deferred Tools  ](https://ai.pydantic.dev/deferred-tools/)
      * [ Built-in Tools  ](https://ai.pydantic.dev/builtin-tools/)
      * [ Common Tools  ](https://ai.pydantic.dev/common-tools/)
      * [ Third-Party Tools  ](https://ai.pydantic.dev/third-party-tools/)
    * Advanced Features
      * [ Image, Audio, Video & Document Input  ](https://ai.pydantic.dev/input/)
      * [ Thinking  ](https://ai.pydantic.dev/thinking/)
      * [ HTTP Request Retries  ](https://ai.pydantic.dev/retries/)
    * MCP
      * [ Overview  ](https://ai.pydantic.dev/mcp/overview/)
      * [ Client  ](https://ai.pydantic.dev/mcp/client/)
      * [ FastMCP Client  ](https://ai.pydantic.dev/mcp/fastmcp-client/)
      * [ Server  ](https://ai.pydantic.dev/mcp/server/)
    * [ Multi-Agent Patterns  ](https://ai.pydantic.dev/multi-agent-applications/)
    * [ Web Chat UI  ](https://ai.pydantic.dev/web/)
    * [ Embeddings  ](https://ai.pydantic.dev/embeddings/)
    * [ Testing  ](https://ai.pydantic.dev/testing/)
  * Pydantic Evals
    * [ Overview  ](https://ai.pydantic.dev/evals/)
    * Getting Started
      * [ Quick Start  ](https://ai.pydantic.dev/evals/quick-start/)
      * Core Concepts  [ Core Concepts  ](https://ai.pydantic.dev/evals/core-concepts/)
        * [ Overview  ](https://ai.pydantic.dev/evals/core-concepts/#overview)
        * [ Unit Testing Analogy  ](https://ai.pydantic.dev/evals/core-concepts/#unit-testing-analogy)
        * [ Dataset  ](https://ai.pydantic.dev/evals/core-concepts/#dataset)
          * [ Key Features  ](https://ai.pydantic.dev/evals/core-concepts/#key-features)
          * [ Dataset-Level vs Case-Level Evaluators  ](https://ai.pydantic.dev/evals/core-concepts/#dataset-level-vs-case-level-evaluators)
        * [ Experiments  ](https://ai.pydantic.dev/evals/core-concepts/#experiments)
          * [ Running an Experiment  ](https://ai.pydantic.dev/evals/core-concepts/#running-an-experiment)
          * [ What Happens During an Experiment  ](https://ai.pydantic.dev/evals/core-concepts/#what-happens-during-an-experiment)
          * [ Multiple Experiments from One Dataset  ](https://ai.pydantic.dev/evals/core-concepts/#multiple-experiments-from-one-dataset)
        * [ Case  ](https://ai.pydantic.dev/evals/core-concepts/#case)
          * [ Case Components  ](https://ai.pydantic.dev/evals/core-concepts/#case-components)
            * [ Inputs  ](https://ai.pydantic.dev/evals/core-concepts/#inputs)
            * [ Expected Output  ](https://ai.pydantic.dev/evals/core-concepts/#expected-output)
            * [ Metadata  ](https://ai.pydantic.dev/evals/core-concepts/#metadata)
            * [ Evaluators  ](https://ai.pydantic.dev/evals/core-concepts/#evaluators)
        * [ Evaluator  ](https://ai.pydantic.dev/evals/core-concepts/#evaluator)
          * [ Evaluator Types  ](https://ai.pydantic.dev/evals/core-concepts/#evaluator-types)
          * [ EvaluatorContext  ](https://ai.pydantic.dev/evals/core-concepts/#evaluatorcontext)
          * [ Multiple Evaluations  ](https://ai.pydantic.dev/evals/core-concepts/#multiple-evaluations)
          * [ Evaluation Reasons  ](https://ai.pydantic.dev/evals/core-concepts/#evaluation-reasons)
        * [ Evaluation Report  ](https://ai.pydantic.dev/evals/core-concepts/#evaluation-report)
          * [ Report Structure  ](https://ai.pydantic.dev/evals/core-concepts/#report-structure)
          * [ ReportCase  ](https://ai.pydantic.dev/evals/core-concepts/#reportcase)
        * [ Data Model Relationships  ](https://ai.pydantic.dev/evals/core-concepts/#data-model-relationships)
          * [ Static Definition  ](https://ai.pydantic.dev/evals/core-concepts/#static-definition)
          * [ Execution (Experiment)  ](https://ai.pydantic.dev/evals/core-concepts/#execution-experiment)
          * [ Results  ](https://ai.pydantic.dev/evals/core-concepts/#results)
          * [ Key Relationships  ](https://ai.pydantic.dev/evals/core-concepts/#key-relationships)
        * [ Next Steps  ](https://ai.pydantic.dev/evals/core-concepts/#next-steps)
    * Evaluators
      * [ Overview  ](https://ai.pydantic.dev/evals/evaluators/overview/)
      * [ Built-in Evaluators  ](https://ai.pydantic.dev/evals/evaluators/built-in/)
      * [ LLM Judge  ](https://ai.pydantic.dev/evals/evaluators/llm-judge/)
      * [ Custom Evaluators  ](https://ai.pydantic.dev/evals/evaluators/custom/)
      * [ Report Evaluators  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/)
      * [ Span-Based  ](https://ai.pydantic.dev/evals/evaluators/span-based/)
    * How-To Guides
      * [ Logfire Integration  ](https://ai.pydantic.dev/evals/how-to/logfire-integration/)
      * [ Dataset Management  ](https://ai.pydantic.dev/evals/how-to/dataset-management/)
      * [ Dataset Serialization  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/)
      * [ Concurrency & Performance  ](https://ai.pydantic.dev/evals/how-to/concurrency/)
      * [ Multi-Run Evaluation  ](https://ai.pydantic.dev/evals/how-to/multi-run/)
      * [ Retry Strategies  ](https://ai.pydantic.dev/evals/how-to/retry-strategies/)
      * [ Metrics & Attributes  ](https://ai.pydantic.dev/evals/how-to/metrics-attributes/)
    * Examples
      * [ Simple Validation  ](https://ai.pydantic.dev/evals/examples/simple-validation/)
  * Pydantic Graph
    * [ Overview  ](https://ai.pydantic.dev/graph/)
    * [ Beta API  ](https://ai.pydantic.dev/graph/beta/)
      * [ Steps  ](https://ai.pydantic.dev/graph/beta/steps/)
      * [ Joins & Reducers  ](https://ai.pydantic.dev/graph/beta/joins/)
      * [ Decisions  ](https://ai.pydantic.dev/graph/beta/decisions/)
      * [ Parallel Execution  ](https://ai.pydantic.dev/graph/beta/parallel/)
  * Integrations
    * [ Debugging & Monitoring with Pydantic Logfire  ](https://ai.pydantic.dev/logfire/)
    * Durable Execution
      * [ Overview  ](https://ai.pydantic.dev/durable_execution/overview/)
      * [ Temporal  ](https://ai.pydantic.dev/durable_execution/temporal/)
      * [ DBOS  ](https://ai.pydantic.dev/durable_execution/dbos/)
      * [ Prefect  ](https://ai.pydantic.dev/durable_execution/prefect/)
    * UI Event Streams
      * [ Overview  ](https://ai.pydantic.dev/ui/overview/)
      * [ AG-UI  ](https://ai.pydantic.dev/ui/ag-ui/)
      * [ Vercel AI  ](https://ai.pydantic.dev/ui/vercel-ai/)
    * [ Agent2Agent (A2A)  ](https://ai.pydantic.dev/a2a/)
  * Related Packages
    * [ clai  ](https://ai.pydantic.dev/cli/)
  * Examples
    * [ Setup  ](https://ai.pydantic.dev/examples/setup/)
    * Getting Started
      * [ Pydantic Model  ](https://ai.pydantic.dev/examples/pydantic-model/)
      * [ Weather agent  ](https://ai.pydantic.dev/examples/weather-agent/)
    * Conversational Agents
      * [ Chat App with FastAPI  ](https://ai.pydantic.dev/examples/chat-app/)
      * [ Bank support  ](https://ai.pydantic.dev/examples/bank-support/)
    * Data & Analytics
      * [ SQL Generation  ](https://ai.pydantic.dev/examples/sql-gen/)
      * [ Data Analyst  ](https://ai.pydantic.dev/examples/data-analyst/)
      * [ RAG  ](https://ai.pydantic.dev/examples/rag/)
    * Streaming
      * [ Stream markdown  ](https://ai.pydantic.dev/examples/stream-markdown/)
      * [ Stream whales  ](https://ai.pydantic.dev/examples/stream-whales/)
    * Complex Workflows
      * [ Flight booking  ](https://ai.pydantic.dev/examples/flight-booking/)
      * [ Question Graph  ](https://ai.pydantic.dev/examples/question-graph/)
    * Business Applications
      * [ Slack Lead Qualifier with Modal  ](https://ai.pydantic.dev/examples/slack-lead-qualifier/)
    * UI Examples
      * [ Agent User Interaction (AG-UI)  ](https://ai.pydantic.dev/examples/ag-ui/)
  * API Reference
    * pydantic_ai
      * [ pydantic_ai.ag_ui  ](https://ai.pydantic.dev/api/ag_ui/)
      * [ pydantic_ai.agent  ](https://ai.pydantic.dev/api/agent/)
      * [ pydantic_ai.builtin_tools  ](https://ai.pydantic.dev/api/builtin_tools/)
      * [ pydantic_ai.common_tools  ](https://ai.pydantic.dev/api/common_tools/)
      * [ pydantic_ai — Concurrency  ](https://ai.pydantic.dev/api/concurrency/)
      * [ pydantic_ai.direct  ](https://ai.pydantic.dev/api/direct/)
      * [ pydantic_ai.durable_exec  ](https://ai.pydantic.dev/api/durable_exec/)
      * [ pydantic_ai.embeddings  ](https://ai.pydantic.dev/api/embeddings/)
      * [ pydantic_ai.exceptions  ](https://ai.pydantic.dev/api/exceptions/)
      * [ pydantic_ai.ext  ](https://ai.pydantic.dev/api/ext/)
      * [ pydantic_ai.format_prompt  ](https://ai.pydantic.dev/api/format_prompt/)
      * [ pydantic_ai.mcp  ](https://ai.pydantic.dev/api/mcp/)
      * [ pydantic_ai.messages  ](https://ai.pydantic.dev/api/messages/)
      * [ pydantic_ai.models.anthropic  ](https://ai.pydantic.dev/api/models/anthropic/)
      * [ pydantic_ai.models  ](https://ai.pydantic.dev/api/models/base/)
      * [ pydantic_ai.models.bedrock  ](https://ai.pydantic.dev/api/models/bedrock/)
      * [ pydantic_ai.models.cerebras  ](https://ai.pydantic.dev/api/models/cerebras/)
      * [ pydantic_ai.models.cohere  ](https://ai.pydantic.dev/api/models/cohere/)
      * [ pydantic_ai.models.fallback  ](https://ai.pydantic.dev/api/models/fallback/)
      * [ pydantic_ai.models.function  ](https://ai.pydantic.dev/api/models/function/)
      * [ pydantic_ai.models.google  ](https://ai.pydantic.dev/api/models/google/)
      * [ pydantic_ai.models.xai  ](https://ai.pydantic.dev/api/models/xai/)
      * [ pydantic_ai.models.groq  ](https://ai.pydantic.dev/api/models/groq/)
      * [ pydantic_ai.models.huggingface  ](https://ai.pydantic.dev/api/models/huggingface/)
      * [ pydantic_ai.models.instrumented  ](https://ai.pydantic.dev/api/models/instrumented/)
      * [ pydantic_ai.models.mcp_sampling  ](https://ai.pydantic.dev/api/models/mcp-sampling/)
      * [ pydantic_ai.models.mistral  ](https://ai.pydantic.dev/api/models/mistral/)
      * [ pydantic_ai.models.openai  ](https://ai.pydantic.dev/api/models/openai/)
      * [ pydantic_ai.models.openrouter  ](https://ai.pydantic.dev/api/models/openrouter/)
      * [ pydantic_ai.models.outlines  ](https://ai.pydantic.dev/api/models/outlines/)
      * [ pydantic_ai.models.test  ](https://ai.pydantic.dev/api/models/test/)
      * [ pydantic_ai.models.wrapper  ](https://ai.pydantic.dev/api/models/wrapper/)
      * [ pydantic_ai.output  ](https://ai.pydantic.dev/api/output/)
      * [ pydantic_ai.profiles  ](https://ai.pydantic.dev/api/profiles/)
      * [ pydantic_ai.providers  ](https://ai.pydantic.dev/api/providers/)
      * [ pydantic_ai.result  ](https://ai.pydantic.dev/api/result/)
      * [ pydantic_ai.retries  ](https://ai.pydantic.dev/api/retries/)
      * [ pydantic_ai.run  ](https://ai.pydantic.dev/api/run/)
      * [ pydantic_ai.settings  ](https://ai.pydantic.dev/api/settings/)
      * [ pydantic_ai.tools  ](https://ai.pydantic.dev/api/tools/)
      * [ pydantic_ai.toolsets  ](https://ai.pydantic.dev/api/toolsets/)
      * [ pydantic_ai.ui.ag_ui  ](https://ai.pydantic.dev/api/ui/ag_ui/)
      * [ pydantic_ai.ui  ](https://ai.pydantic.dev/api/ui/base/)
      * [ pydantic_ai.ui.vercel_ai  ](https://ai.pydantic.dev/api/ui/vercel_ai/)
      * [ pydantic_ai.usage  ](https://ai.pydantic.dev/api/usage/)
    * pydantic_evals
      * [ pydantic_evals.dataset  ](https://ai.pydantic.dev/api/pydantic_evals/dataset/)
      * [ pydantic_evals.evaluators  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/)
      * [ pydantic_evals.reporting  ](https://ai.pydantic.dev/api/pydantic_evals/reporting/)
      * [ pydantic_evals.otel  ](https://ai.pydantic.dev/api/pydantic_evals/otel/)
      * [ pydantic_evals.generation  ](https://ai.pydantic.dev/api/pydantic_evals/generation/)
    * pydantic_graph
      * [ pydantic_graph  ](https://ai.pydantic.dev/api/pydantic_graph/graph/)
      * [ pydantic_graph.nodes  ](https://ai.pydantic.dev/api/pydantic_graph/nodes/)
      * [ pydantic_graph.persistence  ](https://ai.pydantic.dev/api/pydantic_graph/persistence/)
      * [ pydantic_graph.mermaid  ](https://ai.pydantic.dev/api/pydantic_graph/mermaid/)
      * [ pydantic_graph.exceptions  ](https://ai.pydantic.dev/api/pydantic_graph/exceptions/)
      * Beta API
        * [ pydantic_graph.beta  ](https://ai.pydantic.dev/api/pydantic_graph/beta/)
        * [ pydantic_graph.beta.graph  ](https://ai.pydantic.dev/api/pydantic_graph/beta_graph/)
        * [ pydantic_graph.beta.graph_builder  ](https://ai.pydantic.dev/api/pydantic_graph/beta_graph_builder/)
        * [ pydantic_graph.beta.step  ](https://ai.pydantic.dev/api/pydantic_graph/beta_step/)
        * [ pydantic_graph.beta.join  ](https://ai.pydantic.dev/api/pydantic_graph/beta_join/)
        * [ pydantic_graph.beta.decision  ](https://ai.pydantic.dev/api/pydantic_graph/beta_decision/)
        * [ pydantic_graph.beta.node  ](https://ai.pydantic.dev/api/pydantic_graph/beta_node/)
    * fasta2a
      * [ fasta2a  ](https://ai.pydantic.dev/api/fasta2a/)
  * Project
    * [ Contributing  ](https://ai.pydantic.dev/contributing/)
    * [ Upgrade Guide  ](https://ai.pydantic.dev/changelog/)
    * [ Version policy  ](https://ai.pydantic.dev/version-policy/)


  * [ Overview  ](https://ai.pydantic.dev/evals/core-concepts/#overview)
  * [ Unit Testing Analogy  ](https://ai.pydantic.dev/evals/core-concepts/#unit-testing-analogy)
  * [ Dataset  ](https://ai.pydantic.dev/evals/core-concepts/#dataset)
    * [ Key Features  ](https://ai.pydantic.dev/evals/core-concepts/#key-features)
    * [ Dataset-Level vs Case-Level Evaluators  ](https://ai.pydantic.dev/evals/core-concepts/#dataset-level-vs-case-level-evaluators)
  * [ Experiments  ](https://ai.pydantic.dev/evals/core-concepts/#experiments)
    * [ Running an Experiment  ](https://ai.pydantic.dev/evals/core-concepts/#running-an-experiment)
    * [ What Happens During an Experiment  ](https://ai.pydantic.dev/evals/core-concepts/#what-happens-during-an-experiment)
    * [ Multiple Experiments from One Dataset  ](https://ai.pydantic.dev/evals/core-concepts/#multiple-experiments-from-one-dataset)
  * [ Case  ](https://ai.pydantic.dev/evals/core-concepts/#case)
    * [ Case Components  ](https://ai.pydantic.dev/evals/core-concepts/#case-components)
      * [ Inputs  ](https://ai.pydantic.dev/evals/core-concepts/#inputs)
      * [ Expected Output  ](https://ai.pydantic.dev/evals/core-concepts/#expected-output)
      * [ Metadata  ](https://ai.pydantic.dev/evals/core-concepts/#metadata)
      * [ Evaluators  ](https://ai.pydantic.dev/evals/core-concepts/#evaluators)
  * [ Evaluator  ](https://ai.pydantic.dev/evals/core-concepts/#evaluator)
    * [ Evaluator Types  ](https://ai.pydantic.dev/evals/core-concepts/#evaluator-types)
    * [ EvaluatorContext  ](https://ai.pydantic.dev/evals/core-concepts/#evaluatorcontext)
    * [ Multiple Evaluations  ](https://ai.pydantic.dev/evals/core-concepts/#multiple-evaluations)
    * [ Evaluation Reasons  ](https://ai.pydantic.dev/evals/core-concepts/#evaluation-reasons)
  * [ Evaluation Report  ](https://ai.pydantic.dev/evals/core-concepts/#evaluation-report)
    * [ Report Structure  ](https://ai.pydantic.dev/evals/core-concepts/#report-structure)
    * [ ReportCase  ](https://ai.pydantic.dev/evals/core-concepts/#reportcase)
  * [ Data Model Relationships  ](https://ai.pydantic.dev/evals/core-concepts/#data-model-relationships)
    * [ Static Definition  ](https://ai.pydantic.dev/evals/core-concepts/#static-definition)
    * [ Execution (Experiment)  ](https://ai.pydantic.dev/evals/core-concepts/#execution-experiment)
    * [ Results  ](https://ai.pydantic.dev/evals/core-concepts/#results)
    * [ Key Relationships  ](https://ai.pydantic.dev/evals/core-concepts/#key-relationships)
  * [ Next Steps  ](https://ai.pydantic.dev/evals/core-concepts/#next-steps)


  1. [ Pydantic AI  ](https://ai.pydantic.dev/)
  2. [ Pydantic Evals  ](https://ai.pydantic.dev/evals/)
  3. [ Getting Started  ](https://ai.pydantic.dev/evals/quick-start/)


# Core Concepts
This page explains the key concepts in Pydantic Evals and how they work together.
## Overview
Pydantic Evals is built around these core concepts:
  * **[`Dataset`](https://ai.pydantic.dev/api/pydantic_evals/dataset/#pydantic_evals.dataset.Dataset "Dataset")**- A static definition containing test cases and evaluators
  * **[`Case`](https://ai.pydantic.dev/api/pydantic_evals/dataset/#pydantic_evals.dataset.Case "Case



      dataclass
  ")**- A single test scenario with inputs and optional expected outputs
  * **[`Evaluator`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator "Evaluator



      dataclass
  ")**- Logic for scoring or validating individual outputs
  * **[`ReportEvaluator`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ReportEvaluator "ReportEvaluator



      dataclass
  ")**- Logic for analyzing full experiment results (e.g., confusion matrices, accuracy)
  * **Experiment** - The act of running a task function against all cases in a dataset. (This corresponds to a call to `Dataset.evaluate`.)
  * **[`EvaluationReport`](https://ai.pydantic.dev/api/pydantic_evals/reporting/#pydantic_evals.reporting.EvaluationReport "EvaluationReport



      dataclass
  ")**- The results from running an experiment


The key distinction is between:
  * **Definition** (`Dataset` with `Case`s, `Evaluator`s, and `ReportEvaluator`s) - what you want to test
  * **Execution** (Experiment) - running your task against those tests
  * **Results** (`EvaluationReport` with case results and experiment-wide analyses) - what happened during the experiment


## Unit Testing Analogy
A helpful way to think about Pydantic Evals:
Unit Testing | Pydantic Evals
---|---
Test function |  [`Case`](https://ai.pydantic.dev/api/pydantic_evals/dataset/#pydantic_evals.dataset.Case "Case



      dataclass
  ") + [`Evaluator`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator "Evaluator



      dataclass
  ")
Test suite | [`Dataset`](https://ai.pydantic.dev/api/pydantic_evals/dataset/#pydantic_evals.dataset.Dataset "Dataset")
Running tests (`pytest`) |  **Experiment** (`dataset.evaluate(task)`)
Test report | [`EvaluationReport`](https://ai.pydantic.dev/api/pydantic_evals/reporting/#pydantic_evals.reporting.EvaluationReport "EvaluationReport



      dataclass
  ")
`assert` | Evaluator returning `bool`
**Key Difference** : AI systems are probabilistic, so instead of simple pass/fail, evaluations can have:
  * Quantitative scores (0.0 to 1.0)
  * Qualitative labels ("good", "acceptable", "poor")
  * Pass/fail assertions with explanatory reasons


Just like you can run `pytest` multiple times on the same test suite, you can run multiple experiments on the same dataset to compare different implementations or track changes over time.
## Dataset
A [`Dataset`](https://ai.pydantic.dev/api/pydantic_evals/dataset/#pydantic_evals.dataset.Dataset "Dataset") is a collection of test cases and evaluators that define an evaluation suite.
```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import IsInstance

dataset = Dataset(
    name='my_eval_suite',  # Optional name
    cases=[
        Case(inputs='test input', expected_output='test output'),
    ],
    evaluators=[
        IsInstance(type_name='str'),
    ],
)

```

### Key Features
  * **Type-safe** : Generic over `InputsT`, `OutputT`, and `MetadataT` types
  * **Serializable** : Can be saved to/loaded from YAML or JSON files
  * **Evaluable** : Run against any function with matching input/output types


###  `Dataset`-Level vs `Case`-Level Evaluators
Evaluators can be defined at two levels:
  * **`Dataset`-level** : Apply to all cases in the dataset
  * **`Case`-level** : Apply only to specific cases


```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EqualsExpected, IsInstance

dataset = Dataset(
    cases=[
        Case(
            name='special_case',
            inputs='test',
            expected_output='TEST',
            evaluators=[
                # This evaluator only runs for this case
                EqualsExpected(),
            ],
        ),
    ],
    evaluators=[
        # This evaluator runs for ALL cases
        IsInstance(type_name='str'),
    ],
)

```

## Experiments
An **Experiment** is what happens when you execute a task function against all cases in a dataset. This is the bridge between your static test definition (the Dataset) and your results (the EvaluationReport).
### Running an Experiment
You run an experiment by calling [`evaluate()`](https://ai.pydantic.dev/api/pydantic_evals/dataset/#pydantic_evals.dataset.Dataset.evaluate "evaluate



      async
  ") or [`evaluate_sync()`](https://ai.pydantic.dev/api/pydantic_evals/dataset/#pydantic_evals.dataset.Dataset.evaluate_sync "evaluate_sync") on a dataset:
```
from pydantic_evals import Case, Dataset

# Define your dataset (static definition)
dataset = Dataset(
    cases=[
        Case(inputs='hello', expected_output='HELLO'),
        Case(inputs='world', expected_output='WORLD'),
    ],
)

# Define your task
def uppercase_task(text: str) -> str:
    return text.upper()

# Run the experiment (execution)
report = dataset.evaluate_sync(uppercase_task)

```

### What Happens During an Experiment
When you run an experiment:
  1. **Setup** : The dataset loads all cases, evaluators, and report evaluators
  2. **Execution** : For each case:
    1. The task function is called with `case.inputs`
    2. Execution time is measured and OpenTelemetry spans are captured (if `logfire` is configured)
    3. The outputs of the task function for each case are recorded
  3. **Case Evaluation** : For each case output:
    1. All dataset-level evaluators are run
    2. Case-specific evaluators are run (if any)
    3. Results are collected (scores, assertions, labels)
  4. **Report Evaluation** : If report evaluators are configured, they run over the full set of results to produce experiment-wide analyses (confusion matrices, precision-recall curves, scalar metrics, tables, etc.)
  5. **Reporting** : All results are aggregated into an [`EvaluationReport`](https://ai.pydantic.dev/api/pydantic_evals/reporting/#pydantic_evals.reporting.EvaluationReport "EvaluationReport



      dataclass
  "), including both per-case results and experiment-wide analyses


### Multiple Experiments from One Dataset
A key feature of Pydantic Evals is that you can run the same dataset against different task implementations:
```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EqualsExpected

dataset = Dataset(
    cases=[
        Case(inputs='hello', expected_output='HELLO'),
    ],
    evaluators=[EqualsExpected()],
)


# Original implementation
def task_v1(text: str) -> str:
    return text.upper()


# Improved implementation (with exclamation)
def task_v2(text: str) -> str:
    return text.upper() + '!'


# Compare results
report_v1 = dataset.evaluate_sync(task_v1)
report_v2 = dataset.evaluate_sync(task_v2)

avg_v1 = report_v1.averages()
avg_v2 = report_v2.averages()
print(f'V1 pass rate: {avg_v1.assertions if avg_v1 and avg_v1.assertions else 0}')
#> V1 pass rate: 1.0
print(f'V2 pass rate: {avg_v2.assertions if avg_v2 and avg_v2.assertions else 0}')
#> V2 pass rate: 0

```

This allows you to:
  * **Compare implementations** across versions
  * **Track performance** over time
  * **A/B test** different approaches
  * **Validate changes** before deployment


## Case
A [`Case`](https://ai.pydantic.dev/api/pydantic_evals/dataset/#pydantic_evals.dataset.Case "Case



      dataclass
  ") represents a single test scenario with specific inputs and optional expected outputs.
```
from pydantic_evals import Case
from pydantic_evals.evaluators import EqualsExpected

case = Case(
    name='test_uppercase',  # Optional, but recommended for reporting
    inputs='hello world',  # Required: inputs to your task
    expected_output='HELLO WORLD',  # Optional: expected output
    metadata={'category': 'basic'},  # Optional: arbitrary metadata
    evaluators=[EqualsExpected()],  # Optional: case-specific evaluators
)

```

### Case Components
#### Inputs
The inputs to pass to the task being evaluated. Can be any type:
```
from pydantic import BaseModel

from pydantic_evals import Case


class MyInputModel(BaseModel):
    field1: str


# Simple types
Case(inputs='hello')
Case(inputs=42)

# Complex types
Case(inputs={'query': 'What is AI?', 'max_tokens': 100})
Case(inputs=MyInputModel(field1='value'))

```

#### Expected Output
The expected result, used by evaluators like [`EqualsExpected`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EqualsExpected "EqualsExpected



      dataclass
  "):
```
from pydantic_evals import Case

Case(
    inputs='2 + 2',
    expected_output='4',
)

```

If no `expected_output` is provided, evaluators that require it (like `EqualsExpected`) will skip that case.
#### Metadata
Arbitrary data that evaluators can access via [`EvaluatorContext`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext "EvaluatorContext



      dataclass
  "):
```
from pydantic_evals import Case

Case(
    inputs='question',
    metadata={
        'difficulty': 'hard',
        'category': 'math',
        'source': 'exam_2024',
    },
)

```

Metadata is useful for:
  * Filtering cases during analysis
  * Providing context to evaluators
  * Organizing test suites


#### Evaluators
Cases can have their own evaluators that only run for that specific case. This is particularly powerful for building comprehensive evaluation suites where different cases have different requirements - if you could write one evaluator rubric that worked perfectly for all cases, you'd just incorporate it into your agent instructions. Case-specific [`LLMJudge`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.LLMJudge "LLMJudge



      dataclass
  ") evaluators are especially useful for quickly building maintainable golden datasets by describing what "good" looks like for each scenario. See [Case-specific evaluators](https://ai.pydantic.dev/evals/evaluators/overview/#case-specific-evaluators) for a more detailed explanation and examples.
## Evaluator
An [`Evaluator`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator "Evaluator



      dataclass
  ") assesses the output of your task and returns one or more scores, labels, or assertions. Each score, label or assertion can also have an optional string-value reason associated.
### Evaluator Types
Evaluators return different types of results:
Return Type | Purpose | Example
---|---|---
`bool` |  **Assertion** - Pass/fail check |  `True` → ✔, `False` → ✗
`int` or `float` |  **Score** - Numeric quality metric |  `0.95`, `87`
`str` |  **Label** - Categorical result |  `"correct"`, `"hallucination"`
```
from dataclasses import dataclass

from pydantic_evals.evaluators import Evaluator, EvaluatorContext


@dataclass
class ExactMatch(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> bool:
        return ctx.output == ctx.expected_output  # Assertion


@dataclass
class Confidence(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> float:
        # Analyze output and return confidence score
        return 0.95  # Score


@dataclass
class Classifier(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> str:
        if 'error' in ctx.output.lower():
            return 'error'  # Label
        return 'success'

```

Evaluators can also return instances of [`EvaluationReason`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluationReason "EvaluationReason



      dataclass
  "), and dictionaries mapping labels to output values. See the [custom evaluator return types](https://ai.pydantic.dev/evals/evaluators/custom/#return-types) docs for more detail.
### EvaluatorContext
All evaluators receive an [`EvaluatorContext`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext "EvaluatorContext



      dataclass
  ") containing:
  * `name`: Case name (optional)
  * `inputs`: Task inputs
  * `metadata`: Case metadata (optional)
  * `expected_output`: Expected output (optional)
  * `output`: Actual output from task
  * `duration`: Task execution time in seconds
  * `span_tree`: OpenTelemetry spans (if `logfire` is configured)
  * `attributes`: Custom attributes dict
  * `metrics`: Custom metrics dict


### Multiple Evaluations
Evaluators can return multiple results by returning a dictionary:
```
from dataclasses import dataclass

from pydantic_evals.evaluators import Evaluator, EvaluatorContext


@dataclass
class MultiCheck(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> dict[str, bool | float | str]:
        return {
            'is_valid': isinstance(ctx.output, str),  # Assertion
            'length': len(ctx.output),  # Metric
            'category': 'long' if len(ctx.output) > 100 else 'short',  # Label
        }

```

### Evaluation Reasons
Add explanations to your evaluations using [`EvaluationReason`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluationReason "EvaluationReason



      dataclass
  "):
```
from dataclasses import dataclass

from pydantic_evals.evaluators import EvaluationReason, Evaluator, EvaluatorContext


@dataclass
class SmartCheck(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> EvaluationReason:
        if ctx.output == ctx.expected_output:
            return EvaluationReason(
                value=True,
                reason='Exact match with expected output',
            )
        return EvaluationReason(
            value=False,
            reason=f'Expected {ctx.expected_output!r}, got {ctx.output!r}',
        )

```

Reasons appear in reports when using `include_reasons=True`.
## Evaluation Report
An [`EvaluationReport`](https://ai.pydantic.dev/api/pydantic_evals/reporting/#pydantic_evals.reporting.EvaluationReport "EvaluationReport



      dataclass
  ") is the result of running an experiment. It contains all the data from executing your task against the dataset's cases and running all evaluators.
```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EqualsExpected

dataset = Dataset(
    cases=[Case(inputs='hello', expected_output='HELLO')],
    evaluators=[EqualsExpected()],
)


def my_task(text: str) -> str:
    return text.upper()


# Run an experiment
report = dataset.evaluate_sync(my_task)

# Print to console
report.print()
"""
    Evaluation Summary: my_task
┏━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Case ID  ┃ Assertions ┃ Duration ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Case 1   │ ✔          │     10ms │
├──────────┼────────────┼──────────┤
│ Averages │ 100.0% ✔   │     10ms │
└──────────┴────────────┴──────────┘
"""

# Access data programmatically
for case in report.cases:
    print(f'{case.name}: {case.scores}')
    #> Case 1: {}

```

### Report Structure
The [`EvaluationReport`](https://ai.pydantic.dev/api/pydantic_evals/reporting/#pydantic_evals.reporting.EvaluationReport "EvaluationReport



      dataclass
  ") contains:
  * `name`: Experiment name
  * `cases`: List of successful case evaluations
  * `failures`: List of failed executions
  * `analyses`: List of experiment-wide analyses from [report evaluators](https://ai.pydantic.dev/evals/evaluators/report-evaluators/) (confusion matrices, PR curves, scalars, tables)
  * `trace_id`: OpenTelemetry trace ID (optional)
  * `span_id`: OpenTelemetry span ID (optional)


### ReportCase
Each successfulcase result contains:
**Case data:**
  * `name`: Case name
  * `inputs`: Task inputs
  * `metadata`: Case metadata (optional)
  * `expected_output`: Expected output (optional)
  * `output`: Actual output from task


**Evaluation results:**
  * `scores`: Dictionary of numeric scores from evaluators
  * `labels`: Dictionary of categorical labels from evaluators
  * `assertions`: Dictionary of pass/fail assertions from evaluators


**Performance data:**
  * `task_duration`: Task execution time
  * `total_duration`: Total time including evaluators


**Additional data:**
  * `metrics`: Custom metrics dict
  * `attributes`: Custom attributes dict


**Tracing:**
  * `trace_id`: OpenTelemetry trace ID (optional)
  * `span_id`: OpenTelemetry span ID (optional)


**Errors:**
  * `evaluator_failures`: List of evaluator errors


## Data Model Relationships
Here's how the core concepts relate to each other:
### Static Definition
  * A **Dataset** contains:
  * Many **Cases** (test scenarios with inputs and expected outputs)
  * Many **Evaluators** (logic for scoring individual outputs)
  * Many **Report Evaluators** (logic for analyzing full experiment results)


### Execution (Experiment)
When you call `dataset.evaluate(task)`, an **Experiment** runs:
  * The **Task** function is executed against all **Cases** in the **Dataset**
  * All **Evaluators** are run (both dataset-level and case-specific) against each output as appropriate
  * One **EvaluationReport** is produced as the final output


### Results
  * An **EvaluationReport** contains:
  * Results for each **Case** (inputs, outputs, scores, assertions, labels)
  * Experiment-wide **Analyses** from report evaluators (confusion matrices, PR curves, scalars, tables)
  * Summary statistics (averages, pass rates)
  * Performance data (durations)
  * Tracing information (OpenTelemetry spans)


### Key Relationships
  * **One Dataset → Many Experiments** : You can run the same dataset against different task implementations or multiple times to track changes
  * **One Experiment → One Report** : Each time you call `dataset.evaluate(...)`, you get one report
  * **One Experiment → Many Case Results** : The report contains results for every case in the dataset


## Next Steps
  * **[Evaluators Overview](https://ai.pydantic.dev/evals/evaluators/overview/)** - When to use different evaluator types
  * **[Built-in Evaluators](https://ai.pydantic.dev/evals/evaluators/built-in/)** - Complete reference of provided evaluators
  * **[Custom Evaluators](https://ai.pydantic.dev/evals/evaluators/custom/)** - Write your own evaluation logic
  * **[Report Evaluators](https://ai.pydantic.dev/evals/evaluators/report-evaluators/)** - Experiment-wide analyses (confusion matrices, PR curves, etc.)
  * **[Dataset Management](https://ai.pydantic.dev/evals/how-to/dataset-management/)** - Save, load, and generate datasets


© Pydantic Services Inc. 2024 to present
