[ Skip to content ](https://ai.pydantic.dev/evals/#pydantic-evals)
**Join us at the inaugural PyAI Conf in San Francisco on March 10th![Learn More](https://pyai.events/?utm_source=pydantic-ai) **
[ ![logo](https://ai.pydantic.dev/img/logo-white.svg) ](https://ai.pydantic.dev/ "Pydantic AI")
Pydantic AI
Pydantic Evals
Type to start searching
[ pydantic/pydantic-ai
  * v1.63.0
  * 15.1k
  * 1.7k

](https://github.com/pydantic/pydantic-ai "Go to repository")
[ ![logo](https://ai.pydantic.dev/img/logo-white.svg) ](https://ai.pydantic.dev/ "Pydantic AI") Pydantic AI
[ pydantic/pydantic-ai
  * v1.63.0
  * 15.1k
  * 1.7k

](https://github.com/pydantic/pydantic-ai "Go to repository")
  * [ Pydantic AI  ](https://ai.pydantic.dev/)
  * [ Installation  ](https://ai.pydantic.dev/install/)
  * [ Getting Help  ](https://ai.pydantic.dev/help/)
  * [ Troubleshooting  ](https://ai.pydantic.dev/troubleshooting/)
  * [ Pydantic AI Gateway  ](https://ai.pydantic.dev/gateway/)
  * Documentation
    * Core Concepts
      * [ Agents  ](https://ai.pydantic.dev/agent/)
      * [ Dependencies  ](https://ai.pydantic.dev/dependencies/)
      * [ Function Tools  ](https://ai.pydantic.dev/tools/)
      * [ Output  ](https://ai.pydantic.dev/output/)
      * [ Messages and chat history  ](https://ai.pydantic.dev/message-history/)
      * [ Direct Model Requests  ](https://ai.pydantic.dev/direct/)
    * Models & Providers
      * [ Overview  ](https://ai.pydantic.dev/models/overview/)
      * [ OpenAI  ](https://ai.pydantic.dev/models/openai/)
      * [ Anthropic  ](https://ai.pydantic.dev/models/anthropic/)
      * [ Google  ](https://ai.pydantic.dev/models/google/)
      * [ xAI  ](https://ai.pydantic.dev/models/xai/)
      * [ Bedrock  ](https://ai.pydantic.dev/models/bedrock/)
      * [ Cerebras  ](https://ai.pydantic.dev/models/cerebras/)
      * [ Cohere  ](https://ai.pydantic.dev/models/cohere/)
      * [ Groq  ](https://ai.pydantic.dev/models/groq/)
      * [ Hugging Face  ](https://ai.pydantic.dev/models/huggingface/)
      * [ Mistral  ](https://ai.pydantic.dev/models/mistral/)
      * [ OpenRouter  ](https://ai.pydantic.dev/models/openrouter/)
      * [ Outlines  ](https://ai.pydantic.dev/models/outlines/)
    * Tools & Toolsets
      * [ Function Tools  ](https://ai.pydantic.dev/tools/)
      * [ Advanced Tool Features  ](https://ai.pydantic.dev/tools-advanced/)
      * [ Toolsets  ](https://ai.pydantic.dev/toolsets/)
      * [ Deferred Tools  ](https://ai.pydantic.dev/deferred-tools/)
      * [ Built-in Tools  ](https://ai.pydantic.dev/builtin-tools/)
      * [ Common Tools  ](https://ai.pydantic.dev/common-tools/)
      * [ Third-Party Tools  ](https://ai.pydantic.dev/third-party-tools/)
    * Advanced Features
      * [ Image, Audio, Video & Document Input  ](https://ai.pydantic.dev/input/)
      * [ Thinking  ](https://ai.pydantic.dev/thinking/)
      * [ HTTP Request Retries  ](https://ai.pydantic.dev/retries/)
    * MCP
      * [ Overview  ](https://ai.pydantic.dev/mcp/overview/)
      * [ Client  ](https://ai.pydantic.dev/mcp/client/)
      * [ FastMCP Client  ](https://ai.pydantic.dev/mcp/fastmcp-client/)
      * [ Server  ](https://ai.pydantic.dev/mcp/server/)
    * [ Multi-Agent Patterns  ](https://ai.pydantic.dev/multi-agent-applications/)
    * [ Web Chat UI  ](https://ai.pydantic.dev/web/)
    * [ Embeddings  ](https://ai.pydantic.dev/embeddings/)
    * [ Testing  ](https://ai.pydantic.dev/testing/)
  * Pydantic Evals
    * Overview  [ Overview  ](https://ai.pydantic.dev/evals/)
      * [ Design Philosophy  ](https://ai.pydantic.dev/evals/#design-philosophy)
      * [ Quick Navigation  ](https://ai.pydantic.dev/evals/#quick-navigation)
      * [ Code-First Evaluation  ](https://ai.pydantic.dev/evals/#code-first-evaluation)
      * [ Installation  ](https://ai.pydantic.dev/evals/#installation)
      * [ Pydantic Evals Data Model  ](https://ai.pydantic.dev/evals/#pydantic-evals-data-model)
        * [ Data Model Diagram  ](https://ai.pydantic.dev/evals/#data-model-diagram)
        * [ Key Relationships  ](https://ai.pydantic.dev/evals/#key-relationships)
        * [ Data Flow  ](https://ai.pydantic.dev/evals/#data-flow)
      * [ Datasets and Cases  ](https://ai.pydantic.dev/evals/#datasets-and-cases)
      * [ Evaluators  ](https://ai.pydantic.dev/evals/#evaluators)
      * [ Running Experiments  ](https://ai.pydantic.dev/evals/#running-experiments)
      * [ API Reference  ](https://ai.pydantic.dev/evals/#api-reference)
      * [ Next Steps  ](https://ai.pydantic.dev/evals/#next-steps)
    * Getting Started
      * [ Quick Start  ](https://ai.pydantic.dev/evals/quick-start/)
      * [ Core Concepts  ](https://ai.pydantic.dev/evals/core-concepts/)
    * Evaluators
      * [ Overview  ](https://ai.pydantic.dev/evals/evaluators/overview/)
      * [ Built-in Evaluators  ](https://ai.pydantic.dev/evals/evaluators/built-in/)
      * [ LLM Judge  ](https://ai.pydantic.dev/evals/evaluators/llm-judge/)
      * [ Custom Evaluators  ](https://ai.pydantic.dev/evals/evaluators/custom/)
      * [ Report Evaluators  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/)
      * [ Span-Based  ](https://ai.pydantic.dev/evals/evaluators/span-based/)
    * How-To Guides
      * [ Logfire Integration  ](https://ai.pydantic.dev/evals/how-to/logfire-integration/)
      * [ Dataset Management  ](https://ai.pydantic.dev/evals/how-to/dataset-management/)
      * [ Dataset Serialization  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/)
      * [ Concurrency & Performance  ](https://ai.pydantic.dev/evals/how-to/concurrency/)
      * [ Multi-Run Evaluation  ](https://ai.pydantic.dev/evals/how-to/multi-run/)
      * [ Retry Strategies  ](https://ai.pydantic.dev/evals/how-to/retry-strategies/)
      * [ Metrics & Attributes  ](https://ai.pydantic.dev/evals/how-to/metrics-attributes/)
    * Examples
      * [ Simple Validation  ](https://ai.pydantic.dev/evals/examples/simple-validation/)
  * Pydantic Graph
    * [ Overview  ](https://ai.pydantic.dev/graph/)
    * [ Beta API  ](https://ai.pydantic.dev/graph/beta/)
      * [ Steps  ](https://ai.pydantic.dev/graph/beta/steps/)
      * [ Joins & Reducers  ](https://ai.pydantic.dev/graph/beta/joins/)
      * [ Decisions  ](https://ai.pydantic.dev/graph/beta/decisions/)
      * [ Parallel Execution  ](https://ai.pydantic.dev/graph/beta/parallel/)
  * Integrations
    * [ Debugging & Monitoring with Pydantic Logfire  ](https://ai.pydantic.dev/logfire/)
    * Durable Execution
      * [ Overview  ](https://ai.pydantic.dev/durable_execution/overview/)
      * [ Temporal  ](https://ai.pydantic.dev/durable_execution/temporal/)
      * [ DBOS  ](https://ai.pydantic.dev/durable_execution/dbos/)
      * [ Prefect  ](https://ai.pydantic.dev/durable_execution/prefect/)
    * UI Event Streams
      * [ Overview  ](https://ai.pydantic.dev/ui/overview/)
      * [ AG-UI  ](https://ai.pydantic.dev/ui/ag-ui/)
      * [ Vercel AI  ](https://ai.pydantic.dev/ui/vercel-ai/)
    * [ Agent2Agent (A2A)  ](https://ai.pydantic.dev/a2a/)
  * Related Packages
    * [ clai  ](https://ai.pydantic.dev/cli/)
  * Examples
    * [ Setup  ](https://ai.pydantic.dev/examples/setup/)
    * Getting Started
      * [ Pydantic Model  ](https://ai.pydantic.dev/examples/pydantic-model/)
      * [ Weather agent  ](https://ai.pydantic.dev/examples/weather-agent/)
    * Conversational Agents
      * [ Chat App with FastAPI  ](https://ai.pydantic.dev/examples/chat-app/)
      * [ Bank support  ](https://ai.pydantic.dev/examples/bank-support/)
    * Data & Analytics
      * [ SQL Generation  ](https://ai.pydantic.dev/examples/sql-gen/)
      * [ Data Analyst  ](https://ai.pydantic.dev/examples/data-analyst/)
      * [ RAG  ](https://ai.pydantic.dev/examples/rag/)
    * Streaming
      * [ Stream markdown  ](https://ai.pydantic.dev/examples/stream-markdown/)
      * [ Stream whales  ](https://ai.pydantic.dev/examples/stream-whales/)
    * Complex Workflows
      * [ Flight booking  ](https://ai.pydantic.dev/examples/flight-booking/)
      * [ Question Graph  ](https://ai.pydantic.dev/examples/question-graph/)
    * Business Applications
      * [ Slack Lead Qualifier with Modal  ](https://ai.pydantic.dev/examples/slack-lead-qualifier/)
    * UI Examples
      * [ Agent User Interaction (AG-UI)  ](https://ai.pydantic.dev/examples/ag-ui/)
  * API Reference
    * pydantic_ai
      * [ pydantic_ai.ag_ui  ](https://ai.pydantic.dev/api/ag_ui/)
      * [ pydantic_ai.agent  ](https://ai.pydantic.dev/api/agent/)
      * [ pydantic_ai.builtin_tools  ](https://ai.pydantic.dev/api/builtin_tools/)
      * [ pydantic_ai.common_tools  ](https://ai.pydantic.dev/api/common_tools/)
      * [ pydantic_ai — Concurrency  ](https://ai.pydantic.dev/api/concurrency/)
      * [ pydantic_ai.direct  ](https://ai.pydantic.dev/api/direct/)
      * [ pydantic_ai.durable_exec  ](https://ai.pydantic.dev/api/durable_exec/)
      * [ pydantic_ai.embeddings  ](https://ai.pydantic.dev/api/embeddings/)
      * [ pydantic_ai.exceptions  ](https://ai.pydantic.dev/api/exceptions/)
      * [ pydantic_ai.ext  ](https://ai.pydantic.dev/api/ext/)
      * [ pydantic_ai.format_prompt  ](https://ai.pydantic.dev/api/format_prompt/)
      * [ pydantic_ai.mcp  ](https://ai.pydantic.dev/api/mcp/)
      * [ pydantic_ai.messages  ](https://ai.pydantic.dev/api/messages/)
      * [ pydantic_ai.models.anthropic  ](https://ai.pydantic.dev/api/models/anthropic/)
      * [ pydantic_ai.models  ](https://ai.pydantic.dev/api/models/base/)
      * [ pydantic_ai.models.bedrock  ](https://ai.pydantic.dev/api/models/bedrock/)
      * [ pydantic_ai.models.cerebras  ](https://ai.pydantic.dev/api/models/cerebras/)
      * [ pydantic_ai.models.cohere  ](https://ai.pydantic.dev/api/models/cohere/)
      * [ pydantic_ai.models.fallback  ](https://ai.pydantic.dev/api/models/fallback/)
      * [ pydantic_ai.models.function  ](https://ai.pydantic.dev/api/models/function/)
      * [ pydantic_ai.models.google  ](https://ai.pydantic.dev/api/models/google/)
      * [ pydantic_ai.models.xai  ](https://ai.pydantic.dev/api/models/xai/)
      * [ pydantic_ai.models.groq  ](https://ai.pydantic.dev/api/models/groq/)
      * [ pydantic_ai.models.huggingface  ](https://ai.pydantic.dev/api/models/huggingface/)
      * [ pydantic_ai.models.instrumented  ](https://ai.pydantic.dev/api/models/instrumented/)
      * [ pydantic_ai.models.mcp_sampling  ](https://ai.pydantic.dev/api/models/mcp-sampling/)
      * [ pydantic_ai.models.mistral  ](https://ai.pydantic.dev/api/models/mistral/)
      * [ pydantic_ai.models.openai  ](https://ai.pydantic.dev/api/models/openai/)
      * [ pydantic_ai.models.openrouter  ](https://ai.pydantic.dev/api/models/openrouter/)
      * [ pydantic_ai.models.outlines  ](https://ai.pydantic.dev/api/models/outlines/)
      * [ pydantic_ai.models.test  ](https://ai.pydantic.dev/api/models/test/)
      * [ pydantic_ai.models.wrapper  ](https://ai.pydantic.dev/api/models/wrapper/)
      * [ pydantic_ai.output  ](https://ai.pydantic.dev/api/output/)
      * [ pydantic_ai.profiles  ](https://ai.pydantic.dev/api/profiles/)
      * [ pydantic_ai.providers  ](https://ai.pydantic.dev/api/providers/)
      * [ pydantic_ai.result  ](https://ai.pydantic.dev/api/result/)
      * [ pydantic_ai.retries  ](https://ai.pydantic.dev/api/retries/)
      * [ pydantic_ai.run  ](https://ai.pydantic.dev/api/run/)
      * [ pydantic_ai.settings  ](https://ai.pydantic.dev/api/settings/)
      * [ pydantic_ai.tools  ](https://ai.pydantic.dev/api/tools/)
      * [ pydantic_ai.toolsets  ](https://ai.pydantic.dev/api/toolsets/)
      * [ pydantic_ai.ui.ag_ui  ](https://ai.pydantic.dev/api/ui/ag_ui/)
      * [ pydantic_ai.ui  ](https://ai.pydantic.dev/api/ui/base/)
      * [ pydantic_ai.ui.vercel_ai  ](https://ai.pydantic.dev/api/ui/vercel_ai/)
      * [ pydantic_ai.usage  ](https://ai.pydantic.dev/api/usage/)
    * pydantic_evals
      * [ pydantic_evals.dataset  ](https://ai.pydantic.dev/api/pydantic_evals/dataset/)
      * [ pydantic_evals.evaluators  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/)
      * [ pydantic_evals.reporting  ](https://ai.pydantic.dev/api/pydantic_evals/reporting/)
      * [ pydantic_evals.otel  ](https://ai.pydantic.dev/api/pydantic_evals/otel/)
      * [ pydantic_evals.generation  ](https://ai.pydantic.dev/api/pydantic_evals/generation/)
    * pydantic_graph
      * [ pydantic_graph  ](https://ai.pydantic.dev/api/pydantic_graph/graph/)
      * [ pydantic_graph.nodes  ](https://ai.pydantic.dev/api/pydantic_graph/nodes/)
      * [ pydantic_graph.persistence  ](https://ai.pydantic.dev/api/pydantic_graph/persistence/)
      * [ pydantic_graph.mermaid  ](https://ai.pydantic.dev/api/pydantic_graph/mermaid/)
      * [ pydantic_graph.exceptions  ](https://ai.pydantic.dev/api/pydantic_graph/exceptions/)
      * Beta API
        * [ pydantic_graph.beta  ](https://ai.pydantic.dev/api/pydantic_graph/beta/)
        * [ pydantic_graph.beta.graph  ](https://ai.pydantic.dev/api/pydantic_graph/beta_graph/)
        * [ pydantic_graph.beta.graph_builder  ](https://ai.pydantic.dev/api/pydantic_graph/beta_graph_builder/)
        * [ pydantic_graph.beta.step  ](https://ai.pydantic.dev/api/pydantic_graph/beta_step/)
        * [ pydantic_graph.beta.join  ](https://ai.pydantic.dev/api/pydantic_graph/beta_join/)
        * [ pydantic_graph.beta.decision  ](https://ai.pydantic.dev/api/pydantic_graph/beta_decision/)
        * [ pydantic_graph.beta.node  ](https://ai.pydantic.dev/api/pydantic_graph/beta_node/)
    * fasta2a
      * [ fasta2a  ](https://ai.pydantic.dev/api/fasta2a/)
  * Project
    * [ Contributing  ](https://ai.pydantic.dev/contributing/)
    * [ Upgrade Guide  ](https://ai.pydantic.dev/changelog/)
    * [ Version policy  ](https://ai.pydantic.dev/version-policy/)


  * [ Design Philosophy  ](https://ai.pydantic.dev/evals/#design-philosophy)
  * [ Quick Navigation  ](https://ai.pydantic.dev/evals/#quick-navigation)
  * [ Code-First Evaluation  ](https://ai.pydantic.dev/evals/#code-first-evaluation)
  * [ Installation  ](https://ai.pydantic.dev/evals/#installation)
  * [ Pydantic Evals Data Model  ](https://ai.pydantic.dev/evals/#pydantic-evals-data-model)
    * [ Data Model Diagram  ](https://ai.pydantic.dev/evals/#data-model-diagram)
    * [ Key Relationships  ](https://ai.pydantic.dev/evals/#key-relationships)
    * [ Data Flow  ](https://ai.pydantic.dev/evals/#data-flow)
  * [ Datasets and Cases  ](https://ai.pydantic.dev/evals/#datasets-and-cases)
  * [ Evaluators  ](https://ai.pydantic.dev/evals/#evaluators)
  * [ Running Experiments  ](https://ai.pydantic.dev/evals/#running-experiments)
  * [ API Reference  ](https://ai.pydantic.dev/evals/#api-reference)
  * [ Next Steps  ](https://ai.pydantic.dev/evals/#next-steps)


  1. [ Pydantic AI  ](https://ai.pydantic.dev/)
  2. [ Pydantic Evals  ](https://ai.pydantic.dev/evals/)


# Pydantic Evals
**Pydantic Evals** is a powerful evaluation framework for systematically testing and evaluating AI systems, from simple LLM calls to complex multi-agent applications.
## Design Philosophy
Code-First Approach
Pydantic Evals follows a code-first philosophy where all evaluation components are defined in Python. This differs from platforms with web-based configuration. You write and run evals in code, and can write the results to disk or view them in your terminal or in [Pydantic Logfire](https://logfire.pydantic.dev/docs/guides/web-ui/evals/).
Evals are an Emerging Practice
Unlike unit tests, evals are an emerging art/science. Anyone who claims to know exactly how your evals should be defined can safely be ignored. We've designed Pydantic Evals to be flexible and useful without being too opinionated.
## Quick Navigation
**Getting Started:**
  * [Installation](https://ai.pydantic.dev/evals/#installation)
  * [Quick Start](https://ai.pydantic.dev/evals/quick-start/)
  * [Core Concepts](https://ai.pydantic.dev/evals/core-concepts/)


**Evaluators:**
  * [Evaluators Overview](https://ai.pydantic.dev/evals/evaluators/overview/) - Compare evaluator types and learn when to use each approach
  * [Built-in Evaluators](https://ai.pydantic.dev/evals/evaluators/built-in/) - Complete reference for exact match, instance checks, and other ready-to-use evaluators
  * [LLM as a Judge](https://ai.pydantic.dev/evals/evaluators/llm-judge/) - Use LLMs to evaluate subjective qualities, complex criteria, and natural language outputs
  * [Custom Evaluators](https://ai.pydantic.dev/evals/evaluators/custom/) - Implement domain-specific scoring logic and custom evaluation metrics
  * [Span-Based Evaluation](https://ai.pydantic.dev/evals/evaluators/span-based/) - Evaluate internal agent behavior (tool calls, execution flow) using OpenTelemetry traces. Essential for complex agents where correctness depends on _how_ the answer was reached, not just the final output. Also ensures eval assertions align with production telemetry.


**How-To Guides:**
  * [Logfire Integration](https://ai.pydantic.dev/evals/how-to/logfire-integration/) - Visualize results
  * [Dataset Management](https://ai.pydantic.dev/evals/how-to/dataset-management/) - Save, load, generate
  * [Concurrency & Performance](https://ai.pydantic.dev/evals/how-to/concurrency/) - Control parallel execution
  * [Retry Strategies](https://ai.pydantic.dev/evals/how-to/retry-strategies/) - Handle transient failures
  * [Metrics & Attributes](https://ai.pydantic.dev/evals/how-to/metrics-attributes/) - Track custom data


**Examples:**
  * [Simple Validation](https://ai.pydantic.dev/evals/examples/simple-validation/) - Basic example


**Reference:**
  * [API Documentation](https://ai.pydantic.dev/api/pydantic_evals/dataset/)


## Code-First Evaluation
Pydantic Evals follows a **code-first approach** where you define all evaluation components (datasets, experiments, tasks, cases and evaluators) in Python code, or as serialized data loaded by Python code. This differs from platforms with fully web-based configuration.
When you run an _Experiment_ you'll see a progress indicator and can print the results wherever you run your python code (IDE, terminal, etc). You also get a report object back that you can serialize and store or send to a notebook or other application for further visualization and analysis.
If you are using [Pydantic Logfire](https://logfire.pydantic.dev/docs/guides/web-ui/evals/), your experiment results automatically appear in the Logfire web interface for visualization, comparison, and collaborative analysis. Logfire serves as a observability layer - you write and run evals in code, then view and analyze results in the web UI.
## Installation
To install the Pydantic Evals package, run:
[pip](https://ai.pydantic.dev/evals/#__tabbed_1_1)[uv](https://ai.pydantic.dev/evals/#__tabbed_1_2)
```
pip install pydantic-evals

```

```
uv add pydantic-evals

```

`pydantic-evals` does not depend on `pydantic-ai`, but has an optional dependency on `logfire` if you'd like to use OpenTelemetry traces in your evals, or send evaluation results to [logfire](https://pydantic.dev/logfire).
[pip](https://ai.pydantic.dev/evals/#__tabbed_2_1)[uv](https://ai.pydantic.dev/evals/#__tabbed_2_2)
```
pip install 'pydantic-evals[logfire]'

```

```
uv add 'pydantic-evals[logfire]'

```

## Pydantic Evals Data Model
Pydantic Evals is built around a simple data model:
### Data Model Diagram
```
Dataset (1) ──────────── (Many) Case
│                        │
│                        │
└─── (Many) Experiment ──┴─── (Many) Case results
     │
     └─── (1) Task
     │
     └─── (Many) Evaluator

```

### Key Relationships
  1. **Dataset → Cases** : One Dataset contains many Cases
  2. **Dataset → Experiments** : One Dataset can be used across many Experiments over time
  3. **Experiment → Case results** : One Experiment generates results by executing each Case
  4. **Experiment → Task** : One Experiment evaluates one defined Task
  5. **Experiment → Evaluators** : One Experiment uses multiple Evaluators. Dataset-wide Evaluators are run against all Cases, and Case-specific Evaluators against their respective Cases


### Data Flow
  1. **Dataset creation** : Define cases and evaluators in YAML/JSON, or directly in Python
  2. **Experiment execution** : Run `dataset.evaluate_sync(task_function)`
  3. **Cases run** : Each Case is executed against the Task
  4. **Evaluation** : Evaluators score the Task outputs for each Case
  5. **Results** : All Case results are collected into a summary report


A metaphor
A useful metaphor (although not perfect) is to think of evals like a **Unit Testing** framework:
  * **Cases + Evaluators** are your individual unit tests - each one defines a specific scenario you want to test, complete with inputs and expected outcomes. Just like a unit test, a case asks: _"Given this input, does my system produce the right output?"_
  * **Datasets** are like test suites - they are the scaffolding that holds your unit tests together. They group related cases and define shared evaluation criteria that should apply across all tests in the suite.
  * **Experiments** are like running your entire test suite and getting a report. When you execute `dataset.evaluate_sync(my_ai_function)`, you're running all your cases against your AI system and collecting the results - just like running `pytest` and getting a summary of passes, failures, and performance metrics.


The key difference from traditional unit testing is that AI systems are probabilistic. If you're type checking you'll still get a simple pass/fail, but scores for text outputs are likely qualitative and/or categorical, and more open to interpretation.
For a deeper understanding, see [Core Concepts](https://ai.pydantic.dev/evals/core-concepts/).
## Datasets and Cases
In Pydantic Evals, everything begins with [`Dataset`](https://ai.pydantic.dev/api/pydantic_evals/dataset/#pydantic_evals.dataset.Dataset "Dataset")s and [`Case`](https://ai.pydantic.dev/api/pydantic_evals/dataset/#pydantic_evals.dataset.Case "Case



      dataclass
  ")s:
  * **[`Dataset`](https://ai.pydantic.dev/api/pydantic_evals/dataset/#pydantic_evals.dataset.Dataset "Dataset")**: A collection of test Cases designed for the evaluation of a specific task or function
  * **[`Case`](https://ai.pydantic.dev/api/pydantic_evals/dataset/#pydantic_evals.dataset.Case "Case



      dataclass
  ")**: A single test scenario corresponding to Task inputs, with optional expected outputs, metadata, and case-specific evaluators


simple_eval_dataset.py```
from pydantic_evals import Case, Dataset

case1 = Case(
    name='simple_case',
    inputs='What is the capital of France?',
    expected_output='Paris',
    metadata={'difficulty': 'easy'},
)

dataset = Dataset(cases=[case1])

```

_(This example is complete, it can be run "as is")_
See [Dataset Management](https://ai.pydantic.dev/evals/how-to/dataset-management/) to learn about saving, loading, and generating datasets.
## Evaluators
[`Evaluator`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator "Evaluator



      dataclass
  ")s analyze and score the results of your Task when tested against a Case.
These can be deterministic, code-based checks (such as testing model output format with a regex, or checking for the appearance of PII or sensitive data), or they can assess non-deterministic model outputs for qualities like accuracy, precision/recall, hallucinations, or instruction-following.
While both kinds of testing are useful in LLM systems, classical code-based tests are cheaper and easier than tests which require either human or machine review of model outputs.
Pydantic Evals includes several [built-in evaluators](https://ai.pydantic.dev/evals/evaluators/built-in/) and allows you to define [custom evaluators](https://ai.pydantic.dev/evals/evaluators/custom/):
simple_eval_evaluator.py```
from dataclasses import dataclass

from pydantic_evals.evaluators import Evaluator, EvaluatorContext
from pydantic_evals.evaluators.common import IsInstance

from simple_eval_dataset import dataset

dataset.add_evaluator(IsInstance(type_name='str'))  [](https://ai.pydantic.dev/evals/#__code_6_annotation_1)


@dataclass
class MyEvaluator(Evaluator):
    async def evaluate(self, ctx: EvaluatorContext[str, str]) -> float:  [](https://ai.pydantic.dev/evals/#__code_6_annotation_2)
        if ctx.output == ctx.expected_output:
            return 1.0
        elif (
            isinstance(ctx.output, str)
            and ctx.expected_output.lower() in ctx.output.lower()
        ):
            return 0.8
        else:
            return 0.0


dataset.add_evaluator(MyEvaluator())

```

_(This example is complete, it can be run "as is")_
Learn more:
  * [Evaluators Overview](https://ai.pydantic.dev/evals/evaluators/overview/) - When to use different types
  * [Built-in Evaluators](https://ai.pydantic.dev/evals/evaluators/built-in/) - Complete reference
  * [LLM Judge](https://ai.pydantic.dev/evals/evaluators/llm-judge/) - Using LLMs as evaluators
  * [Custom Evaluators](https://ai.pydantic.dev/evals/evaluators/custom/) - Write your own logic
  * [Span-Based Evaluation](https://ai.pydantic.dev/evals/evaluators/span-based/) - Analyze execution traces


## Running Experiments
Performing evaluations involves running a task against all cases in a dataset, also known as running an "experiment".
Putting the above two examples together and using the more declarative `evaluators` kwarg to [`Dataset`](https://ai.pydantic.dev/api/pydantic_evals/dataset/#pydantic_evals.dataset.Dataset "Dataset"):
simple_eval_complete.py```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import Evaluator, EvaluatorContext, IsInstance

case1 = Case(  [](https://ai.pydantic.dev/evals/#__code_7_annotation_1)
    name='simple_case',
    inputs='What is the capital of France?',
    expected_output='Paris',
    metadata={'difficulty': 'easy'},
)


class MyEvaluator(Evaluator[str, str]):
    def evaluate(self, ctx: EvaluatorContext[str, str]) -> float:
        if ctx.output == ctx.expected_output:
            return 1.0
        elif (
            isinstance(ctx.output, str)
            and ctx.expected_output.lower() in ctx.output.lower()
        ):
            return 0.8
        else:
            return 0.0


dataset = Dataset(
    cases=[case1],
    evaluators=[IsInstance(type_name='str'), MyEvaluator()],  [](https://ai.pydantic.dev/evals/#__code_7_annotation_2)
)


async def guess_city(question: str) -> str:  [](https://ai.pydantic.dev/evals/#__code_7_annotation_3)
    return 'Paris'


report = dataset.evaluate_sync(guess_city)  [](https://ai.pydantic.dev/evals/#__code_7_annotation_4)
report.print(include_input=True, include_output=True, include_durations=False)  [](https://ai.pydantic.dev/evals/#__code_7_annotation_5)
"""
                              Evaluation Summary: guess_city
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Case ID     ┃ Inputs                         ┃ Outputs ┃ Scores            ┃ Assertions ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ simple_case │ What is the capital of France? │ Paris   │ MyEvaluator: 1.00 │ ✔          │
├─────────────┼────────────────────────────────┼─────────┼───────────────────┼────────────┤
│ Averages    │                                │         │ MyEvaluator: 1.00 │ 100.0% ✔   │
└─────────────┴────────────────────────────────┴─────────┴───────────────────┴────────────┘
"""

```

_(This example is complete, it can be run "as is")_
See [Quick Start](https://ai.pydantic.dev/evals/quick-start/) for more examples and [Concurrency & Performance](https://ai.pydantic.dev/evals/how-to/concurrency/) to learn about controlling parallel execution.
## API Reference
For comprehensive coverage of all classes, methods, and configuration options, see the detailed [API Reference documentation](https://ai.pydantic.dev/api/pydantic_evals/dataset/).
## Next Steps
  1. **Start with simple evaluations** using [Quick Start](https://ai.pydantic.dev/evals/quick-start/)
  2. **Understand the data model** with [Core Concepts](https://ai.pydantic.dev/evals/core-concepts/)
  3. **Explore built-in evaluators** in [Built-in Evaluators](https://ai.pydantic.dev/evals/evaluators/built-in/)
  4. **Integrate with Logfire** for visualization: [Logfire Integration](https://ai.pydantic.dev/evals/how-to/logfire-integration/)
  5. **Build comprehensive test suites** with [Dataset Management](https://ai.pydantic.dev/evals/how-to/dataset-management/)
  6. **Implement custom evaluators** for domain-specific metrics: [Custom Evaluators](https://ai.pydantic.dev/evals/evaluators/custom/)


© Pydantic Services Inc. 2024 to present
