[ Skip to content ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#dataset-serialization)
**Join us at the inaugural PyAI Conf in San Francisco on March 10th![Learn More](https://pyai.events/?utm_source=pydantic-ai) **
[ ![logo](https://ai.pydantic.dev/img/logo-white.svg) ](https://ai.pydantic.dev/ "Pydantic AI")
Pydantic AI
Dataset Serialization
Type to start searching
[ pydantic/pydantic-ai
  * v1.63.0
  * 15.1k
  * 1.7k

](https://github.com/pydantic/pydantic-ai "Go to repository")
[ ![logo](https://ai.pydantic.dev/img/logo-white.svg) ](https://ai.pydantic.dev/ "Pydantic AI") Pydantic AI
[ pydantic/pydantic-ai
  * v1.63.0
  * 15.1k
  * 1.7k

](https://github.com/pydantic/pydantic-ai "Go to repository")
  * [ Pydantic AI  ](https://ai.pydantic.dev/)
  * [ Installation  ](https://ai.pydantic.dev/install/)
  * [ Getting Help  ](https://ai.pydantic.dev/help/)
  * [ Troubleshooting  ](https://ai.pydantic.dev/troubleshooting/)
  * [ Pydantic AI Gateway  ](https://ai.pydantic.dev/gateway/)
  * Documentation
    * Core Concepts
      * [ Agents  ](https://ai.pydantic.dev/agent/)
      * [ Dependencies  ](https://ai.pydantic.dev/dependencies/)
      * [ Function Tools  ](https://ai.pydantic.dev/tools/)
      * [ Output  ](https://ai.pydantic.dev/output/)
      * [ Messages and chat history  ](https://ai.pydantic.dev/message-history/)
      * [ Direct Model Requests  ](https://ai.pydantic.dev/direct/)
    * Models & Providers
      * [ Overview  ](https://ai.pydantic.dev/models/overview/)
      * [ OpenAI  ](https://ai.pydantic.dev/models/openai/)
      * [ Anthropic  ](https://ai.pydantic.dev/models/anthropic/)
      * [ Google  ](https://ai.pydantic.dev/models/google/)
      * [ xAI  ](https://ai.pydantic.dev/models/xai/)
      * [ Bedrock  ](https://ai.pydantic.dev/models/bedrock/)
      * [ Cerebras  ](https://ai.pydantic.dev/models/cerebras/)
      * [ Cohere  ](https://ai.pydantic.dev/models/cohere/)
      * [ Groq  ](https://ai.pydantic.dev/models/groq/)
      * [ Hugging Face  ](https://ai.pydantic.dev/models/huggingface/)
      * [ Mistral  ](https://ai.pydantic.dev/models/mistral/)
      * [ OpenRouter  ](https://ai.pydantic.dev/models/openrouter/)
      * [ Outlines  ](https://ai.pydantic.dev/models/outlines/)
    * Tools & Toolsets
      * [ Function Tools  ](https://ai.pydantic.dev/tools/)
      * [ Advanced Tool Features  ](https://ai.pydantic.dev/tools-advanced/)
      * [ Toolsets  ](https://ai.pydantic.dev/toolsets/)
      * [ Deferred Tools  ](https://ai.pydantic.dev/deferred-tools/)
      * [ Built-in Tools  ](https://ai.pydantic.dev/builtin-tools/)
      * [ Common Tools  ](https://ai.pydantic.dev/common-tools/)
      * [ Third-Party Tools  ](https://ai.pydantic.dev/third-party-tools/)
    * Advanced Features
      * [ Image, Audio, Video & Document Input  ](https://ai.pydantic.dev/input/)
      * [ Thinking  ](https://ai.pydantic.dev/thinking/)
      * [ HTTP Request Retries  ](https://ai.pydantic.dev/retries/)
    * MCP
      * [ Overview  ](https://ai.pydantic.dev/mcp/overview/)
      * [ Client  ](https://ai.pydantic.dev/mcp/client/)
      * [ FastMCP Client  ](https://ai.pydantic.dev/mcp/fastmcp-client/)
      * [ Server  ](https://ai.pydantic.dev/mcp/server/)
    * [ Multi-Agent Patterns  ](https://ai.pydantic.dev/multi-agent-applications/)
    * [ Web Chat UI  ](https://ai.pydantic.dev/web/)
    * [ Embeddings  ](https://ai.pydantic.dev/embeddings/)
    * [ Testing  ](https://ai.pydantic.dev/testing/)
  * Pydantic Evals
    * [ Overview  ](https://ai.pydantic.dev/evals/)
    * Getting Started
      * [ Quick Start  ](https://ai.pydantic.dev/evals/quick-start/)
      * [ Core Concepts  ](https://ai.pydantic.dev/evals/core-concepts/)
    * Evaluators
      * [ Overview  ](https://ai.pydantic.dev/evals/evaluators/overview/)
      * [ Built-in Evaluators  ](https://ai.pydantic.dev/evals/evaluators/built-in/)
      * [ LLM Judge  ](https://ai.pydantic.dev/evals/evaluators/llm-judge/)
      * [ Custom Evaluators  ](https://ai.pydantic.dev/evals/evaluators/custom/)
      * [ Report Evaluators  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/)
      * [ Span-Based  ](https://ai.pydantic.dev/evals/evaluators/span-based/)
    * How-To Guides
      * [ Logfire Integration  ](https://ai.pydantic.dev/evals/how-to/logfire-integration/)
      * [ Dataset Management  ](https://ai.pydantic.dev/evals/how-to/dataset-management/)
      * Dataset Serialization  [ Dataset Serialization  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/)
        * [ Overview  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#overview)
        * [ YAML Format  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#yaml-format)
          * [ Basic Example  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#basic-example)
          * [ YAML Output  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#yaml-output)
          * [ JSON Schema for IDEs  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#json-schema-for-ides)
          * [ Loading from YAML  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#loading-from-yaml)
        * [ JSON Format  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#json-format)
          * [ Basic Example  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#basic-example_1)
          * [ JSON Output  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#json-output)
          * [ Loading from JSON  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#loading-from-json)
        * [ Schema Generation  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#schema-generation)
          * [ Automatic Schema Creation  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#automatic-schema-creation)
          * [ Custom Schema Location  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#custom-schema-location)
          * [ Schema Path Templates  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#schema-path-templates)
          * [ Manual Schema Generation  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#manual-schema-generation)
        * [ Custom Evaluators  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#custom-evaluators)
          * [ Requirements  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#requirements)
          * [ Complete Example  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#complete-example)
          * [ Saved YAML  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#saved-yaml)
          * [ Loading with Custom Evaluators  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#loading-with-custom-evaluators)
        * [ Evaluator Serialization Formats  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#evaluator-serialization-formats)
          * [ 1. Name Only (No Parameters)  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#1-name-only-no-parameters)
          * [ 2. Single Parameter (Short Form)  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#2-single-parameter-short-form)
          * [ 3. Multiple Parameters (Dict Form)  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#3-multiple-parameters-dict-form)
        * [ Format Comparison  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#format-comparison)
        * [ Advanced: Evaluator Serialization Name  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#advanced-evaluator-serialization-name)
        * [ Troubleshooting  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#troubleshooting)
          * [ Schema Not Found in IDE  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#schema-not-found-in-ide)
          * [ Custom Evaluator Not Found  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#custom-evaluator-not-found)
          * [ Format Inference Failed  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#format-inference-failed)
          * [ Schema Generation Error  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#schema-generation-error)
        * [ Next Steps  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#next-steps)
      * [ Concurrency & Performance  ](https://ai.pydantic.dev/evals/how-to/concurrency/)
      * [ Multi-Run Evaluation  ](https://ai.pydantic.dev/evals/how-to/multi-run/)
      * [ Retry Strategies  ](https://ai.pydantic.dev/evals/how-to/retry-strategies/)
      * [ Metrics & Attributes  ](https://ai.pydantic.dev/evals/how-to/metrics-attributes/)
    * Examples
      * [ Simple Validation  ](https://ai.pydantic.dev/evals/examples/simple-validation/)
  * Pydantic Graph
    * [ Overview  ](https://ai.pydantic.dev/graph/)
    * [ Beta API  ](https://ai.pydantic.dev/graph/beta/)
      * [ Steps  ](https://ai.pydantic.dev/graph/beta/steps/)
      * [ Joins & Reducers  ](https://ai.pydantic.dev/graph/beta/joins/)
      * [ Decisions  ](https://ai.pydantic.dev/graph/beta/decisions/)
      * [ Parallel Execution  ](https://ai.pydantic.dev/graph/beta/parallel/)
  * Integrations
    * [ Debugging & Monitoring with Pydantic Logfire  ](https://ai.pydantic.dev/logfire/)
    * Durable Execution
      * [ Overview  ](https://ai.pydantic.dev/durable_execution/overview/)
      * [ Temporal  ](https://ai.pydantic.dev/durable_execution/temporal/)
      * [ DBOS  ](https://ai.pydantic.dev/durable_execution/dbos/)
      * [ Prefect  ](https://ai.pydantic.dev/durable_execution/prefect/)
    * UI Event Streams
      * [ Overview  ](https://ai.pydantic.dev/ui/overview/)
      * [ AG-UI  ](https://ai.pydantic.dev/ui/ag-ui/)
      * [ Vercel AI  ](https://ai.pydantic.dev/ui/vercel-ai/)
    * [ Agent2Agent (A2A)  ](https://ai.pydantic.dev/a2a/)
  * Related Packages
    * [ clai  ](https://ai.pydantic.dev/cli/)
  * Examples
    * [ Setup  ](https://ai.pydantic.dev/examples/setup/)
    * Getting Started
      * [ Pydantic Model  ](https://ai.pydantic.dev/examples/pydantic-model/)
      * [ Weather agent  ](https://ai.pydantic.dev/examples/weather-agent/)
    * Conversational Agents
      * [ Chat App with FastAPI  ](https://ai.pydantic.dev/examples/chat-app/)
      * [ Bank support  ](https://ai.pydantic.dev/examples/bank-support/)
    * Data & Analytics
      * [ SQL Generation  ](https://ai.pydantic.dev/examples/sql-gen/)
      * [ Data Analyst  ](https://ai.pydantic.dev/examples/data-analyst/)
      * [ RAG  ](https://ai.pydantic.dev/examples/rag/)
    * Streaming
      * [ Stream markdown  ](https://ai.pydantic.dev/examples/stream-markdown/)
      * [ Stream whales  ](https://ai.pydantic.dev/examples/stream-whales/)
    * Complex Workflows
      * [ Flight booking  ](https://ai.pydantic.dev/examples/flight-booking/)
      * [ Question Graph  ](https://ai.pydantic.dev/examples/question-graph/)
    * Business Applications
      * [ Slack Lead Qualifier with Modal  ](https://ai.pydantic.dev/examples/slack-lead-qualifier/)
    * UI Examples
      * [ Agent User Interaction (AG-UI)  ](https://ai.pydantic.dev/examples/ag-ui/)
  * API Reference
    * pydantic_ai
      * [ pydantic_ai.ag_ui  ](https://ai.pydantic.dev/api/ag_ui/)
      * [ pydantic_ai.agent  ](https://ai.pydantic.dev/api/agent/)
      * [ pydantic_ai.builtin_tools  ](https://ai.pydantic.dev/api/builtin_tools/)
      * [ pydantic_ai.common_tools  ](https://ai.pydantic.dev/api/common_tools/)
      * [ pydantic_ai — Concurrency  ](https://ai.pydantic.dev/api/concurrency/)
      * [ pydantic_ai.direct  ](https://ai.pydantic.dev/api/direct/)
      * [ pydantic_ai.durable_exec  ](https://ai.pydantic.dev/api/durable_exec/)
      * [ pydantic_ai.embeddings  ](https://ai.pydantic.dev/api/embeddings/)
      * [ pydantic_ai.exceptions  ](https://ai.pydantic.dev/api/exceptions/)
      * [ pydantic_ai.ext  ](https://ai.pydantic.dev/api/ext/)
      * [ pydantic_ai.format_prompt  ](https://ai.pydantic.dev/api/format_prompt/)
      * [ pydantic_ai.mcp  ](https://ai.pydantic.dev/api/mcp/)
      * [ pydantic_ai.messages  ](https://ai.pydantic.dev/api/messages/)
      * [ pydantic_ai.models.anthropic  ](https://ai.pydantic.dev/api/models/anthropic/)
      * [ pydantic_ai.models  ](https://ai.pydantic.dev/api/models/base/)
      * [ pydantic_ai.models.bedrock  ](https://ai.pydantic.dev/api/models/bedrock/)
      * [ pydantic_ai.models.cerebras  ](https://ai.pydantic.dev/api/models/cerebras/)
      * [ pydantic_ai.models.cohere  ](https://ai.pydantic.dev/api/models/cohere/)
      * [ pydantic_ai.models.fallback  ](https://ai.pydantic.dev/api/models/fallback/)
      * [ pydantic_ai.models.function  ](https://ai.pydantic.dev/api/models/function/)
      * [ pydantic_ai.models.google  ](https://ai.pydantic.dev/api/models/google/)
      * [ pydantic_ai.models.xai  ](https://ai.pydantic.dev/api/models/xai/)
      * [ pydantic_ai.models.groq  ](https://ai.pydantic.dev/api/models/groq/)
      * [ pydantic_ai.models.huggingface  ](https://ai.pydantic.dev/api/models/huggingface/)
      * [ pydantic_ai.models.instrumented  ](https://ai.pydantic.dev/api/models/instrumented/)
      * [ pydantic_ai.models.mcp_sampling  ](https://ai.pydantic.dev/api/models/mcp-sampling/)
      * [ pydantic_ai.models.mistral  ](https://ai.pydantic.dev/api/models/mistral/)
      * [ pydantic_ai.models.openai  ](https://ai.pydantic.dev/api/models/openai/)
      * [ pydantic_ai.models.openrouter  ](https://ai.pydantic.dev/api/models/openrouter/)
      * [ pydantic_ai.models.outlines  ](https://ai.pydantic.dev/api/models/outlines/)
      * [ pydantic_ai.models.test  ](https://ai.pydantic.dev/api/models/test/)
      * [ pydantic_ai.models.wrapper  ](https://ai.pydantic.dev/api/models/wrapper/)
      * [ pydantic_ai.output  ](https://ai.pydantic.dev/api/output/)
      * [ pydantic_ai.profiles  ](https://ai.pydantic.dev/api/profiles/)
      * [ pydantic_ai.providers  ](https://ai.pydantic.dev/api/providers/)
      * [ pydantic_ai.result  ](https://ai.pydantic.dev/api/result/)
      * [ pydantic_ai.retries  ](https://ai.pydantic.dev/api/retries/)
      * [ pydantic_ai.run  ](https://ai.pydantic.dev/api/run/)
      * [ pydantic_ai.settings  ](https://ai.pydantic.dev/api/settings/)
      * [ pydantic_ai.tools  ](https://ai.pydantic.dev/api/tools/)
      * [ pydantic_ai.toolsets  ](https://ai.pydantic.dev/api/toolsets/)
      * [ pydantic_ai.ui.ag_ui  ](https://ai.pydantic.dev/api/ui/ag_ui/)
      * [ pydantic_ai.ui  ](https://ai.pydantic.dev/api/ui/base/)
      * [ pydantic_ai.ui.vercel_ai  ](https://ai.pydantic.dev/api/ui/vercel_ai/)
      * [ pydantic_ai.usage  ](https://ai.pydantic.dev/api/usage/)
    * pydantic_evals
      * [ pydantic_evals.dataset  ](https://ai.pydantic.dev/api/pydantic_evals/dataset/)
      * [ pydantic_evals.evaluators  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/)
      * [ pydantic_evals.reporting  ](https://ai.pydantic.dev/api/pydantic_evals/reporting/)
      * [ pydantic_evals.otel  ](https://ai.pydantic.dev/api/pydantic_evals/otel/)
      * [ pydantic_evals.generation  ](https://ai.pydantic.dev/api/pydantic_evals/generation/)
    * pydantic_graph
      * [ pydantic_graph  ](https://ai.pydantic.dev/api/pydantic_graph/graph/)
      * [ pydantic_graph.nodes  ](https://ai.pydantic.dev/api/pydantic_graph/nodes/)
      * [ pydantic_graph.persistence  ](https://ai.pydantic.dev/api/pydantic_graph/persistence/)
      * [ pydantic_graph.mermaid  ](https://ai.pydantic.dev/api/pydantic_graph/mermaid/)
      * [ pydantic_graph.exceptions  ](https://ai.pydantic.dev/api/pydantic_graph/exceptions/)
      * Beta API
        * [ pydantic_graph.beta  ](https://ai.pydantic.dev/api/pydantic_graph/beta/)
        * [ pydantic_graph.beta.graph  ](https://ai.pydantic.dev/api/pydantic_graph/beta_graph/)
        * [ pydantic_graph.beta.graph_builder  ](https://ai.pydantic.dev/api/pydantic_graph/beta_graph_builder/)
        * [ pydantic_graph.beta.step  ](https://ai.pydantic.dev/api/pydantic_graph/beta_step/)
        * [ pydantic_graph.beta.join  ](https://ai.pydantic.dev/api/pydantic_graph/beta_join/)
        * [ pydantic_graph.beta.decision  ](https://ai.pydantic.dev/api/pydantic_graph/beta_decision/)
        * [ pydantic_graph.beta.node  ](https://ai.pydantic.dev/api/pydantic_graph/beta_node/)
    * fasta2a
      * [ fasta2a  ](https://ai.pydantic.dev/api/fasta2a/)
  * Project
    * [ Contributing  ](https://ai.pydantic.dev/contributing/)
    * [ Upgrade Guide  ](https://ai.pydantic.dev/changelog/)
    * [ Version policy  ](https://ai.pydantic.dev/version-policy/)


  * [ Overview  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#overview)
  * [ YAML Format  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#yaml-format)
    * [ Basic Example  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#basic-example)
    * [ YAML Output  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#yaml-output)
    * [ JSON Schema for IDEs  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#json-schema-for-ides)
    * [ Loading from YAML  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#loading-from-yaml)
  * [ JSON Format  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#json-format)
    * [ Basic Example  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#basic-example_1)
    * [ JSON Output  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#json-output)
    * [ Loading from JSON  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#loading-from-json)
  * [ Schema Generation  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#schema-generation)
    * [ Automatic Schema Creation  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#automatic-schema-creation)
    * [ Custom Schema Location  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#custom-schema-location)
    * [ Schema Path Templates  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#schema-path-templates)
    * [ Manual Schema Generation  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#manual-schema-generation)
  * [ Custom Evaluators  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#custom-evaluators)
    * [ Requirements  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#requirements)
    * [ Complete Example  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#complete-example)
    * [ Saved YAML  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#saved-yaml)
    * [ Loading with Custom Evaluators  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#loading-with-custom-evaluators)
  * [ Evaluator Serialization Formats  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#evaluator-serialization-formats)
    * [ 1. Name Only (No Parameters)  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#1-name-only-no-parameters)
    * [ 2. Single Parameter (Short Form)  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#2-single-parameter-short-form)
    * [ 3. Multiple Parameters (Dict Form)  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#3-multiple-parameters-dict-form)
  * [ Format Comparison  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#format-comparison)
  * [ Advanced: Evaluator Serialization Name  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#advanced-evaluator-serialization-name)
  * [ Troubleshooting  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#troubleshooting)
    * [ Schema Not Found in IDE  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#schema-not-found-in-ide)
    * [ Custom Evaluator Not Found  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#custom-evaluator-not-found)
    * [ Format Inference Failed  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#format-inference-failed)
    * [ Schema Generation Error  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#schema-generation-error)
  * [ Next Steps  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/#next-steps)


  1. [ Pydantic AI  ](https://ai.pydantic.dev/)
  2. [ Pydantic Evals  ](https://ai.pydantic.dev/evals/)
  3. [ How-To Guides  ](https://ai.pydantic.dev/evals/how-to/logfire-integration/)


# Dataset Serialization
Learn how to save and load datasets in different formats, with support for custom evaluators and IDE integration.
## Overview
Pydantic Evals supports serializing datasets to files in two formats:
  * **YAML** (`.yaml`, `.yml`) - Human-readable, great for version control
  * **JSON** (`.json`) - Structured, machine-readable


Both formats support: - Automatic JSON schema generation for IDE autocomplete and validation - Custom evaluator serialization/deserialization - Type-safe loading with generic parameters
## YAML Format
YAML is the recommended format for most use cases due to its readability and compact syntax.
### Basic Example
```
from typing import Any

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EqualsExpected, IsInstance

# Create a dataset with typed parameters
dataset = Dataset[str, str, Any](
    name='my_tests',
    cases=[
        Case(
            name='test_1',
            inputs='hello',
            expected_output='HELLO',
        ),
    ],
    evaluators=[
        IsInstance(type_name='str'),
        EqualsExpected(),
    ],
)

# Save to YAML
dataset.to_file('my_tests.yaml')

```

This creates two files:
  1. **`my_tests.yaml`**- The dataset
  2. **`my_tests_schema.json`**- JSON schema for IDE support


### YAML Output
```
# yaml-language-server: $schema=my_tests_schema.json
name: my_tests
cases:
- name: test_1
  inputs: hello
  expected_output: HELLO
evaluators:
- IsInstance: str
- EqualsExpected

```

### JSON Schema for IDEs
The first line references the schema file:
```
# yaml-language-server: $schema=my_tests_schema.json

```

This enables: - ✅ **Autocomplete** in VS Code, PyCharm, and other editors - ✅ **Inline validation** while editing - ✅ **Documentation tooltips** for fields - ✅ **Error highlighting** for invalid data
Editor Support
The `yaml-language-server` comment is supported by:
  * VS Code (with YAML extension)
  * JetBrains IDEs (PyCharm, IntelliJ, etc.)
  * Most editors with YAML language server support


See the [YAML Language Server docs](https://github.com/redhat-developer/yaml-language-server#using-inlined-schema) for more details.
### Loading from YAML
```
from pathlib import Path
from typing import Any

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EqualsExpected, IsInstance

# First create and save the dataset
Path('my_tests.yaml').parent.mkdir(exist_ok=True)
dataset = Dataset[str, str, Any](
    name='my_tests',
    cases=[Case(name='test_1', inputs='hello', expected_output='HELLO')],
    evaluators=[IsInstance(type_name='str'), EqualsExpected()],
)
dataset.to_file('my_tests.yaml')

# Load the dataset with type parameters
dataset = Dataset[str, str, Any].from_file('my_tests.yaml')


def my_task(text: str) -> str:
    return text.upper()


# Run evaluation
report = dataset.evaluate_sync(my_task)

```

## JSON Format
JSON format is useful for programmatic generation or when strict structure is required.
### Basic Example
```
from typing import Any

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EqualsExpected

dataset = Dataset[str, str, Any](
    name='my_tests',
    cases=[
        Case(name='test_1', inputs='hello', expected_output='HELLO'),
    ],
    evaluators=[EqualsExpected()],
)

# Save to JSON
dataset.to_file('my_tests.json')

```

### JSON Output
```
{
  "$schema": "my_tests_schema.json",
  "name": "my_tests",
  "cases": [
    {
      "name": "test_1",
      "inputs": "hello",
      "expected_output": "HELLO"
    }
  ],
  "evaluators": [
    "EqualsExpected"
  ]
}

```

The `$schema` key at the top enables IDE support similar to YAML.
### Loading from JSON
```
from typing import Any

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EqualsExpected

# First create and save the dataset
dataset = Dataset[str, str, Any](
    name='my_tests',
    cases=[Case(name='test_1', inputs='hello', expected_output='HELLO')],
    evaluators=[EqualsExpected()],
)
dataset.to_file('my_tests.json')

# Load from JSON
dataset = Dataset[str, str, Any].from_file('my_tests.json')

```

## Schema Generation
### Automatic Schema Creation
By default, `to_file()` creates a JSON schema file alongside your dataset:
```
from typing import Any

from pydantic_evals import Case, Dataset

dataset = Dataset[str, str, Any](cases=[Case(inputs='test')])

# Creates both my_tests.yaml AND my_tests_schema.json
dataset.to_file('my_tests.yaml')

```

### Custom Schema Location
```
from pathlib import Path
from typing import Any

from pydantic_evals import Case, Dataset

dataset = Dataset[str, str, Any](cases=[Case(inputs='test')])

# Create directories
Path('data').mkdir(exist_ok=True)

# Custom schema filename (relative to dataset file location)
dataset.to_file(
    'data/my_tests.yaml',
    schema_path='my_schema.json',
)

# No schema file
dataset.to_file('my_tests.yaml', schema_path=None)

```

### Schema Path Templates
Use `{stem}` to reference the dataset filename:
```
from typing import Any

from pydantic_evals import Case, Dataset

dataset = Dataset[str, str, Any](cases=[Case(inputs='test')])

# Creates: my_tests.yaml and my_tests.schema.json
dataset.to_file(
    'my_tests.yaml',
    schema_path='{stem}.schema.json',
)

```

### Manual Schema Generation
Generate a schema without saving the dataset:
```
import json
from typing import Any

from pydantic_evals import Dataset

# Get schema as dictionary for a specific dataset type
schema = Dataset[str, str, Any].model_json_schema_with_evaluators()

# Save manually
with open('custom_schema.json', 'w', encoding='utf-8') as f:
    json.dump(schema, f, indent=2)

```

## Custom Evaluators
Custom evaluators require special handling during serialization and deserialization.
### Requirements
Custom evaluators must:
  1. Be decorated with `@dataclass`
  2. Inherit from `Evaluator`
  3. Be passed to both `to_file()` and `from_file()`


### Complete Example
```
from dataclasses import dataclass
from typing import Any

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import Evaluator, EvaluatorContext


@dataclass
class CustomThreshold(Evaluator):
    """Check if output length exceeds a threshold."""

    min_length: int
    max_length: int = 100

    def evaluate(self, ctx: EvaluatorContext) -> bool:
        length = len(str(ctx.output))
        return self.min_length <= length <= self.max_length


# Create dataset with custom evaluator
dataset = Dataset[str, str, Any](
    cases=[
        Case(
            name='test_length',
            inputs='example',
            expected_output='long result',
            evaluators=[
                CustomThreshold(min_length=5, max_length=20),
            ],
        ),
    ],
)

# Save with custom evaluator types
dataset.to_file(
    'dataset.yaml',
    custom_evaluator_types=[CustomThreshold],
)

```

### Saved YAML
```
# yaml-language-server: $schema=dataset_schema.json
cases:
- name: test_length
  inputs: example
  expected_output: long result
  evaluators:
  - CustomThreshold:
      min_length: 5
      max_length: 20

```

### Loading with Custom Evaluators
```
from dataclasses import dataclass
from typing import Any

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import Evaluator, EvaluatorContext


@dataclass
class CustomThreshold(Evaluator):
    """Check if output length exceeds a threshold."""

    min_length: int
    max_length: int = 100

    def evaluate(self, ctx: EvaluatorContext) -> bool:
        length = len(str(ctx.output))
        return self.min_length <= length <= self.max_length


# First create and save the dataset
dataset = Dataset[str, str, Any](
    cases=[
        Case(
            name='test_length',
            inputs='example',
            expected_output='long result',
            evaluators=[CustomThreshold(min_length=5, max_length=20)],
        ),
    ],
)
dataset.to_file('dataset.yaml', custom_evaluator_types=[CustomThreshold])

# Load with custom evaluator registry
dataset = Dataset[str, str, Any].from_file(
    'dataset.yaml',
    custom_evaluator_types=[CustomThreshold],
)

```

Important
You must pass `custom_evaluator_types` to **both** `to_file()` and `from_file()`.
  * `to_file()`: Includes the evaluator in the JSON schema
  * `from_file()`: Registers the evaluator for deserialization


## Evaluator Serialization Formats
Evaluators can be serialized in three forms:
### 1. Name Only (No Parameters)
```
evaluators:
- EqualsExpected
- IsInstance: str  # Using default parameter

```

### 2. Single Parameter (Short Form)
```
evaluators:
- IsInstance: str
- Contains: "required text"
- MaxDuration: 2.0

```

### 3. Multiple Parameters (Dict Form)
```
evaluators:
- CustomThreshold:
    min_length: 5
    max_length: 20
- LLMJudge:
    rubric: "Response is accurate"
    model: "openai:gpt-5"
    include_input: true

```

## Format Comparison
Feature | YAML | JSON
---|---|---
Human readable | ✅ Excellent | ⚠️ Good
Comments | ✅ Yes | ❌ No
Compact | ✅ Yes | ⚠️ Verbose
Machine parsing | ✅ Good | ✅ Excellent
IDE support | ✅ Yes | ✅ Yes
Version control | ✅ Clean diffs | ⚠️ Noisy diffs
**Recommendation** : Use YAML for most cases, JSON for programmatic generation.
## Advanced: Evaluator Serialization Name
Customize how your evaluator appears in serialized files:
```
from dataclasses import dataclass

from pydantic_evals.evaluators import Evaluator, EvaluatorContext


@dataclass
class VeryLongDescriptiveEvaluatorName(Evaluator):
    @classmethod
    def get_serialization_name(cls) -> str:
        return 'ShortName'

    def evaluate(self, ctx: EvaluatorContext) -> bool:
        return True

```

In YAML:
```
evaluators:
- ShortName  # Instead of VeryLongDescriptiveEvaluatorName

```

## Troubleshooting
### Schema Not Found in IDE
**Problem** : YAML file doesn't show autocomplete
**Solutions** :
  1. **Check the schema path** in the first line of YAML:
```
# yaml-language-server: $schema=correct_schema_name.json

```

  2. **Verify schema file exists** in the same directory
  3. **Restart the language server** in your IDE
  4. **Install YAML extension** (VS Code: "YAML" by Red Hat)


### Custom Evaluator Not Found
**Problem** : `ValueError: Unknown evaluator name: 'CustomEvaluator'`
**Solution** : Pass `custom_evaluator_types` when loading:
```
from dataclasses import dataclass
from typing import Any

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import Evaluator, EvaluatorContext


@dataclass
class CustomEvaluator(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> bool:
        return True


# First create and save with custom evaluator
dataset = Dataset[str, str, Any](
    cases=[Case(inputs='test', evaluators=[CustomEvaluator()])],
)
dataset.to_file('tests.yaml', custom_evaluator_types=[CustomEvaluator])

# Load with custom evaluator types
dataset = Dataset[str, str, Any].from_file(
    'tests.yaml',
    custom_evaluator_types=[CustomEvaluator],  # Required!
)

```

### Format Inference Failed
**Problem** : `ValueError: Cannot infer format from extension`
**Solution** : Specify format explicitly:
```
from typing import Any

from pydantic_evals import Case, Dataset

dataset = Dataset[str, str, Any](cases=[Case(inputs='test')])

# Explicit format for unusual extensions
dataset.to_file('data.txt', fmt='yaml')
dataset_loaded = Dataset[str, str, Any].from_file('data.txt', fmt='yaml')

```

### Schema Generation Error
**Problem** : Custom evaluator causes schema generation to fail
**Solution** : Ensure evaluator is a proper dataclass:
```
from dataclasses import dataclass

from pydantic_evals.evaluators import Evaluator, EvaluatorContext


# ✅ Correct
@dataclass
class MyEvaluator(Evaluator):
    value: int

    def evaluate(self, ctx: EvaluatorContext) -> bool:
        return True


# ❌ Wrong: Missing @dataclass
class BadEvaluator(Evaluator):
    def __init__(self, value: int):
        self.value = value

    def evaluate(self, ctx: EvaluatorContext) -> bool:
        return True

```

## Next Steps
  * **[Dataset Management](https://ai.pydantic.dev/evals/how-to/dataset-management/)** - Creating and organizing datasets
  * **[Custom Evaluators](https://ai.pydantic.dev/evals/evaluators/custom/)** - Write custom evaluation logic
  * **[Core Concepts](https://ai.pydantic.dev/evals/core-concepts/)** - Understand the data model


© Pydantic Services Inc. 2024 to present
