[ Skip to content ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#report-evaluators)
**Join us at the inaugural PyAI Conf in San Francisco on March 10th![Learn More](https://pyai.events/?utm_source=pydantic-ai) **
[ ![logo](https://ai.pydantic.dev/img/logo-white.svg) ](https://ai.pydantic.dev/ "Pydantic AI")
Pydantic AI
Report Evaluators
Type to start searching
[ pydantic/pydantic-ai
  * v1.63.0
  * 15.1k
  * 1.7k

](https://github.com/pydantic/pydantic-ai "Go to repository")
[ ![logo](https://ai.pydantic.dev/img/logo-white.svg) ](https://ai.pydantic.dev/ "Pydantic AI") Pydantic AI
[ pydantic/pydantic-ai
  * v1.63.0
  * 15.1k
  * 1.7k

](https://github.com/pydantic/pydantic-ai "Go to repository")
  * [ Pydantic AI  ](https://ai.pydantic.dev/)
  * [ Installation  ](https://ai.pydantic.dev/install/)
  * [ Getting Help  ](https://ai.pydantic.dev/help/)
  * [ Troubleshooting  ](https://ai.pydantic.dev/troubleshooting/)
  * [ Pydantic AI Gateway  ](https://ai.pydantic.dev/gateway/)
  * Documentation
    * Core Concepts
      * [ Agents  ](https://ai.pydantic.dev/agent/)
      * [ Dependencies  ](https://ai.pydantic.dev/dependencies/)
      * [ Function Tools  ](https://ai.pydantic.dev/tools/)
      * [ Output  ](https://ai.pydantic.dev/output/)
      * [ Messages and chat history  ](https://ai.pydantic.dev/message-history/)
      * [ Direct Model Requests  ](https://ai.pydantic.dev/direct/)
    * Models & Providers
      * [ Overview  ](https://ai.pydantic.dev/models/overview/)
      * [ OpenAI  ](https://ai.pydantic.dev/models/openai/)
      * [ Anthropic  ](https://ai.pydantic.dev/models/anthropic/)
      * [ Google  ](https://ai.pydantic.dev/models/google/)
      * [ xAI  ](https://ai.pydantic.dev/models/xai/)
      * [ Bedrock  ](https://ai.pydantic.dev/models/bedrock/)
      * [ Cerebras  ](https://ai.pydantic.dev/models/cerebras/)
      * [ Cohere  ](https://ai.pydantic.dev/models/cohere/)
      * [ Groq  ](https://ai.pydantic.dev/models/groq/)
      * [ Hugging Face  ](https://ai.pydantic.dev/models/huggingface/)
      * [ Mistral  ](https://ai.pydantic.dev/models/mistral/)
      * [ OpenRouter  ](https://ai.pydantic.dev/models/openrouter/)
      * [ Outlines  ](https://ai.pydantic.dev/models/outlines/)
    * Tools & Toolsets
      * [ Function Tools  ](https://ai.pydantic.dev/tools/)
      * [ Advanced Tool Features  ](https://ai.pydantic.dev/tools-advanced/)
      * [ Toolsets  ](https://ai.pydantic.dev/toolsets/)
      * [ Deferred Tools  ](https://ai.pydantic.dev/deferred-tools/)
      * [ Built-in Tools  ](https://ai.pydantic.dev/builtin-tools/)
      * [ Common Tools  ](https://ai.pydantic.dev/common-tools/)
      * [ Third-Party Tools  ](https://ai.pydantic.dev/third-party-tools/)
    * Advanced Features
      * [ Image, Audio, Video & Document Input  ](https://ai.pydantic.dev/input/)
      * [ Thinking  ](https://ai.pydantic.dev/thinking/)
      * [ HTTP Request Retries  ](https://ai.pydantic.dev/retries/)
    * MCP
      * [ Overview  ](https://ai.pydantic.dev/mcp/overview/)
      * [ Client  ](https://ai.pydantic.dev/mcp/client/)
      * [ FastMCP Client  ](https://ai.pydantic.dev/mcp/fastmcp-client/)
      * [ Server  ](https://ai.pydantic.dev/mcp/server/)
    * [ Multi-Agent Patterns  ](https://ai.pydantic.dev/multi-agent-applications/)
    * [ Web Chat UI  ](https://ai.pydantic.dev/web/)
    * [ Embeddings  ](https://ai.pydantic.dev/embeddings/)
    * [ Testing  ](https://ai.pydantic.dev/testing/)
  * Pydantic Evals
    * [ Overview  ](https://ai.pydantic.dev/evals/)
    * Getting Started
      * [ Quick Start  ](https://ai.pydantic.dev/evals/quick-start/)
      * [ Core Concepts  ](https://ai.pydantic.dev/evals/core-concepts/)
    * Evaluators
      * [ Overview  ](https://ai.pydantic.dev/evals/evaluators/overview/)
      * [ Built-in Evaluators  ](https://ai.pydantic.dev/evals/evaluators/built-in/)
      * [ LLM Judge  ](https://ai.pydantic.dev/evals/evaluators/llm-judge/)
      * [ Custom Evaluators  ](https://ai.pydantic.dev/evals/evaluators/custom/)
      * Report Evaluators  [ Report Evaluators  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/)
        * [ How Report Evaluators Work  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#how-report-evaluators-work)
        * [ Using Report Evaluators  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#using-report-evaluators)
        * [ Built-in Report Evaluators  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#built-in-report-evaluators)
          * [ ConfusionMatrixEvaluator  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#confusionmatrixevaluator)
          * [ PrecisionRecallEvaluator  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#precisionrecallevaluator)
          * [ ROCAUCEvaluator  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#rocaucevaluator)
          * [ KolmogorovSmirnovEvaluator  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#kolmogorovsmirnovevaluator)
        * [ Custom Report Evaluators  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#custom-report-evaluators)
          * [ ReportEvaluatorContext  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#reportevaluatorcontext)
          * [ Return Types  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#return-types)
            * [ ScalarResult  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#scalarresult)
            * [ TableResult  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#tableresult)
            * [ ConfusionMatrix  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#confusionmatrix)
            * [ PrecisionRecall  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#precisionrecall)
            * [ LinePlot  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#lineplot)
          * [ Returning Multiple Analyses  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#returning-multiple-analyses)
          * [ Async Report Evaluators  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#async-report-evaluators)
        * [ Serialization  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#serialization)
        * [ Viewing Analyses in Logfire  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#viewing-analyses-in-logfire)
        * [ Complete Example  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#complete-example)
        * [ Next Steps  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#next-steps)
      * [ Span-Based  ](https://ai.pydantic.dev/evals/evaluators/span-based/)
    * How-To Guides
      * [ Logfire Integration  ](https://ai.pydantic.dev/evals/how-to/logfire-integration/)
      * [ Dataset Management  ](https://ai.pydantic.dev/evals/how-to/dataset-management/)
      * [ Dataset Serialization  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/)
      * [ Concurrency & Performance  ](https://ai.pydantic.dev/evals/how-to/concurrency/)
      * [ Multi-Run Evaluation  ](https://ai.pydantic.dev/evals/how-to/multi-run/)
      * [ Retry Strategies  ](https://ai.pydantic.dev/evals/how-to/retry-strategies/)
      * [ Metrics & Attributes  ](https://ai.pydantic.dev/evals/how-to/metrics-attributes/)
    * Examples
      * [ Simple Validation  ](https://ai.pydantic.dev/evals/examples/simple-validation/)
  * Pydantic Graph
    * [ Overview  ](https://ai.pydantic.dev/graph/)
    * [ Beta API  ](https://ai.pydantic.dev/graph/beta/)
      * [ Steps  ](https://ai.pydantic.dev/graph/beta/steps/)
      * [ Joins & Reducers  ](https://ai.pydantic.dev/graph/beta/joins/)
      * [ Decisions  ](https://ai.pydantic.dev/graph/beta/decisions/)
      * [ Parallel Execution  ](https://ai.pydantic.dev/graph/beta/parallel/)
  * Integrations
    * [ Debugging & Monitoring with Pydantic Logfire  ](https://ai.pydantic.dev/logfire/)
    * Durable Execution
      * [ Overview  ](https://ai.pydantic.dev/durable_execution/overview/)
      * [ Temporal  ](https://ai.pydantic.dev/durable_execution/temporal/)
      * [ DBOS  ](https://ai.pydantic.dev/durable_execution/dbos/)
      * [ Prefect  ](https://ai.pydantic.dev/durable_execution/prefect/)
    * UI Event Streams
      * [ Overview  ](https://ai.pydantic.dev/ui/overview/)
      * [ AG-UI  ](https://ai.pydantic.dev/ui/ag-ui/)
      * [ Vercel AI  ](https://ai.pydantic.dev/ui/vercel-ai/)
    * [ Agent2Agent (A2A)  ](https://ai.pydantic.dev/a2a/)
  * Related Packages
    * [ clai  ](https://ai.pydantic.dev/cli/)
  * Examples
    * [ Setup  ](https://ai.pydantic.dev/examples/setup/)
    * Getting Started
      * [ Pydantic Model  ](https://ai.pydantic.dev/examples/pydantic-model/)
      * [ Weather agent  ](https://ai.pydantic.dev/examples/weather-agent/)
    * Conversational Agents
      * [ Chat App with FastAPI  ](https://ai.pydantic.dev/examples/chat-app/)
      * [ Bank support  ](https://ai.pydantic.dev/examples/bank-support/)
    * Data & Analytics
      * [ SQL Generation  ](https://ai.pydantic.dev/examples/sql-gen/)
      * [ Data Analyst  ](https://ai.pydantic.dev/examples/data-analyst/)
      * [ RAG  ](https://ai.pydantic.dev/examples/rag/)
    * Streaming
      * [ Stream markdown  ](https://ai.pydantic.dev/examples/stream-markdown/)
      * [ Stream whales  ](https://ai.pydantic.dev/examples/stream-whales/)
    * Complex Workflows
      * [ Flight booking  ](https://ai.pydantic.dev/examples/flight-booking/)
      * [ Question Graph  ](https://ai.pydantic.dev/examples/question-graph/)
    * Business Applications
      * [ Slack Lead Qualifier with Modal  ](https://ai.pydantic.dev/examples/slack-lead-qualifier/)
    * UI Examples
      * [ Agent User Interaction (AG-UI)  ](https://ai.pydantic.dev/examples/ag-ui/)
  * API Reference
    * pydantic_ai
      * [ pydantic_ai.ag_ui  ](https://ai.pydantic.dev/api/ag_ui/)
      * [ pydantic_ai.agent  ](https://ai.pydantic.dev/api/agent/)
      * [ pydantic_ai.builtin_tools  ](https://ai.pydantic.dev/api/builtin_tools/)
      * [ pydantic_ai.common_tools  ](https://ai.pydantic.dev/api/common_tools/)
      * [ pydantic_ai — Concurrency  ](https://ai.pydantic.dev/api/concurrency/)
      * [ pydantic_ai.direct  ](https://ai.pydantic.dev/api/direct/)
      * [ pydantic_ai.durable_exec  ](https://ai.pydantic.dev/api/durable_exec/)
      * [ pydantic_ai.embeddings  ](https://ai.pydantic.dev/api/embeddings/)
      * [ pydantic_ai.exceptions  ](https://ai.pydantic.dev/api/exceptions/)
      * [ pydantic_ai.ext  ](https://ai.pydantic.dev/api/ext/)
      * [ pydantic_ai.format_prompt  ](https://ai.pydantic.dev/api/format_prompt/)
      * [ pydantic_ai.mcp  ](https://ai.pydantic.dev/api/mcp/)
      * [ pydantic_ai.messages  ](https://ai.pydantic.dev/api/messages/)
      * [ pydantic_ai.models.anthropic  ](https://ai.pydantic.dev/api/models/anthropic/)
      * [ pydantic_ai.models  ](https://ai.pydantic.dev/api/models/base/)
      * [ pydantic_ai.models.bedrock  ](https://ai.pydantic.dev/api/models/bedrock/)
      * [ pydantic_ai.models.cerebras  ](https://ai.pydantic.dev/api/models/cerebras/)
      * [ pydantic_ai.models.cohere  ](https://ai.pydantic.dev/api/models/cohere/)
      * [ pydantic_ai.models.fallback  ](https://ai.pydantic.dev/api/models/fallback/)
      * [ pydantic_ai.models.function  ](https://ai.pydantic.dev/api/models/function/)
      * [ pydantic_ai.models.google  ](https://ai.pydantic.dev/api/models/google/)
      * [ pydantic_ai.models.xai  ](https://ai.pydantic.dev/api/models/xai/)
      * [ pydantic_ai.models.groq  ](https://ai.pydantic.dev/api/models/groq/)
      * [ pydantic_ai.models.huggingface  ](https://ai.pydantic.dev/api/models/huggingface/)
      * [ pydantic_ai.models.instrumented  ](https://ai.pydantic.dev/api/models/instrumented/)
      * [ pydantic_ai.models.mcp_sampling  ](https://ai.pydantic.dev/api/models/mcp-sampling/)
      * [ pydantic_ai.models.mistral  ](https://ai.pydantic.dev/api/models/mistral/)
      * [ pydantic_ai.models.openai  ](https://ai.pydantic.dev/api/models/openai/)
      * [ pydantic_ai.models.openrouter  ](https://ai.pydantic.dev/api/models/openrouter/)
      * [ pydantic_ai.models.outlines  ](https://ai.pydantic.dev/api/models/outlines/)
      * [ pydantic_ai.models.test  ](https://ai.pydantic.dev/api/models/test/)
      * [ pydantic_ai.models.wrapper  ](https://ai.pydantic.dev/api/models/wrapper/)
      * [ pydantic_ai.output  ](https://ai.pydantic.dev/api/output/)
      * [ pydantic_ai.profiles  ](https://ai.pydantic.dev/api/profiles/)
      * [ pydantic_ai.providers  ](https://ai.pydantic.dev/api/providers/)
      * [ pydantic_ai.result  ](https://ai.pydantic.dev/api/result/)
      * [ pydantic_ai.retries  ](https://ai.pydantic.dev/api/retries/)
      * [ pydantic_ai.run  ](https://ai.pydantic.dev/api/run/)
      * [ pydantic_ai.settings  ](https://ai.pydantic.dev/api/settings/)
      * [ pydantic_ai.tools  ](https://ai.pydantic.dev/api/tools/)
      * [ pydantic_ai.toolsets  ](https://ai.pydantic.dev/api/toolsets/)
      * [ pydantic_ai.ui.ag_ui  ](https://ai.pydantic.dev/api/ui/ag_ui/)
      * [ pydantic_ai.ui  ](https://ai.pydantic.dev/api/ui/base/)
      * [ pydantic_ai.ui.vercel_ai  ](https://ai.pydantic.dev/api/ui/vercel_ai/)
      * [ pydantic_ai.usage  ](https://ai.pydantic.dev/api/usage/)
    * pydantic_evals
      * [ pydantic_evals.dataset  ](https://ai.pydantic.dev/api/pydantic_evals/dataset/)
      * [ pydantic_evals.evaluators  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/)
      * [ pydantic_evals.reporting  ](https://ai.pydantic.dev/api/pydantic_evals/reporting/)
      * [ pydantic_evals.otel  ](https://ai.pydantic.dev/api/pydantic_evals/otel/)
      * [ pydantic_evals.generation  ](https://ai.pydantic.dev/api/pydantic_evals/generation/)
    * pydantic_graph
      * [ pydantic_graph  ](https://ai.pydantic.dev/api/pydantic_graph/graph/)
      * [ pydantic_graph.nodes  ](https://ai.pydantic.dev/api/pydantic_graph/nodes/)
      * [ pydantic_graph.persistence  ](https://ai.pydantic.dev/api/pydantic_graph/persistence/)
      * [ pydantic_graph.mermaid  ](https://ai.pydantic.dev/api/pydantic_graph/mermaid/)
      * [ pydantic_graph.exceptions  ](https://ai.pydantic.dev/api/pydantic_graph/exceptions/)
      * Beta API
        * [ pydantic_graph.beta  ](https://ai.pydantic.dev/api/pydantic_graph/beta/)
        * [ pydantic_graph.beta.graph  ](https://ai.pydantic.dev/api/pydantic_graph/beta_graph/)
        * [ pydantic_graph.beta.graph_builder  ](https://ai.pydantic.dev/api/pydantic_graph/beta_graph_builder/)
        * [ pydantic_graph.beta.step  ](https://ai.pydantic.dev/api/pydantic_graph/beta_step/)
        * [ pydantic_graph.beta.join  ](https://ai.pydantic.dev/api/pydantic_graph/beta_join/)
        * [ pydantic_graph.beta.decision  ](https://ai.pydantic.dev/api/pydantic_graph/beta_decision/)
        * [ pydantic_graph.beta.node  ](https://ai.pydantic.dev/api/pydantic_graph/beta_node/)
    * fasta2a
      * [ fasta2a  ](https://ai.pydantic.dev/api/fasta2a/)
  * Project
    * [ Contributing  ](https://ai.pydantic.dev/contributing/)
    * [ Upgrade Guide  ](https://ai.pydantic.dev/changelog/)
    * [ Version policy  ](https://ai.pydantic.dev/version-policy/)


  * [ How Report Evaluators Work  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#how-report-evaluators-work)
  * [ Using Report Evaluators  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#using-report-evaluators)
  * [ Built-in Report Evaluators  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#built-in-report-evaluators)
    * [ ConfusionMatrixEvaluator  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#confusionmatrixevaluator)
    * [ PrecisionRecallEvaluator  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#precisionrecallevaluator)
    * [ ROCAUCEvaluator  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#rocaucevaluator)
    * [ KolmogorovSmirnovEvaluator  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#kolmogorovsmirnovevaluator)
  * [ Custom Report Evaluators  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#custom-report-evaluators)
    * [ ReportEvaluatorContext  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#reportevaluatorcontext)
    * [ Return Types  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#return-types)
      * [ ScalarResult  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#scalarresult)
      * [ TableResult  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#tableresult)
      * [ ConfusionMatrix  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#confusionmatrix)
      * [ PrecisionRecall  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#precisionrecall)
      * [ LinePlot  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#lineplot)
    * [ Returning Multiple Analyses  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#returning-multiple-analyses)
    * [ Async Report Evaluators  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#async-report-evaluators)
  * [ Serialization  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#serialization)
  * [ Viewing Analyses in Logfire  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#viewing-analyses-in-logfire)
  * [ Complete Example  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#complete-example)
  * [ Next Steps  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#next-steps)


  1. [ Pydantic AI  ](https://ai.pydantic.dev/)
  2. [ Pydantic Evals  ](https://ai.pydantic.dev/evals/)
  3. [ Evaluators  ](https://ai.pydantic.dev/evals/evaluators/overview/)


# Report Evaluators
Report evaluators analyze entire experiment results rather than individual cases. Use them to compute experiment-wide statistics like confusion matrices, precision-recall curves, accuracy scores, or custom summary tables.
## How Report Evaluators Work
Regular [evaluators](https://ai.pydantic.dev/evals/evaluators/overview/) run once per case and assess individual outputs. Report evaluators run once per experiment _after_ all cases have been evaluated, receiving the full [`EvaluationReport`](https://ai.pydantic.dev/api/pydantic_evals/reporting/#pydantic_evals.reporting.EvaluationReport "EvaluationReport



      dataclass
  ") as input.
```
Cases executed → Case evaluators run → Report evaluators run → Final report

```

Results from report evaluators are stored as **analyses** on the report and, when Logfire is configured, are attached to the experiment span as structured attributes for visualization.
## Using Report Evaluators
Pass report evaluators to `Dataset` via the `report_evaluators` parameter:
```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import ConfusionMatrixEvaluator


def my_classifier(text: str) -> str:
    text = text.lower()
    if 'cat' in text or 'meow' in text:
        return 'cat'
    elif 'dog' in text or 'bark' in text:
        return 'dog'
    return 'unknown'


dataset = Dataset(
    cases=[
        Case(name='cat', inputs='The cat goes meow', expected_output='cat'),
        Case(name='dog', inputs='The dog barks', expected_output='dog'),
    ],
    report_evaluators=[
        ConfusionMatrixEvaluator(
            predicted_from='output',
            expected_from='expected_output',
            title='Animal Classification',
        ),
    ],
)

report = dataset.evaluate_sync(my_classifier)
# report.analyses contains the ConfusionMatrix result

```

## Built-in Report Evaluators
### ConfusionMatrixEvaluator
Builds a confusion matrix comparing predicted vs expected labels across all cases.
```
from pydantic_evals.evaluators import ConfusionMatrixEvaluator

ConfusionMatrixEvaluator(
    predicted_from='output',
    expected_from='expected_output',
    title='My Confusion Matrix',
)

```

**Parameters:**
Parameter | Type | Default | Description
---|---|---|---
`predicted_from` | `'expected_output' \| 'output' \| 'metadata' \| 'labels'` | `'output'` | Source for predicted values
`predicted_key` | `str \| None` | `None` | Key to extract when using `metadata` or `labels`
`expected_from` | `'expected_output' \| 'output' \| 'metadata' \| 'labels'` | `'expected_output'` | Source for expected/true values
`expected_key` | `str \| None` | `None` | Key to extract when using `metadata` or `labels`
`title` | `str` | `'Confusion Matrix'` | Title shown in reports
**Returns:** [`ConfusionMatrix`](https://ai.pydantic.dev/api/pydantic_evals/reporting/#pydantic_evals.reporting.ConfusionMatrix "ConfusionMatrix")
**Data Sources:**
  * `'output'` — the task's actual output (converted to string)
  * `'expected_output'` — the case's expected output (converted to string)
  * `'metadata'` — a value from the case's metadata dict (requires `key`)
  * `'labels'` — a label result from a case-level evaluator (requires `key`)


**Example — classification with expected outputs:**
```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import ConfusionMatrixEvaluator

dataset = Dataset(
    cases=[
        Case(inputs='meow', expected_output='cat'),
        Case(inputs='woof', expected_output='dog'),
        Case(inputs='chirp', expected_output='bird'),
    ],
    report_evaluators=[
        ConfusionMatrixEvaluator(
            predicted_from='output',
            expected_from='expected_output',
        ),
    ],
)

```

**Example — using evaluator labels:**
If a case-level evaluator produces a label like `predicted_class`, you can reference it:
```
from dataclasses import dataclass

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import (
    ConfusionMatrixEvaluator,
    Evaluator,
    EvaluatorContext,
)


@dataclass
class ClassifyOutput(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> dict[str, str]:
        # Classify the output into a category
        return {'predicted_class': categorize(ctx.output)}


def categorize(output: str) -> str:
    return 'positive' if 'good' in output.lower() else 'negative'


dataset = Dataset(
    cases=[Case(inputs='test', expected_output='positive')],
    evaluators=[ClassifyOutput()],
    report_evaluators=[
        ConfusionMatrixEvaluator(
            predicted_from='labels',
            predicted_key='predicted_class',
            expected_from='expected_output',
        ),
    ],
)

```

* * *
### PrecisionRecallEvaluator
Computes a precision-recall curve with AUC (area under the curve) from numeric scores and binary ground-truth labels.
```
from pydantic_evals.evaluators import PrecisionRecallEvaluator

PrecisionRecallEvaluator(
    score_from='scores',
    score_key='confidence',
    positive_from='assertions',
    positive_key='is_correct',
)

```

**Parameters:**
Parameter | Type | Default | Description
---|---|---|---
`score_key` | `str` | _(required)_ | Key in scores or metrics dict
`positive_from` | `'expected_output' \| 'assertions' \| 'labels'` | _(required)_ | Source for ground-truth binary labels
`positive_key` | `str \| None` | `None` | Key in assertions or labels dict
`score_from` | `'scores' \| 'metrics'` | `'scores'` | Source for numeric scores
`title` | `str` | `'Precision-Recall Curve'` | Title shown in reports
`n_thresholds` | `int` | `100` | Number of threshold points on the curve
**Returns:** [`PrecisionRecall`](https://ai.pydantic.dev/api/pydantic_evals/reporting/#pydantic_evals.reporting.PrecisionRecall "PrecisionRecall") + [`ScalarResult`](https://ai.pydantic.dev/api/pydantic_evals/reporting/#pydantic_evals.reporting.ScalarResult "ScalarResult") (AUC)
The AUC is computed at full resolution (using every unique score as a threshold) for accuracy, then the curve points are downsampled to `n_thresholds` for display. The AUC is returned both on the curve (for chart rendering) and as a separate `ScalarResult` for querying and sorting.
**Score Sources:**
  * `'scores'` — a numeric score from a case-level evaluator (looked up by `score_key`)
  * `'metrics'` — a custom metric set during task execution (looked up by `score_key`)


**Positive Sources:**
  * `'assertions'` — a boolean assertion from a case-level evaluator (looked up by `positive_key`)
  * `'labels'` — a label result cast to boolean (looked up by `positive_key`)
  * `'expected_output'` — the case's expected output cast to boolean


**Example:**
```
from dataclasses import dataclass
from typing import Any

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import (
    Evaluator,
    EvaluatorContext,
    PrecisionRecallEvaluator,
)


@dataclass
class ConfidenceEvaluator(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> dict[str, Any]:
        confidence = calculate_confidence(ctx.output)
        return {
            'confidence': confidence,      # numeric score
            'is_correct': ctx.output == ctx.expected_output,  # boolean assertion
        }


def calculate_confidence(output: str) -> float:
    return 0.85  # placeholder


dataset = Dataset(
    cases=[
        Case(inputs='test 1', expected_output='cat'),
        Case(inputs='test 2', expected_output='dog'),
    ],
    evaluators=[ConfidenceEvaluator()],
    report_evaluators=[
        PrecisionRecallEvaluator(
            score_from='scores',
            score_key='confidence',
            positive_from='assertions',
            positive_key='is_correct',
        ),
    ],
)

```

* * *
### ROCAUCEvaluator
Computes an ROC (Receiver Operating Characteristic) curve and AUC from numeric scores and binary ground-truth labels. The ROC curve plots the True Positive Rate against the False Positive Rate at various threshold values, with a dashed random-baseline diagonal for reference.
```
from pydantic_evals.evaluators import ROCAUCEvaluator

ROCAUCEvaluator(
    score_key='confidence',
    positive_from='assertions',
    positive_key='is_correct',
)

```

**Parameters:**
Parameter | Type | Default | Description
---|---|---|---
`score_key` | `str` | _(required)_ | Key in scores or metrics dict
`positive_from` | `'expected_output' \| 'assertions' \| 'labels'` | _(required)_ | Source for ground-truth binary labels
`positive_key` | `str \| None` | `None` | Key in assertions or labels dict
`score_from` | `'scores' \| 'metrics'` | `'scores'` | Source for numeric scores
`title` | `str` | `'ROC Curve'` | Title shown in reports
`n_thresholds` | `int` | `100` | Number of threshold points on the curve
**Returns:** [`LinePlot`](https://ai.pydantic.dev/api/pydantic_evals/reporting/#pydantic_evals.reporting.LinePlot "LinePlot") + [`ScalarResult`](https://ai.pydantic.dev/api/pydantic_evals/reporting/#pydantic_evals.reporting.ScalarResult "ScalarResult") (AUC)
The AUC is computed at full resolution. The chart includes a dashed "Random" baseline diagonal from (0, 0) to (1, 1) for visual comparison.
**Score and Positive Sources:** Same as [`PrecisionRecallEvaluator`](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#precisionrecallevaluator).
* * *
### KolmogorovSmirnovEvaluator
Computes a Kolmogorov-Smirnov plot and KS statistic from numeric scores and binary ground-truth labels. The KS plot shows the empirical CDFs (cumulative distribution functions) of the score distribution for positive and negative cases. The KS statistic is the maximum vertical distance between the two CDFs — higher values indicate better class separation.
```
from pydantic_evals.evaluators import KolmogorovSmirnovEvaluator

KolmogorovSmirnovEvaluator(
    score_key='confidence',
    positive_from='assertions',
    positive_key='is_correct',
)

```

**Parameters:**
Parameter | Type | Default | Description
---|---|---|---
`score_key` | `str` | _(required)_ | Key in scores or metrics dict
`positive_from` | `'expected_output' \| 'assertions' \| 'labels'` | _(required)_ | Source for ground-truth binary labels
`positive_key` | `str \| None` | `None` | Key in assertions or labels dict
`score_from` | `'scores' \| 'metrics'` | `'scores'` | Source for numeric scores
`title` | `str` | `'KS Plot'` | Title shown in reports
`n_thresholds` | `int` | `100` | Number of threshold points on the curve
**Returns:** [`LinePlot`](https://ai.pydantic.dev/api/pydantic_evals/reporting/#pydantic_evals.reporting.LinePlot "LinePlot") + [`ScalarResult`](https://ai.pydantic.dev/api/pydantic_evals/reporting/#pydantic_evals.reporting.ScalarResult "ScalarResult") (KS Statistic)
**Score and Positive Sources:** Same as [`PrecisionRecallEvaluator`](https://ai.pydantic.dev/evals/evaluators/report-evaluators/#precisionrecallevaluator).
* * *
## Custom Report Evaluators
Write custom report evaluators by inheriting from [`ReportEvaluator`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ReportEvaluator "ReportEvaluator



      dataclass
  ") and implementing the `evaluate` method:
```
from dataclasses import dataclass

from pydantic_evals.evaluators import ReportEvaluator, ReportEvaluatorContext
from pydantic_evals.reporting.analyses import ScalarResult


@dataclass
class AccuracyEvaluator(ReportEvaluator):
    """Computes overall accuracy as a scalar metric."""

    def evaluate(self, ctx: ReportEvaluatorContext) -> ScalarResult:
        cases = ctx.report.cases
        if not cases:
            return ScalarResult(title='Accuracy', value=0.0, unit='%')

        correct = sum(
            1 for case in cases
            if case.output == case.expected_output
        )
        accuracy = correct / len(cases) * 100
        return ScalarResult(title='Accuracy', value=accuracy, unit='%')

```

### ReportEvaluatorContext
The context passed to `evaluate()` contains:
  * `ctx.name` — the experiment name
  * `ctx.report` — the full [`EvaluationReport`](https://ai.pydantic.dev/api/pydantic_evals/reporting/#pydantic_evals.reporting.EvaluationReport "EvaluationReport



      dataclass
  ") with all case results
  * `ctx.experiment_metadata` — optional experiment-level metadata dict


Through `ctx.report.cases`, you can access each case's inputs, outputs, expected outputs, scores, labels, assertions, metrics, and attributes.
### Return Types
Report evaluators must return a `ReportAnalysis` or a `list[ReportAnalysis]`. The available analysis types are:
#### ScalarResult
A single numeric statistic:
```
from pydantic_evals.reporting.analyses import ScalarResult

ScalarResult(
    title='Accuracy',
    value=93.3,
    unit='%',
    description='Percentage of correctly classified cases.',
)

```

Field | Type | Description
---|---|---
`title` | `str` | Display name
`value` | `float \| int` | The numeric value
`unit` | `str \| None` | Optional unit label (e.g., `'%'`, `'ms'`)
`description` | `str \| None` | Optional longer description
* * *
#### TableResult
A generic table of data:
```
from pydantic_evals.reporting.analyses import TableResult

TableResult(
    title='Per-Class Metrics',
    columns=['Class', 'Precision', 'Recall', 'F1'],
    rows=[
        ['cat', 0.95, 0.90, 0.924],
        ['dog', 0.88, 0.92, 0.899],
    ],
    description='Precision, recall, and F1 per class.',
)

```

Field | Type | Description
---|---|---
`title` | `str` | Display name
`columns` | `list[str]` | Column headers
`rows` | `list[list[str \| int \| float \| bool \| None]]` | Row data
`description` | `str \| None` | Optional longer description
* * *
#### ConfusionMatrix
A confusion matrix (typically produced by `ConfusionMatrixEvaluator`, but can be constructed directly):
```
from pydantic_evals.reporting.analyses import ConfusionMatrix

ConfusionMatrix(
    title='Sentiment',
    class_labels=['positive', 'negative', 'neutral'],
    matrix=[
        [45, 3, 2],   # expected=positive
        [5, 40, 5],   # expected=negative
        [1, 2, 47],   # expected=neutral
    ],
)

```

Field | Type | Description
---|---|---
`title` | `str` | Display name
`class_labels` | `list[str]` | Ordered labels for both axes
`matrix` | `list[list[int]]` |  `matrix[expected][predicted]` = count
`description` | `str \| None` | Optional longer description
* * *
#### PrecisionRecall
Precision-recall curve data (typically produced by `PrecisionRecallEvaluator`):
Field | Type | Description
---|---|---
`title` | `str` | Display name
`curves` | `list[PrecisionRecallCurve]` | One or more curves
`description` | `str \| None` | Optional longer description
Each `PrecisionRecallCurve` contains a `name`, a list of `PrecisionRecallPoint`s (with `threshold`, `precision`, `recall`), and an optional `auc` value.
* * *
#### LinePlot
A generic XY line chart with labeled axes, supporting multiple curves. Use this for ROC curves, KS plots, calibration curves, or any custom line chart:
```
from pydantic_evals.reporting.analyses import LinePlot, LinePlotCurve, LinePlotPoint

LinePlot(
    title='ROC Curve',
    x_label='False Positive Rate',
    y_label='True Positive Rate',
    x_range=(0, 1),
    y_range=(0, 1),
    curves=[
        LinePlotCurve(
            name='Model (AUC: 0.95)',
            points=[LinePlotPoint(x=0.0, y=0.0), LinePlotPoint(x=0.1, y=0.8), LinePlotPoint(x=1.0, y=1.0)],
        ),
        LinePlotCurve(
            name='Random',
            points=[LinePlotPoint(x=0, y=0), LinePlotPoint(x=1, y=1)],
            style='dashed',
        ),
    ],
)

```

Field | Type | Description
---|---|---
`title` | `str` | Display name
`x_label` | `str` | Label for the x-axis
`y_label` | `str` | Label for the y-axis
`x_range` | `tuple[float, float] \| None` | Optional fixed range for x-axis
`y_range` | `tuple[float, float] \| None` | Optional fixed range for y-axis
`curves` | `list[LinePlotCurve]` | One or more curves to plot
`description` | `str \| None` | Optional longer description
Each `LinePlotCurve` contains a `name`, a list of `LinePlotPoint`s (with `x`, `y`), an optional `style` (`'solid'` or `'dashed'`), and an optional `step` interpolation mode (`'start'`, `'middle'`, or `'end'`) for step functions like empirical CDFs.
`LinePlot` is the recommended return type for custom curve-based evaluators — any evaluator that returns a `LinePlot` will be rendered as a line chart in the Logfire UI without requiring any frontend changes.
### Returning Multiple Analyses
A single report evaluator can return multiple analyses by returning a list:
```
from dataclasses import dataclass

from pydantic_evals.evaluators import ReportEvaluator, ReportEvaluatorContext
from pydantic_evals.reporting.analyses import ReportAnalysis, ScalarResult, TableResult


@dataclass
class ClassificationSummary(ReportEvaluator):
    """Produces both a scalar accuracy and a per-class metrics table."""

    def evaluate(self, ctx: ReportEvaluatorContext) -> list[ReportAnalysis]:
        cases = ctx.report.cases
        if not cases:
            return []

        labels = sorted({str(c.expected_output) for c in cases if c.expected_output})

        # Scalar: overall accuracy
        correct = sum(1 for c in cases if c.output == c.expected_output)
        accuracy = ScalarResult(
            title='Accuracy', value=correct / len(cases) * 100, unit='%'
        )

        # Table: per-class breakdown
        rows = []
        for label in labels:
            tp = sum(1 for c in cases if str(c.output) == label and str(c.expected_output) == label)
            fp = sum(1 for c in cases if str(c.output) == label and str(c.expected_output) != label)
            fn = sum(1 for c in cases if str(c.output) != label and str(c.expected_output) == label)
            p = tp / (tp + fp) if (tp + fp) > 0 else 0.0
            r = tp / (tp + fn) if (tp + fn) > 0 else 0.0
            f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0.0
            rows.append([label, round(p, 3), round(r, 3), round(f1, 3)])

        table = TableResult(
            title='Per-Class Metrics',
            columns=['Class', 'Precision', 'Recall', 'F1'],
            rows=rows,
        )

        return [accuracy, table]

```

### Async Report Evaluators
Report evaluators support async `evaluate` methods, handled automatically via `evaluate_async`:
```
from dataclasses import dataclass

from pydantic_evals.evaluators import ReportEvaluator, ReportEvaluatorContext
from pydantic_evals.reporting.analyses import ScalarResult


@dataclass
class AsyncAccuracy(ReportEvaluator):
    async def evaluate(self, ctx: ReportEvaluatorContext) -> ScalarResult:
        # Can use async I/O here (e.g., call an external API)
        cases = ctx.report.cases
        correct = sum(1 for c in cases if c.output == c.expected_output)
        return ScalarResult(
            title='Accuracy',
            value=correct / len(cases) * 100 if cases else 0.0,
            unit='%',
        )

```

## Serialization
Report evaluators are serialized to and from YAML/JSON dataset files using the same format as case-level evaluators. This means datasets with report evaluators can be fully round-tripped through file serialization.
**Example YAML dataset with report evaluators:**
```
# yaml-language-server: $schema=./test_cases_schema.json
name: classifier_eval
cases:
  - name: cat_test
    inputs: The cat meows
    expected_output: cat
  - name: dog_test
    inputs: The dog barks
    expected_output: dog
report_evaluators:
  - ConfusionMatrixEvaluator
  - PrecisionRecallEvaluator:
      score_key: confidence
      positive_from: assertions
      positive_key: is_correct

```

Built-in report evaluators (`ConfusionMatrixEvaluator`, `PrecisionRecallEvaluator`, `ROCAUCEvaluator`, `KolmogorovSmirnovEvaluator`) are recognized automatically. For custom report evaluators, pass them via `custom_report_evaluator_types`:
```
from pydantic_evals import Dataset

dataset = Dataset[str, str, None].from_file(
    'test_cases.yaml',
    custom_report_evaluator_types=[MyCustomReportEvaluator],
)

```

Similarly, when saving a dataset with custom report evaluators, pass them to `to_file` so the JSON schema includes them:
```
dataset.to_file(
    'test_cases.yaml',
    custom_report_evaluator_types=[MyCustomReportEvaluator],
)

```

## Viewing Analyses in Logfire
When [Logfire is configured](https://ai.pydantic.dev/evals/how-to/logfire-integration/), analyses are automatically attached to the experiment span as the `logfire.experiment.analyses` attribute. The Logfire UI renders them as interactive visualizations:
  * **Confusion matrices** are displayed as heatmaps
  * **Precision-recall curves** are rendered as line charts with AUC in the legend
  * **Line plots** (ROC curves, KS plots, etc.) are rendered as line charts with configurable axes
  * **Scalar results** are shown as labeled values
  * **Tables** are rendered as formatted data tables


When comparing multiple experiments in the Logfire Evals view, analyses of the same type are displayed side by side for easy comparison.
## Complete Example
A full example combining case-level evaluators with report evaluators:
```
from dataclasses import dataclass
from typing import Any

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import (
    ConfusionMatrixEvaluator,
    Evaluator,
    EvaluatorContext,
    KolmogorovSmirnovEvaluator,
    PrecisionRecallEvaluator,
    ReportEvaluator,
    ReportEvaluatorContext,
    ROCAUCEvaluator,
)
from pydantic_evals.reporting.analyses import ScalarResult


def my_classifier(text: str) -> str:
    text = text.lower()
    if 'cat' in text or 'meow' in text:
        return 'cat'
    elif 'dog' in text or 'bark' in text:
        return 'dog'
    elif 'bird' in text or 'chirp' in text:
        return 'bird'
    return 'unknown'


# Case-level evaluator: runs per case
@dataclass
class ConfidenceEvaluator(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> dict[str, Any]:
        confidence = compute_confidence(ctx.output, ctx.inputs)
        is_correct = ctx.output == ctx.expected_output
        return {
            'confidence': confidence,
            'is_correct': is_correct,
        }


def compute_confidence(output: str, inputs: str) -> float:
    return 0.85  # placeholder


# Report-level evaluator: runs once over the full report
@dataclass
class AccuracyEvaluator(ReportEvaluator):
    def evaluate(self, ctx: ReportEvaluatorContext) -> ScalarResult:
        cases = ctx.report.cases
        correct = sum(1 for c in cases if c.output == c.expected_output)
        return ScalarResult(
            title='Accuracy',
            value=correct / len(cases) * 100 if cases else 0.0,
            unit='%',
        )


dataset = Dataset(
    cases=[
        Case(inputs='The cat meows', expected_output='cat'),
        Case(inputs='The dog barks', expected_output='dog'),
        Case(inputs='A bird chirps', expected_output='bird'),
    ],
    evaluators=[ConfidenceEvaluator()],
    report_evaluators=[
        ConfusionMatrixEvaluator(
            predicted_from='output',
            expected_from='expected_output',
            title='Animal Classification',
        ),
        PrecisionRecallEvaluator(
            score_from='scores',
            score_key='confidence',
            positive_from='assertions',
            positive_key='is_correct',
        ),
        ROCAUCEvaluator(
            score_from='scores',
            score_key='confidence',
            positive_from='assertions',
            positive_key='is_correct',
        ),
        KolmogorovSmirnovEvaluator(
            score_from='scores',
            score_key='confidence',
            positive_from='assertions',
            positive_key='is_correct',
        ),
        AccuracyEvaluator(),
    ],
)

report = dataset.evaluate_sync(my_classifier)

# Access analyses programmatically
for analysis in report.analyses:
    print(f'{analysis.type}: {analysis.title}')
    #> confusion_matrix: Animal Classification
    #> precision_recall: Precision-Recall Curve
    #> scalar: Precision-Recall Curve AUC
    #> line_plot: ROC Curve
    #> scalar: ROC Curve AUC
    #> line_plot: KS Plot
    #> scalar: KS Statistic
    #> scalar: Accuracy

```

## Next Steps
  * **[Built-in Evaluators](https://ai.pydantic.dev/evals/evaluators/built-in/)** — Case-level evaluator reference
  * **[Custom Evaluators](https://ai.pydantic.dev/evals/evaluators/custom/)** — Writing case-level evaluators
  * **[Logfire Integration](https://ai.pydantic.dev/evals/how-to/logfire-integration/)** — Viewing analyses in the Logfire UI


© Pydantic Services Inc. 2024 to present
