[ Skip to content ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evalsevaluators)
**Join us at the inaugural PyAI Conf in San Francisco on March 10th![Learn More](https://pyai.events/?utm_source=pydantic-ai) **
[ ![logo](https://ai.pydantic.dev/img/logo-white.svg) ](https://ai.pydantic.dev/ "Pydantic AI")
Pydantic AI
pydantic_evals.evaluators
Type to start searching
[ pydantic/pydantic-ai
  * v1.63.0
  * 15.1k
  * 1.7k

](https://github.com/pydantic/pydantic-ai "Go to repository")
[ ![logo](https://ai.pydantic.dev/img/logo-white.svg) ](https://ai.pydantic.dev/ "Pydantic AI") Pydantic AI
[ pydantic/pydantic-ai
  * v1.63.0
  * 15.1k
  * 1.7k

](https://github.com/pydantic/pydantic-ai "Go to repository")
  * [ Pydantic AI  ](https://ai.pydantic.dev/)
  * [ Installation  ](https://ai.pydantic.dev/install/)
  * [ Getting Help  ](https://ai.pydantic.dev/help/)
  * [ Troubleshooting  ](https://ai.pydantic.dev/troubleshooting/)
  * [ Pydantic AI Gateway  ](https://ai.pydantic.dev/gateway/)
  * Documentation
    * Core Concepts
      * [ Agents  ](https://ai.pydantic.dev/agent/)
      * [ Dependencies  ](https://ai.pydantic.dev/dependencies/)
      * [ Function Tools  ](https://ai.pydantic.dev/tools/)
      * [ Output  ](https://ai.pydantic.dev/output/)
      * [ Messages and chat history  ](https://ai.pydantic.dev/message-history/)
      * [ Direct Model Requests  ](https://ai.pydantic.dev/direct/)
    * Models & Providers
      * [ Overview  ](https://ai.pydantic.dev/models/overview/)
      * [ OpenAI  ](https://ai.pydantic.dev/models/openai/)
      * [ Anthropic  ](https://ai.pydantic.dev/models/anthropic/)
      * [ Google  ](https://ai.pydantic.dev/models/google/)
      * [ xAI  ](https://ai.pydantic.dev/models/xai/)
      * [ Bedrock  ](https://ai.pydantic.dev/models/bedrock/)
      * [ Cerebras  ](https://ai.pydantic.dev/models/cerebras/)
      * [ Cohere  ](https://ai.pydantic.dev/models/cohere/)
      * [ Groq  ](https://ai.pydantic.dev/models/groq/)
      * [ Hugging Face  ](https://ai.pydantic.dev/models/huggingface/)
      * [ Mistral  ](https://ai.pydantic.dev/models/mistral/)
      * [ OpenRouter  ](https://ai.pydantic.dev/models/openrouter/)
      * [ Outlines  ](https://ai.pydantic.dev/models/outlines/)
    * Tools & Toolsets
      * [ Function Tools  ](https://ai.pydantic.dev/tools/)
      * [ Advanced Tool Features  ](https://ai.pydantic.dev/tools-advanced/)
      * [ Toolsets  ](https://ai.pydantic.dev/toolsets/)
      * [ Deferred Tools  ](https://ai.pydantic.dev/deferred-tools/)
      * [ Built-in Tools  ](https://ai.pydantic.dev/builtin-tools/)
      * [ Common Tools  ](https://ai.pydantic.dev/common-tools/)
      * [ Third-Party Tools  ](https://ai.pydantic.dev/third-party-tools/)
    * Advanced Features
      * [ Image, Audio, Video & Document Input  ](https://ai.pydantic.dev/input/)
      * [ Thinking  ](https://ai.pydantic.dev/thinking/)
      * [ HTTP Request Retries  ](https://ai.pydantic.dev/retries/)
    * MCP
      * [ Overview  ](https://ai.pydantic.dev/mcp/overview/)
      * [ Client  ](https://ai.pydantic.dev/mcp/client/)
      * [ FastMCP Client  ](https://ai.pydantic.dev/mcp/fastmcp-client/)
      * [ Server  ](https://ai.pydantic.dev/mcp/server/)
    * [ Multi-Agent Patterns  ](https://ai.pydantic.dev/multi-agent-applications/)
    * [ Web Chat UI  ](https://ai.pydantic.dev/web/)
    * [ Embeddings  ](https://ai.pydantic.dev/embeddings/)
    * [ Testing  ](https://ai.pydantic.dev/testing/)
  * Pydantic Evals
    * [ Overview  ](https://ai.pydantic.dev/evals/)
    * Getting Started
      * [ Quick Start  ](https://ai.pydantic.dev/evals/quick-start/)
      * [ Core Concepts  ](https://ai.pydantic.dev/evals/core-concepts/)
    * Evaluators
      * [ Overview  ](https://ai.pydantic.dev/evals/evaluators/overview/)
      * [ Built-in Evaluators  ](https://ai.pydantic.dev/evals/evaluators/built-in/)
      * [ LLM Judge  ](https://ai.pydantic.dev/evals/evaluators/llm-judge/)
      * [ Custom Evaluators  ](https://ai.pydantic.dev/evals/evaluators/custom/)
      * [ Report Evaluators  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/)
      * [ Span-Based  ](https://ai.pydantic.dev/evals/evaluators/span-based/)
    * How-To Guides
      * [ Logfire Integration  ](https://ai.pydantic.dev/evals/how-to/logfire-integration/)
      * [ Dataset Management  ](https://ai.pydantic.dev/evals/how-to/dataset-management/)
      * [ Dataset Serialization  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/)
      * [ Concurrency & Performance  ](https://ai.pydantic.dev/evals/how-to/concurrency/)
      * [ Multi-Run Evaluation  ](https://ai.pydantic.dev/evals/how-to/multi-run/)
      * [ Retry Strategies  ](https://ai.pydantic.dev/evals/how-to/retry-strategies/)
      * [ Metrics & Attributes  ](https://ai.pydantic.dev/evals/how-to/metrics-attributes/)
    * Examples
      * [ Simple Validation  ](https://ai.pydantic.dev/evals/examples/simple-validation/)
  * Pydantic Graph
    * [ Overview  ](https://ai.pydantic.dev/graph/)
    * [ Beta API  ](https://ai.pydantic.dev/graph/beta/)
      * [ Steps  ](https://ai.pydantic.dev/graph/beta/steps/)
      * [ Joins & Reducers  ](https://ai.pydantic.dev/graph/beta/joins/)
      * [ Decisions  ](https://ai.pydantic.dev/graph/beta/decisions/)
      * [ Parallel Execution  ](https://ai.pydantic.dev/graph/beta/parallel/)
  * Integrations
    * [ Debugging & Monitoring with Pydantic Logfire  ](https://ai.pydantic.dev/logfire/)
    * Durable Execution
      * [ Overview  ](https://ai.pydantic.dev/durable_execution/overview/)
      * [ Temporal  ](https://ai.pydantic.dev/durable_execution/temporal/)
      * [ DBOS  ](https://ai.pydantic.dev/durable_execution/dbos/)
      * [ Prefect  ](https://ai.pydantic.dev/durable_execution/prefect/)
    * UI Event Streams
      * [ Overview  ](https://ai.pydantic.dev/ui/overview/)
      * [ AG-UI  ](https://ai.pydantic.dev/ui/ag-ui/)
      * [ Vercel AI  ](https://ai.pydantic.dev/ui/vercel-ai/)
    * [ Agent2Agent (A2A)  ](https://ai.pydantic.dev/a2a/)
  * Related Packages
    * [ clai  ](https://ai.pydantic.dev/cli/)
  * Examples
    * [ Setup  ](https://ai.pydantic.dev/examples/setup/)
    * Getting Started
      * [ Pydantic Model  ](https://ai.pydantic.dev/examples/pydantic-model/)
      * [ Weather agent  ](https://ai.pydantic.dev/examples/weather-agent/)
    * Conversational Agents
      * [ Chat App with FastAPI  ](https://ai.pydantic.dev/examples/chat-app/)
      * [ Bank support  ](https://ai.pydantic.dev/examples/bank-support/)
    * Data & Analytics
      * [ SQL Generation  ](https://ai.pydantic.dev/examples/sql-gen/)
      * [ Data Analyst  ](https://ai.pydantic.dev/examples/data-analyst/)
      * [ RAG  ](https://ai.pydantic.dev/examples/rag/)
    * Streaming
      * [ Stream markdown  ](https://ai.pydantic.dev/examples/stream-markdown/)
      * [ Stream whales  ](https://ai.pydantic.dev/examples/stream-whales/)
    * Complex Workflows
      * [ Flight booking  ](https://ai.pydantic.dev/examples/flight-booking/)
      * [ Question Graph  ](https://ai.pydantic.dev/examples/question-graph/)
    * Business Applications
      * [ Slack Lead Qualifier with Modal  ](https://ai.pydantic.dev/examples/slack-lead-qualifier/)
    * UI Examples
      * [ Agent User Interaction (AG-UI)  ](https://ai.pydantic.dev/examples/ag-ui/)
  * API Reference
    * pydantic_ai
      * [ pydantic_ai.ag_ui  ](https://ai.pydantic.dev/api/ag_ui/)
      * [ pydantic_ai.agent  ](https://ai.pydantic.dev/api/agent/)
      * [ pydantic_ai.builtin_tools  ](https://ai.pydantic.dev/api/builtin_tools/)
      * [ pydantic_ai.common_tools  ](https://ai.pydantic.dev/api/common_tools/)
      * [ pydantic_ai — Concurrency  ](https://ai.pydantic.dev/api/concurrency/)
      * [ pydantic_ai.direct  ](https://ai.pydantic.dev/api/direct/)
      * [ pydantic_ai.durable_exec  ](https://ai.pydantic.dev/api/durable_exec/)
      * [ pydantic_ai.embeddings  ](https://ai.pydantic.dev/api/embeddings/)
      * [ pydantic_ai.exceptions  ](https://ai.pydantic.dev/api/exceptions/)
      * [ pydantic_ai.ext  ](https://ai.pydantic.dev/api/ext/)
      * [ pydantic_ai.format_prompt  ](https://ai.pydantic.dev/api/format_prompt/)
      * [ pydantic_ai.mcp  ](https://ai.pydantic.dev/api/mcp/)
      * [ pydantic_ai.messages  ](https://ai.pydantic.dev/api/messages/)
      * [ pydantic_ai.models.anthropic  ](https://ai.pydantic.dev/api/models/anthropic/)
      * [ pydantic_ai.models  ](https://ai.pydantic.dev/api/models/base/)
      * [ pydantic_ai.models.bedrock  ](https://ai.pydantic.dev/api/models/bedrock/)
      * [ pydantic_ai.models.cerebras  ](https://ai.pydantic.dev/api/models/cerebras/)
      * [ pydantic_ai.models.cohere  ](https://ai.pydantic.dev/api/models/cohere/)
      * [ pydantic_ai.models.fallback  ](https://ai.pydantic.dev/api/models/fallback/)
      * [ pydantic_ai.models.function  ](https://ai.pydantic.dev/api/models/function/)
      * [ pydantic_ai.models.google  ](https://ai.pydantic.dev/api/models/google/)
      * [ pydantic_ai.models.xai  ](https://ai.pydantic.dev/api/models/xai/)
      * [ pydantic_ai.models.groq  ](https://ai.pydantic.dev/api/models/groq/)
      * [ pydantic_ai.models.huggingface  ](https://ai.pydantic.dev/api/models/huggingface/)
      * [ pydantic_ai.models.instrumented  ](https://ai.pydantic.dev/api/models/instrumented/)
      * [ pydantic_ai.models.mcp_sampling  ](https://ai.pydantic.dev/api/models/mcp-sampling/)
      * [ pydantic_ai.models.mistral  ](https://ai.pydantic.dev/api/models/mistral/)
      * [ pydantic_ai.models.openai  ](https://ai.pydantic.dev/api/models/openai/)
      * [ pydantic_ai.models.openrouter  ](https://ai.pydantic.dev/api/models/openrouter/)
      * [ pydantic_ai.models.outlines  ](https://ai.pydantic.dev/api/models/outlines/)
      * [ pydantic_ai.models.test  ](https://ai.pydantic.dev/api/models/test/)
      * [ pydantic_ai.models.wrapper  ](https://ai.pydantic.dev/api/models/wrapper/)
      * [ pydantic_ai.output  ](https://ai.pydantic.dev/api/output/)
      * [ pydantic_ai.profiles  ](https://ai.pydantic.dev/api/profiles/)
      * [ pydantic_ai.providers  ](https://ai.pydantic.dev/api/providers/)
      * [ pydantic_ai.result  ](https://ai.pydantic.dev/api/result/)
      * [ pydantic_ai.retries  ](https://ai.pydantic.dev/api/retries/)
      * [ pydantic_ai.run  ](https://ai.pydantic.dev/api/run/)
      * [ pydantic_ai.settings  ](https://ai.pydantic.dev/api/settings/)
      * [ pydantic_ai.tools  ](https://ai.pydantic.dev/api/tools/)
      * [ pydantic_ai.toolsets  ](https://ai.pydantic.dev/api/toolsets/)
      * [ pydantic_ai.ui.ag_ui  ](https://ai.pydantic.dev/api/ui/ag_ui/)
      * [ pydantic_ai.ui  ](https://ai.pydantic.dev/api/ui/base/)
      * [ pydantic_ai.ui.vercel_ai  ](https://ai.pydantic.dev/api/ui/vercel_ai/)
      * [ pydantic_ai.usage  ](https://ai.pydantic.dev/api/usage/)
    * pydantic_evals
      * [ pydantic_evals.dataset  ](https://ai.pydantic.dev/api/pydantic_evals/dataset/)
      * pydantic_evals.evaluators  [ pydantic_evals.evaluators  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/)
        * [ evaluators  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators)
        * [ Contains  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Contains)
        * [ Equals  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Equals)
        * [ EqualsExpected  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EqualsExpected)
        * [ HasMatchingSpan  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.HasMatchingSpan)
        * [ IsInstance  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.IsInstance)
        * [ LLMJudge  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.LLMJudge)
        * [ MaxDuration  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.MaxDuration)
        * [ OutputConfig  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.OutputConfig)
        * [ EvaluatorContext  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext)
          * [ name  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext.name)
          * [ inputs  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext.inputs)
          * [ metadata  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext.metadata)
          * [ expected_output  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext.expected_output)
          * [ output  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext.output)
          * [ duration  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext.duration)
          * [ attributes  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext.attributes)
          * [ metrics  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext.metrics)
          * [ span_tree  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext.span_tree)
        * [ EvaluationReason  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluationReason)
        * [ EvaluationResult  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluationResult)
          * [ downcast  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluationResult.downcast)
        * [ Evaluator  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator)
          * [ name  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator.name)
          * [ get_default_evaluation_name  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator.get_default_evaluation_name)
          * [ evaluate  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator.evaluate)
          * [ evaluate_sync  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator.evaluate_sync)
          * [ evaluate_async  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator.evaluate_async)
        * [ EvaluatorFailure  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorFailure)
        * [ EvaluatorOutput  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorOutput)
        * [ EvaluatorSpec  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorSpec)
          * [ name  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorSpec.name)
          * [ arguments  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorSpec.arguments)
          * [ args  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorSpec.args)
          * [ kwargs  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorSpec.kwargs)
          * [ deserialize  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorSpec.deserialize)
          * [ serialize  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorSpec.serialize)
        * [ ConfusionMatrixEvaluator  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ConfusionMatrixEvaluator)
        * [ KolmogorovSmirnovEvaluator  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.KolmogorovSmirnovEvaluator)
        * [ PrecisionRecallEvaluator  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.PrecisionRecallEvaluator)
        * [ ROCAUCEvaluator  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ROCAUCEvaluator)
        * [ ReportEvaluator  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ReportEvaluator)
          * [ evaluate  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ReportEvaluator.evaluate)
          * [ evaluate_async  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ReportEvaluator.evaluate_async)
        * [ ReportEvaluatorContext  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ReportEvaluatorContext)
          * [ name  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ReportEvaluatorContext.name)
          * [ report  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ReportEvaluatorContext.report)
          * [ experiment_metadata  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ReportEvaluatorContext.experiment_metadata)
        * [ llm_as_a_judge  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.llm_as_a_judge)
        * [ GradingOutput  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.llm_as_a_judge.GradingOutput)
        * [ judge_output  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.llm_as_a_judge.judge_output)
        * [ judge_input_output  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.llm_as_a_judge.judge_input_output)
        * [ judge_input_output_expected  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.llm_as_a_judge.judge_input_output_expected)
        * [ judge_output_expected  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.llm_as_a_judge.judge_output_expected)
        * [ set_default_judge_model  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.llm_as_a_judge.set_default_judge_model)
      * [ pydantic_evals.reporting  ](https://ai.pydantic.dev/api/pydantic_evals/reporting/)
      * [ pydantic_evals.otel  ](https://ai.pydantic.dev/api/pydantic_evals/otel/)
      * [ pydantic_evals.generation  ](https://ai.pydantic.dev/api/pydantic_evals/generation/)
    * pydantic_graph
      * [ pydantic_graph  ](https://ai.pydantic.dev/api/pydantic_graph/graph/)
      * [ pydantic_graph.nodes  ](https://ai.pydantic.dev/api/pydantic_graph/nodes/)
      * [ pydantic_graph.persistence  ](https://ai.pydantic.dev/api/pydantic_graph/persistence/)
      * [ pydantic_graph.mermaid  ](https://ai.pydantic.dev/api/pydantic_graph/mermaid/)
      * [ pydantic_graph.exceptions  ](https://ai.pydantic.dev/api/pydantic_graph/exceptions/)
      * Beta API
        * [ pydantic_graph.beta  ](https://ai.pydantic.dev/api/pydantic_graph/beta/)
        * [ pydantic_graph.beta.graph  ](https://ai.pydantic.dev/api/pydantic_graph/beta_graph/)
        * [ pydantic_graph.beta.graph_builder  ](https://ai.pydantic.dev/api/pydantic_graph/beta_graph_builder/)
        * [ pydantic_graph.beta.step  ](https://ai.pydantic.dev/api/pydantic_graph/beta_step/)
        * [ pydantic_graph.beta.join  ](https://ai.pydantic.dev/api/pydantic_graph/beta_join/)
        * [ pydantic_graph.beta.decision  ](https://ai.pydantic.dev/api/pydantic_graph/beta_decision/)
        * [ pydantic_graph.beta.node  ](https://ai.pydantic.dev/api/pydantic_graph/beta_node/)
    * fasta2a
      * [ fasta2a  ](https://ai.pydantic.dev/api/fasta2a/)
  * Project
    * [ Contributing  ](https://ai.pydantic.dev/contributing/)
    * [ Upgrade Guide  ](https://ai.pydantic.dev/changelog/)
    * [ Version policy  ](https://ai.pydantic.dev/version-policy/)


  * [ evaluators  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators)
  * [ Contains  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Contains)
  * [ Equals  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Equals)
  * [ EqualsExpected  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EqualsExpected)
  * [ HasMatchingSpan  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.HasMatchingSpan)
  * [ IsInstance  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.IsInstance)
  * [ LLMJudge  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.LLMJudge)
  * [ MaxDuration  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.MaxDuration)
  * [ OutputConfig  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.OutputConfig)
  * [ EvaluatorContext  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext)
    * [ name  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext.name)
    * [ inputs  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext.inputs)
    * [ metadata  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext.metadata)
    * [ expected_output  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext.expected_output)
    * [ output  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext.output)
    * [ duration  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext.duration)
    * [ attributes  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext.attributes)
    * [ metrics  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext.metrics)
    * [ span_tree  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext.span_tree)
  * [ EvaluationReason  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluationReason)
  * [ EvaluationResult  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluationResult)
    * [ downcast  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluationResult.downcast)
  * [ Evaluator  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator)
    * [ name  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator.name)
    * [ get_default_evaluation_name  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator.get_default_evaluation_name)
    * [ evaluate  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator.evaluate)
    * [ evaluate_sync  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator.evaluate_sync)
    * [ evaluate_async  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator.evaluate_async)
  * [ EvaluatorFailure  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorFailure)
  * [ EvaluatorOutput  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorOutput)
  * [ EvaluatorSpec  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorSpec)
    * [ name  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorSpec.name)
    * [ arguments  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorSpec.arguments)
    * [ args  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorSpec.args)
    * [ kwargs  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorSpec.kwargs)
    * [ deserialize  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorSpec.deserialize)
    * [ serialize  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorSpec.serialize)
  * [ ConfusionMatrixEvaluator  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ConfusionMatrixEvaluator)
  * [ KolmogorovSmirnovEvaluator  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.KolmogorovSmirnovEvaluator)
  * [ PrecisionRecallEvaluator  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.PrecisionRecallEvaluator)
  * [ ROCAUCEvaluator  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ROCAUCEvaluator)
  * [ ReportEvaluator  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ReportEvaluator)
    * [ evaluate  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ReportEvaluator.evaluate)
    * [ evaluate_async  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ReportEvaluator.evaluate_async)
  * [ ReportEvaluatorContext  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ReportEvaluatorContext)
    * [ name  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ReportEvaluatorContext.name)
    * [ report  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ReportEvaluatorContext.report)
    * [ experiment_metadata  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ReportEvaluatorContext.experiment_metadata)
  * [ llm_as_a_judge  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.llm_as_a_judge)
  * [ GradingOutput  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.llm_as_a_judge.GradingOutput)
  * [ judge_output  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.llm_as_a_judge.judge_output)
  * [ judge_input_output  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.llm_as_a_judge.judge_input_output)
  * [ judge_input_output_expected  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.llm_as_a_judge.judge_input_output_expected)
  * [ judge_output_expected  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.llm_as_a_judge.judge_output_expected)
  * [ set_default_judge_model  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.llm_as_a_judge.set_default_judge_model)


  1. [ Pydantic AI  ](https://ai.pydantic.dev/)
  2. [ API Reference  ](https://ai.pydantic.dev/api/ag_ui/)
  3. [ pydantic_evals  ](https://ai.pydantic.dev/api/pydantic_evals/dataset/)


# `pydantic_evals.evaluators`
###  Contains `dataclass`
Bases: `Evaluator[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator "Evaluator



      dataclass
   \(pydantic_evals.evaluators.evaluator.Evaluator\)")[object[](https://docs.python.org/3/library/functions.html#object), object[](https://docs.python.org/3/library/functions.html#object), object[](https://docs.python.org/3/library/functions.html#object)]`
Check if the output contains the expected output.
For strings, checks if expected_output is a substring of output. For lists/tuples, checks if expected_output is in output. For dicts, checks if all key-value pairs in expected_output are in output. For model-like types (BaseModel, dataclasses), converts to a dict and checks key-value pairs.
Note: case_sensitive only applies when both the value and output are strings.
Source code in `pydantic_evals/pydantic_evals/evaluators/common.py`
```
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
```
| ```
@dataclass(repr=False)
class Contains(Evaluator[object, object, object]):
    """Check if the output contains the expected output.

    For strings, checks if expected_output is a substring of output.
    For lists/tuples, checks if expected_output is in output.
    For dicts, checks if all key-value pairs in expected_output are in output.
    For model-like types (BaseModel, dataclasses), converts to a dict and checks key-value pairs.

    Note: case_sensitive only applies when both the value and output are strings.
    """

    value: Any
    case_sensitive: bool = True
    as_strings: bool = False
    evaluation_name: str | None = field(default=None)

    def evaluate(
        self,
        ctx: EvaluatorContext[object, object, object],
    ) -> EvaluationReason:
        # Convert objects to strings if requested
        failure_reason: str | None = None
        as_strings = self.as_strings or (isinstance(self.value, str) and isinstance(ctx.output, str))
        if as_strings:
            output_str = str(ctx.output)
            expected_str = str(self.value)

            if not self.case_sensitive:
                output_str = output_str.lower()
                expected_str = expected_str.lower()

            failure_reason: str | None = None
            if expected_str not in output_str:
                output_trunc = _truncated_repr(output_str, max_length=100)
                expected_trunc = _truncated_repr(expected_str, max_length=100)
                failure_reason = f'Output string {output_trunc} does not contain expected string {expected_trunc}'
            return EvaluationReason(value=failure_reason is None, reason=failure_reason)

        try:
            # Handle different collection types
            output_type = type(ctx.output)
            output_is_model_like = is_model_like(output_type)
            if isinstance(ctx.output, dict) or output_is_model_like:
                if output_is_model_like:
                    adapter: TypeAdapter[Any] = TypeAdapter(output_type)
                    output_dict = adapter.dump_python(ctx.output)  # pyright: ignore[reportUnknownMemberType]
                else:
                    # Cast to Any to avoid type checking issues
                    output_dict = cast(dict[Any, Any], ctx.output)  # pyright: ignore[reportUnknownMemberType]

                if isinstance(self.value, dict):
                    # Cast to Any to avoid type checking issues
                    expected_dict = cast(dict[Any, Any], self.value)  # pyright: ignore[reportUnknownMemberType]
                    for k in expected_dict:
                        if k not in output_dict:
                            k_trunc = _truncated_repr(k, max_length=30)
                            failure_reason = f'Output does not contain expected key {k_trunc}'
                            break
                        elif output_dict[k] != expected_dict[k]:
                            k_trunc = _truncated_repr(k, max_length=30)
                            output_v_trunc = _truncated_repr(output_dict[k], max_length=100)
                            expected_v_trunc = _truncated_repr(expected_dict[k], max_length=100)
                            failure_reason = (
                                f'Output has different value for key {k_trunc}: {output_v_trunc} != {expected_v_trunc}'
                            )
                            break
                else:
                    if self.value not in output_dict:
                        output_trunc = _truncated_repr(output_dict, max_length=200)
                        failure_reason = f'Output {output_trunc} does not contain provided value as a key'
            elif self.value not in ctx.output:  # pyright: ignore[reportOperatorIssue]  # will be handled by except block
                output_trunc = _truncated_repr(ctx.output, max_length=200)
                failure_reason = f'Output {output_trunc} does not contain provided value'
        except (TypeError, ValueError) as e:
            failure_reason = f'Containment check failed: {e}'

        return EvaluationReason(value=failure_reason is None, reason=failure_reason)

```

---|---
###  Equals `dataclass`
Bases: `Evaluator[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator "Evaluator



      dataclass
   \(pydantic_evals.evaluators.evaluator.Evaluator\)")[object[](https://docs.python.org/3/library/functions.html#object), object[](https://docs.python.org/3/library/functions.html#object), object[](https://docs.python.org/3/library/functions.html#object)]`
Check if the output exactly equals the provided value.
Source code in `pydantic_evals/pydantic_evals/evaluators/common.py`
```
30
31
32
33
34
35
36
37
38
```
| ```
@dataclass(repr=False)
class Equals(Evaluator[object, object, object]):
    """Check if the output exactly equals the provided value."""

    value: Any
    evaluation_name: str | None = field(default=None)

    def evaluate(self, ctx: EvaluatorContext[object, object, object]) -> bool:
        return ctx.output == self.value

```

---|---
###  EqualsExpected `dataclass`
Bases: `Evaluator[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator "Evaluator



      dataclass
   \(pydantic_evals.evaluators.evaluator.Evaluator\)")[object[](https://docs.python.org/3/library/functions.html#object), object[](https://docs.python.org/3/library/functions.html#object), object[](https://docs.python.org/3/library/functions.html#object)]`
Check if the output exactly equals the expected output.
Source code in `pydantic_evals/pydantic_evals/evaluators/common.py`
```
41
42
43
44
45
46
47
48
49
50
```
| ```
@dataclass(repr=False)
class EqualsExpected(Evaluator[object, object, object]):
    """Check if the output exactly equals the expected output."""

    evaluation_name: str | None = field(default=None)

    def evaluate(self, ctx: EvaluatorContext[object, object, object]) -> bool | dict[str, bool]:
        if ctx.expected_output is None:
            return {}  # Only compare if expected output is provided
        return ctx.output == ctx.expected_output

```

---|---
###  HasMatchingSpan `dataclass`
Bases: `Evaluator[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator "Evaluator



      dataclass
   \(pydantic_evals.evaluators.evaluator.Evaluator\)")[object[](https://docs.python.org/3/library/functions.html#object), object[](https://docs.python.org/3/library/functions.html#object), object[](https://docs.python.org/3/library/functions.html#object)]`
Check if the span tree contains a span that matches the specified query.
Source code in `pydantic_evals/pydantic_evals/evaluators/common.py`
```
269
270
271
272
273
274
275
276
277
278
279
280
```
| ```
@dataclass(repr=False)
class HasMatchingSpan(Evaluator[object, object, object]):
    """Check if the span tree contains a span that matches the specified query."""

    query: SpanQuery
    evaluation_name: str | None = field(default=None)

    def evaluate(
        self,
        ctx: EvaluatorContext[object, object, object],
    ) -> bool:
        return ctx.span_tree.any(self.query)

```

---|---
###  IsInstance `dataclass`
Bases: `Evaluator[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator "Evaluator



      dataclass
   \(pydantic_evals.evaluators.evaluator.Evaluator\)")[object[](https://docs.python.org/3/library/functions.html#object), object[](https://docs.python.org/3/library/functions.html#object), object[](https://docs.python.org/3/library/functions.html#object)]`
Check if the output is an instance of a type with the given name.
Source code in `pydantic_evals/pydantic_evals/evaluators/common.py`
```
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
```
| ```
@dataclass(repr=False)
class IsInstance(Evaluator[object, object, object]):
    """Check if the output is an instance of a type with the given name."""

    type_name: str
    evaluation_name: str | None = field(default=None)

    def evaluate(self, ctx: EvaluatorContext[object, object, object]) -> EvaluationReason:
        output = ctx.output
        for cls in type(output).__mro__:
            if cls.__name__ == self.type_name or cls.__qualname__ == self.type_name:
                return EvaluationReason(value=True)

        reason = f'output is of type {type(output).__name__}'
        if type(output).__qualname__ != type(output).__name__:
            reason += f' (qualname: {type(output).__qualname__})'
        return EvaluationReason(value=False, reason=reason)

```

---|---
###  LLMJudge `dataclass`
Bases: `Evaluator[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator "Evaluator



      dataclass
   \(pydantic_evals.evaluators.evaluator.Evaluator\)")[object[](https://docs.python.org/3/library/functions.html#object), object[](https://docs.python.org/3/library/functions.html#object), object[](https://docs.python.org/3/library/functions.html#object)]`
Judge whether the output of a language model meets the criteria of a provided rubric.
If you do not specify a model, it uses the default model for judging. This starts as 'openai:gpt-5.2', but can be overridden by calling [`set_default_judge_model`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.llm_as_a_judge.set_default_judge_model "set_default_judge_model").
Source code in `pydantic_evals/pydantic_evals/evaluators/common.py`
```
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
```
| ```
@dataclass(repr=False)
class LLMJudge(Evaluator[object, object, object]):
    """Judge whether the output of a language model meets the criteria of a provided rubric.

    If you do not specify a model, it uses the default model for judging. This starts as 'openai:gpt-5.2', but can be
    overridden by calling [`set_default_judge_model`][pydantic_evals.evaluators.llm_as_a_judge.set_default_judge_model].
    """

    rubric: str
    model: models.Model | models.KnownModelName | str | None = None
    include_input: bool = False
    include_expected_output: bool = False
    model_settings: ModelSettings | None = None
    score: OutputConfig | Literal[False] = False
    assertion: OutputConfig | Literal[False] = field(default_factory=lambda: OutputConfig(include_reason=True))

    async def evaluate(
        self,
        ctx: EvaluatorContext[object, object, object],
    ) -> EvaluatorOutput:
        if self.include_input:
            if self.include_expected_output:
                from .llm_as_a_judge import judge_input_output_expected

                grading_output = await judge_input_output_expected(
                    ctx.inputs, ctx.output, ctx.expected_output, self.rubric, self.model, self.model_settings
                )
            else:
                from .llm_as_a_judge import judge_input_output

                grading_output = await judge_input_output(
                    ctx.inputs, ctx.output, self.rubric, self.model, self.model_settings
                )
        else:
            if self.include_expected_output:
                from .llm_as_a_judge import judge_output_expected

                grading_output = await judge_output_expected(
                    ctx.output, ctx.expected_output, self.rubric, self.model, self.model_settings
                )
            else:
                from .llm_as_a_judge import judge_output

                grading_output = await judge_output(ctx.output, self.rubric, self.model, self.model_settings)

        output: dict[str, EvaluationScalar | EvaluationReason] = {}
        include_both = self.score is not False and self.assertion is not False
        evaluation_name = self.get_default_evaluation_name()

        if self.score is not False:
            default_name = f'{evaluation_name}_score' if include_both else evaluation_name
            _update_combined_output(output, grading_output.score, grading_output.reason, self.score, default_name)

        if self.assertion is not False:
            default_name = f'{evaluation_name}_pass' if include_both else evaluation_name
            _update_combined_output(output, grading_output.pass_, grading_output.reason, self.assertion, default_name)

        return output

    def build_serialization_arguments(self):
        result = super().build_serialization_arguments()
        # always serialize the model as a string when present; use its name if it's a KnownModelName
        if (model := result.get('model')) and isinstance(model, models.Model):  # pragma: no branch
            result['model'] = model.model_id

        # Note: this may lead to confusion if you try to serialize-then-deserialize with a custom model.
        # I expect that is rare enough to be worth not solving yet, but common enough that we probably will want to
        # solve it eventually. I'm imagining some kind of model registry, but don't want to work out the details yet.
        return result

```

---|---
###  MaxDuration `dataclass`
Bases: `Evaluator[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.Evaluator "Evaluator



      dataclass
   \(pydantic_evals.evaluators.evaluator.Evaluator\)")[object[](https://docs.python.org/3/library/functions.html#object), object[](https://docs.python.org/3/library/functions.html#object), object[](https://docs.python.org/3/library/functions.html#object)]`
Check if the execution time is under the specified maximum.
Source code in `pydantic_evals/pydantic_evals/evaluators/common.py`
```
163
164
165
166
167
168
169
170
171
172
173
174
```
| ```
@dataclass(repr=False)
class MaxDuration(Evaluator[object, object, object]):
    """Check if the execution time is under the specified maximum."""

    seconds: float | timedelta

    def evaluate(self, ctx: EvaluatorContext[object, object, object]) -> bool:
        duration = timedelta(seconds=ctx.duration)
        seconds = self.seconds
        if not isinstance(seconds, timedelta):
            seconds = timedelta(seconds=seconds)
        return duration <= seconds

```

---|---
###  OutputConfig
Bases: `TypedDict[](https://typing-extensions.readthedocs.io/en/latest/index.html#typing_extensions.TypedDict "typing_extensions.TypedDict")`
Configuration for the score and assertion outputs of the LLMJudge evaluator.
Source code in `pydantic_evals/pydantic_evals/evaluators/common.py`
```
177
178
179
180
181
```
| ```
class OutputConfig(TypedDict, total=False):
    """Configuration for the score and assertion outputs of the LLMJudge evaluator."""

    evaluation_name: str
    include_reason: bool

```

---|---
###  EvaluatorContext `dataclass`
Bases: `Generic[](https://docs.python.org/3/library/typing.html#typing.Generic "typing.Generic")[InputsT, OutputT, MetadataT]`
Context for evaluating a task execution.
An instance of this class is the sole input to all Evaluators. It contains all the information needed to evaluate the task execution, including inputs, outputs, metadata, and telemetry data.
Evaluators use this context to access the task inputs, actual output, expected output, and other information when evaluating the result of the task execution.
Example:
```
from dataclasses import dataclass

from pydantic_evals.evaluators import Evaluator, EvaluatorContext


@dataclass
class ExactMatch(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> bool:
        # Use the context to access task inputs, outputs, and expected outputs
        return ctx.output == ctx.expected_output

```

Source code in `pydantic_evals/pydantic_evals/evaluators/context.py`
```
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
```
| ```
@dataclass(kw_only=True)
class EvaluatorContext(Generic[InputsT, OutputT, MetadataT]):
    """Context for evaluating a task execution.

    An instance of this class is the sole input to all Evaluators. It contains all the information
    needed to evaluate the task execution, including inputs, outputs, metadata, and telemetry data.

    Evaluators use this context to access the task inputs, actual output, expected output, and other
    information when evaluating the result of the task execution.

    Example:
```python
    from dataclasses import dataclass

    from pydantic_evals.evaluators import Evaluator, EvaluatorContext


    @dataclass
    class ExactMatch(Evaluator):
        def evaluate(self, ctx: EvaluatorContext) -> bool:
            # Use the context to access task inputs, outputs, and expected outputs
            return ctx.output == ctx.expected_output
```
    """

    name: str | None
    """The name of the case."""
    inputs: InputsT
    """The inputs provided to the task for this case."""
    metadata: MetadataT | None
    """Metadata associated with the case, if provided. May be None if no metadata was specified."""
    expected_output: OutputT | None
    """The expected output for the case, if provided. May be None if no expected output was specified."""

    output: OutputT
    """The actual output produced by the task for this case."""
    duration: float
    """The duration of the task run for this case."""
    _span_tree: SpanTree | SpanTreeRecordingError = field(repr=False)
    """The span tree for the task run for this case.

    This will be `None` if `logfire.configure` has not been called.
    """

    attributes: dict[str, Any]
    """Attributes associated with the task run for this case.

    These can be set by calling `pydantic_evals.dataset.set_eval_attribute` in any code executed
    during the evaluation task."""
    metrics: dict[str, int | float]
    """Metrics associated with the task run for this case.

    These can be set by calling `pydantic_evals.dataset.increment_eval_metric` in any code executed
    during the evaluation task."""

    @property
    def span_tree(self) -> SpanTree:
        """Get the `SpanTree` for this task execution.

        The span tree is a graph where each node corresponds to an OpenTelemetry span recorded during the task
        execution, including timing information and any custom spans created during execution.

        Returns:
            The span tree for the task execution.

        Raises:
            SpanTreeRecordingError: If spans were not captured during execution of the task, e.g. due to not having
                the necessary dependencies installed.
        """
        if isinstance(self._span_tree, SpanTreeRecordingError):
            # In this case, there was a reason we couldn't record the SpanTree. We raise that now
            raise self._span_tree
        return self._span_tree

```

---|---
####  name `instance-attribute`
```
name: str[](https://docs.python.org/3/library/stdtypes.html#str) | None

```

The name of the case.
####  inputs `instance-attribute`
```
inputs: InputsT

```

The inputs provided to the task for this case.
####  metadata `instance-attribute`
```
metadata: MetadataT | None

```

Metadata associated with the case, if provided. May be None if no metadata was specified.
####  expected_output `instance-attribute`
```
expected_output: OutputT | None

```

The expected output for the case, if provided. May be None if no expected output was specified.
####  output `instance-attribute`
```
output: OutputT

```

The actual output produced by the task for this case.
####  duration `instance-attribute`
```
duration: float[](https://docs.python.org/3/library/functions.html#float)

```

The duration of the task run for this case.
####  attributes `instance-attribute`
```
attributes: dict[](https://docs.python.org/3/library/stdtypes.html#dict)[str[](https://docs.python.org/3/library/stdtypes.html#str), Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any")]

```

Attributes associated with the task run for this case.
These can be set by calling `pydantic_evals.dataset.set_eval_attribute` in any code executed during the evaluation task.
####  metrics `instance-attribute`
```
metrics: dict[](https://docs.python.org/3/library/stdtypes.html#dict)[str[](https://docs.python.org/3/library/stdtypes.html#str), int[](https://docs.python.org/3/library/functions.html#int) | float[](https://docs.python.org/3/library/functions.html#float)]

```

Metrics associated with the task run for this case.
These can be set by calling `pydantic_evals.dataset.increment_eval_metric` in any code executed during the evaluation task.
####  span_tree `property`
```
span_tree: SpanTree[](https://ai.pydantic.dev/api/pydantic_evals/otel/#pydantic_evals.otel.SpanTree "SpanTree



      dataclass
   \(pydantic_evals.otel.span_tree.SpanTree\)")

```

Get the `SpanTree` for this task execution.
The span tree is a graph where each node corresponds to an OpenTelemetry span recorded during the task execution, including timing information and any custom spans created during execution.
Returns:
Type | Description
---|---
`SpanTree[](https://ai.pydantic.dev/api/pydantic_evals/otel/#pydantic_evals.otel.SpanTree "SpanTree



      dataclass
   \(pydantic_evals.otel.span_tree.SpanTree\)")` |  The span tree for the task execution.
Raises:
Type | Description
---|---
`SpanTreeRecordingError` |  If spans were not captured during execution of the task, e.g. due to not having the necessary dependencies installed.
###  EvaluationReason `dataclass`
The result of running an evaluator with an optional explanation.
Contains a scalar value and an optional "reason" explaining the value.
Parameters:
Name | Type | Description | Default
---|---|---|---
`value` |  `EvaluationScalar` |  The scalar result of the evaluation (boolean, integer, float, or string). |  _required_
`reason` |  `str[](https://docs.python.org/3/library/stdtypes.html#str) | None` |  An optional explanation of the evaluation result. |  `None`
Source code in `pydantic_evals/pydantic_evals/evaluators/evaluator.py`
```
33
34
35
36
37
38
39
40
41
42
43
44
45
```
| ```
@dataclass
class EvaluationReason:
    """The result of running an evaluator with an optional explanation.

    Contains a scalar value and an optional "reason" explaining the value.

    Args:
        value: The scalar result of the evaluation (boolean, integer, float, or string).
        reason: An optional explanation of the evaluation result.
    """

    value: EvaluationScalar
    reason: str | None = None

```

---|---
###  EvaluationResult `dataclass`
Bases: `Generic[](https://docs.python.org/3/library/typing.html#typing.Generic "typing.Generic")[EvaluationScalarT]`
The details of an individual evaluation result.
Contains the name, value, reason, and source evaluator for a single evaluation.
Parameters:
Name | Type | Description | Default
---|---|---|---
`name` |  `str[](https://docs.python.org/3/library/stdtypes.html#str)` |  The name of the evaluation. |  _required_
`value` |  `EvaluationScalarT` |  The scalar result of the evaluation. |  _required_
`reason` |  `str[](https://docs.python.org/3/library/stdtypes.html#str) | None` |  An optional explanation of the evaluation result. |  _required_
`source` |  `EvaluatorSpec[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorSpec "EvaluatorSpec \(pydantic_evals.evaluators.spec.EvaluatorSpec\)")` |  The spec of the evaluator that produced this result. |  _required_
Source code in `pydantic_evals/pydantic_evals/evaluators/evaluator.py`
```
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
```
| ```
@dataclass
class EvaluationResult(Generic[EvaluationScalarT]):
    """The details of an individual evaluation result.

    Contains the name, value, reason, and source evaluator for a single evaluation.

    Args:
        name: The name of the evaluation.
        value: The scalar result of the evaluation.
        reason: An optional explanation of the evaluation result.
        source: The spec of the evaluator that produced this result.
    """

    name: str
    value: EvaluationScalarT
    reason: str | None
    source: EvaluatorSpec

    def downcast(self, *value_types: type[T]) -> EvaluationResult[T] | None:
        """Attempt to downcast this result to a more specific type.

        Args:
            *value_types: The types to check the value against.

        Returns:
            A downcast version of this result if the value is an instance of one of the given types,
            otherwise None.
        """
        # Check if value matches any of the target types, handling bool as a special case
        for value_type in value_types:
            if isinstance(self.value, value_type):
                # Only match bool with explicit bool type
                if isinstance(self.value, bool) and value_type is not bool:
                    continue
                return cast(EvaluationResult[T], self)
        return None

```

---|---
####  downcast
```
downcast(
    *value_types: type[](https://docs.python.org/3/library/functions.html#type)[T],
) -> EvaluationResult[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluationResult "EvaluationResult



      dataclass
   \(pydantic_evals.evaluators.evaluator.EvaluationResult\)")[T] | None

```

Attempt to downcast this result to a more specific type.
Parameters:
Name | Type | Description | Default
---|---|---|---
`*value_types` |  `type[](https://docs.python.org/3/library/functions.html#type)[T]` |  The types to check the value against. |  `()`
Returns:
Type | Description
---|---
`EvaluationResult[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluationResult "EvaluationResult



      dataclass
   \(pydantic_evals.evaluators.evaluator.EvaluationResult\)")[T] | None` |  A downcast version of this result if the value is an instance of one of the given types,
`EvaluationResult[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluationResult "EvaluationResult



      dataclass
   \(pydantic_evals.evaluators.evaluator.EvaluationResult\)")[T] | None` |  otherwise None.
Source code in `pydantic_evals/pydantic_evals/evaluators/evaluator.py`
```
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
```
| ```
def downcast(self, *value_types: type[T]) -> EvaluationResult[T] | None:
    """Attempt to downcast this result to a more specific type.

    Args:
        *value_types: The types to check the value against.

    Returns:
        A downcast version of this result if the value is an instance of one of the given types,
        otherwise None.
    """
    # Check if value matches any of the target types, handling bool as a special case
    for value_type in value_types:
        if isinstance(self.value, value_type):
            # Only match bool with explicit bool type
            if isinstance(self.value, bool) and value_type is not bool:
                continue
            return cast(EvaluationResult[T], self)
    return None

```

---|---
###  Evaluator `dataclass`
Bases: `BaseEvaluator`, `Generic[](https://docs.python.org/3/library/typing.html#typing.Generic "typing.Generic")[InputsT, OutputT, MetadataT]`
Base class for all evaluators.
Evaluators can assess the performance of a task in a variety of ways, as a function of the EvaluatorContext.
Subclasses must implement the `evaluate` method. Note it can be defined with either `def` or `async def`.
Example:
```
from dataclasses import dataclass

from pydantic_evals.evaluators import Evaluator, EvaluatorContext


@dataclass
class ExactMatch(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> bool:
        return ctx.output == ctx.expected_output

```

Source code in `pydantic_evals/pydantic_evals/evaluators/evaluator.py`
```
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
```
| ```
@dataclass(repr=False)
class Evaluator(BaseEvaluator, Generic[InputsT, OutputT, MetadataT]):
    """Base class for all evaluators.

    Evaluators can assess the performance of a task in a variety of ways, as a function of the EvaluatorContext.

    Subclasses must implement the `evaluate` method. Note it can be defined with either `def` or `async def`.

    Example:
```python
    from dataclasses import dataclass

    from pydantic_evals.evaluators import Evaluator, EvaluatorContext


    @dataclass
    class ExactMatch(Evaluator):
        def evaluate(self, ctx: EvaluatorContext) -> bool:
            return ctx.output == ctx.expected_output
```
    """

    @classmethod
    @deprecated('`name` has been renamed, use `get_serialization_name` instead.')
    def name(cls) -> str:
        """`name` has been renamed, use `get_serialization_name` instead."""
        return cls.get_serialization_name()

    def get_default_evaluation_name(self) -> str:
        """Return the default name to use in reports for the output of this evaluator.

        By default, if the evaluator has an attribute called `evaluation_name` of type string, that will be used.
        Otherwise, the serialization name of the evaluator (which is usually the class name) will be used.

        This can be overridden to get a more descriptive name in evaluation reports, e.g. using instance information.

        Note that evaluators that return a mapping of results will always use the keys of that mapping as the names
        of the associated evaluation results.
        """
        evaluation_name = getattr(self, 'evaluation_name', None)
        if isinstance(evaluation_name, str):
            # If the evaluator has an attribute `name` of type string, use that
            return evaluation_name

        return self.get_serialization_name()

    @abstractmethod
    def evaluate(
        self, ctx: EvaluatorContext[InputsT, OutputT, MetadataT]
    ) -> EvaluatorOutput | Awaitable[EvaluatorOutput]:  # pragma: no cover
        """Evaluate the task output in the given context.

        This is the main evaluation method that subclasses must implement. It can be either synchronous
        or asynchronous, returning either an EvaluatorOutput directly or an Awaitable[EvaluatorOutput].

        Args:
            ctx: The context containing the inputs, outputs, and metadata for evaluation.

        Returns:
            The evaluation result, which can be a scalar value, an EvaluationReason, or a mapping
            of evaluation names to either of those. Can be returned either synchronously or as an
            awaitable for asynchronous evaluation.
        """
        raise NotImplementedError('You must implement `evaluate`.')

    def evaluate_sync(self, ctx: EvaluatorContext[InputsT, OutputT, MetadataT]) -> EvaluatorOutput:
        """Run the evaluator synchronously, handling both sync and async implementations.

        This method ensures synchronous execution by running any async evaluate implementation
        to completion using run_until_complete.

        Args:
            ctx: The context containing the inputs, outputs, and metadata for evaluation.

        Returns:
            The evaluation result, which can be a scalar value, an EvaluationReason, or a mapping
            of evaluation names to either of those.
        """
        output = self.evaluate(ctx)
        if inspect.iscoroutine(output):  # pragma: no cover
            return get_event_loop().run_until_complete(output)
        else:
            return cast(EvaluatorOutput, output)

    async def evaluate_async(self, ctx: EvaluatorContext[InputsT, OutputT, MetadataT]) -> EvaluatorOutput:
        """Run the evaluator asynchronously, handling both sync and async implementations.

        This method ensures asynchronous execution by properly awaiting any async evaluate
        implementation. For synchronous implementations, it returns the result directly.

        Args:
            ctx: The context containing the inputs, outputs, and metadata for evaluation.

        Returns:
            The evaluation result, which can be a scalar value, an EvaluationReason, or a mapping
            of evaluation names to either of those.
        """
        # Note: If self.evaluate is synchronous, but you need to prevent this from blocking, override this method with:
        # return await anyio.to_thread.run_sync(self.evaluate, ctx)
        output = self.evaluate(ctx)
        if inspect.iscoroutine(output):
            return await output
        else:
            return cast(EvaluatorOutput, output)

```

---|---
####  name `classmethod` `deprecated`
```
name() -> str[](https://docs.python.org/3/library/stdtypes.html#str)

```

Deprecated
`name` has been renamed, use `get_serialization_name` instead.
`name` has been renamed, use `get_serialization_name` instead.
Source code in `pydantic_evals/pydantic_evals/evaluators/evaluator.py`
```
140
141
142
143
144
```
| ```
@classmethod
@deprecated('`name` has been renamed, use `get_serialization_name` instead.')
def name(cls) -> str:
    """`name` has been renamed, use `get_serialization_name` instead."""
    return cls.get_serialization_name()

```

---|---
####  get_default_evaluation_name
```
get_default_evaluation_name() -> str[](https://docs.python.org/3/library/stdtypes.html#str)

```

Return the default name to use in reports for the output of this evaluator.
By default, if the evaluator has an attribute called `evaluation_name` of type string, that will be used. Otherwise, the serialization name of the evaluator (which is usually the class name) will be used.
This can be overridden to get a more descriptive name in evaluation reports, e.g. using instance information.
Note that evaluators that return a mapping of results will always use the keys of that mapping as the names of the associated evaluation results.
Source code in `pydantic_evals/pydantic_evals/evaluators/evaluator.py`
```
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
```
| ```
def get_default_evaluation_name(self) -> str:
    """Return the default name to use in reports for the output of this evaluator.

    By default, if the evaluator has an attribute called `evaluation_name` of type string, that will be used.
    Otherwise, the serialization name of the evaluator (which is usually the class name) will be used.

    This can be overridden to get a more descriptive name in evaluation reports, e.g. using instance information.

    Note that evaluators that return a mapping of results will always use the keys of that mapping as the names
    of the associated evaluation results.
    """
    evaluation_name = getattr(self, 'evaluation_name', None)
    if isinstance(evaluation_name, str):
        # If the evaluator has an attribute `name` of type string, use that
        return evaluation_name

    return self.get_serialization_name()

```

---|---
####  evaluate `abstractmethod`
```
evaluate(
    ctx: EvaluatorContext[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext "EvaluatorContext



      dataclass
   \(pydantic_evals.evaluators.context.EvaluatorContext\)")[InputsT, OutputT, MetadataT],
) -> EvaluatorOutput[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorOutput "EvaluatorOutput



      module-attribute
   \(pydantic_evals.evaluators.evaluator.EvaluatorOutput\)") | Awaitable[](https://docs.python.org/3/library/collections.abc.html#collections.abc.Awaitable "collections.abc.Awaitable")[EvaluatorOutput[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorOutput "EvaluatorOutput



      module-attribute
   \(pydantic_evals.evaluators.evaluator.EvaluatorOutput\)")]

```

Evaluate the task output in the given context.
This is the main evaluation method that subclasses must implement. It can be either synchronous or asynchronous, returning either an EvaluatorOutput directly or an Awaitable[EvaluatorOutput].
Parameters:
Name | Type | Description | Default
---|---|---|---
`ctx` |  `EvaluatorContext[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext "EvaluatorContext



      dataclass
   \(pydantic_evals.evaluators.context.EvaluatorContext\)")[InputsT, OutputT, MetadataT]` |  The context containing the inputs, outputs, and metadata for evaluation. |  _required_
Returns:
Type | Description
---|---
`EvaluatorOutput[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorOutput "EvaluatorOutput



      module-attribute
   \(pydantic_evals.evaluators.evaluator.EvaluatorOutput\)") | Awaitable[](https://docs.python.org/3/library/collections.abc.html#collections.abc.Awaitable "collections.abc.Awaitable")[EvaluatorOutput[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorOutput "EvaluatorOutput



      module-attribute
   \(pydantic_evals.evaluators.evaluator.EvaluatorOutput\)")]` |  The evaluation result, which can be a scalar value, an EvaluationReason, or a mapping
`EvaluatorOutput[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorOutput "EvaluatorOutput



      module-attribute
   \(pydantic_evals.evaluators.evaluator.EvaluatorOutput\)") | Awaitable[](https://docs.python.org/3/library/collections.abc.html#collections.abc.Awaitable "collections.abc.Awaitable")[EvaluatorOutput[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorOutput "EvaluatorOutput



      module-attribute
   \(pydantic_evals.evaluators.evaluator.EvaluatorOutput\)")]` |  of evaluation names to either of those. Can be returned either synchronously or as an
`EvaluatorOutput[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorOutput "EvaluatorOutput



      module-attribute
   \(pydantic_evals.evaluators.evaluator.EvaluatorOutput\)") | Awaitable[](https://docs.python.org/3/library/collections.abc.html#collections.abc.Awaitable "collections.abc.Awaitable")[EvaluatorOutput[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorOutput "EvaluatorOutput



      module-attribute
   \(pydantic_evals.evaluators.evaluator.EvaluatorOutput\)")]` |  awaitable for asynchronous evaluation.
Source code in `pydantic_evals/pydantic_evals/evaluators/evaluator.py`
```
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
```
| ```
@abstractmethod
def evaluate(
    self, ctx: EvaluatorContext[InputsT, OutputT, MetadataT]
) -> EvaluatorOutput | Awaitable[EvaluatorOutput]:  # pragma: no cover
    """Evaluate the task output in the given context.

    This is the main evaluation method that subclasses must implement. It can be either synchronous
    or asynchronous, returning either an EvaluatorOutput directly or an Awaitable[EvaluatorOutput].

    Args:
        ctx: The context containing the inputs, outputs, and metadata for evaluation.

    Returns:
        The evaluation result, which can be a scalar value, an EvaluationReason, or a mapping
        of evaluation names to either of those. Can be returned either synchronously or as an
        awaitable for asynchronous evaluation.
    """
    raise NotImplementedError('You must implement `evaluate`.')

```

---|---
####  evaluate_sync
```
evaluate_sync(
    ctx: EvaluatorContext[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext "EvaluatorContext



      dataclass
   \(pydantic_evals.evaluators.context.EvaluatorContext\)")[InputsT, OutputT, MetadataT],
) -> EvaluatorOutput[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorOutput "EvaluatorOutput



      module-attribute
   \(pydantic_evals.evaluators.evaluator.EvaluatorOutput\)")

```

Run the evaluator synchronously, handling both sync and async implementations.
This method ensures synchronous execution by running any async evaluate implementation to completion using run_until_complete.
Parameters:
Name | Type | Description | Default
---|---|---|---
`ctx` |  `EvaluatorContext[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext "EvaluatorContext



      dataclass
   \(pydantic_evals.evaluators.context.EvaluatorContext\)")[InputsT, OutputT, MetadataT]` |  The context containing the inputs, outputs, and metadata for evaluation. |  _required_
Returns:
Type | Description
---|---
`EvaluatorOutput[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorOutput "EvaluatorOutput



      module-attribute
   \(pydantic_evals.evaluators.evaluator.EvaluatorOutput\)")` |  The evaluation result, which can be a scalar value, an EvaluationReason, or a mapping
`EvaluatorOutput[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorOutput "EvaluatorOutput



      module-attribute
   \(pydantic_evals.evaluators.evaluator.EvaluatorOutput\)")` |  of evaluation names to either of those.
Source code in `pydantic_evals/pydantic_evals/evaluators/evaluator.py`
```
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
```
| ```
def evaluate_sync(self, ctx: EvaluatorContext[InputsT, OutputT, MetadataT]) -> EvaluatorOutput:
    """Run the evaluator synchronously, handling both sync and async implementations.

    This method ensures synchronous execution by running any async evaluate implementation
    to completion using run_until_complete.

    Args:
        ctx: The context containing the inputs, outputs, and metadata for evaluation.

    Returns:
        The evaluation result, which can be a scalar value, an EvaluationReason, or a mapping
        of evaluation names to either of those.
    """
    output = self.evaluate(ctx)
    if inspect.iscoroutine(output):  # pragma: no cover
        return get_event_loop().run_until_complete(output)
    else:
        return cast(EvaluatorOutput, output)

```

---|---
####  evaluate_async `async`
```
evaluate_async(
    ctx: EvaluatorContext[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext "EvaluatorContext



      dataclass
   \(pydantic_evals.evaluators.context.EvaluatorContext\)")[InputsT, OutputT, MetadataT],
) -> EvaluatorOutput[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorOutput "EvaluatorOutput



      module-attribute
   \(pydantic_evals.evaluators.evaluator.EvaluatorOutput\)")

```

Run the evaluator asynchronously, handling both sync and async implementations.
This method ensures asynchronous execution by properly awaiting any async evaluate implementation. For synchronous implementations, it returns the result directly.
Parameters:
Name | Type | Description | Default
---|---|---|---
`ctx` |  `EvaluatorContext[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorContext "EvaluatorContext



      dataclass
   \(pydantic_evals.evaluators.context.EvaluatorContext\)")[InputsT, OutputT, MetadataT]` |  The context containing the inputs, outputs, and metadata for evaluation. |  _required_
Returns:
Type | Description
---|---
`EvaluatorOutput[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorOutput "EvaluatorOutput



      module-attribute
   \(pydantic_evals.evaluators.evaluator.EvaluatorOutput\)")` |  The evaluation result, which can be a scalar value, an EvaluationReason, or a mapping
`EvaluatorOutput[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorOutput "EvaluatorOutput



      module-attribute
   \(pydantic_evals.evaluators.evaluator.EvaluatorOutput\)")` |  of evaluation names to either of those.
Source code in `pydantic_evals/pydantic_evals/evaluators/evaluator.py`
```
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
```
| ```
async def evaluate_async(self, ctx: EvaluatorContext[InputsT, OutputT, MetadataT]) -> EvaluatorOutput:
    """Run the evaluator asynchronously, handling both sync and async implementations.

    This method ensures asynchronous execution by properly awaiting any async evaluate
    implementation. For synchronous implementations, it returns the result directly.

    Args:
        ctx: The context containing the inputs, outputs, and metadata for evaluation.

    Returns:
        The evaluation result, which can be a scalar value, an EvaluationReason, or a mapping
        of evaluation names to either of those.
    """
    # Note: If self.evaluate is synchronous, but you need to prevent this from blocking, override this method with:
    # return await anyio.to_thread.run_sync(self.evaluate, ctx)
    output = self.evaluate(ctx)
    if inspect.iscoroutine(output):
        return await output
    else:
        return cast(EvaluatorOutput, output)

```

---|---
###  EvaluatorFailure `dataclass`
Represents a failure raised during the execution of an evaluator.
Source code in `pydantic_evals/pydantic_evals/evaluators/evaluator.py`
```
 97
 98
 99
100
101
102
103
104
```
| ```
@dataclass
class EvaluatorFailure:
    """Represents a failure raised during the execution of an evaluator."""

    name: str
    error_message: str
    error_stacktrace: str
    source: EvaluatorSpec

```

---|---
###  EvaluatorOutput `module-attribute`
```
EvaluatorOutput = (
    EvaluationScalar
    | EvaluationReason[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluationReason "EvaluationReason



      dataclass
   \(pydantic_evals.evaluators.evaluator.EvaluationReason\)")
    | Mapping[](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping "collections.abc.Mapping")[str[](https://docs.python.org/3/library/stdtypes.html#str), EvaluationScalar | EvaluationReason[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluationReason "EvaluationReason



      dataclass
   \(pydantic_evals.evaluators.evaluator.EvaluationReason\)")]
)

```

Type for the output of an evaluator, which can be a scalar, an EvaluationReason, or a mapping of names to either.
###  EvaluatorSpec
Bases: `BaseModel[](https://docs.pydantic.dev/latest/api/base_model/#pydantic.BaseModel "pydantic.BaseModel")`
The specification of an evaluator to be run.
This class is used to represent evaluators in a serializable format, supporting various short forms for convenience when defining evaluators in YAML or JSON dataset files.
In particular, each of the following forms is supported for specifying an evaluator with name `MyEvaluator`: * `'MyEvaluator'` - Just the (string) name of the Evaluator subclass is used if its `__init__` takes no arguments * `{'MyEvaluator': first_arg}` - A single argument is passed as the first positional argument to `MyEvaluator.__init__` * `{'MyEvaluator': {k1: v1, k2: v2}}` - Multiple kwargs are passed to `MyEvaluator.__init__`
Source code in `pydantic_evals/pydantic_evals/evaluators/spec.py`
```
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
```
| ```
class EvaluatorSpec(BaseModel):
    """The specification of an evaluator to be run.

    This class is used to represent evaluators in a serializable format, supporting various
    short forms for convenience when defining evaluators in YAML or JSON dataset files.

    In particular, each of the following forms is supported for specifying an evaluator with name `MyEvaluator`:
    * `'MyEvaluator'` - Just the (string) name of the Evaluator subclass is used if its `__init__` takes no arguments
    * `{'MyEvaluator': first_arg}` - A single argument is passed as the first positional argument to `MyEvaluator.__init__`
    * `{'MyEvaluator': {k1: v1, k2: v2}}` - Multiple kwargs are passed to `MyEvaluator.__init__`
    """

    name: str
    """The name of the evaluator class; should be the value returned by `EvaluatorClass.get_serialization_name()`"""

    arguments: None | tuple[Any] | dict[str, Any]
    """The arguments to pass to the evaluator's constructor.

    Can be None (no arguments), a tuple (a single positional argument), or a dict (keyword arguments).
    """

    @property
    def args(self) -> tuple[Any, ...]:
        """Get the positional arguments for the evaluator.

        Returns:
            A tuple of positional arguments if arguments is a tuple, otherwise an empty tuple.
        """
        if isinstance(self.arguments, tuple):
            return self.arguments
        return ()

    @property
    def kwargs(self) -> dict[str, Any]:
        """Get the keyword arguments for the evaluator.

        Returns:
            A dictionary of keyword arguments if arguments is a dict, otherwise an empty dict.
        """
        if isinstance(self.arguments, dict):
            return self.arguments
        return {}

    @model_validator(mode='wrap')
    @classmethod
    def deserialize(cls, value: Any, handler: ModelWrapValidatorHandler[EvaluatorSpec]) -> EvaluatorSpec:
        """Deserialize an EvaluatorSpec from various formats.

        This validator handles the various short forms of evaluator specifications,
        converting them to a consistent EvaluatorSpec instance.

        Args:
            value: The value to deserialize.
            handler: The validator handler.

        Returns:
            The deserialized EvaluatorSpec.

        Raises:
            ValidationError: If the value cannot be deserialized.
        """
        try:
            result = handler(value)
            return result
        except ValidationError as exc:
            try:
                deserialized = _SerializedEvaluatorSpec.model_validate(value)
            except ValidationError:
                raise exc  # raise the original error
            return deserialized.to_evaluator_spec()

    @model_serializer(mode='wrap')
    def serialize(self, handler: SerializerFunctionWrapHandler, info: SerializationInfo) -> Any:
        """Serialize using the appropriate short-form if possible.

        Returns:
            The serialized evaluator specification, using the shortest form possible:
            - Just the name if there are no arguments
            - {name: first_arg} if there's a single positional argument
            - {name: {kwargs}} if there are multiple (keyword) arguments
        """
        if isinstance(info.context, dict) and info.context.get('use_short_form'):  # pyright: ignore[reportUnknownMemberType]
            if self.arguments is None:
                return self.name
            elif isinstance(self.arguments, tuple):
                return {self.name: self.arguments[0]}
            else:
                return {self.name: self.arguments}
        else:
            return handler(self)

```

---|---
####  name `instance-attribute`
```
name: str[](https://docs.python.org/3/library/stdtypes.html#str)

```

The name of the evaluator class; should be the value returned by `EvaluatorClass.get_serialization_name()`
####  arguments `instance-attribute`
```
arguments: None | tuple[](https://docs.python.org/3/library/stdtypes.html#tuple)[Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any")] | dict[](https://docs.python.org/3/library/stdtypes.html#dict)[str[](https://docs.python.org/3/library/stdtypes.html#str), Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any")]

```

The arguments to pass to the evaluator's constructor.
Can be None (no arguments), a tuple (a single positional argument), or a dict (keyword arguments).
####  args `property`
```
args: tuple[](https://docs.python.org/3/library/stdtypes.html#tuple)[Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any"), ...]

```

Get the positional arguments for the evaluator.
Returns:
Type | Description
---|---
`tuple[](https://docs.python.org/3/library/stdtypes.html#tuple)[Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any"), ...]` |  A tuple of positional arguments if arguments is a tuple, otherwise an empty tuple.
####  kwargs `property`
```
kwargs: dict[](https://docs.python.org/3/library/stdtypes.html#dict)[str[](https://docs.python.org/3/library/stdtypes.html#str), Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any")]

```

Get the keyword arguments for the evaluator.
Returns:
Type | Description
---|---
`dict[](https://docs.python.org/3/library/stdtypes.html#dict)[str[](https://docs.python.org/3/library/stdtypes.html#str), Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any")]` |  A dictionary of keyword arguments if arguments is a dict, otherwise an empty dict.
####  deserialize `classmethod`
```
deserialize(
    value: Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any"),
    handler: ModelWrapValidatorHandler[](https://docs.pydantic.dev/latest/api/functional_validators/#pydantic.functional_validators.ModelWrapValidatorHandler "pydantic.ModelWrapValidatorHandler")[EvaluatorSpec[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorSpec "EvaluatorSpec \(pydantic_evals.evaluators.spec.EvaluatorSpec\)")],
) -> EvaluatorSpec[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorSpec "EvaluatorSpec \(pydantic_evals.evaluators.spec.EvaluatorSpec\)")

```

Deserialize an EvaluatorSpec from various formats.
This validator handles the various short forms of evaluator specifications, converting them to a consistent EvaluatorSpec instance.
Parameters:
Name | Type | Description | Default
---|---|---|---
`value` |  `Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any")` |  The value to deserialize. |  _required_
`handler` |  `ModelWrapValidatorHandler[](https://docs.pydantic.dev/latest/api/functional_validators/#pydantic.functional_validators.ModelWrapValidatorHandler "pydantic.ModelWrapValidatorHandler")[EvaluatorSpec[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorSpec "EvaluatorSpec \(pydantic_evals.evaluators.spec.EvaluatorSpec\)")]` |  The validator handler. |  _required_
Returns:
Type | Description
---|---
`EvaluatorSpec[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.EvaluatorSpec "EvaluatorSpec \(pydantic_evals.evaluators.spec.EvaluatorSpec\)")` |  The deserialized EvaluatorSpec.
Raises:
Type | Description
---|---
`ValidationError` |  If the value cannot be deserialized.
Source code in `pydantic_evals/pydantic_evals/evaluators/spec.py`
```
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
```
| ```
@model_validator(mode='wrap')
@classmethod
def deserialize(cls, value: Any, handler: ModelWrapValidatorHandler[EvaluatorSpec]) -> EvaluatorSpec:
    """Deserialize an EvaluatorSpec from various formats.

    This validator handles the various short forms of evaluator specifications,
    converting them to a consistent EvaluatorSpec instance.

    Args:
        value: The value to deserialize.
        handler: The validator handler.

    Returns:
        The deserialized EvaluatorSpec.

    Raises:
        ValidationError: If the value cannot be deserialized.
    """
    try:
        result = handler(value)
        return result
    except ValidationError as exc:
        try:
            deserialized = _SerializedEvaluatorSpec.model_validate(value)
        except ValidationError:
            raise exc  # raise the original error
        return deserialized.to_evaluator_spec()

```

---|---
####  serialize
```
serialize(
    handler: SerializerFunctionWrapHandler,
    info: SerializationInfo[](https://docs.pydantic.dev/latest/api/pydantic_core_schema/#pydantic_core.core_schema.SerializationInfo "pydantic_core.core_schema.SerializationInfo"),
) -> Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any")

```

Serialize using the appropriate short-form if possible.
Returns:
Type | Description
---|---
`Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any")` |  The serialized evaluator specification, using the shortest form possible:
`Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any")` |
  * Just the name if there are no arguments


`Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any")` |
  * {name: first_arg} if there's a single positional argument


`Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any")` |
  * {name: {kwargs}} if there are multiple (keyword) arguments


Source code in `pydantic_evals/pydantic_evals/evaluators/spec.py`
```
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
```
| ```
@model_serializer(mode='wrap')
def serialize(self, handler: SerializerFunctionWrapHandler, info: SerializationInfo) -> Any:
    """Serialize using the appropriate short-form if possible.

    Returns:
        The serialized evaluator specification, using the shortest form possible:
        - Just the name if there are no arguments
        - {name: first_arg} if there's a single positional argument
        - {name: {kwargs}} if there are multiple (keyword) arguments
    """
    if isinstance(info.context, dict) and info.context.get('use_short_form'):  # pyright: ignore[reportUnknownMemberType]
        if self.arguments is None:
            return self.name
        elif isinstance(self.arguments, tuple):
            return {self.name: self.arguments[0]}
        else:
            return {self.name: self.arguments}
    else:
        return handler(self)

```

---|---
###  ConfusionMatrixEvaluator `dataclass`
Bases: `ReportEvaluator[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ReportEvaluator "ReportEvaluator



      dataclass
   \(pydantic_evals.evaluators.report_evaluator.ReportEvaluator\)")`
Computes a confusion matrix from case data.
Source code in `pydantic_evals/pydantic_evals/evaluators/report_common.py`
```
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
```
| ```
@dataclass(repr=False)
class ConfusionMatrixEvaluator(ReportEvaluator):
    """Computes a confusion matrix from case data."""

    predicted_from: Literal['expected_output', 'output', 'metadata', 'labels'] = 'output'
    predicted_key: str | None = None

    expected_from: Literal['expected_output', 'output', 'metadata', 'labels'] = 'expected_output'
    expected_key: str | None = None

    title: str = 'Confusion Matrix'

    def evaluate(self, ctx: ReportEvaluatorContext[Any, Any, Any]) -> ConfusionMatrix:
        predicted: list[str] = []
        expected: list[str] = []

        for case in ctx.report.cases:
            pred = self._extract(case, self.predicted_from, self.predicted_key)
            exp = self._extract(case, self.expected_from, self.expected_key)
            if pred is None or exp is None:
                continue
            predicted.append(pred)
            expected.append(exp)

        all_labels = sorted(set(predicted) | set(expected))
        label_to_idx = {label: i for i, label in enumerate(all_labels)}
        matrix = [[0] * len(all_labels) for _ in all_labels]

        for e, p in zip(expected, predicted):
            matrix[label_to_idx[e]][label_to_idx[p]] += 1

        return ConfusionMatrix(
            title=self.title,
            class_labels=all_labels,
            matrix=matrix,
        )

    @staticmethod
    def _extract(
        case: ReportCase[Any, Any, Any],
        from_: Literal['expected_output', 'output', 'metadata', 'labels'],
        key: str | None,
    ) -> str | None:
        if from_ == 'expected_output':
            return str(case.expected_output) if case.expected_output is not None else None
        elif from_ == 'output':
            return str(case.output) if case.output is not None else None
        elif from_ == 'metadata':
            if key is not None:
                if isinstance(case.metadata, dict):
                    metadata_dict = cast(dict[str, Any], case.metadata)  # pyright: ignore[reportUnknownMemberType]
                    val = metadata_dict.get(key)
                    return str(val) if val is not None else None
                return None  # key requested but metadata isn't a dict — skip this case
            return str(case.metadata) if case.metadata is not None else None
        elif from_ == 'labels':
            if key is None:
                raise ValueError("'key' is required when from_='labels'")
            label_result = case.labels.get(key)
            return label_result.value if label_result else None
        assert_never(from_)

```

---|---
###  KolmogorovSmirnovEvaluator `dataclass`
Bases: `ReportEvaluator[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ReportEvaluator "ReportEvaluator



      dataclass
   \(pydantic_evals.evaluators.report_evaluator.ReportEvaluator\)")`
Computes a Kolmogorov-Smirnov plot and statistic from case data.
Plots the empirical CDFs of the score distribution for positive and negative cases, and computes the KS statistic (maximum vertical distance between the two CDFs).
Returns a `LinePlot` with the two CDF curves and a `ScalarResult` with the KS statistic.
Source code in `pydantic_evals/pydantic_evals/evaluators/report_common.py`
```
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
```
| ```
@dataclass(repr=False)
class KolmogorovSmirnovEvaluator(ReportEvaluator):
    """Computes a Kolmogorov-Smirnov plot and statistic from case data.

    Plots the empirical CDFs of the score distribution for positive and negative cases,
    and computes the KS statistic (maximum vertical distance between the two CDFs).

    Returns a `LinePlot` with the two CDF curves and a `ScalarResult` with the KS statistic.
    """

    score_key: str
    positive_from: Literal['expected_output', 'assertions', 'labels']
    positive_key: str | None = None

    score_from: Literal['scores', 'metrics'] = 'scores'

    title: str = 'KS Plot'
    n_thresholds: int = 100

    def evaluate(self, ctx: ReportEvaluatorContext[Any, Any, Any]) -> list[ReportAnalysis]:
        scored_cases = _extract_scored_cases(
            ctx.report.cases, self.score_key, self.score_from, self.positive_from, self.positive_key
        )

        empty_result: list[ReportAnalysis] = [
            LinePlot(
                title=self.title,
                x_label='Score',
                y_label='Cumulative Probability',
                y_range=(0, 1),
                curves=[],
            ),
            ScalarResult(title='KS Statistic', value=float('nan')),
        ]
        if not scored_cases:
            return empty_result

        pos_scores = sorted(s for s, p in scored_cases if p)
        neg_scores = sorted(s for s, p in scored_cases if not p)

        if not pos_scores or not neg_scores:
            return empty_result

        # Compute CDFs at all unique scores using binary search
        all_scores = sorted({s for s, _ in scored_cases})
        # Start both CDFs at y=0 at the minimum score
        pos_cdf: list[tuple[float, float]] = [(all_scores[0], 0.0)]
        neg_cdf: list[tuple[float, float]] = [(all_scores[0], 0.0)]
        ks_stat = 0.0

        for score in all_scores:
            pos_val = bisect_right(pos_scores, score) / len(pos_scores)
            neg_val = bisect_right(neg_scores, score) / len(neg_scores)
            pos_cdf.append((score, pos_val))
            neg_cdf.append((score, neg_val))
            ks_stat = max(ks_stat, abs(pos_val - neg_val))

        # Downsample for display
        display_pos = _downsample(pos_cdf, self.n_thresholds)
        display_neg = _downsample(neg_cdf, self.n_thresholds)

        pos_curve = LinePlotCurve(
            name='Positive',
            points=[LinePlotPoint(x=s, y=v) for s, v in display_pos],
            step='end',
        )
        neg_curve = LinePlotCurve(
            name='Negative',
            points=[LinePlotPoint(x=s, y=v) for s, v in display_neg],
            step='end',
        )

        return [
            LinePlot(
                title=self.title,
                x_label='Score',
                y_label='Cumulative Probability',
                y_range=(0, 1),
                curves=[pos_curve, neg_curve],
            ),
            ScalarResult(title='KS Statistic', value=ks_stat),
        ]

```

---|---
###  PrecisionRecallEvaluator `dataclass`
Bases: `ReportEvaluator[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ReportEvaluator "ReportEvaluator



      dataclass
   \(pydantic_evals.evaluators.report_evaluator.ReportEvaluator\)")`
Computes a precision-recall curve from case data.
Returns both a `PrecisionRecall` chart and a `ScalarResult` with the AUC value. The AUC is computed at full resolution (every unique score threshold) for accuracy, while the chart points are downsampled to `n_thresholds` for display.
Source code in `pydantic_evals/pydantic_evals/evaluators/report_common.py`
```
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
```
| ```
@dataclass(repr=False)
class PrecisionRecallEvaluator(ReportEvaluator):
    """Computes a precision-recall curve from case data.

    Returns both a `PrecisionRecall` chart and a `ScalarResult` with the AUC value.
    The AUC is computed at full resolution (every unique score threshold) for accuracy,
    while the chart points are downsampled to `n_thresholds` for display.
    """

    score_key: str
    positive_from: Literal['expected_output', 'assertions', 'labels']
    positive_key: str | None = None

    score_from: Literal['scores', 'metrics'] = 'scores'

    title: str = 'Precision-Recall Curve'
    n_thresholds: int = 100

    def evaluate(self, ctx: ReportEvaluatorContext[Any, Any, Any]) -> list[ReportAnalysis]:
        scored_cases = _extract_scored_cases(
            ctx.report.cases, self.score_key, self.score_from, self.positive_from, self.positive_key
        )

        if not scored_cases:
            return [
                PrecisionRecall(title=self.title, curves=[]),
                ScalarResult(title=f'{self.title} AUC', value=float('nan')),
            ]

        total_positives = sum(1 for _, p in scored_cases if p)

        # Compute precision/recall at every unique score for exact AUC
        unique_thresholds = sorted({s for s, _ in scored_cases}, reverse=True)
        # Start with anchor at (recall=0, precision=1) — the "no predictions" point
        max_score = unique_thresholds[0]
        all_points: list[PrecisionRecallPoint] = [PrecisionRecallPoint(threshold=max_score, precision=1.0, recall=0.0)]
        for threshold in unique_thresholds:
            tp = sum(1 for s, p in scored_cases if s >= threshold and p)
            fp = sum(1 for s, p in scored_cases if s >= threshold and not p)
            fn = total_positives - tp
            precision = tp / (tp + fp) if (tp + fp) > 0 else 1.0
            recall = tp / (fn + tp) if (fn + tp) > 0 else 0.0
            all_points.append(PrecisionRecallPoint(threshold=threshold, precision=precision, recall=recall))

        # Exact AUC from the full-resolution points (anchor included)
        auc_points = [(p.recall, p.precision) for p in all_points]
        auc = _trapezoidal_auc(auc_points)

        # Downsample for display
        if len(all_points) <= self.n_thresholds or self.n_thresholds <= 1:
            display_points = all_points
        else:
            indices = sorted(
                {int(i * (len(all_points) - 1) / (self.n_thresholds - 1)) for i in range(self.n_thresholds)}
            )
            display_points = [all_points[i] for i in indices]

        curve = PrecisionRecallCurve(name=ctx.name, points=display_points, auc=auc)
        return [
            PrecisionRecall(title=self.title, curves=[curve]),
            ScalarResult(title=f'{self.title} AUC', value=auc),
        ]

```

---|---
###  ROCAUCEvaluator `dataclass`
Bases: `ReportEvaluator[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ReportEvaluator "ReportEvaluator



      dataclass
   \(pydantic_evals.evaluators.report_evaluator.ReportEvaluator\)")`
Computes an ROC curve and AUC from case data.
Returns a `LinePlot` with the ROC curve (plus a dashed random-baseline diagonal) and a `ScalarResult` with the AUC value.
Source code in `pydantic_evals/pydantic_evals/evaluators/report_common.py`
```
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
```
| ```
@dataclass(repr=False)
class ROCAUCEvaluator(ReportEvaluator):
    """Computes an ROC curve and AUC from case data.

    Returns a `LinePlot` with the ROC curve (plus a dashed random-baseline diagonal)
    and a `ScalarResult` with the AUC value.
    """

    score_key: str
    positive_from: Literal['expected_output', 'assertions', 'labels']
    positive_key: str | None = None

    score_from: Literal['scores', 'metrics'] = 'scores'

    title: str = 'ROC Curve'
    n_thresholds: int = 100

    def evaluate(self, ctx: ReportEvaluatorContext[Any, Any, Any]) -> list[ReportAnalysis]:
        scored_cases = _extract_scored_cases(
            ctx.report.cases, self.score_key, self.score_from, self.positive_from, self.positive_key
        )

        empty_result: list[ReportAnalysis] = [
            LinePlot(
                title=self.title,
                x_label='False Positive Rate',
                y_label='True Positive Rate',
                x_range=(0, 1),
                y_range=(0, 1),
                curves=[],
            ),
            ScalarResult(title=f'{self.title} AUC', value=float('nan')),
        ]
        if not scored_cases:
            return empty_result

        total_positives = sum(1 for _, p in scored_cases if p)
        total_negatives = len(scored_cases) - total_positives

        if total_positives == 0 or total_negatives == 0:
            return empty_result

        # Compute TPR/FPR at every unique score for exact AUC
        unique_thresholds = sorted({s for s, _ in scored_cases}, reverse=True)
        all_fpr_tpr: list[tuple[float, float]] = [(0.0, 0.0)]
        for threshold in unique_thresholds:
            tp = sum(1 for s, p in scored_cases if s >= threshold and p)
            fp = sum(1 for s, p in scored_cases if s >= threshold and not p)
            tpr = tp / total_positives
            fpr = fp / total_negatives
            all_fpr_tpr.append((fpr, tpr))
        all_fpr_tpr.sort()

        # Exact AUC
        auc = _trapezoidal_auc(all_fpr_tpr)

        # Downsample for display
        downsampled = _downsample(all_fpr_tpr, self.n_thresholds)

        roc_curve = LinePlotCurve(
            name=f'{ctx.name} (AUC: {auc:.3f})',
            points=[LinePlotPoint(x=fpr, y=tpr) for fpr, tpr in downsampled],
        )
        baseline = LinePlotCurve(
            name='Random',
            points=[LinePlotPoint(x=0, y=0), LinePlotPoint(x=1, y=1)],
            style='dashed',
        )

        return [
            LinePlot(
                title=self.title,
                x_label='False Positive Rate',
                y_label='True Positive Rate',
                x_range=(0, 1),
                y_range=(0, 1),
                curves=[roc_curve, baseline],
            ),
            ScalarResult(title=f'{self.title} AUC', value=auc),
        ]

```

---|---
###  ReportEvaluator `dataclass`
Bases: `BaseEvaluator`, `Generic[](https://docs.python.org/3/library/typing.html#typing.Generic "typing.Generic")[InputsT, OutputT, MetadataT]`
Base class for experiment-wide evaluators that analyze full reports.
Unlike case-level Evaluators which assess individual task outputs, ReportEvaluators see all case results together and produce experiment-wide analyses like confusion matrices, precision-recall curves, or scalar statistics.
Source code in `pydantic_evals/pydantic_evals/evaluators/report_evaluator.py`
```
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
```
| ```
@dataclass(repr=False)
class ReportEvaluator(BaseEvaluator, Generic[InputsT, OutputT, MetadataT]):
    """Base class for experiment-wide evaluators that analyze full reports.

    Unlike case-level Evaluators which assess individual task outputs,
    ReportEvaluators see all case results together and produce
    experiment-wide analyses like confusion matrices, precision-recall curves,
    or scalar statistics.
    """

    @abstractmethod
    def evaluate(
        self, ctx: ReportEvaluatorContext[InputsT, OutputT, MetadataT]
    ) -> ReportAnalysis | list[ReportAnalysis] | Awaitable[ReportAnalysis | list[ReportAnalysis]]:
        """Evaluate the full report and return experiment-wide analysis/analyses."""
        ...

    async def evaluate_async(
        self, ctx: ReportEvaluatorContext[InputsT, OutputT, MetadataT]
    ) -> ReportAnalysis | list[ReportAnalysis]:
        """Evaluate, handling both sync and async implementations."""
        output = self.evaluate(ctx)
        if inspect.iscoroutine(output):
            return await output
        return cast('ReportAnalysis | list[ReportAnalysis]', output)

```

---|---
####  evaluate `abstractmethod`
```
evaluate(
    ctx: ReportEvaluatorContext[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ReportEvaluatorContext "ReportEvaluatorContext



      dataclass
   \(pydantic_evals.evaluators.report_evaluator.ReportEvaluatorContext\)")[
        InputsT, OutputT, MetadataT
    ],
) -> (
    ReportAnalysis[](https://ai.pydantic.dev/api/pydantic_evals/reporting/#pydantic_evals.reporting.ReportAnalysis "ReportAnalysis



      module-attribute
   \(pydantic_evals.reporting.analyses.ReportAnalysis\)")
    | list[](https://docs.python.org/3/library/stdtypes.html#list)[ReportAnalysis[](https://ai.pydantic.dev/api/pydantic_evals/reporting/#pydantic_evals.reporting.ReportAnalysis "ReportAnalysis



      module-attribute
   \(pydantic_evals.reporting.analyses.ReportAnalysis\)")]
    | Awaitable[](https://docs.python.org/3/library/collections.abc.html#collections.abc.Awaitable "collections.abc.Awaitable")[ReportAnalysis[](https://ai.pydantic.dev/api/pydantic_evals/reporting/#pydantic_evals.reporting.ReportAnalysis "ReportAnalysis



      module-attribute
   \(pydantic_evals.reporting.analyses.ReportAnalysis\)") | list[](https://docs.python.org/3/library/stdtypes.html#list)[ReportAnalysis[](https://ai.pydantic.dev/api/pydantic_evals/reporting/#pydantic_evals.reporting.ReportAnalysis "ReportAnalysis



      module-attribute
   \(pydantic_evals.reporting.analyses.ReportAnalysis\)")]]
)

```

Evaluate the full report and return experiment-wide analysis/analyses.
Source code in `pydantic_evals/pydantic_evals/evaluators/report_evaluator.py`
```
49
50
51
52
53
54
```
| ```
@abstractmethod
def evaluate(
    self, ctx: ReportEvaluatorContext[InputsT, OutputT, MetadataT]
) -> ReportAnalysis | list[ReportAnalysis] | Awaitable[ReportAnalysis | list[ReportAnalysis]]:
    """Evaluate the full report and return experiment-wide analysis/analyses."""
    ...

```

---|---
####  evaluate_async `async`
```
evaluate_async(
    ctx: ReportEvaluatorContext[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.ReportEvaluatorContext "ReportEvaluatorContext



      dataclass
   \(pydantic_evals.evaluators.report_evaluator.ReportEvaluatorContext\)")[
        InputsT, OutputT, MetadataT
    ],
) -> ReportAnalysis[](https://ai.pydantic.dev/api/pydantic_evals/reporting/#pydantic_evals.reporting.ReportAnalysis "ReportAnalysis



      module-attribute
   \(pydantic_evals.reporting.analyses.ReportAnalysis\)") | list[](https://docs.python.org/3/library/stdtypes.html#list)[ReportAnalysis[](https://ai.pydantic.dev/api/pydantic_evals/reporting/#pydantic_evals.reporting.ReportAnalysis "ReportAnalysis



      module-attribute
   \(pydantic_evals.reporting.analyses.ReportAnalysis\)")]

```

Evaluate, handling both sync and async implementations.
Source code in `pydantic_evals/pydantic_evals/evaluators/report_evaluator.py`
```
56
57
58
59
60
61
62
63
```
| ```
async def evaluate_async(
    self, ctx: ReportEvaluatorContext[InputsT, OutputT, MetadataT]
) -> ReportAnalysis | list[ReportAnalysis]:
    """Evaluate, handling both sync and async implementations."""
    output = self.evaluate(ctx)
    if inspect.iscoroutine(output):
        return await output
    return cast('ReportAnalysis | list[ReportAnalysis]', output)

```

---|---
###  ReportEvaluatorContext `dataclass`
Bases: `Generic[](https://docs.python.org/3/library/typing.html#typing.Generic "typing.Generic")[InputsT, OutputT, MetadataT]`
Context for report-level evaluation, containing the full experiment results.
Source code in `pydantic_evals/pydantic_evals/evaluators/report_evaluator.py`
```
27
28
29
30
31
32
33
34
35
36
```
| ```
@dataclass(kw_only=True)
class ReportEvaluatorContext(Generic[InputsT, OutputT, MetadataT]):
    """Context for report-level evaluation, containing the full experiment results."""

    name: str
    """The experiment name."""
    report: EvaluationReport[InputsT, OutputT, MetadataT]
    """The full evaluation report."""
    experiment_metadata: dict[str, Any] | None
    """Experiment-level metadata."""

```

---|---
####  name `instance-attribute`
```
name: str[](https://docs.python.org/3/library/stdtypes.html#str)

```

The experiment name.
####  report `instance-attribute`
```
report: EvaluationReport[](https://ai.pydantic.dev/api/pydantic_evals/reporting/#pydantic_evals.reporting.EvaluationReport "EvaluationReport



      dataclass
   \(pydantic_evals.reporting.EvaluationReport\)")[InputsT, OutputT, MetadataT]

```

The full evaluation report.
####  experiment_metadata `instance-attribute`
```
experiment_metadata: dict[](https://docs.python.org/3/library/stdtypes.html#dict)[str[](https://docs.python.org/3/library/stdtypes.html#str), Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any")] | None

```

Experiment-level metadata.
###  GradingOutput
Bases: `BaseModel[](https://docs.pydantic.dev/latest/api/base_model/#pydantic.BaseModel "pydantic.BaseModel")`
The output of a grading operation.
Source code in `pydantic_evals/pydantic_evals/evaluators/llm_as_a_judge.py`
```
27
28
29
30
31
32
```
| ```
class GradingOutput(BaseModel, populate_by_name=True):
    """The output of a grading operation."""

    reason: str
    pass_: bool = Field(validation_alias='pass', serialization_alias='pass')
    score: float

```

---|---
###  judge_output `async`
```
judge_output(
    output: Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any"),
    rubric: str[](https://docs.python.org/3/library/stdtypes.html#str),
    model: Model[](https://ai.pydantic.dev/api/models/base/#pydantic_ai.models.Model "Model \(pydantic_ai.models.Model\)") | KnownModelName[](https://ai.pydantic.dev/api/models/base/#pydantic_ai.models.KnownModelName "KnownModelName



      module-attribute
   \(pydantic_ai.models.KnownModelName\)") | str[](https://docs.python.org/3/library/stdtypes.html#str) | None = None,
    model_settings: ModelSettings[](https://ai.pydantic.dev/api/settings/#pydantic_ai.settings.ModelSettings "ModelSettings \(pydantic_ai.settings.ModelSettings\)") | None = None,
) -> GradingOutput[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.llm_as_a_judge.GradingOutput "GradingOutput \(pydantic_evals.evaluators.llm_as_a_judge.GradingOutput\)")

```

Judge the output of a model based on a rubric.
If the model is not specified, a default model is used. The default model starts as 'openai:gpt-5.2', but this can be changed using the `set_default_judge_model` function.
Source code in `pydantic_evals/pydantic_evals/evaluators/llm_as_a_judge.py`
```
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
```
| ```
async def judge_output(
    output: Any,
    rubric: str,
    model: models.Model | models.KnownModelName | str | None = None,
    model_settings: ModelSettings | None = None,
) -> GradingOutput:
    """Judge the output of a model based on a rubric.

    If the model is not specified, a default model is used. The default model starts as 'openai:gpt-5.2',
    but this can be changed using the `set_default_judge_model` function.
    """
    user_prompt = _build_prompt(output=output, rubric=rubric)
    return (
        await _judge_output_agent.run(user_prompt, model=model or _default_model, model_settings=model_settings)
    ).output

```

---|---
###  judge_input_output `async`
```
judge_input_output(
    inputs: Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any"),
    output: Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any"),
    rubric: str[](https://docs.python.org/3/library/stdtypes.html#str),
    model: Model[](https://ai.pydantic.dev/api/models/base/#pydantic_ai.models.Model "Model \(pydantic_ai.models.Model\)") | KnownModelName[](https://ai.pydantic.dev/api/models/base/#pydantic_ai.models.KnownModelName "KnownModelName



      module-attribute
   \(pydantic_ai.models.KnownModelName\)") | str[](https://docs.python.org/3/library/stdtypes.html#str) | None = None,
    model_settings: ModelSettings[](https://ai.pydantic.dev/api/settings/#pydantic_ai.settings.ModelSettings "ModelSettings \(pydantic_ai.settings.ModelSettings\)") | None = None,
) -> GradingOutput[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.llm_as_a_judge.GradingOutput "GradingOutput \(pydantic_evals.evaluators.llm_as_a_judge.GradingOutput\)")

```

Judge the output of a model based on the inputs and a rubric.
If the model is not specified, a default model is used. The default model starts as 'openai:gpt-5.2', but this can be changed using the `set_default_judge_model` function.
Source code in `pydantic_evals/pydantic_evals/evaluators/llm_as_a_judge.py`
```
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
```
| ```
async def judge_input_output(
    inputs: Any,
    output: Any,
    rubric: str,
    model: models.Model | models.KnownModelName | str | None = None,
    model_settings: ModelSettings | None = None,
) -> GradingOutput:
    """Judge the output of a model based on the inputs and a rubric.

    If the model is not specified, a default model is used. The default model starts as 'openai:gpt-5.2',
    but this can be changed using the `set_default_judge_model` function.
    """
    user_prompt = _build_prompt(inputs=inputs, output=output, rubric=rubric)

    return (
        await _judge_input_output_agent.run(user_prompt, model=model or _default_model, model_settings=model_settings)
    ).output

```

---|---
###  judge_input_output_expected `async`
```
judge_input_output_expected(
    inputs: Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any"),
    output: Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any"),
    expected_output: Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any"),
    rubric: str[](https://docs.python.org/3/library/stdtypes.html#str),
    model: Model[](https://ai.pydantic.dev/api/models/base/#pydantic_ai.models.Model "Model \(pydantic_ai.models.Model\)") | KnownModelName[](https://ai.pydantic.dev/api/models/base/#pydantic_ai.models.KnownModelName "KnownModelName



      module-attribute
   \(pydantic_ai.models.KnownModelName\)") | str[](https://docs.python.org/3/library/stdtypes.html#str) | None = None,
    model_settings: ModelSettings[](https://ai.pydantic.dev/api/settings/#pydantic_ai.settings.ModelSettings "ModelSettings \(pydantic_ai.settings.ModelSettings\)") | None = None,
) -> GradingOutput[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.llm_as_a_judge.GradingOutput "GradingOutput \(pydantic_evals.evaluators.llm_as_a_judge.GradingOutput\)")

```

Judge the output of a model based on the inputs and a rubric.
If the model is not specified, a default model is used. The default model starts as 'openai:gpt-5.2', but this can be changed using the `set_default_judge_model` function.
Source code in `pydantic_evals/pydantic_evals/evaluators/llm_as_a_judge.py`
```
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
```
| ```
async def judge_input_output_expected(
    inputs: Any,
    output: Any,
    expected_output: Any,
    rubric: str,
    model: models.Model | models.KnownModelName | str | None = None,
    model_settings: ModelSettings | None = None,
) -> GradingOutput:
    """Judge the output of a model based on the inputs and a rubric.

    If the model is not specified, a default model is used. The default model starts as 'openai:gpt-5.2',
    but this can be changed using the `set_default_judge_model` function.
    """
    user_prompt = _build_prompt(inputs=inputs, output=output, rubric=rubric, expected_output=expected_output)

    return (
        await _judge_input_output_expected_agent.run(
            user_prompt, model=model or _default_model, model_settings=model_settings
        )
    ).output

```

---|---
###  judge_output_expected `async`
```
judge_output_expected(
    output: Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any"),
    expected_output: Any[](https://docs.python.org/3/library/typing.html#typing.Any "typing.Any"),
    rubric: str[](https://docs.python.org/3/library/stdtypes.html#str),
    model: Model[](https://ai.pydantic.dev/api/models/base/#pydantic_ai.models.Model "Model \(pydantic_ai.models.Model\)") | KnownModelName[](https://ai.pydantic.dev/api/models/base/#pydantic_ai.models.KnownModelName "KnownModelName



      module-attribute
   \(pydantic_ai.models.KnownModelName\)") | str[](https://docs.python.org/3/library/stdtypes.html#str) | None = None,
    model_settings: ModelSettings[](https://ai.pydantic.dev/api/settings/#pydantic_ai.settings.ModelSettings "ModelSettings \(pydantic_ai.settings.ModelSettings\)") | None = None,
) -> GradingOutput[](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.llm_as_a_judge.GradingOutput "GradingOutput \(pydantic_evals.evaluators.llm_as_a_judge.GradingOutput\)")

```

Judge the output of a model based on the expected output, output, and a rubric.
If the model is not specified, a default model is used. The default model starts as 'openai:gpt-5.2', but this can be changed using the `set_default_judge_model` function.
Source code in `pydantic_evals/pydantic_evals/evaluators/llm_as_a_judge.py`
```
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
```
| ```
async def judge_output_expected(
    output: Any,
    expected_output: Any,
    rubric: str,
    model: models.Model | models.KnownModelName | str | None = None,
    model_settings: ModelSettings | None = None,
) -> GradingOutput:
    """Judge the output of a model based on the expected output, output, and a rubric.

    If the model is not specified, a default model is used. The default model starts as 'openai:gpt-5.2',
    but this can be changed using the `set_default_judge_model` function.
    """
    user_prompt = _build_prompt(output=output, rubric=rubric, expected_output=expected_output)
    return (
        await _judge_output_expected_agent.run(
            user_prompt, model=model or _default_model, model_settings=model_settings
        )
    ).output

```

---|---
###  set_default_judge_model
```
set_default_judge_model(
    model: Model[](https://ai.pydantic.dev/api/models/base/#pydantic_ai.models.Model "Model \(pydantic_ai.models.Model\)") | KnownModelName[](https://ai.pydantic.dev/api/models/base/#pydantic_ai.models.KnownModelName "KnownModelName



      module-attribute
   \(pydantic_ai.models.KnownModelName\)"),
) -> None

```

Set the default model used for judging.
This model is used if `None` is passed to the `model` argument of `judge_output` and `judge_input_output`.
Source code in `pydantic_evals/pydantic_evals/evaluators/llm_as_a_judge.py`
```
205
206
207
208
209
210
211
```
| ```
def set_default_judge_model(model: models.Model | models.KnownModelName) -> None:
    """Set the default model used for judging.

    This model is used if `None` is passed to the `model` argument of `judge_output` and `judge_input_output`.
    """
    global _default_model
    _default_model = model

```

---|---
© Pydantic Services Inc. 2024 to present
