[ Skip to content ](https://ai.pydantic.dev/evals/evaluators/span-based/#span-based-evaluation)
**Join us at the inaugural PyAI Conf in San Francisco on March 10th![Learn More](https://pyai.events/?utm_source=pydantic-ai) **
[ ![logo](https://ai.pydantic.dev/img/logo-white.svg) ](https://ai.pydantic.dev/ "Pydantic AI")
Pydantic AI
Span-Based
Type to start searching
[ pydantic/pydantic-ai
  * v1.63.0
  * 15.1k
  * 1.7k

](https://github.com/pydantic/pydantic-ai "Go to repository")
[ ![logo](https://ai.pydantic.dev/img/logo-white.svg) ](https://ai.pydantic.dev/ "Pydantic AI") Pydantic AI
[ pydantic/pydantic-ai
  * v1.63.0
  * 15.1k
  * 1.7k

](https://github.com/pydantic/pydantic-ai "Go to repository")
  * [ Pydantic AI  ](https://ai.pydantic.dev/)
  * [ Installation  ](https://ai.pydantic.dev/install/)
  * [ Getting Help  ](https://ai.pydantic.dev/help/)
  * [ Troubleshooting  ](https://ai.pydantic.dev/troubleshooting/)
  * [ Pydantic AI Gateway  ](https://ai.pydantic.dev/gateway/)
  * Documentation
    * Core Concepts
      * [ Agents  ](https://ai.pydantic.dev/agent/)
      * [ Dependencies  ](https://ai.pydantic.dev/dependencies/)
      * [ Function Tools  ](https://ai.pydantic.dev/tools/)
      * [ Output  ](https://ai.pydantic.dev/output/)
      * [ Messages and chat history  ](https://ai.pydantic.dev/message-history/)
      * [ Direct Model Requests  ](https://ai.pydantic.dev/direct/)
    * Models & Providers
      * [ Overview  ](https://ai.pydantic.dev/models/overview/)
      * [ OpenAI  ](https://ai.pydantic.dev/models/openai/)
      * [ Anthropic  ](https://ai.pydantic.dev/models/anthropic/)
      * [ Google  ](https://ai.pydantic.dev/models/google/)
      * [ xAI  ](https://ai.pydantic.dev/models/xai/)
      * [ Bedrock  ](https://ai.pydantic.dev/models/bedrock/)
      * [ Cerebras  ](https://ai.pydantic.dev/models/cerebras/)
      * [ Cohere  ](https://ai.pydantic.dev/models/cohere/)
      * [ Groq  ](https://ai.pydantic.dev/models/groq/)
      * [ Hugging Face  ](https://ai.pydantic.dev/models/huggingface/)
      * [ Mistral  ](https://ai.pydantic.dev/models/mistral/)
      * [ OpenRouter  ](https://ai.pydantic.dev/models/openrouter/)
      * [ Outlines  ](https://ai.pydantic.dev/models/outlines/)
    * Tools & Toolsets
      * [ Function Tools  ](https://ai.pydantic.dev/tools/)
      * [ Advanced Tool Features  ](https://ai.pydantic.dev/tools-advanced/)
      * [ Toolsets  ](https://ai.pydantic.dev/toolsets/)
      * [ Deferred Tools  ](https://ai.pydantic.dev/deferred-tools/)
      * [ Built-in Tools  ](https://ai.pydantic.dev/builtin-tools/)
      * [ Common Tools  ](https://ai.pydantic.dev/common-tools/)
      * [ Third-Party Tools  ](https://ai.pydantic.dev/third-party-tools/)
    * Advanced Features
      * [ Image, Audio, Video & Document Input  ](https://ai.pydantic.dev/input/)
      * [ Thinking  ](https://ai.pydantic.dev/thinking/)
      * [ HTTP Request Retries  ](https://ai.pydantic.dev/retries/)
    * MCP
      * [ Overview  ](https://ai.pydantic.dev/mcp/overview/)
      * [ Client  ](https://ai.pydantic.dev/mcp/client/)
      * [ FastMCP Client  ](https://ai.pydantic.dev/mcp/fastmcp-client/)
      * [ Server  ](https://ai.pydantic.dev/mcp/server/)
    * [ Multi-Agent Patterns  ](https://ai.pydantic.dev/multi-agent-applications/)
    * [ Web Chat UI  ](https://ai.pydantic.dev/web/)
    * [ Embeddings  ](https://ai.pydantic.dev/embeddings/)
    * [ Testing  ](https://ai.pydantic.dev/testing/)
  * Pydantic Evals
    * [ Overview  ](https://ai.pydantic.dev/evals/)
    * Getting Started
      * [ Quick Start  ](https://ai.pydantic.dev/evals/quick-start/)
      * [ Core Concepts  ](https://ai.pydantic.dev/evals/core-concepts/)
    * Evaluators
      * [ Overview  ](https://ai.pydantic.dev/evals/evaluators/overview/)
      * [ Built-in Evaluators  ](https://ai.pydantic.dev/evals/evaluators/built-in/)
      * [ LLM Judge  ](https://ai.pydantic.dev/evals/evaluators/llm-judge/)
      * [ Custom Evaluators  ](https://ai.pydantic.dev/evals/evaluators/custom/)
      * [ Report Evaluators  ](https://ai.pydantic.dev/evals/evaluators/report-evaluators/)
      * Span-Based  [ Span-Based  ](https://ai.pydantic.dev/evals/evaluators/span-based/)
        * [ Overview  ](https://ai.pydantic.dev/evals/evaluators/span-based/#overview)
          * [ Why Span-Based Evaluation?  ](https://ai.pydantic.dev/evals/evaluators/span-based/#why-span-based-evaluation)
          * [ Real-World Scenarios  ](https://ai.pydantic.dev/evals/evaluators/span-based/#real-world-scenarios)
          * [ How It Works  ](https://ai.pydantic.dev/evals/evaluators/span-based/#how-it-works)
        * [ Basic Usage  ](https://ai.pydantic.dev/evals/evaluators/span-based/#basic-usage)
        * [ HasMatchingSpan Evaluator  ](https://ai.pydantic.dev/evals/evaluators/span-based/#hasmatchingspan-evaluator)
        * [ SpanQuery Reference  ](https://ai.pydantic.dev/evals/evaluators/span-based/#spanquery-reference)
          * [ Name Conditions  ](https://ai.pydantic.dev/evals/evaluators/span-based/#name-conditions)
          * [ Attribute Conditions  ](https://ai.pydantic.dev/evals/evaluators/span-based/#attribute-conditions)
          * [ Duration Conditions  ](https://ai.pydantic.dev/evals/evaluators/span-based/#duration-conditions)
          * [ Logical Operators  ](https://ai.pydantic.dev/evals/evaluators/span-based/#logical-operators)
          * [ Child/Descendant Conditions  ](https://ai.pydantic.dev/evals/evaluators/span-based/#childdescendant-conditions)
          * [ Ancestor/Depth Conditions  ](https://ai.pydantic.dev/evals/evaluators/span-based/#ancestordepth-conditions)
          * [ Stop Recursing  ](https://ai.pydantic.dev/evals/evaluators/span-based/#stop-recursing)
        * [ Practical Examples  ](https://ai.pydantic.dev/evals/evaluators/span-based/#practical-examples)
          * [ Verify Tool Usage  ](https://ai.pydantic.dev/evals/evaluators/span-based/#verify-tool-usage)
          * [ Check Multiple Tools  ](https://ai.pydantic.dev/evals/evaluators/span-based/#check-multiple-tools)
          * [ Performance Assertions  ](https://ai.pydantic.dev/evals/evaluators/span-based/#performance-assertions)
          * [ Error Detection  ](https://ai.pydantic.dev/evals/evaluators/span-based/#error-detection)
          * [ Complex Behavioral Checks  ](https://ai.pydantic.dev/evals/evaluators/span-based/#complex-behavioral-checks)
        * [ Custom Evaluators with SpanTree  ](https://ai.pydantic.dev/evals/evaluators/span-based/#custom-evaluators-with-spantree)
          * [ SpanTree API  ](https://ai.pydantic.dev/evals/evaluators/span-based/#spantree-api)
          * [ SpanNode Properties  ](https://ai.pydantic.dev/evals/evaluators/span-based/#spannode-properties)
        * [ Debugging Span Queries  ](https://ai.pydantic.dev/evals/evaluators/span-based/#debugging-span-queries)
          * [ View Spans in Logfire  ](https://ai.pydantic.dev/evals/evaluators/span-based/#view-spans-in-logfire)
          * [ Print Span Tree  ](https://ai.pydantic.dev/evals/evaluators/span-based/#print-span-tree)
          * [ Query Testing  ](https://ai.pydantic.dev/evals/evaluators/span-based/#query-testing)
        * [ Use Cases  ](https://ai.pydantic.dev/evals/evaluators/span-based/#use-cases)
          * [ RAG System Verification  ](https://ai.pydantic.dev/evals/evaluators/span-based/#rag-system-verification)
          * [ Multi-Agent Systems  ](https://ai.pydantic.dev/evals/evaluators/span-based/#multi-agent-systems)
          * [ Tool Usage Patterns  ](https://ai.pydantic.dev/evals/evaluators/span-based/#tool-usage-patterns)
        * [ Best Practices  ](https://ai.pydantic.dev/evals/evaluators/span-based/#best-practices)
        * [ Next Steps  ](https://ai.pydantic.dev/evals/evaluators/span-based/#next-steps)
    * How-To Guides
      * [ Logfire Integration  ](https://ai.pydantic.dev/evals/how-to/logfire-integration/)
      * [ Dataset Management  ](https://ai.pydantic.dev/evals/how-to/dataset-management/)
      * [ Dataset Serialization  ](https://ai.pydantic.dev/evals/how-to/dataset-serialization/)
      * [ Concurrency & Performance  ](https://ai.pydantic.dev/evals/how-to/concurrency/)
      * [ Multi-Run Evaluation  ](https://ai.pydantic.dev/evals/how-to/multi-run/)
      * [ Retry Strategies  ](https://ai.pydantic.dev/evals/how-to/retry-strategies/)
      * [ Metrics & Attributes  ](https://ai.pydantic.dev/evals/how-to/metrics-attributes/)
    * Examples
      * [ Simple Validation  ](https://ai.pydantic.dev/evals/examples/simple-validation/)
  * Pydantic Graph
    * [ Overview  ](https://ai.pydantic.dev/graph/)
    * [ Beta API  ](https://ai.pydantic.dev/graph/beta/)
      * [ Steps  ](https://ai.pydantic.dev/graph/beta/steps/)
      * [ Joins & Reducers  ](https://ai.pydantic.dev/graph/beta/joins/)
      * [ Decisions  ](https://ai.pydantic.dev/graph/beta/decisions/)
      * [ Parallel Execution  ](https://ai.pydantic.dev/graph/beta/parallel/)
  * Integrations
    * [ Debugging & Monitoring with Pydantic Logfire  ](https://ai.pydantic.dev/logfire/)
    * Durable Execution
      * [ Overview  ](https://ai.pydantic.dev/durable_execution/overview/)
      * [ Temporal  ](https://ai.pydantic.dev/durable_execution/temporal/)
      * [ DBOS  ](https://ai.pydantic.dev/durable_execution/dbos/)
      * [ Prefect  ](https://ai.pydantic.dev/durable_execution/prefect/)
    * UI Event Streams
      * [ Overview  ](https://ai.pydantic.dev/ui/overview/)
      * [ AG-UI  ](https://ai.pydantic.dev/ui/ag-ui/)
      * [ Vercel AI  ](https://ai.pydantic.dev/ui/vercel-ai/)
    * [ Agent2Agent (A2A)  ](https://ai.pydantic.dev/a2a/)
  * Related Packages
    * [ clai  ](https://ai.pydantic.dev/cli/)
  * Examples
    * [ Setup  ](https://ai.pydantic.dev/examples/setup/)
    * Getting Started
      * [ Pydantic Model  ](https://ai.pydantic.dev/examples/pydantic-model/)
      * [ Weather agent  ](https://ai.pydantic.dev/examples/weather-agent/)
    * Conversational Agents
      * [ Chat App with FastAPI  ](https://ai.pydantic.dev/examples/chat-app/)
      * [ Bank support  ](https://ai.pydantic.dev/examples/bank-support/)
    * Data & Analytics
      * [ SQL Generation  ](https://ai.pydantic.dev/examples/sql-gen/)
      * [ Data Analyst  ](https://ai.pydantic.dev/examples/data-analyst/)
      * [ RAG  ](https://ai.pydantic.dev/examples/rag/)
    * Streaming
      * [ Stream markdown  ](https://ai.pydantic.dev/examples/stream-markdown/)
      * [ Stream whales  ](https://ai.pydantic.dev/examples/stream-whales/)
    * Complex Workflows
      * [ Flight booking  ](https://ai.pydantic.dev/examples/flight-booking/)
      * [ Question Graph  ](https://ai.pydantic.dev/examples/question-graph/)
    * Business Applications
      * [ Slack Lead Qualifier with Modal  ](https://ai.pydantic.dev/examples/slack-lead-qualifier/)
    * UI Examples
      * [ Agent User Interaction (AG-UI)  ](https://ai.pydantic.dev/examples/ag-ui/)
  * API Reference
    * pydantic_ai
      * [ pydantic_ai.ag_ui  ](https://ai.pydantic.dev/api/ag_ui/)
      * [ pydantic_ai.agent  ](https://ai.pydantic.dev/api/agent/)
      * [ pydantic_ai.builtin_tools  ](https://ai.pydantic.dev/api/builtin_tools/)
      * [ pydantic_ai.common_tools  ](https://ai.pydantic.dev/api/common_tools/)
      * [ pydantic_ai — Concurrency  ](https://ai.pydantic.dev/api/concurrency/)
      * [ pydantic_ai.direct  ](https://ai.pydantic.dev/api/direct/)
      * [ pydantic_ai.durable_exec  ](https://ai.pydantic.dev/api/durable_exec/)
      * [ pydantic_ai.embeddings  ](https://ai.pydantic.dev/api/embeddings/)
      * [ pydantic_ai.exceptions  ](https://ai.pydantic.dev/api/exceptions/)
      * [ pydantic_ai.ext  ](https://ai.pydantic.dev/api/ext/)
      * [ pydantic_ai.format_prompt  ](https://ai.pydantic.dev/api/format_prompt/)
      * [ pydantic_ai.mcp  ](https://ai.pydantic.dev/api/mcp/)
      * [ pydantic_ai.messages  ](https://ai.pydantic.dev/api/messages/)
      * [ pydantic_ai.models.anthropic  ](https://ai.pydantic.dev/api/models/anthropic/)
      * [ pydantic_ai.models  ](https://ai.pydantic.dev/api/models/base/)
      * [ pydantic_ai.models.bedrock  ](https://ai.pydantic.dev/api/models/bedrock/)
      * [ pydantic_ai.models.cerebras  ](https://ai.pydantic.dev/api/models/cerebras/)
      * [ pydantic_ai.models.cohere  ](https://ai.pydantic.dev/api/models/cohere/)
      * [ pydantic_ai.models.fallback  ](https://ai.pydantic.dev/api/models/fallback/)
      * [ pydantic_ai.models.function  ](https://ai.pydantic.dev/api/models/function/)
      * [ pydantic_ai.models.google  ](https://ai.pydantic.dev/api/models/google/)
      * [ pydantic_ai.models.xai  ](https://ai.pydantic.dev/api/models/xai/)
      * [ pydantic_ai.models.groq  ](https://ai.pydantic.dev/api/models/groq/)
      * [ pydantic_ai.models.huggingface  ](https://ai.pydantic.dev/api/models/huggingface/)
      * [ pydantic_ai.models.instrumented  ](https://ai.pydantic.dev/api/models/instrumented/)
      * [ pydantic_ai.models.mcp_sampling  ](https://ai.pydantic.dev/api/models/mcp-sampling/)
      * [ pydantic_ai.models.mistral  ](https://ai.pydantic.dev/api/models/mistral/)
      * [ pydantic_ai.models.openai  ](https://ai.pydantic.dev/api/models/openai/)
      * [ pydantic_ai.models.openrouter  ](https://ai.pydantic.dev/api/models/openrouter/)
      * [ pydantic_ai.models.outlines  ](https://ai.pydantic.dev/api/models/outlines/)
      * [ pydantic_ai.models.test  ](https://ai.pydantic.dev/api/models/test/)
      * [ pydantic_ai.models.wrapper  ](https://ai.pydantic.dev/api/models/wrapper/)
      * [ pydantic_ai.output  ](https://ai.pydantic.dev/api/output/)
      * [ pydantic_ai.profiles  ](https://ai.pydantic.dev/api/profiles/)
      * [ pydantic_ai.providers  ](https://ai.pydantic.dev/api/providers/)
      * [ pydantic_ai.result  ](https://ai.pydantic.dev/api/result/)
      * [ pydantic_ai.retries  ](https://ai.pydantic.dev/api/retries/)
      * [ pydantic_ai.run  ](https://ai.pydantic.dev/api/run/)
      * [ pydantic_ai.settings  ](https://ai.pydantic.dev/api/settings/)
      * [ pydantic_ai.tools  ](https://ai.pydantic.dev/api/tools/)
      * [ pydantic_ai.toolsets  ](https://ai.pydantic.dev/api/toolsets/)
      * [ pydantic_ai.ui.ag_ui  ](https://ai.pydantic.dev/api/ui/ag_ui/)
      * [ pydantic_ai.ui  ](https://ai.pydantic.dev/api/ui/base/)
      * [ pydantic_ai.ui.vercel_ai  ](https://ai.pydantic.dev/api/ui/vercel_ai/)
      * [ pydantic_ai.usage  ](https://ai.pydantic.dev/api/usage/)
    * pydantic_evals
      * [ pydantic_evals.dataset  ](https://ai.pydantic.dev/api/pydantic_evals/dataset/)
      * [ pydantic_evals.evaluators  ](https://ai.pydantic.dev/api/pydantic_evals/evaluators/)
      * [ pydantic_evals.reporting  ](https://ai.pydantic.dev/api/pydantic_evals/reporting/)
      * [ pydantic_evals.otel  ](https://ai.pydantic.dev/api/pydantic_evals/otel/)
      * [ pydantic_evals.generation  ](https://ai.pydantic.dev/api/pydantic_evals/generation/)
    * pydantic_graph
      * [ pydantic_graph  ](https://ai.pydantic.dev/api/pydantic_graph/graph/)
      * [ pydantic_graph.nodes  ](https://ai.pydantic.dev/api/pydantic_graph/nodes/)
      * [ pydantic_graph.persistence  ](https://ai.pydantic.dev/api/pydantic_graph/persistence/)
      * [ pydantic_graph.mermaid  ](https://ai.pydantic.dev/api/pydantic_graph/mermaid/)
      * [ pydantic_graph.exceptions  ](https://ai.pydantic.dev/api/pydantic_graph/exceptions/)
      * Beta API
        * [ pydantic_graph.beta  ](https://ai.pydantic.dev/api/pydantic_graph/beta/)
        * [ pydantic_graph.beta.graph  ](https://ai.pydantic.dev/api/pydantic_graph/beta_graph/)
        * [ pydantic_graph.beta.graph_builder  ](https://ai.pydantic.dev/api/pydantic_graph/beta_graph_builder/)
        * [ pydantic_graph.beta.step  ](https://ai.pydantic.dev/api/pydantic_graph/beta_step/)
        * [ pydantic_graph.beta.join  ](https://ai.pydantic.dev/api/pydantic_graph/beta_join/)
        * [ pydantic_graph.beta.decision  ](https://ai.pydantic.dev/api/pydantic_graph/beta_decision/)
        * [ pydantic_graph.beta.node  ](https://ai.pydantic.dev/api/pydantic_graph/beta_node/)
    * fasta2a
      * [ fasta2a  ](https://ai.pydantic.dev/api/fasta2a/)
  * Project
    * [ Contributing  ](https://ai.pydantic.dev/contributing/)
    * [ Upgrade Guide  ](https://ai.pydantic.dev/changelog/)
    * [ Version policy  ](https://ai.pydantic.dev/version-policy/)


  * [ Overview  ](https://ai.pydantic.dev/evals/evaluators/span-based/#overview)
    * [ Why Span-Based Evaluation?  ](https://ai.pydantic.dev/evals/evaluators/span-based/#why-span-based-evaluation)
    * [ Real-World Scenarios  ](https://ai.pydantic.dev/evals/evaluators/span-based/#real-world-scenarios)
    * [ How It Works  ](https://ai.pydantic.dev/evals/evaluators/span-based/#how-it-works)
  * [ Basic Usage  ](https://ai.pydantic.dev/evals/evaluators/span-based/#basic-usage)
  * [ HasMatchingSpan Evaluator  ](https://ai.pydantic.dev/evals/evaluators/span-based/#hasmatchingspan-evaluator)
  * [ SpanQuery Reference  ](https://ai.pydantic.dev/evals/evaluators/span-based/#spanquery-reference)
    * [ Name Conditions  ](https://ai.pydantic.dev/evals/evaluators/span-based/#name-conditions)
    * [ Attribute Conditions  ](https://ai.pydantic.dev/evals/evaluators/span-based/#attribute-conditions)
    * [ Duration Conditions  ](https://ai.pydantic.dev/evals/evaluators/span-based/#duration-conditions)
    * [ Logical Operators  ](https://ai.pydantic.dev/evals/evaluators/span-based/#logical-operators)
    * [ Child/Descendant Conditions  ](https://ai.pydantic.dev/evals/evaluators/span-based/#childdescendant-conditions)
    * [ Ancestor/Depth Conditions  ](https://ai.pydantic.dev/evals/evaluators/span-based/#ancestordepth-conditions)
    * [ Stop Recursing  ](https://ai.pydantic.dev/evals/evaluators/span-based/#stop-recursing)
  * [ Practical Examples  ](https://ai.pydantic.dev/evals/evaluators/span-based/#practical-examples)
    * [ Verify Tool Usage  ](https://ai.pydantic.dev/evals/evaluators/span-based/#verify-tool-usage)
    * [ Check Multiple Tools  ](https://ai.pydantic.dev/evals/evaluators/span-based/#check-multiple-tools)
    * [ Performance Assertions  ](https://ai.pydantic.dev/evals/evaluators/span-based/#performance-assertions)
    * [ Error Detection  ](https://ai.pydantic.dev/evals/evaluators/span-based/#error-detection)
    * [ Complex Behavioral Checks  ](https://ai.pydantic.dev/evals/evaluators/span-based/#complex-behavioral-checks)
  * [ Custom Evaluators with SpanTree  ](https://ai.pydantic.dev/evals/evaluators/span-based/#custom-evaluators-with-spantree)
    * [ SpanTree API  ](https://ai.pydantic.dev/evals/evaluators/span-based/#spantree-api)
    * [ SpanNode Properties  ](https://ai.pydantic.dev/evals/evaluators/span-based/#spannode-properties)
  * [ Debugging Span Queries  ](https://ai.pydantic.dev/evals/evaluators/span-based/#debugging-span-queries)
    * [ View Spans in Logfire  ](https://ai.pydantic.dev/evals/evaluators/span-based/#view-spans-in-logfire)
    * [ Print Span Tree  ](https://ai.pydantic.dev/evals/evaluators/span-based/#print-span-tree)
    * [ Query Testing  ](https://ai.pydantic.dev/evals/evaluators/span-based/#query-testing)
  * [ Use Cases  ](https://ai.pydantic.dev/evals/evaluators/span-based/#use-cases)
    * [ RAG System Verification  ](https://ai.pydantic.dev/evals/evaluators/span-based/#rag-system-verification)
    * [ Multi-Agent Systems  ](https://ai.pydantic.dev/evals/evaluators/span-based/#multi-agent-systems)
    * [ Tool Usage Patterns  ](https://ai.pydantic.dev/evals/evaluators/span-based/#tool-usage-patterns)
  * [ Best Practices  ](https://ai.pydantic.dev/evals/evaluators/span-based/#best-practices)
  * [ Next Steps  ](https://ai.pydantic.dev/evals/evaluators/span-based/#next-steps)


  1. [ Pydantic AI  ](https://ai.pydantic.dev/)
  2. [ Pydantic Evals  ](https://ai.pydantic.dev/evals/)
  3. [ Evaluators  ](https://ai.pydantic.dev/evals/evaluators/overview/)


# Span-Based Evaluation
Evaluate AI system behavior by analyzing OpenTelemetry spans captured during execution.
Requires Logfire
Span-based evaluation requires `logfire` to be installed and configured:
```
pip install 'pydantic-evals[logfire]'

```

## Overview
Span-based evaluation enables you to evaluate **how** your AI system executes, not just **what** it produces. This is essential for complex agents where ensuring the desired behavior depends on the execution path taken, not just the final output.
### Why Span-Based Evaluation?
Traditional evaluators assess task inputs and outputs. For simple tasks, this may be sufficient—if the output is correct, the task succeeded. But for complex multi-step agents, the _process_ matters as much as the result:
  * **A correct answer reached incorrectly** - An agent might produce the right output by accident (e.g., guessing, using cached data when it should have searched, calling the wrong tools but getting lucky)
  * **Verification of required behaviors** - You need to ensure specific tools were called, certain code paths executed, or particular patterns followed
  * **Performance and efficiency** - The agent should reach the answer efficiently, without unnecessary tool calls, infinite loops, or excessive retries
  * **Safety and compliance** - Critical to verify that dangerous operations weren't attempted, sensitive data wasn't accessed inappropriately, or guardrails weren't bypassed


### Real-World Scenarios
Span-based evaluation is particularly valuable for:
  * **RAG systems** - Verify documents were retrieved and reranked before generation, not just that the answer included citations
  * **Multi-agent coordination** - Ensure the orchestrator delegated to the right specialist agents in the correct order
  * **Tool-calling agents** - Confirm specific tools were used (or avoided), and in the expected sequence
  * **Debugging and regression testing** - Catch behavioral regressions where outputs remain correct but the internal logic deteriorates
  * **Production alignment** - Ensure your evaluation assertions operate on the same telemetry data captured in production, so eval insights directly translate to production monitoring


### How It Works
When you configure logfire (`logfire.configure()`), Pydantic Evals captures all OpenTelemetry spans generated during task execution. You can then write evaluators that assert conditions on:
  * **Which tools were called** - `HasMatchingSpan(query={'name_contains': 'search_tool'})`
  * **Code paths executed** - Verify specific functions ran or particular branches taken
  * **Timing characteristics** - Check that operations complete within SLA bounds
  * **Error conditions** - Detect retries, fallbacks, or specific failure modes
  * **Execution structure** - Verify parent-child relationships, delegation patterns, or execution order


This creates a fundamentally different evaluation paradigm: you're testing behavioral contracts, not just input-output relationships.
## Basic Usage
```
import logfire

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import HasMatchingSpan

# Configure logfire to capture spans
logfire.configure(send_to_logfire='if-token-present')

dataset = Dataset(
    cases=[Case(inputs='test')],
    evaluators=[
        # Check that database was queried
        HasMatchingSpan(
            query={'name_contains': 'database_query'},
            evaluation_name='used_database',
        ),
    ],
)

```

## HasMatchingSpan Evaluator
The [`HasMatchingSpan`](https://ai.pydantic.dev/api/pydantic_evals/evaluators/#pydantic_evals.evaluators.HasMatchingSpan "HasMatchingSpan



      dataclass
  ") evaluator checks if any span matches a query:
```
from pydantic_evals.evaluators import HasMatchingSpan

HasMatchingSpan(
    query={'name_contains': 'test'},
    evaluation_name='span_check',
)

```

**Returns:** `bool` - `True` if any span matches the query
## SpanQuery Reference
A [`SpanQuery`](https://ai.pydantic.dev/api/pydantic_evals/otel/#pydantic_evals.otel.SpanQuery "SpanQuery") is a dictionary with query conditions:
### Name Conditions
Match spans by name:
```
# Exact name match
{'name_equals': 'search_database'}

# Contains substring
{'name_contains': 'tool_call'}

# Regex pattern
{'name_matches_regex': r'llm_call_\d+'}

```

### Attribute Conditions
Match spans with specific attributes:
```
# Has specific attribute values
{'has_attributes': {'operation': 'search', 'status': 'success'}}

# Has attribute keys (any value)
{'has_attribute_keys': ['user_id', 'request_id']}

```

### Duration Conditions
Match based on execution time:
```
from datetime import timedelta

# Minimum duration
{'min_duration': 1.0}  # seconds
{'min_duration': timedelta(seconds=1)}

# Maximum duration
{'max_duration': 5.0}  # seconds
{'max_duration': timedelta(seconds=5)}

# Range
{'min_duration': 0.5, 'max_duration': 2.0}

```

### Logical Operators
Combine conditions:
```
# NOT
{'not_': {'name_contains': 'error'}}

# AND (all must match)
{'and_': [
    {'name_contains': 'tool'},
    {'max_duration': 1.0},
]}

# OR (any must match)
{'or_': [
    {'name_equals': 'search'},
    {'name_equals': 'query'},
]}

```

### Child/Descendant Conditions
Query relationships between spans:
```
# Count direct children
{'min_child_count': 1}
{'max_child_count': 5}

# Some child matches query
{'some_child_has': {'name_contains': 'retry'}}

# All children match query
{'all_children_have': {'max_duration': 0.5}}

# No children match query
{'no_child_has': {'has_attributes': {'error': True}}}

# Descendant queries (recursive)
{'min_descendant_count': 5}
{'some_descendant_has': {'name_contains': 'api_call'}}

```

### Ancestor/Depth Conditions
Query span hierarchy:
```
# Depth (root spans have depth 0)
{'min_depth': 1}  # Not a root span
{'max_depth': 2}  # At most 2 levels deep

# Ancestor queries
{'some_ancestor_has': {'name_equals': 'agent_run'}}
{'all_ancestors_have': {'max_duration': 10.0}}
{'no_ancestor_has': {'has_attributes': {'error': True}}}

```

### Stop Recursing
Control recursive queries:
```
{
    'some_descendant_has': {'name_contains': 'expensive'},
    'stop_recursing_when': {'name_equals': 'boundary'},
}
# Only search descendants until hitting a span named 'boundary'

```

## Practical Examples
### Verify Tool Usage
Check that specific tools were called:
```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import HasMatchingSpan

dataset = Dataset(
    cases=[Case(inputs='test')],
    evaluators=[
        # Must call search tool
        HasMatchingSpan(
            query={'name_contains': 'search_tool'},
            evaluation_name='used_search',
        ),

        # Must NOT call dangerous tool
        HasMatchingSpan(
            query={'not_': {'name_contains': 'delete_database'}},
            evaluation_name='safe_execution',
        ),
    ],
)

```

### Check Multiple Tools
Verify a sequence of operations:
```
from pydantic_evals.evaluators import HasMatchingSpan

evaluators = [
    HasMatchingSpan(
        query={'name_contains': 'retrieve_context'},
        evaluation_name='retrieved_context',
    ),
    HasMatchingSpan(
        query={'name_contains': 'generate_response'},
        evaluation_name='generated_response',
    ),
    HasMatchingSpan(
        query={'and_': [
            {'name_contains': 'cite'},
            {'has_attribute_keys': ['source_id']},
        ]},
        evaluation_name='added_citations',
    ),
]

```

### Performance Assertions
Ensure operations meet latency requirements:
```
from pydantic_evals.evaluators import HasMatchingSpan

evaluators = [
    # Database queries should be fast
    HasMatchingSpan(
        query={'and_': [
            {'name_contains': 'database'},
            {'max_duration': 0.1},  # 100ms max
        ]},
        evaluation_name='fast_db_queries',
    ),

    # Overall should complete quickly
    HasMatchingSpan(
        query={'and_': [
            {'name_equals': 'task_execution'},
            {'max_duration': 2.0},
        ]},
        evaluation_name='within_sla',
    ),
]

```

### Error Detection
Check for error conditions:
```
from pydantic_evals.evaluators import HasMatchingSpan

evaluators = [
    # No errors occurred
    HasMatchingSpan(
        query={'not_': {'has_attributes': {'error': True}}},
        evaluation_name='no_errors',
    ),

    # Retries happened
    HasMatchingSpan(
        query={'name_contains': 'retry'},
        evaluation_name='had_retries',
    ),

    # Fallback was used
    HasMatchingSpan(
        query={'name_contains': 'fallback_model'},
        evaluation_name='used_fallback',
    ),
]

```

### Complex Behavioral Checks
Verify sophisticated behavior patterns:
```
from pydantic_evals.evaluators import HasMatchingSpan

evaluators = [
    # Agent delegated to sub-agent
    HasMatchingSpan(
        query={'and_': [
            {'name_contains': 'agent'},
            {'some_child_has': {'name_contains': 'delegate'}},
        ]},
        evaluation_name='used_delegation',
    ),

    # Made multiple LLM calls with retries
    HasMatchingSpan(
        query={'and_': [
            {'name_contains': 'llm_call'},
            {'some_descendant_has': {'name_contains': 'retry'}},
            {'min_descendant_count': 3},
        ]},
        evaluation_name='retry_pattern',
    ),
]

```

## Custom Evaluators with SpanTree
For more complex span analysis, write custom evaluators:
```
from dataclasses import dataclass

from pydantic_evals.evaluators import Evaluator, EvaluatorContext


@dataclass
class CustomSpanCheck(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> dict[str, bool | int]:
        span_tree = ctx.span_tree

        # Find specific spans
        llm_spans = span_tree.find(lambda node: 'llm' in node.name)
        tool_spans = span_tree.find(lambda node: 'tool' in node.name)

        # Calculate metrics
        total_llm_time = sum(
            span.duration.total_seconds() for span in llm_spans
        )

        return {
            'used_llm': len(llm_spans) > 0,
            'used_tools': len(tool_spans) > 0,
            'tool_count': len(tool_spans),
            'llm_fast': total_llm_time < 2.0,
        }

```

### SpanTree API
The [`SpanTree`](https://ai.pydantic.dev/api/pydantic_evals/otel/#pydantic_evals.otel.SpanTree "SpanTree



      dataclass
  ") provides methods for span analysis:
```
from pydantic_evals.otel import SpanTree


# Example API (requires span_tree from context)
def example_api(span_tree: SpanTree) -> None:
    span_tree.find(lambda n: True)  # Find all matching nodes
    span_tree.any({'name_contains': 'test'})  # Check if any span matches
    span_tree.all({'name_contains': 'test'})  # Check if all spans match
    span_tree.count({'name_contains': 'test'})  # Count matching spans

    # Iteration
    for node in span_tree:
        print(node.name, node.duration, node.attributes)

```

### SpanNode Properties
Each [`SpanNode`](https://ai.pydantic.dev/api/pydantic_evals/otel/#pydantic_evals.otel.SpanNode "SpanNode



      dataclass
  ") has:
```
from pydantic_evals.otel import SpanNode


# Example properties (requires node from context)
def example_properties(node: SpanNode) -> None:
    _ = node.name  # Span name
    _ = node.duration  # timedelta
    _ = node.attributes  # dict[str, AttributeValue]
    _ = node.start_timestamp  # datetime
    _ = node.end_timestamp  # datetime
    _ = node.children  # list[SpanNode]
    _ = node.descendants  # list[SpanNode] (recursive)
    _ = node.ancestors  # list[SpanNode]
    _ = node.parent  # SpanNode | None

```

## Debugging Span Queries
### View Spans in Logfire
If you're sending data to Logfire, you can view all spans in the web UI to understand the trace structure.
### Print Span Tree
```
from dataclasses import dataclass

from pydantic_evals.evaluators import Evaluator, EvaluatorContext


@dataclass
class DebugSpans(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> bool:
        for node in ctx.span_tree:
            print(f"{'  ' * len(node.ancestors)}{node.name} ({node.duration})")
        return True

```

### Query Testing
Test queries incrementally:
```
from pydantic_evals.evaluators import HasMatchingSpan

# Start simple
query = {'name_contains': 'tool'}

# Add conditions gradually
query = {'and_': [
    {'name_contains': 'tool'},
    {'max_duration': 1.0},
]}

# Test in evaluator
HasMatchingSpan(query=query, evaluation_name='test')

```

## Use Cases
### RAG System Verification
Verify retrieval-augmented generation workflow:
```
from pydantic_evals.evaluators import HasMatchingSpan

evaluators = [
    # Retrieved documents
    HasMatchingSpan(
        query={'name_contains': 'vector_search'},
        evaluation_name='retrieved_docs',
    ),

    # Reranked results
    HasMatchingSpan(
        query={'name_contains': 'rerank'},
        evaluation_name='reranked_results',
    ),

    # Generated with context
    HasMatchingSpan(
        query={'and_': [
            {'name_contains': 'generate'},
            {'has_attribute_keys': ['context_ids']},
        ]},
        evaluation_name='used_context',
    ),
]

```

### Multi-Agent Systems
Verify agent coordination:
```
from pydantic_evals.evaluators import HasMatchingSpan

evaluators = [
    # Master agent ran
    HasMatchingSpan(
        query={'name_equals': 'master_agent'},
        evaluation_name='master_ran',
    ),

    # Delegated to specialist
    HasMatchingSpan(
        query={'and_': [
            {'name_contains': 'specialist_agent'},
            {'some_ancestor_has': {'name_equals': 'master_agent'}},
        ]},
        evaluation_name='delegated_correctly',
    ),

    # No circular delegation
    HasMatchingSpan(
        query={'not_': {'and_': [
            {'name_contains': 'agent'},
            {'some_descendant_has': {'name_contains': 'agent'}},
            {'some_ancestor_has': {'name_contains': 'agent'}},
        ]}},
        evaluation_name='no_circular_delegation',
    ),
]

```

### Tool Usage Patterns
Verify intelligent tool selection:
```
from pydantic_evals.evaluators import HasMatchingSpan

evaluators = [
    # Used search before answering
    HasMatchingSpan(
        query={'and_': [
            {'name_contains': 'search'},
            {'some_ancestor_has': {'name_contains': 'answer'}},
        ]},
        evaluation_name='searched_before_answering',
    ),

    # Limited tool calls (no loops)
    HasMatchingSpan(
        query={'and_': [
            {'name_contains': 'tool'},
            {'max_child_count': 5},
        ]},
        evaluation_name='reasonable_tool_usage',
    ),
]

```

## Best Practices
  1. **Start Simple** : Begin with basic name queries, add complexity as needed
  2. **Use Descriptive Names** : Name your spans well in your application code
  3. **Test Queries** : Verify queries work before running full evaluations
  4. **Combine with Other Evaluators** : Use span checks alongside output validation
  5. **Document Expectations** : Comment why specific spans should/shouldn't exist


## Next Steps
  * **[Logfire Integration](https://ai.pydantic.dev/evals/how-to/logfire-integration/)** - Set up Logfire for span capture
  * **[Custom Evaluators](https://ai.pydantic.dev/evals/evaluators/custom/)** - Write advanced span analysis
  * **[Built-in Evaluators](https://ai.pydantic.dev/evals/evaluators/built-in/)** - Other evaluator types


© Pydantic Services Inc. 2024 to present
