---
title: Tracing
---

## Overview ##

`judgeval`'s tracing module allows you to view your LLM application's execution from **end-to-end**. 

Using tracing, you can:
- Gain observability into **every layer of your agentic system**, from database queries to tool calling and text generation.
- Measure the performance of **each system component in any way** you want to measure it. For instance:
    - Catch regressions in **retrieval quality, factuality, answer relevance**, and 10+ other [**research-backed metrics**](/evaluation/scorers/introduction).
    - Quantify the **quality of each tool call** your agent makes
    - Track the latency of each system component
    - Count the token usage of each LLM generation
- Export your workflow runs to the Judgment platform for **real-time analysis** or as a dataset for [**offline experimentation**](/evaluation/introduction).


## Tracing Your Workflow ##

Setting up tracing with `judgeval` takes two simple steps:

### 1. Initialize a tracer with your API keys and project name

<CodeGroup>
```Python Python
from judgeval.tracer import Tracer

# loads from JUDGMENT_API_KEY and JUDGMENT_ORG_ID env vars
judgment = Tracer(project_name="my_project")
```
```Typescript Typescript
import { Tracer } from 'judgeval';

// The getInstance method ensures a singleton Tracer is used
const judgment = Tracer.getInstance({ projectName: "my_project" });
```
</CodeGroup>

<Note>
    The [Judgment tracer](/api_reference/trace) is a singleton object that should be shared across your application. 
    Your project name will be used to organize your traces in one place on the Judgment platform.
</Note>


### 2. Wrap your workflow components

`judgeval` provides wrapping mechanisms for your workflow components:

#### `wrap()` ####
The `wrap()` function goes over your LLM client (e.g. OpenAI, Anthropic, etc.) and captures metadata surrounding your LLM calls, such as:
- Latency
- Token usage
- Prompt/Completion
- Model name

Here's an example of using `wrap()` on an OpenAI client:
<CodeGroup>
```Python Python
from openai import OpenAI
from judgeval.tracer import wrap

client = wrap(OpenAI())
```
```Typescript Typescript
import OpenAI from 'openai';
import { wrap } from 'judgeval';

const client = wrap(new OpenAI());
```
</CodeGroup>

<Note>
    When using OpenAI streaming with a wrapped client, you need to explicitly enable token usage tracking by setting `stream_options={"include_usage": True}`. Otherwise, token counts won't be captured for streaming calls.

    ```python
    # Enable token counting with streaming
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "Hello"}],
        stream=True,
        stream_options={"include_usage": True}  # Required for token counting
    )
    ```
</Note>

#### `@observe` (Python) / `observe()` (Typescript) ####
The `@observe` decorator (Python) or the `observe()` higher-order function (Typescript) wraps your functions/tools and captures metadata surrounding your function calls, such as:
- Latency
- Input/Output
- Span type (e.g. `retriever`, `tool`, `LLM call`, etc.)

With automatic deep tracing (enabled by default), you only need to observe top-level functions, and all nested function calls will be automatically traced. This significantly reduces the amount of instrumentation needed in your code.

Here's an example of using the observer mechanism:
<CodeGroup>
```Python Python
# loads from JUDGMENT_API_KEY env var
judgment = Tracer(project_name="my_project")  # Deep tracing is enabled by default

# With automatic deep tracing, you only need to observe the top-level function
@judgment.observe(span_type="function")
def my_workflow():
    # All function calls inside will be automatically traced
    my_tool()
    print("Hello world!")

# This function will be automatically traced when called from my_workflow
def my_tool():
    print("Tool executed!")

```
```Typescript Typescript
import { Tracer } from 'judgeval';

const judgment = Tracer.getInstance({ projectName: "my_project" });

async function myTool(): Promise<void> {
    console.log("Hello world!");
}

const observedMyTool = judgment.observe({ spanType: "tool" })(myTool);

// You would then call observedMyTool() instead of myTool()
// await observedMyTool();
```
</CodeGroup>

<Note>
    `span_type` is a string that you can use to categorize and organize your trace spans. 
    Span types are displayed on the trace UI to easily navigate a visualization of your workflow. 
    Common span types include `tool`, `function`, `retriever`, `database`, `web search`, etc.
</Note>

#### Putting it all Together
Here's a complete example of using judgeval's tracing mechanisms with automatic deep tracing:
<CodeGroup>
```Python Python
from judgeval.tracer import Tracer, wrap
from openai import OpenAI

openai_client = wrap(OpenAI())
# loads from JUDGMENT_API_KEY and JUDGMENT_ORG_ID env vars
judgment = Tracer(project_name="my_project")  # Deep tracing is enabled by default

# Define a tool function that will be automatically traced when called from main
def my_tool():
    return "Hello world!"

# With automatic deep tracing, only the main function needs to be observed
@judgment.observe(span_type="function")
def my_llm_call():
    message = my_tool()  # This tool call will be automatically traced
    res = openai_client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": message}]
    )
    return res.choices[0].message.content

# This implicitly starts a trace if one isn't active
# and saves it upon completion or error.
main_result = my_llm_call()
```
```Typescript Typescript
import { Tracer, wrap } from 'judgeval';
import OpenAI from 'openai';

const judgment = Tracer.getInstance({ projectName: "my_project" });
const openaiClient = wrap(new OpenAI());

async function myTool(): Promise<string> {
    return "Hello world!";
}
const observedMyTool = judgment.observe({ spanType: "tool" })(myTool);

async function myLlmCall(): Promise<string | null | undefined> {
    const message = await observedMyTool();
    const res = await openaiClient.chat.completions.create({
        model: "gpt-4o",
        messages: [{ role: "user", content: message }],
    });
    return res.choices[0]?.message?.content;
}
const observedLlmCall = judgment.observe({ spanType: "function" })(myLlmCall);

// Calling the observed function implicitly starts and saves the trace
async function runImplicitTrace() {
    try {
        const result = await observedLlmCall();
        console.log("Implicit trace completed:", result);
    } catch (error) {
        console.error("Implicit trace failed:", error);
        // Trace saved automatically by observe on completion/error
    }
}

runImplicitTrace();
```
</CodeGroup>

And the trace will appear on the Judgment platform as follows:

![Alt text](/images/basic_trace_example.png "Basic Trace Example")

### Using Streaming with Token Counting

When using streaming responses with a wrapped client, you need to explicitly enable token usage tracking:

<CodeGroup>
```Python Python
@judgment.observe(span_type="function")
def my_llm_streaming_call():
    # Enable token counting with streaming API calls
    stream = openai_client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "Write a poem"}],
        stream=True,
        stream_options={"include_usage": True}  # Required for token counting
    )
    
    # Process the stream
    full_response = ""
    for chunk in stream:
        if chunk.choices and chunk.choices[0].delta and chunk.choices[0].delta.content:
            content = chunk.choices[0].delta.content
            full_response += content
            print(content, end="", flush=True)
    
    return full_response
```
```Typescript Typescript
const observedStreamingCall = judgment.observe({ spanType: "function" })(async () => {
    // Enable token counting with streaming API calls
    const stream = await openaiClient.chat.completions.create({
        model: "gpt-4o",
        messages: [{ role: "user", content: "Write a poem" }],
        stream: true,
        stream_options: { include_usage: true }  // Required for token counting
    });
    
    // Process the stream
    let fullResponse = "";
    for await (const chunk of stream) {
        if (chunk.choices[0]?.delta?.content) {
            const content = chunk.choices[0].delta.content;
            fullResponse += content;
            process.stdout.write(content);
        }
    }
    
    return fullResponse;
});
```
</CodeGroup>

<Warning>
  Without setting `stream_options={"include_usage": True}`, token counts will not be captured for streaming API calls, and your usage metrics in traces will be incomplete.
</Warning>

### 3. Running Production Evaluations

Optionally, you can run asynchronous evaluations directly inside your traces.

This enables you to run evaluations on your **production data in real-time**, which can be useful for:
- **Guardrailing your production system** against quality regressions (hallucinations, toxic responses, revealing private data, etc.).
- Exporting production data for **offline experimentation** (e.g for A/B testing your workflow versions on relevant use cases).
- Getting **actionable insights** on how to fix common failure modes in your workflow (e.g. missing knowledge base info, suboptimal prompts, etc.). 

To execute an asynchronous evaluation, you can use the `trace.asyncEvaluate()` method (Typescript) or `judgment.async_evaluate()` (Python, assuming it operates on the currently active trace).

<CodeGroup>
```Python Python
from judgeval.tracer import Tracer
from judgeval.scorers import AnswerRelevancyScorer

judgment = Tracer(project_name="my_project")

@judgment.observe(span_type="function")
def main():
    query = "What is the capital of France?"
    res = "The capital of France is Paris."  # Replace with your workflow logic
    
    # Assumes this runs within an active trace context implicitly started by @observe
    judgment.async_evaluate(
        scorers=[AnswerRelevancyScorer(threshold=1.0)],
        input=query,
        actual_output=res,
        model="gpt-4o",
    )
    return res

main() # Call the observed function
```
```Typescript Typescript
import { Tracer, AnswerRelevancyScorer } from 'judgeval';

const judgment = Tracer.getInstance({ projectName: "my_project" });

async function mainWithEval(): Promise<string> {
    const query = "What is the capital of France?";
    const res = "The capital of France is Paris."; // Replace with your workflow logic

    const currentTrace = judgment.getCurrentTrace();
    if (currentTrace) {
        await currentTrace.asyncEvaluate(
            [new AnswerRelevancyScorer(1.0)], 
            {
                input: query,
                actualOutput: res,
                model: "gpt-4o"
            }
        );
    } else {
        console.warn("No active trace for async evaluation.");
    }
    return res;
}

// Explicitly start a trace for the evaluation
async function runMainWithEvalTrace() {
    const trace = judgment.startTrace("main_eval_trace");
    const observedMainWithEval = judgment.observe({ spanType: "function" })(mainWithEval);
    try {
        const result = await observedMainWithEval();
        console.log("Trace with eval completed:", result);
        await trace.save();
    } catch (error) {
        console.error("Trace with eval failed:", error);
        await trace.save();
    }
}

runMainWithEvalTrace();
```
</CodeGroup>

<Tip>
    Your async evaluations will be logged to the Judgment platform as part of the original trace and 
    a new evaluation will be created on the Judgment platform.
</Tip>

## Example: Music Recommendation Agent

In this video, we'll walk through all of the topics covered in this guide by tracing over a simple OpenAI API-based music recommendation agent.

<iframe 
    width="560" 
    height="315" 
    src="https://www.youtube.com/embed/7g0fut06UxQ"
    title="YouTube video player" 
    frameborder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" 
    referrerpolicy="strict-origin-when-cross-origin" 
    allowfullscreen
></iframe>


## Advanced: Customizing Traces Using the Context Manager (Python) / Explicit Trace Client (Typescript) ##

In Python, if you need to customize your tracing context beyond the implicit behavior of `@observe`, you can use the `with judgment.trace()` context manager.
In Typescript, you achieve similar control by explicitly creating a `TraceClient` instance using `judgment.startTrace()` and manually calling methods like `save()` or `print()` on it.

The explicit trace client allows you to **save or print the state of the trace at any point in the workflow**.
This is useful for debugging or exporting any state of your workflow to run an evaluation from!

<Tip>
    Any functions wrapped with `judgment.observe()` called within the scope where the `TraceClient` is active will automatically be associated with that trace.
</Tip>

Here's an example of using explicit trace management:
<CodeGroup>
```Python Python
from judgeval.tracer import Tracer, wrap
from openai import OpenAI

judgment = Tracer(project_name="my_project")
client = wrap(OpenAI())

@judgment.observe(span_type="tool")
def my_tool():
    return "Hello world!"

def main():
    with judgment.trace(name="my_workflow") as trace:
        res = client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": f"{my_tool()}"}]
        )
    
        trace.print()  # prints the state of the trace to console
        trace.save()  # saves the current state of the trace to the Judgment platform
        # Note: Python trace context likely saves automatically on exit

    return res.choices[0].message.content
```
```Typescript Typescript
import { Tracer, wrap } from 'judgeval';
import OpenAI from 'openai';

const judgment = Tracer.getInstance({ projectName: "my_project" });
const client = wrap(new OpenAI());

async function myTool(): Promise<string> {
    return "Hello world!";
}
const observedMyTool = judgment.observe({ spanType: "tool" })(myTool);

async function main() {
    // Start the trace explicitly
    const trace = judgment.startTrace("my_workflow"); 
    let resultMessage: string | null | undefined;

    try {
        const toolOutput = await observedMyTool(); // observedMyTool is associated with the active trace
        const res = await client.chat.completions.create({
            model: "gpt-4o",
            messages: [{ role: "user", content: toolOutput }],
        });
        resultMessage = res.choices[0]?.message?.content;

        trace.print(); // Prints the trace to console
        await trace.save(); // Saves the trace to the Judgment platform
        
    } catch (error) {
        console.error("Workflow failed:", error);
        await trace.save(); // Still save trace on error
    } finally {
        // Ensure save is called if not already done (optional, depends on desired behavior)
        // await trace.save(); // Could be redundant if already saved in try/catch
    }

    return resultMessage;
}

main();
```
</CodeGroup>

<Warning>
    In Python, the `with judgment.trace()` context manager should only be used if you need fine-grained control 
    over the trace lifecycle. In Typescript, explicit management via `startTrace()` and `trace.save()` is the standard way to gain this control.
    In most simple cases in Python, the `@observe` decorator is sufficient.
</Warning>