---
title: Integrating Judgeval with LangGraph
---
Integrating Judgeval with LangGraph allows for detailed tracing and evaluation of your graph workflows. By adding either the `JudgevalCallbackHandler` (synchronous) or `AsyncJudgevalCallbackHandler` (asynchronous) to your LangGraph invocation, you can automatically trace node executions, tool calls, and LLM interactions within your graph using Judgeval.

<Note>
We expect you to already be familiar with Judgeval and its core concepts. If not, please refer to the [Getting Started](/getting_started) guide.
</Note>

## Judgeval Callback Handlers

Judgeval provides two callback handlers for LangGraph integration:

1. `JudgevalCallbackHandler`: For synchronous LangGraph workflows (`graph.invoke`)
2. `AsyncJudgevalCallbackHandler`: For asynchronous LangGraph workflows (`graph.ainvoke`)

Both handlers automatically capture:

*   Visits to each node in your graph.
*   Calls made to any tools integrated with your LangGraph agents or nodes.
*   Interactions with Language Models (LLMs).

The handler instances store metadata about the execution, accessible after the graph run:

- `executed_nodes`: A list of node names that were executed.
- `executed_tools`: A list of tool names that were called.
- `executed_node_tools`: A list detailing the sequence of node and tool executions, formatted like `['node_A', 'node_A:tool_X', 'node_B']`.

<Tip>
When using either callback handler, you do not need to add `@observe` decorators to your LangGraph nodes or tools for tracing purposes.
</Tip>

## Triggering Evaluations

You can trigger Judgeval evaluations directly from within your graph nodes. This associates the evaluation results with the specific node's execution span in the trace. The recommended way is to use the `add_evaluation_to_state` helper function:

```python
# Inside your LangGraph node function
from judgeval.integrations.langgraph import add_evaluation_to_state
from judgeval.scorers import AnswerRelevancyScorer # Or other scorers

# For synchronous nodes:
def my_node_function(state: State) -> State:
    # ... your node logic ...
    user_input = "some input"
    llm_output = "some output"
    model_name = "gpt-4"

    add_evaluation_to_state(
        state=state,
        scorers=[AnswerRelevancyScorer(threshold=0.7)],
        input=user_input,
        actual_output=llm_output,
        model=model_name
    )
    # ... potentially modify state further ...
    return state

# For asynchronous nodes:
async def my_async_node_function(state: State) -> State:
    # ... your node logic ...
    user_input = "some input"
    llm_output = "some output"
    model_name = "gpt-4"

    add_evaluation_to_state(
        state=state,
        scorers=[AnswerRelevancyScorer(threshold=0.7)],
        input=user_input,
        actual_output=llm_output,
        model=model_name
    )
    # ... potentially modify state further ...
    return state
```

Alternatively, you can manually add an `EvaluationConfig` object to your node's output state under the key `_judgeval_eval`. The handler will detect this and trigger the evaluation.

## Example Workflows

### Synchronous Workflow

```python
import os
from typing import TypedDict, Sequence
from dotenv import load_dotenv
from langchain_core.messages import HumanMessage
from langgraph.graph import StateGraph, END
from judgeval.common.tracer import Tracer
from judgeval.integrations.langgraph import JudgevalCallbackHandler # <-- Import sync handler
from judgeval.scorers import AnswerRelevancyScorer # Or other scorers
from judgeval.data import Example

# Define your state
class State(TypedDict):
    messages: Sequence[HumanMessage]
    # ... other state fields

# Load environment variables
load_dotenv()
PROJECT_NAME = "my-langgraph-project"

# Initialize Tracer and Handler
judgment = Tracer(api_key=os.getenv("JUDGMENT_API_KEY"), project_name=PROJECT_NAME)
handler = JudgevalCallbackHandler(judgment) # <-- Use sync handler

# Define your node functions
def node_1(state: State):
    # ... node logic ...
    # Optionally add evaluation here using add_evaluation_to_state(state, ...)
    return state

def node_2(state: State):
    # ... node logic ...
    return state

# Build the graph
graph_builder = StateGraph(State)
graph_builder.add_node("node_1", node_1)
graph_builder.add_node("node_2", node_2)
graph_builder.set_entry_point("node_1")
graph_builder.add_edge("node_1", "node_2")
graph_builder.add_edge("node_2", END)
graph = graph_builder.compile()

# Run the graph synchronously, passing the handler in config
def run_graph():
    initial_state = {"messages": [HumanMessage(content="Hello!")]}
    config_with_callbacks = {"callbacks": [handler]} # <-- Pass handler here
    
    final_state = graph.invoke(initial_state, config=config_with_callbacks) # <-- Use invoke

    # Accessing the handler attributes after execution
    print("Executed Nodes:", handler.executed_nodes)
    print("Executed Tools:", handler.executed_tools) # Will be empty if no tools used
    print("Node/Tool Flow:", handler.executed_node_tools)

    print("Final State:", final_state)

# Run the function
if __name__ == "__main__":
    run_graph()
```

### Asynchronous Workflow

```python
import os
import asyncio
from typing import TypedDict, Sequence
from dotenv import load_dotenv
from langchain_core.messages import HumanMessage
from langgraph.graph import StateGraph, END
from judgeval.common.tracer import Tracer
from judgeval.integrations.langgraph import AsyncJudgevalCallbackHandler # <-- Import async handler
from judgeval.scorers import AnswerRelevancyScorer # Or other scorers
from judgeval.data import Example

# Define your state
class State(TypedDict):
    messages: Sequence[HumanMessage]
    # ... other state fields

# Load environment variables
load_dotenv()
PROJECT_NAME = "my-langgraph-project"

# Initialize Tracer and Handler
judgment = Tracer(api_key=os.getenv("JUDGMENT_API_KEY"), project_name=PROJECT_NAME)
handler = AsyncJudgevalCallbackHandler(judgment) # <-- Use async handler

# Define your node functions (async)
async def node_1(state: State):
    # ... node logic ...
    # Optionally add evaluation here using add_evaluation_to_state(state, ...)
    return state

async def node_2(state: State):
    # ... node logic ...
    return state

# Build the graph
graph_builder = StateGraph(State)
graph_builder.add_node("node_1", node_1)
graph_builder.add_node("node_2", node_2)
graph_builder.set_entry_point("node_1")
graph_builder.add_edge("node_1", "node_2")
graph_builder.add_edge("node_2", END)
graph = graph_builder.compile()

# Run the graph asynchronously, passing the handler in config
async def run_graph():
    initial_state = {"messages": [HumanMessage(content="Hello!")]}
    config_with_callbacks = {"callbacks": [handler]} # <-- Pass handler here
    
    final_state = await graph.ainvoke(initial_state, config=config_with_callbacks) # <-- Use ainvoke

    # Accessing the handler attributes after execution
    print("Executed Nodes:", handler.executed_nodes)
    print("Executed Tools:", handler.executed_tools) # Will be empty if no tools used
    print("Node/Tool Flow:", handler.executed_node_tools)

    print("Final State:", final_state)

# Run the async function
if __name__ == "__main__":
    asyncio.run(run_graph())
```

View some of our demo code for more detailed examples.

- [Basic Workflow](https://github.com/JudgmentLabs/judgment-cookbook/blob/main/integrations/langgraph/basic.py)
- [Human in the Loop](https://github.com/JudgmentLabs/judgment-cookbook/blob/main/integrations/langgraph/human_in_the_loop/human_in_the_loop.py)
