---
title: Getting Started
description: "This guide will help you learn the essential components of `judgeval`."
---

# Installation 

<CodeGroup>
```Shell Python
pip install judgeval
```
```Shell Typescript
npm install judgeval
```
</CodeGroup>

Judgeval runs evaluations that you can manage inside the library. Additionally, you should analyze and manage your evaluations, datasets, and metrics on 
the natively-integrated [Judgment Platform](https://app.judgmentlabs.ai/register), an all-in-one suite for LLM system evaluation.

<Tip>
Our team is always making new releases of the `judgeval` package! 
To get the latest Python version, run `pip install --upgrade judgeval`. 
To get the latest Typescript version, run `npm update judgeval`.
You can follow our latest updates via our [GitHub](https://github.com/judgmentlabs).
</Tip>

# Judgment API Keys

Our API keys allow you to access the `JudgmentClient` and `Tracer` which enable you to track your agents and run evaluations on 
Judgment Labs' infrastructure, access our state-of-the-art judge models, and manage your evaluations/datasets on the Judgment Platform. 

To get your account and organization API keys, create an account on the [Judgment Platform](https://app.judgmentlabs.ai/register). 

```
export JUDGMENT_API_KEY="your_key_here"
export JUDGMENT_ORG_ID="your_org_id_here"
```

<Note>
For assistance with your registration and setup, such as dealing with sensitive data that has to reside in your private VPCs, 
feel free to [get in touch with our team](mailto:contact@judgmentlabs.ai).
</Note>


# Create Your First Experiment

<CodeGroup>
```Python Python
from judgeval import JudgmentClient
from judgeval.data import Example
from judgeval.scorers import FaithfulnessScorer

client = JudgmentClient()

example = Example(
    input="What if these shoes don't fit?",
    actual_output="We offer a 30-day full refund at no extra cost.",
    retrieval_context=["All customers are eligible for a 30 day full refund at no extra cost."],
)

scorer = FaithfulnessScorer(threshold=0.5)
results = client.run_evaluation(
    examples=[example],
    scorers=[scorer],
    model="gpt-4o",
)
print(results)
```
```Typescript Typescript
import { JudgmentClient, ExampleBuilder, FaithfulnessScorer, logger } from 'judgeval';

async function runFirstExperiment() { // Wrap in async function for await
    const client = JudgmentClient.getInstance();

    const example = new ExampleBuilder()
        .input("What if these shoes don't fit?")
        .actualOutput("We offer a 30-day full refund at no extra cost.")
        .context(["All customers are eligible for a 30 day full refund at no extra cost."])
        .build();

    const scorer = new FaithfulnessScorer(0.5);
    const results = await client.evaluate({
        examples: [example],
        scorers: [scorer],
        model: "gpt-4o",
        projectName: "my-first-ts-project", // Example project name
        evalName: "first-ts-experiment"   // Example eval name
    });
    logger.print(results);
}

runFirstExperiment(); // Call the async function
```
</CodeGroup>

Congratulations! Your evaluation should have passed. Let's break down what happened.

- The variable `input` mimics a user input and `actual_output` is a placeholder for what your LLM system returns based on the input.
- The variable `retrieval_context` (Python) or `context` (Typescript) represents the retrieved context from your RAG knowledge base.
- `FaithfulnessScorer(threshold=0.5)` is a scorer that checks if the output is hallucinated relative to the retrieved context.
    - <Note>The threshold is used in the context of [unit testing](/evaluation/unit_testing).</Note>
- We chose `gpt-4o` as our judge model to measure faithfulness. Judgment Labs offers ANY judge model for your evaluation needs. 
Consider trying out our state-of-the-art [Osiris judge models](https://cs191.stanford.edu/projects/Shan,%20Alexander_NLP%20191W.pdf) for your next evaluation!

<Tip>
To learn more about using the Judgment Client to run evaluations, click [here](/api_reference/judgment_client).
</Tip>

# Create Your First Trace
`judgeval` traces enable you to monitor your LLM systems in online **development and production** stages. 
Traces enable you to track your LLM system's flow end-to-end and measure:
- LLM costs
- Workflow latency
- Quality metrics, such as hallucination, retrieval quality, and more.


<CodeGroup>
```Python Python
from judgeval.common.tracer import Tracer, wrap
from openai import OpenAI

client = wrap(OpenAI())
judgment = Tracer(project_name="my_project")  # Deep tracing is enabled by default

# With automatic deep tracing, you only need to observe the top-level function
@judgment.observe(span_type="function")
def main():
    # my_tool will be automatically traced without @observe
    task_input = my_tool()
    res = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": f"{task_input}"}]
    )
    return res.choices[0].message.content

# No @observe needed - automatically traced when called from main
def my_tool():
    return "Hello world!"

# Calling the observed function implicitly starts and saves the trace
main()
```
```Typescript Typescript
import { Tracer, wrap } from 'judgeval';
import OpenAI from 'openai';

// Ensure JUDGMENT_API_KEY and JUDGMENT_ORG_ID are set in your environment
const client = wrap(new OpenAI());
const judgment = Tracer.getInstance({ projectName: "my_project" });

// Define the function to be traced
async function myTool(): Promise<string> {
    return "Hello world!";
}
// Wrap the function using the observe method
const observedMyTool = judgment.observe({ spanType: "tool" })(myTool);

// Define the main function
async function main(): Promise<string | null | undefined> {
    const taskInput = await observedMyTool();
    const res = await client.chat.completions.create({
        model: "gpt-4o",
        messages: [{ role: "user", content: taskInput }],
    });
    return res.choices[0]?.message?.content;
}

// Wrap the main function
const observedMain = judgment.observe({ spanType: "function" })(main);

// Calling the observed function implicitly starts and saves the trace
async function runImplicitTrace() {
    try {
        const result = await observedMain();
        console.log("Implicit trace completed successfully:", result);
    } catch (error) {
        console.error("Implicit trace failed:", error);
        // The trace should still be saved automatically on error by observe
    }
}

runImplicitTrace();
```
</CodeGroup>

With automatic deep tracing, you only need to observe top-level functions, and all nested function calls will be automatically traced. This significantly reduces the amount of instrumentation needed in your code.

Congratulations! You've just created your first trace. It should look like this:

<div style={{display: 'flex', justifyContent: 'center'}}>
  ![Alt text](/images/trace_ss.png "Image of a RAG pipeline trace")
</div>

There are many benefits of monitoring your LLM systems with `judgeval` tracing, including:
- Debugging LLM workflows in seconds with full observability
- Using production workflow data to create experimental datasets for future improvement/optimization
- Tracking and creating Slack/Email alerts on **any metric** (e.g. latency, cost, hallucination, etc.)

<Tip>
To learn more about `judgeval`'s tracing module, click [here](/tracing/introduction).
</Tip>

# Create Your First Online Evaluation

In addition to tracing, `judgeval` allows you to run online evaluations on your LLM systems. This enables you to:
- Catch real-time quality regressions to take action before customers are impacted
- Gain insights into your agent performance in real-world scenarios

To run an online evaluation, you can simply add one line of code to your existing trace:

<CodeGroup>
```Python Python
from judgeval.common.tracer import Tracer, wrap
from judgeval.scorers import AnswerRelevancyScorer
from openai import OpenAI

client = wrap(OpenAI())
judgment = Tracer(project_name="my_project")

@judgment.observe(span_type="tool")
def my_tool():
    return "Hello world!"

@judgment.observe(span_type="function")
def main():
    task_input = my_tool()
    res = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": f"{task_input}"}]
    ).choices[0].message.content

    # In Python, this likely operates on the implicit trace context
    judgment.async_evaluate(
        scorers=[AnswerRelevancyScorer(threshold=0.5)],
        input=task_input,
        actual_output=res,
        model="gpt-4o"
    )

    return res

main()
```
```Typescript Typescript
import { Tracer, wrap, AnswerRelevancyScorer } from 'judgeval';
import OpenAI from 'openai';

// Ensure JUDGMENT_API_KEY and JUDGMENT_ORG_ID are set in your environment
const client = wrap(new OpenAI());
const judgment = Tracer.getInstance({ projectName: "my_project" });

// Define the function to be traced
async function myTool(): Promise<string> {
    return "Hello world!";
}
// Wrap the function using the observe method
const observedMyTool = judgment.observe({ spanType: "tool" })(myTool);

// Define the main function with async evaluation
async function mainWithEval(): Promise<string | null | undefined> {
    const taskInput = await observedMyTool();
    const res = await client.chat.completions.create({
        model: "gpt-4o",
        messages: [{ role: "user", content: taskInput }],
    });
    const actualOutput = res.choices[0]?.message?.content;

    if (actualOutput) {
        // Get the current trace instance from the Tracer singleton
        const currentTrace = judgment.getCurrentTrace(); 
        if (currentTrace) {
             // Run async evaluation within the current trace context
            await currentTrace.asyncEvaluate(
                [new AnswerRelevancyScorer(0.5)],
                {
                    input: taskInput,
                    actualOutput: actualOutput,
                    model: "gpt-4o"
                }
            );
        } else {
            console.warn("Could not find active trace to run async evaluation.");
        }
    }
    return actualOutput;
}

// Still need explicit trace management here because asyncEvaluate is on the TraceClient
async function runMainWithEval() {
    const trace = judgment.startTrace("main_trace_with_eval"); // Start a trace
    try {
        // Wrap the main function call within the trace's context
        const observedMain = judgment.observe({ spanType: "function" })(mainWithEval);
        const result = await observedMain();
        console.log("Trace with eval completed successfully:", result);
        await trace.save(); // Save the trace
    } catch (error) {
        console.error("Trace with eval failed:", error);
        await trace.save(); // Save the trace even on error
    }
}

runMainWithEval();
```
</CodeGroup>

Online evaluations are automatically logged to the Judgment Platform as part of your traces. You can view them by navigating to your trace and clicking on the 
trace span that contains the online evaluation. If there is a quality regression, the UI will display an alert, like this: 

<div style={{display: 'flex', justifyContent: 'center'}}>
  ![Alt text](/images/online_eval_fault.png "Image of an alert on the Judgment Platform")
</div>

# Optimizing Your LLM System

Evaluation and monitoring are the building blocks for optimizing LLM systems. Measuring the quality of your LLM workflows 
allows you to **compare design iterations** and ultimately find the **optimal set of prompts, models, RAG architectures, etc.** that 
make your LLM excel in your production use cases. 

**A typical experimental setup might look like this:**

1. Create a new Project in the Judgment platform by either running an evaluation from the SDK or via the platform UI. 
This will help you keep track of all evaluations and traces for different iterations of your LLM system.

<Note> 
A Project keeps track of Experiments and Traces relating to a specific workflow. Each Experiment contains a set of Scorers that have been run on a set of Examples.
</Note>

2. You can create separate Experiments for different iterations of your LLM system, allowing you to independently test each component of your LLM system.

<Tip>
You can try different models (e.g. `gpt-4o`, `claude-3-5-sonnet`, etc.) and prompt templates in each Experiment to find the 
optimal setup for your LLM system. 
</Tip>


# Next Steps

Congratulations! You've just finished getting started with `judgeval` and the Judgment Platform. 

For a deeper dive into using `judgeval`, learn more about [experiments](/evaluation/introduction), [unit testing](/evaluation/unit_testing), and [monitoring](/monitoring/introduction)!