---
title: Getting Started
description: "This guide will help you learn the essential components of `judgeval`."
---

# Installation 

`pip install judgeval`

Judgeval runs evaluations that you can manage inside the library. Additionally, you should analyze and manage your evaluations, datasets, and metrics on 
the natively-integrated [Judgment Platform](https://app.judgmentlabs.ai/register), an all-in-one suite for LLM system evaluation.

<Tip>
Our team is always making new releases of the `judgeval` package! To get the latest version, run `pip install --upgrade judgeval`. 
You can follow our latest updates via our [GitHub](https://github.com/judgmentlabs/judgeval).
</Tip>

# Judgment API Keys

Our API keys allow you to access the `JudgmentClient` and `Tracer` which enable you to track your agents and run evaluations on 
Judgment Labs' infrastructure, access our state-of-the-art judge models, and manage your evaluations/datasets on the Judgment Platform. 

To get your account and organization API keys, create an account on the [Judgment Platform](https://app.judgmentlabs.ai/register). 

```
export JUDGMENT_API_KEY="your_key_here"
export JUDGMENT_ORG_ID="your_org_id_here"
```

<Note>
For assistance with your registration and setup, such as dealing with sensitive data that has to reside in your private VPCs, 
feel free to [get in touch with our team](mailto:contact@judgmentlabs.ai).
</Note>


# Create Your First Experiment

```python sample_eval.py
from judgeval import JudgmentClient
from judgeval.data import Example
from judgeval.scorers import FaithfulnessScorer

client = JudgmentClient()

example = Example(
    input="What if these shoes don't fit?",
    actual_output="We offer a 30-day full refund at no extra cost.",
    retrieval_context=["All customers are eligible for a 30 day full refund at no extra cost."],
)

scorer = FaithfulnessScorer(threshold=0.5)
results = client.run_evaluation(
    examples=[example],
    scorers=[scorer],
    model="gpt-4o",
)
print(results)
```

Congratulations! Your evaluation should have passed. Let's break down what happened.

- The variable `input` mimics a user input and `actual_output` is a placeholder for what your LLM system returns based on the input.
- The variable `retrieval_context` represents the retrieved context from your RAG knowledge base.
- `FaithfulnessScorer(threshold=0.5)` is a scorer that checks if the output is hallucinated relative to the retrieved context.
    - <Note>The threshold is used in the context of [unit testing](/evaluation/unit_testing).</Note>
- We chose `gpt-4o` as our judge model to measure faithfulness. Judgment Labs offers ANY judge model for your evaluation needs. 
Consider trying out our state-of-the-art Osiris judge models for your next evaluation!

<Tip>
To learn more about using the Judgment Client to run evaluations, click [here](/api_reference/judgment_client).
</Tip>

# Create Your First Trace
`judgeval` traces enable you to monitor your LLM systems in online **development and production** stages. 
Traces enable you to track your LLM system's flow end-to-end and measure:
- LLM costs
- Workflow latency
- Quality metrics, such as hallucination, retrieval quality, and more.


```python trace_example.py
from judgeval.common.tracer import Tracer, wrap
from openai import OpenAI

client = wrap(OpenAI())
judgment = Tracer(project_name="my_project")

@judgment.observe(span_type="tool")
def my_tool():
    return "Hello world!"

@judgment.observe(span_type="function")
def main():
    task_input = my_tool()
    res = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": f"{task_input}"}]
    )
    return res.choices[0].message.content
```

Congratulations! You've just created your first trace. It should look like this:

<div style={{display: 'flex', justifyContent: 'center'}}>
  ![Alt text](/images/trace_ss.png "Image of a RAG pipeline trace")
</div>

There are many benefits of monitoring your LLM systems with `judgeval` tracing, including:
- Debugging LLM workflows in seconds with full observability
- Using production workflow data to create experimental datasets for future improvement/optimization
- Tracking and creating Slack/Email alerts on **any metric** (e.g. latency, cost, hallucination, etc.)

<Tip>
To learn more about `judgeval`'s tracing module, click [here](/tracing/introduction).
</Tip>

# Create Your First Online Evaluation

In addition to tracing, `judgeval` allows you to run online evaluations on your LLM systems. This enables you to:
- Catch real-time quality regressions to take action before customers are impacted
- Gain insights into your agent performance in real-world scenarios

To run an online evaluation, you can simply add one line of code to your existing trace:

```python trace_example.py
from judgeval.common.tracer import Tracer, wrap
from judgeval.scorers import AnswerRelevancyScorer
from openai import OpenAI

client = wrap(OpenAI())
judgment = Tracer(project_name="my_project")

@judgment.observe(span_type="tool")
def my_tool():
    return "Hello world!"

@judgment.observe(span_type="function")
def main():
    task_input = my_tool()
    res = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": f"{task_input}"}]
    ).choices[0].message.content

    judgment.async_evaluate(
        scorers=[AnswerRelevancyScorer(threshold=0.5)],
        input=task_input,
        actual_output=res,
        model="gpt-4o"
    )

    return res
```

Online evaluations are automatically logged to the Judgment Platform as part of your traces. You can view them by navigating to your trace and clicking on the 
trace span that contains the online evaluation. If there is a quality regression, the UI will display an alert, like this: 

<div style={{display: 'flex', justifyContent: 'center'}}>
  ![Alt text](/images/online_eval_fault.png "Image of an alert on the Judgment Platform")
</div>

{/*
# Create Your First Scorer
`judgeval` offers three kinds of LLM scorers for your evaluation needs: ready-made, classifier scorers, and custom scorers.

## Ready-made Scorers
Judgment Labs provides default implementations of 10+ research-backed metrics covering evaluation needs ranging from hallucination detection to RAG retrieval quality. To create a ready-made scorer, just import it directly from `judgeval.scorers`:

```python scorer_example.py
from judgeval import JudgmentClient
from judgeval.data import Example
from judgeval.scorers import FaithfulnessScorer

client = JudgmentClient()
example = Example(
    input="...",
    actual_output="...",
    retrieval_context=["..."],
)
scorer = FaithfulnessScorer(threshold=0.5)

results = client.run_evaluation(
    examples=[example],
    scorers=[scorer],
    model="gpt-4o",
)
print(results)
```

<Note>
For a complete list of ready-made scorers, see the [scorers docs](/evaluation/scorers).
</Note>

## Classifier Scorers
`judgeval` allows you to create custom scorers using natural language. These can range from simple judges to powerful evaluators for your LLM systems.

```python classifier_scorer.py
from judgeval.scorers import ClassifierScorer

classifier_scorer = ClassifierScorer(
    name="Tone Scorer",
    threshold=0.9,
    conversation=[
        {
            "role": "system", 
            "content": "Is the response positive (Y/N)? The response is: {{actual_output}}."
        }
    ],
    options={"Y": 1, "N": 0}
)
```

To learn more about `ClassifierScorer`s, click [here](/evaluation/scorers/classifier_scorer).

## Custom Scorers
If you find that none of the ready-made scorers or classifier scorers fit your needs, you can easily create your own custom scorer. 
These can be as simple or complex as you need them to be and **_do not_** have to use an LLM judge model. 
Here's an example of computing BLEU scores:

```python bleu_custom_scorer.py
import sacrebleu
from judgeval.scorers import JudgevalScorer

class BLEUScorer(JudgevalScorer):
    def __init__(self, threshold: float = 0.5):
        super().__init__(score_type="BLEU", threshold=threshold)

    def score_example(self, example: Example) -> float:
        reference = example.expected_output
        candidate = example.actual_output

        score = sacrebleu.sentence_bleu(candidate, [reference]).score
        self.score = score
        return score

    # Async implementation of score_example(). If you have no async logic, you can
    # just use the synchronous implementation.
    async def a_score_example(self, example: Example) -> float:
        return self.score_example(example)

    def success_check(self) -> bool:
        return self.score >= self.threshold

    @property
    def __name__(self):
        return "BLEU"

# example usage 
example = Example("input"="...", "actual_output"="...", "expected_output"="...")
scorer = BLEUScorer()
results = scorer.score_example(example)
print(results)
```

<Note>
For a deeper dive into making custom scorers, see the custom scorers [page](/evaluation/scorers/custom_scorers).
</Note>

## Running Multiple Scorers Simultaneously

If you're interested in measuring multiple metrics at once, you can group scorers together when running evaluations, 
regardless of the type of scorer.

```python multiple_scorers.py
from judgeval import JudgmentClient
from judgeval.scorers import FaithfulnessScorer, SummarizationScorer

client = JudgmentClient()

faithfulness_scorer = FaithfulnessScorer(threshold=0.5)
summarization_scorer = SummarizationScorer(threshold=0.8)

results = client.run_evaluation(
    examples=[example],
    scorers=[faithfulness_scorer, summarization_scorer],
    model="gpt-4o",
)
```

# Create Your First Dataset 
In most cases, you will not be running evaluations on a single example; instead, you will be scoring your LLM system on a dataset. 
Judgeval allows you to create datasets, save them, and run evaluations on them. 
An `EvalDataset` is a collection of `Example`s.

```python create_dataset.py
from judgeval.data import Example, EvalDataset

example1 = Example(input="...", actual_output="...")
example2 = Example(input="...", actual_output="...")

dataset = EvalDataset(examples=[example1, example2])
```

Then, you can run evaluations on the dataset:

```python evaluate_dataset.py
...

client = JudgmentClient()
scorer = FaithfulnessScorer(threshold=0.5)
results = client.evaluate_dataset(
    dataset=dataset,
    scorers=[scorer],
    model="Qwen/Qwen2.5-72B-Instruct-Turbo",
)
```


# Using Judgment Labs Platform
 
When scaling your evaluations beyond a few examples, Judgment's platform allows you to manage your **evaluations, 
datasets, and scorers** in a single place. 

To get started, create a Judgment account by emailing us at `contact@judgmentlabs.ai`. 
We'll get you set up with a login and you'll be able to:
- Run evaluations directly on Judgment's platform
- Track and inspect evaluations with an **intuitive UI**
- Compare your evaluations **across iterations** of your LLM system.
- Manage your datasets and scorers
- Monitor your LLM systems in production, gaining insights into your LLM system's performance in **real-world scenarios.**

<Note> 
Click [here](/judgment/introduction) to learn more about Judgment Labs' platform. 
</Note>

## Running Evaluations on Judgment

Work in progress!

## Managing Datasets

Work in progress!

*/}

# Optimizing Your LLM System

Evaluation and monitoring are the building blocks for optimizing LLM systems. Measuring the quality of your LLM workflows 
allows you to **compare design iterations** and ultimately find the **optimal set of prompts, models, RAG architectures, etc.** that 
make your LLM excel in your production use cases. 

**A typical experimental setup might look like this:**

1. Create a new Project in the Judgment platform by either running an evaluation from the SDK or via the platform UI. 
This will help you keep track of all evaluations and traces for different iterations of your LLM system.

<Note> 
A Project keeps track of Experiments and Traces relating to a specific workflow. Each Experiment contains a set of Scorers that have been run on a set of Examples.
</Note>

2. You can create separate Experiments for different iterations of your LLM system, allowing you to independently test each component of your LLM system.

<Tip>
You can try different models (e.g. `gpt-4o`, `claude-3-5-sonnet`, etc.) and prompt templates in each Experiment to find the 
optimal setup for your LLM system. 
</Tip>


# Next Steps

Congratulations! You've just finished getting started with `judgeval` and the Judgment Platform. 

For a deeper dive into using `judgeval`, learn more about [experiments](/evaluation/introduction), [unit testing](/evaluation/unit_testing), and [monitoring](/monitoring/introduction)!
