---
title: Getting Started
description: "This guide will help you learn the essential components of `judgeval`."
---

# Installation 

`pip install judgeval`

Judgeval runs evaluations on your local machine. However, you may find it easier to directly run 
evaluations using Judgment Labs' models or via the [Judgment Platform](https://app.judgmentlabs.ai/login), 
an all-in-one suite for LLM system evaluation.

# Making a Judgment Key

Creating a Judgment API key allows you to access the `JudgmentClient`, enabling you to run evaluations on Judgment Labs' infrastructure,
access our state-of-the-art judge models, and manage your evaluations/datasets on the Judgment Platform. 

Once you have a key, you can set the environment variable `JUDGMENT_API_KEY` to your key. 
This allows the `JudgmentClient` to authenticate your requests to the Judgment API.

```
export JUDGMENT_API_KEY="your_key_here"
```

To receive a key, please email us at `contact@judgmentlabs.ai`.


<Note>
Running evaluations on Judgment Labs' infrastructure is recommended for 
large-scale evaluations. [Contact us](mailto:contact@judgmentlabs.ai) if you're dealing with 
sensitive data that has to reside in your private VPCs.
</Note>

# Create your first evaluation

```python sample_eval.py
from judgeval import JudgmentClient
from judgeval.data import Example
from judgeval.scorers import FaithfulnessScorer

client = JudgmentClient()

example = Example(
    input="What if these shoes don't fit?",
    actual_output="We offer a 30-day full refund at no extra cost.",
    retrieval_context=["All customers are eligible for a 30 day full refund at no extra cost."],
)

scorer = FaithfulnessScorer(threshold=0.5)
results = client.run_evaluation(
    examples=[example],
    scorers=[scorer],
    model="gpt-4o",
)
print(results)
```

Congratulations! Your evaluation should have passed. Let's break down what happened.

- The variable `input` mimics a user input and `actual_output` is a placeholder for what your LLM system returns based on the input.
- The variable `retrieval_context` represents the retrieved context from your knowledge base and `FaithfulnessScorer(threshold=0.5)` 
is a scorer that checks if the output is hallucinated relative to the retrieved context.
- Scorers give values betweeen 0 - 1 and we set the threshold for this scorer to 0.5 in the context of a unit test. If you are interested measuring rather than testing, you can ignore this threshold and reference the `score` field alone.
- We chose `gpt-4o` as our judge model for faithfulness. Judgment Labs offers ANY judge model for your evaluation needs. Consider trying out our state-of-the-art judge models for your next evaluation!

# Create Your First Scorer
`judgeval` offers three kinds of LLM scorers for your evaluation needs: ready-made, classifier scorers, and custom scorers.

## Ready-made Scorers
Judgment Labs provides default implementations of 10+ research-backed metrics covering evaluation needs ranging from hallucination detection to RAG retrieval quality. To create a ready-made scorer, just import it directly from `judgeval.scorers`:

```python scorer_example.py
from judgeval import JudgmentClient
from judgeval.data import Example
from judgeval.scorers import FaithfulnessScorer

client = JudgmentClient()
example = Example(
    input="...",
    actual_output="...",
    retrieval_context=["..."],
)
scorer = FaithfulnessScorer(threshold=0.5)

results = client.run_evaluation(
    examples=[example],
    scorers=[scorer],
    model="gpt-4o",
)
print(results)
```

<Note>
For a complete list of ready-made scorers, see the [scorers docs](/evaluation/scorers).
</Note>

## Classifier Scorers
`judgeval` allows you to create custom scorers using natural language. These can range from simple judges to powerful evaluators for your LLM systems.

```python classifier_scorer.py
from judgeval.scorers import ClassifierScorer

classifier_scorer = ClassifierScorer(
    name="Tone Scorer",
    threshold=0.9,
    conversation=[
        {
            "role": "system", 
            "content": "Is the response positive (Y/N)? The response is: {{actual_output}}."
        }
    ],
    options={"Y": 1, "N": 0}
)
```

To learn more about `ClassifierScorer`s, click [here](/evaluation/scorers/classifier_scorer).

## Custom Scorers
If you find that none of the ready-made scorers or classifier scorers fit your needs, you can easily create your own custom scorer. 
These can be as simple or complex as you need them to be and **_do not_** have to use an LLM judge model. 
Here's an example of computing BLEU scores:

```python bleu_custom_scorer.py
import sacrebleu
from judgeval.scorers import CustomScorer 

class BLEUScorer(CustomScorer):
    def __init__(self, threshold: float = 0.5):
        super().__init__(score_type="BLEU", threshold=threshold)

    def score_example(self, example: Example) -> float:
        reference = example.expected_output
        candidate = example.actual_output

        score = sacrebleu.sentence_bleu(candidate, [reference]).score
        self.score = score
        return score

    # Async implementation of score_example(). If you have no async logic, you can
    # just use the synchronous implementation.
    async def a_score_example(self, example: Example) -> float:
        return self.score_example(example)

    def success_check(self) -> bool:
        return self.score >= self.threshold

    @property
    def __name__(self):
        return "BLEU"

# example usage 
example = Example("input"="...", "actual_output"="...", "expected_output"="...")
scorer = BLEUScorer()
results = scorer.score_example(example)
print(results)
```

<Note>
For a deeper dive into making custom scorers, see the custom scorers [page](/evaluation/scorers/custom_scorers).
</Note>

## Running Multiple Scorers Simultaneously

If you're interested in measuring multiple metrics at once, you can group scorers together when running evaluations, 
regardless of the type of scorer.

```python multiple_scorers.py
from judgeval import JudgmentClient
from judgeval.scorers import FaithfulnessScorer, SummarizationScorer

client = JudgmentClient()

faithfulness_scorer = FaithfulnessScorer(threshold=0.5)
summarization_scorer = SummarizationScorer(threshold=0.8)

results = client.run_evaluation(
    examples=[example],
    scorers=[faithfulness_scorer, summarization_scorer],
    model="gpt-4o",
)
```

# Create Your First Dataset 
In most cases, you will not be running evaluations on a single example; instead, you will be scoring your LLM system on a dataset. 
Judgeval allows you to create datasets, save them, and run evaluations on them. 
An `EvalDataset` is a collection of `Example`s and/or `GroundTruthExample`s.

<Note>
A `GroundTruthExample` is an `Example` that has no `actual_output` field since it will be generated at test time.
</Note>

```python create_dataset.py
from judgeval.data import Example, GroundTruthExample, EvalDataset

example1 = Example(input="...", actual_output="...")
example2 = Example(input="...", actual_output="...")

dataset = EvalDataset(examples=[example1, example2])
```

Then, you can run evaluations on the dataset:

```python evaluate_dataset.py
...

client = JudgmentClient()
scorer = FaithfulnessScorer(threshold=0.5)
results = client.evaluate_dataset(
    dataset=dataset,
    scorers=[scorer],
    model="QWEN",
)
```


# Using Judgment Labs Platform
 
When scaling your evaluations beyond a few examples, Judgment's platform allows you to manage your **evaluations, 
datasets, and scorers** in a single place. 

To get started, create a Judgment account by emailing us at `contact@judgmentlabs.ai`. 
We'll get you set up with a login and you'll be able to:
- Run evaluations directly on Judgment's platform
- Track and inspect evaluations with an **intuitive UI**
- Compare your evaluations **across iterations** of your LLM system.
- Manage your datasets and scorers
- Monitor your LLM systems in production, gaining insights into your LLM system's performance in **real-world scenarios.**

<Note> 
Click [here](/judgment/introduction) to learn more about Judgment Labs' platform. 
</Note>

## Running Evaluations on Judgment

Work in progress!

## Managing Datasets

Work in progress!

## Optimizing Your LLM System

Evaluation is a **prerequisite** for optimizing your LLM systems. Measuring the quality of your LLM workflows 
allows you to **compare design iterations** and ultimately find the **optimal set of prompts, models, RAG architectures, etc.** that 
make your LLM perform best. 

**A typical experimental setup might look like this:**

1. Create a new `Project` in the Judgment platform by either running an evaluation from the SDK or via the platform UI. 
This will help you keep track of all evaluations for different iterations of your LLM system.

<Note> 
A `Project` keeps track of `Evaluation Run`s in your project. Each `Evaluation Run` contains a set of `Scorer`s that have been run on a set of `Example`s.
</Note>

2. You can create separate `Evaluation Run`s for different iterations of your LLM system, allowing you to independently test each component of your LLM system.

<Tip>
You can try different models (e.g. `gpt-4o`, `claude-3-5-sonnet`, etc.) and prompt templates in each `Evaluation Run` to find the 
optimal setup for your LLM system. 
</Tip>

## Monitoring LLM Systems in Production

Beyond experimenting and measuring historical performance, `judgeval` supports monitoring your LLM systems in **production**. 
Using our `tracing` module, you can **track your LLM system outputs from end to end**, allowing you to visualize the flow of your LLM system. 
Additionally, you can **enable evaluations to run in real-time** using Judgment's state-of-the-art judge models. 

<div style={{display: 'flex', justifyContent: 'center'}}>
  ![Alt text](/images/trace_screenshot.png "Image of a RAG pipeline trace")
</div>

There are many benefits of monitoring your LLM systems in production with `judgeval`, including:
- Detecting hallucinations and other quality issues **before they reach your customers**
- Automatically creating experimental datasets from your **real-world production cases** for future improvement/optimization
- Track and create alerts on **any metric** (e.g. latency, cost, hallucination, etc.)

<Tip>
To learn more about `judgeval`'s tracing module, click [here](/tracing/introduction).
</Tip>
