---
title: Introduction
---

## Overview

Scorers act as measurement tools for evaluating LLM systems based on specific criteria. 
`judgeval` comes with a set of **10+ built-in scorers** that you can easily start with.

<Tip>
We're always adding new scorers to `judgeval`. If you have a suggestion, please [let us know](mailto:contact@judgmentlabs.ai)!
</Tip>

Scorers execute on `Example`s and `EvalDataset`s, producing a **numerical score**. 
This enables you to **use evaluations as unit tests** by setting a `threshold` to determine whether an evaluation was successful or not. 

## Categories of Scorers
`judgeval` supports three categories of scorers. 
- **Default Scorers**: built-in scorers that are ready to use
- **Custom Scorers**: Powerful scorers that you can tailor to your own LLM system
- **Classifier Scorers**: A special custom scorer that evaluates your LLM system using a natural language criteria

In this section, we'll cover each kind of scorer and how to use them.

### Default Scorers
Most of the built-in scorers in `judgeval` are **LLM judges**, meaning they use LLMs to evaluate your LLM system. 
This is intentional since **LLM evaluations are flexible, scalable, and strongly correlate with human evaluation**.

`judgeval`'s default scorers have been meticulously crafted by our research team based on leading work in the LLM evaluation community. 

Our implementations are described on their respective documentation pages. 
Judgment implementations of default scorers are backed by **leading industry/academic research** and are preferable to other implementations because:
- They are meticulously prompt-engineered to maximize evaluation quality and consistency
- Provide a chain of thought for evaluation scores, so you can double-check the evaluation quality
- Can be run using any LLM, including Judgment's **state-of-the-art LLM judges** developed in collaboration with **Stanford's AI Lab**.

Here are the default scorers in `judgeval`:
- [Answer Correctness](/evaluation/scorers/answer_correctness)
- [Answer Relevancy](/evaluation/scorers/answer_relevancy)
- [Comparison](/evaluation/scorers/comparison)
- [Contextual Precision](/evaluation/scorers/contextual_precision)
- [Contextual Recall](/evaluation/scorers/contextual_recall)
- [Contextual Relevancy](/evaluation/scorers/contextual_relevancy)
- [Execution Order](/evaluation/scorers/execution_order)
- [Faithfulness](/evaluation/scorers/faithfulness)
- [Groundedness](/evaluation/scorers/groundedness)
- [JSON Correctness](/evaluation/scorers/json_correctness)
- [Summarization](/evaluation/scorers/summarization)


### Agentic Scorers

Agentic scorers are designed to evaluate multiple steps together within your LLM system—rather than evaluating one step at a time. They are especially useful for multi-step workflows, such as agents, chains, or graphs, where the quality of the overall process matters more than individual outputs.

By grouping related steps into a single evaluation, agentic scorers allow you to:
- Measure end-to-end correctness
- Evaluate goal completion across multiple steps
- Assess coherence and strategy in long chains of reasoning

This makes them ideal for agentic systems, where individual tool calls or LLM responses should be judged as part of a larger goal-oriented behavior.

Here are the agentic scorers in `judgeval`:
- [Derailment](/evaluation/scorers/agent/derailment)

### Custom Scorers

If you find that none of the default scorers meet your evaluation needs, setting up a custom scorer is easy with `judgeval`.
You can create a custom scorer by inheritng from the `JudgevalScorer` class and implementing three methods:
- `score_example()`: produces a score for a single `Example`.
- `a_score_example()`: async version of `score_example()`. You may use the same implementation logic as `score_example()`.
- `_success_check()`: determines whether an evaluation was successful.

Custom scorers can be as simple or complex as you want, and **do not need to use LLMs**. 
For sample implementations, check out the [Custom Scorers](/evaluation/scorers/custom_scorers) documentation page.


### Classifier Scorers

Classifier scorers are a special type of custom scorer that can evaluate your LLM system using a natural language criteria. 

They either be defined using our judgeval SDK or using the Judgment Platform directly. For more information, check out the [Classifier Scorers](/evaluation/scorers/classifier_scorer) documentation page.

## Running Scorers

All scorers in `judgeval` can be run uniformly through the `JudgmentClient`. All scorers are set to run in async mode by default in order to support parallelized evaluations for large datasets.

```python run_scorer.py
...

client = JudgmentClient()
results = client.run_evaluation(
    examples=[example],
    scorers=[scorer],
    model="gpt-4o",
)
```

<Tip>  
To learn about how a certain default scorer works, check out its documentation page for a deep dive into how scores are calculated and what fields are required.
</Tip>
