---
title: Classifier Scorers
description: ""
---

A `ClassifierScorer` is a powerful tool for evaluating your LLM system using natural language criteria. 
Classifier scorers are great for prototyping new evaluation criteria on a small set of examples before using them to benchmark your workflows at scale.

## Creating a Classifier Scorer 

### `judgeval` SDK

You can create a `ClassifierScorer` by providing a natural language description of your evaluation task/criteria and a set of choices that an LLM judge can choose from when evaluating an example. 
Here's an example of creating a `ClassifierScorer` that determines if a response is friendly or not:

```python friendliness_scorer.py

from judgeval.scorers import ClassifierScorer

friendliness_scorer = ClassifierScorer(
    name="Friendliness Scorer",
    threshold=1.0,
    conversation=[
        {
            "role": "system", 
            "content": "Is the response positive (Y/N)? The response is: {{actual_output}}."
        }
    ],
    options={"Y": 1, "N": 0}
)
```

<Tip>
You can put variables from [`Example`s](/evaluation/data_examples) into your `conversation` by using the mustache `{{variable_name}}` syntax.
</Tip>

### `Judgment` Platform

1. Navigate to the `Scorers` tab in the Judgment platform. You'll find this on via the sidebar on the left.
2. Click the `Create Scorer` button in the top right corner.

![Alt text](/images/create_scorer.png "Optional title")

3. Here, you can create a custom scorer by using a criteria in natural language, supplying custom arguments from the [`Example`](evaluation/data_examples) class. 
Then, you supply a set of **choices** the scorer can select from when evaluating an example. Finally, you can test your scorer on samples in our playground.

4. Once you're finished, you can save the scorer and use it in your evaluation runs just like any other scorer in `judgeval`.

#### Example 

Here's an example of building a similar `ClassifierScorer` that checks if the LLM's tone is too aggressive. 

![Alt text](/images/create_aggressive_scorer.png "Optional title")


## Using a Classifier Scorer

Classifer scorers can be used in the same way as any other scorer in `judgeval`. 
They can also be run in conjunction with other scorers in a single evaluation run!

```python run_classifier_scorer.py
...

results = client.run_evaluation(
    examples=[example1],
    scorers=[friendliness_scorer],
    model="gpt-4o"
)
```

### Saving Classifier Scorers 

Whether you create a `ClassifierScorer` via the `judgeval` SDK or the Judgment platform, you can save it to the `Judgment` platform for reuse in future evaluations. 
- If you create a `ClassifierScorer` via the `judgeval` SDK, you can save it by calling `client.push_classifier_scorer()`. 
- Similarly, you can load a `ClassifierScorer` by calling `client.fetch_classifier_scorer()`. 
- Each `ClassifierScorer` has a **unique slug** that you can use to identify it. 

```python 
from judgeval import JudgmentClient

client = JudgmentClient()

# Saving a ClassifierScorer from SDK to platform
friendliness_slug = client.push_classifier_scorer(friendliness_scorer)

# Loading a ClassifierScorer from platform to SDK
classifier_scorer = client.fetch_classifier_scorer("classifier-scorer-slug")
```

TODO add image of slugs on the platform
