# Scorers {#sec-scorers}

## Overview

Scorers evaluate whether solvers were successful in finding the right `output` for the `target` defined in the dataset, and in what measure. Scorers generally take one of the following forms:

1.  Extracting a specific answer out of a model's completion output using a variety of heuristics.

2.  Applying a text similarity algorithm to see if the model's completion is close to what is set out in the `target`.

3.  Using another model to assess whether the model's completion satisfies a description of the ideal answer in `target`.

4.  Using another rubric entirely (e.g. did the model produce a valid version of a file format, etc.)

Scorers also define one or more metrics which are used to aggregate scores (e.g. `accuracy()` which computes what percentage of scores are correct, or `mean()` which provides an average for scores that exist on a continuum).

## Built-In Scorers

Inspect includes some simple text matching scorers as well as a couple of model graded scorers. Built in scorers can be imported from the `inspect_ai.scorer` module. Below is a summary of these scorers. There is not (yet) reference documentation on these functions so the best way to learn about how they can be customised, etc. is to use the **Go to Definition** command in your source editor.

-   `includes()`

    Determine whether the `target` from the `Sample` appears anywhere inside the model output. Can be case sensitive or insensitive (defaults to the latter).

-   `match()`

    Determine whether the `target` from the `Sample` appears at the beginning or end of model output (defaults to looking at the end). Has options for ignoring case, white-space, and punctuation (all are ignored by default).

-   `pattern()`

    Extract the answer from model output using a regular expression.

-   `answer()`

    Scorer for model output that preceded answers with "ANSWER: ". Can extract letters,
    words, or the remainder of the line.

-   `model_graded_qa()`

    Have another model assess whether the model output is a correct answer based on the grading guidance contained in `target`. Has a built-in template that can be customized.

-   `model_graded_fact()`

    Have another model assess whether the model output contains a fact that is set out in `target`. This is a more narrow assessment than `model_graded_qa()`, and is used when model output is too complex to be assessed using a simple `match()` or `pattern()` scorer.

Scorers provide one or more built-in metrics (each of the scorers above provides `accuracy` as a metric). You can also provide your own custom metrics in `Task` definitions. For example:

``` python
Task(
    dataset=dataset,
    plan=[
        system_message(SYSTEM_MESSAGE),
        multiple_choice()
    ],
    scorer=match(),
    metrics=[custom_metric()]
)
```

## Custom Scorers

Let's take a look at the source code for a couple of the built in scorers as a jumping off point for implementing your own scorers. If you are working on custom scorers, you should also review the [Scorer Workflow](#sec-scorer-workflow) section below for tips on optimising your development process.

::: {.callout-note appearance="simple"}
When creating custom scorers, it's critical that you understand Inspect's concurrency model. More specifically, if your scorer is doing non-trivial work (e.g. calling REST APIs, executing external processes, etc.) please review [Eval Tuning](#sec-eval-tuning) before proceeding.
:::

### Example: Includes

``` python
@scorer(metrics=[accuracy()])
def includes(ignore_case: bool = True):

    async def score(state: TaskState, target: Target):

        # check for correct
        value = state.output.completion
        answer = target.text
        if ignore_case:
            correct = value.lower().rfind(answer.lower()) != -1
        else:
            correct = value.rfind(answer) != -1

        # return score
        return Score(
            value = "C" if correct else "I",
            explanation=state.output.completion
        )

    return score
```

A few things to note about this implementation:

1.  The function applies the `@scorer` decorator—this registers the `Scorer` with Inspect, making it possible to capture its name and parameters for logging, as well as make it callable from a configuration file (e.g. a YAML specification of an eval).
2.  The `score()` function is declared as `async`. This is so that it can participate in Inspect's optimised scheduling for expensive model generation calls (this scorer doesn't call a model but others will).
3.  We make use of the `text` property on the `Target`. This is a convenience property to get a simple text value out of the `Target` (as targets can technically be a list of strings).
4.  We provide the full model completion as the explanation for the score (`explanation` is optional, but often useful to refer to during evaluation development).

### Example: Model Graded QA

Here's the code for the `model_graded_qa()` scorer:

``` python
MODEL_GRADED_QA_TEMPLATE = f"""
You are assessing a submitted answer on a given task based on a
criterion. Here is the data:

[BEGIN DATA]
***
[Task]: {{question}}
***
[Submission]: {{answer}}
***
[Criterion]: {{criterion}}
***
[END DATA]

Does the submission meet the criterion?

Reasoning:
"""

@scorer(metrics=[accuracy()])
def model_graded_qa(
    template: str = MODEL_GRADED_QA_TEMPLATE,
    extractor: Callable[[ModelOutput], str] | None = None,
    model: str | Model | None = None,
) -> Scorer:
    # resolve model
    grader_model = get_model(model)

    # resolve grading template
    template = resource(template)

    # provide default scoring function if required
    extractor = extractor if extractor else extract_grade

    async def score(state: TaskState, target: Target) -> Score:
        # format the scoring template
        score_prompt = template.format(
            question=state.input_text,
            answer=state.output.completion,
            criterion=target.text,
        )

        # query the model for the score
        score = await grader_model.generate(score_prompt)

        # return score (reduced by extractor) with explanation
        return Score(
            value=extractor(score),
            explanation=score.completion,
        )

    return score

def extract_grade(output: ModelOutput) -> str:
    text: str = output.completion
    match = re.search("Grade: .", text)
    if match is None:
        raise ValueError("No grade found in model output.")
    return text[match.end() - 1]
```

Note that the call to `model_grader.generate()` is done with `await`—this is critical to ensure that the scorer participates correctly in the scheduling of generation work.

There is one other thing to note: we use the `input_text` property of the `TaskState` to access a string version of the original user input to substitute it into the grading template. Using the `input_text` has two benefits: (1) It is guaranteed to cover the original input from the dataset (rather than a transformed prompt in `messages`); and (2) It normalises the input to a string (as it could have been a message list).

## Metrics

Each scorer provides one or more built-in metrics (typically `accuracy` and `bootstrap_std`). In addition, you can specify other metrics (either built-in or custom) to compute when defining a `Task`:

``` python
Task(
    dataset=dataset,
    plan=[
        system_message(SYSTEM_MESSAGE),
        multiple_choice()
    ],
    scorer=match(),
    metrics=[custom_metric()]
)
```

### Built-In Metrics

Inspect includes some simple built in metrics for calculating accuracy, mean, etc. Built in metrics can be imported from the `inspect_ai.scorer` module. Below is a summary of these metrics. There is not (yet) reference documentation on these functions so the best way to learn about how they can be customised, etc. is to use the **Go to Definition** command in your source editor.

-   `accuracy()`

    Compute proportion of total answers which are correct. For correct/incorrect scores assigned 1 or 0, can optionally assign 0.5 for partially correct answers.

-   `mean()`

    Mean of all scores.

-   `var()`

    Variance over all scores.

-   `bootstrap_std()`

    Standard deviation of a bootstrapped estimate of the mean. 1000 samples are taken by default (modify this using the `num_samples` option).

### Custom Metrics

You can also add your own metrics with `@metric` decorated functions. For example, here is the implementation of the variance metric:

``` python
import numpy as np

from inspect_ai.scorer import Metric, Score, metric

def var() -> Metric:
    """Compute variance over all scores."""

    def metric(scores: list[Score]) -> float:
        return np.var([score.as_float() for score in scores]).item()

    return metric
```

## Workflow {#sec-scorer-workflow}

### Score Command

By default, model output in evaluations is automatically scored. However, you can separate generation and scoring by using the `--no-score` option. For example:

``` bash
inspect eval popularity.py --model openai/gpt-4 --no-score
```

You can score an evaluation previously run this way using the `inspect score` command:

``` bash
# score last eval
inspect score popularity.py

# score specific log file
inspect score popularity.py ./logs/2024-02-23_task_gpt-4_TUhnCn473c6.json
```

::: callout-tip
Using a distinct scoring step is particularly useful during scorer development, as it bypasses the entire generation phase, saving lots of time and inference costs.
:::

### Log Overwriting

By default, `inspect score` overwrites the file it scores. If don't want to overwrite target files, pass the `--no-overwrite` flag:

``` bash
inspect score popularity.py --no-overwrite
```

When specifying `--no-overwrite`, a `-scored` suffix will be added to the original log file name:

``` bash
./logs/2024-02-23_task_gpt-4_TUhnCn473c6-scored.json
```

Note that the `--no-overwrite` flag does not apply to log files that already have the `-scored` suffix—those files are always overwritten by `inspect score`. If you plan on scoring multiple times and you want to save each scoring output, you will want to copy the log to another location before re-scoring.

### Python API

If you are exploring the performance of different scorers, you might find it more useful to call the `score()` function using varying scorers or scorer options. For example:

``` python
log = eval(popularity, model="openai/gpt-4")[0]

grader_models = [
    "openai/gpt-4",
    "anthropic/claude-3-opus-20240229",
    "google/gemini-1.0-pro",
    "mistral/mistral-large-latest"
]

scoring_logs = [score(log, model_graded_qa(model=model)) 
                for model in grader_models]

plot_results(scoring_logs)
```