# Evals: Evaluating LLM Outputs

!!! mira ""

    <div align="center">
        If you haven't already, we recommend first reading the section on [Response Models](./response_models.md)
    </div>

Evaluating the outputs of Large Language Models (LLMs) is a crucial step in developing robust and reliable AI applications. This section covers various approaches to evaluating LLM outputs, including using LLMs as evaluators as well as implementing hardcoded evaluation criteria.

## What are "Evals"?

Evals, short for evaluations, are methods used to assess the quality, accuracy, and appropriateness of LLM outputs. These evaluations can range from simple checks to complex, multi-faceted assessments. The choice of evaluation method depends on the specific requirements of your application and the nature of the LLM outputs you're working with.

!!! warning "Avoid General Evals"

    The following documentation uses examples that are more general in their evaluation criteria. It is extremely important that you tailor your own evaluations to your specific task. While general evaluation templates can act as a good way to get started, we do not recommend relying on such criteria to evaluate the quality of your outputs. Instead, focus on engineering your evaluations such that they match your specific task and criteria to maximize the chance you are successfully measuring quality.

## LLM Evaluators

One powerful approach to evaluating LLM outputs is to use other LLMs as evaluators. This method leverages the language understanding capabilities of LLMs to perform nuanced evaluations that might be difficult to achieve with hardcoded criteria.

!!! mira ""

    {% for method, method_title in zip(prompt_writing_methods, prompt_writing_method_titles) %}
    === "{{ method_title }}"

        {% for provider in supported_llm_providers %}
        === "{{ provider }}"

            {% if method == "string_template" %}
            ```python hl_lines="6-7 20-25 40 47"
            {% elif method == "messages" %}
            ```python hl_lines="8-9 24-29 44 51"
            {% elif method == "base_message_param" %}
            ```python hl_lines="8-9 26-31 47 54"
            {% else %}
            ```python hl_lines="8-9 23-28 42 49"
            {% endif %}
            --8<-- "examples/learn/evals/llm/{{ provider | provider_dir }}/{{ method }}.py"
            ```
        {% endfor %}

    {% endfor %}

In this example we

1. Define an `Eval` response model for structuring the evaluators output to include its `reasoning` and `score`.
2. Create an LLM evaluator using the `call` decorator, carefully engineering the response to define the scoring criteria clearly.
3. Run the evaluator on sample text (in place of text generated by a different LLM call).

!!! tip "Include all relevant information"

    When evaluating LLM outputs with LLMs, it's important to provide the LLM evaluator with all relevant information. For example, you'll want to include the input to the original call along with the generated text.

### Panel of Judges

Rather than using a single model to evaluate your LLM outputs, you can use multiple models as a panel of judges to get a more varied, balanced assessment.

Let's take a look at an example using OpenAI and Anthropic:

!!! mira ""

    {% for method, method_title in zip(prompt_writing_methods, prompt_writing_method_titles) %}
    === "{{ method_title }}"


        {% if method == "string_template" %}
        ```python hl_lines="34-37 40-43 49-50"
        {% elif method == "messages" %}
        ```python hl_lines="39-42 45-48 54-55"
        {% elif method == "base_message_param" %}
        ```python hl_lines="42-45 48-51 57-58"
        {% else %}
        ```python hl_lines="37-40 43-46 52-53"
        {% endif %}
        --8<-- "examples/learn/evals/panel/{{ method }}.py"
        ```

    {% endfor %}

We are taking advantage of [provider-agnostic prompts](./calls.md#provider-agnostic-usage) in this example to easily call multiple providers with the same prompt. Of course, you can always engineer each judge specifically for a given provider instead.

!!! tip "Async for parallel evaluations"

    We highly recommend using [parallel asynchronous calls](./async.md#parallel-async-calls) to run your evaluations more quickly since each call can (and should) be run in parallel.

## Hardcoded Evaluation Criteria

While LLM-based evaluations are powerful, there are cases where simpler, hardcoded criteria can be more appropriate. These methods are particularly useful for evaluating specific, well-defined aspects of LLM outputs.

Here are a few examples of such hardcoded evaluations:

!!! mira ""

    === "Exact Match"

        ```python hl_lines="2"
        --8<-- "examples/learn/evals/hardcoded/exact_match.py"
        ```

    === "Recall and Precision"

        ```python hl_lines="7-8"
        --8<-- "examples/learn/evals/hardcoded/recall_and_precision.py"
        ```

    === "Regular Expression"

        ```python hl_lines="5-6"
        --8<-- "examples/learn/evals/hardcoded/regex.py"
        ```

## Next Steps

By leveraging a combination of LLM-based evaluations and hardcoded criteria, you can create robust and nuanced evaluation systems for LLM outputs. Remember to continually refine your approach based on the specific needs of your application and the evolving capabilities of language models.

Next, we recommend taking a look at our [evaluation tutorials](../tutorials/evals/evaluating_web_search_agent.ipynb)
