"""Core metric architecture for evaLLM.

![status](../../../badges/status-evallm-metrics.svg)

This package provides the metric contract, composition model, registration/discovery,
and configuration-schema machinery used by evaLLM's evaluation pipeline.

!!! abstract
    `evallm.metrics` defines the metric subsystem end-to-end: `base` specifies the
    shared metric contract and traversal helpers, `composite` provides recursive
    multi-metric aggregation, `config` handles runtime schema definition and
    validation, `registry` provides discovery and registration, and `builtin`
    contains reference implementations shipped with evaLLM.

## Design Decisions

:material-source-branch: **Decision**: Use a *Composite* tree for multi-metric execution.
:   :material-lightbulb-on-outline: **Motivation**: evaLLM must run all configured
    metrics and merge their outputs for the same input record, not short-circuit
    through handlers.

:material-source-branch: **Decision**: Enforce tree invariants with identity-based membership.
:   :material-lightbulb-on-outline: **Motivation**: `CompositeMetric.add/remove`
    operates on node identity (`is`), not class equivalence, so duplicate classes are
    allowed while preserving strict parent ownership and cycle-free structure.

:material-source-branch: **Decision**: Validate metric value type at class-definition time.
:   :material-lightbulb-on-outline: **Motivation**: `BaseMetric[T]` captures `T`
    on subclass creation, verifies it is serializable, and validates `evaluate`
    signature, failing fast for invalid metric implementations.

:material-source-branch: **Decision**: Keep metric configuration declarative and strict.
:   :material-lightbulb-on-outline: **Motivation**: `config_schema` defines runtime
    parameters; validation happens in `BaseMetric.__init__`/`from_config`, and
    `DefaultSchema` forbids parameters when none are declared.

:material-source-branch: **Decision**: Discover metrics through an import-time catalog.
:   :material-lightbulb-on-outline: **Motivation**: `@register_metric` writes classes
    into a `Catalog` implementation (`MetricsCatalog` by default), enabling
    pluggability without hard-coded metric lists.

:material-source-branch: **Decision**: Distinguish class identity from result-field identity.
:   :material-lightbulb-on-outline: **Motivation**: `name` identifies the metric class
    in the registry, while `result_key` names one metric instance's output field.
    This allows multiple differently-configured instances of the same class.

## Composite Invariants

The composite implementation enforces the following invariants:

- **Tree shape**: add/remove operations do not create cycles.
- **Single parent**: each metric instance has at most one parent at a time.
- **Identity-based membership**: node identity, not class equality, determines
   whether two metrics are "the same node" in the tree.

## Extension Path for Custom Metrics

1. Subclass `BaseMetric[T]` with a serializable value type.
2. Implement `evaluate(text) -> dict[str, T]`.
3. Optionally define `config_schema` (or use `@config_schema`) for runtime parameters.
4. Optionally register with `@register_metric` to make the metric discoverable.

???+ example annotate
    A custom metric that computes token lengths.

    ```python
    from evallm.metrics.base import BaseMetric
    from evallm.metrics.registry import register_metric


    @register_metric(name="token_lengths")  # (1)!
    class CustomMetric(
        BaseMetric[list[int]]  # (2)!
    ):
        def evaluate(self, text: str) -> dict[str, list[int]]:
            res: list[int]  # (3)!

            if text is None or text.strip() == "":  # (4)!
                res = []
            else:  # (5)!
                res = [len(tok) for tok in text.split()]

            return self.result(res)  # (6)!
    ```

1.  or simply [`@register_metric`][evallm.metrics.registry.register_metric] to use the
    metric class' name as the metric's name in the catalog.
2.  parametrize [`BaseMetric`][evallm.metrics.base.BaseMetric] with the desired value
    type (in this case, `list[int]`).
3.  Ensure that the computed value's type matches the metric's declared
    [`value_type`][evallm.metrics.base.BaseMetric.value_type].
4.  Make sure to handle edge cases!
5.  Implement the evaluation logic
6.  Return the computed value in a dictionary, keyed by the metric's
    [`result_key`][evallm.metrics.base.BaseMetric.result_key] using the
    [`result(value)`][evallm.metrics.base.BaseMetric.result] helper method.

### 1. Parametrizing BaseMetric with a value type

When creating a custom metric, the first question you need to answer is: _"What type
of value do I want my metric to return?"_

This could be any JSON-serializable type, such as a string, number, boolean, list, or
dictionary (see
[`SerializableValueType`][evallm.metrics.types.SerializableValueType]
for the full list of supported types).

The metric's value type is then specified by parametrizing the `BaseMetric[T]` generic
class with the desired type (i.e. binding your desired type to `T`).

### 2. Implementing the evaluate method

In order for your metric to be used in an evaluation pipeline, you need to tell evaLLM
how to compute its value from an input string. At the least, this involves:

- handling edge cases (how should an empty or `None` input be evaluated?),
- defining the core evaluation logic,
- ensuring that the computed value's type matches with the type you declared in
    [step 1](#1-parametrizing-`BaseMetric`-with-a-value-type), and
- returning it in a dictionary, keyed by the metric's
    [`result_key`][evallm.metrics.base.BaseMetric.result_key] (the
    [`result(value)`][evallm.metrics.base.BaseMetric.result] helper
    does this for you).

### 3. Registering your metric

Finally, if you want your metric to be discoverable by evaLLM's evaluation pipelines,
decorate your metric class with the
[`@register_metric`][evallm.metrics.registry.register_metric] decorator.
By default, this will register your metric in the
[`MetricsCatalog`][evallm.metrics.registry.MetricsCatalog] under the name
of your metric class (use the `name` argument of the decorator to specify a custom name
).
"""

__all__ = []
