"""CLI for evaLLM

![status](badges/status-cli-main.svg)

## CLI Usage

### Evaluate a dataset

=== "Recommended: use a config file"
    ```bash
    evallm run --config ./run.yaml
    ```

=== "Alternative: use flags only"
    ```bash
    evallm run \\
      --input-file ./data/dataset.jsonl \\
      --text-key text \\
      --record-id-key id \\
      --metric CountsMetric
    ```

    This works best for metrics that require no runtime configuration, or for
    preset forms.

### Configuration model

`evallm run` needs two top-level sections:

1. `dataset`
    - `input_file` (required): path to JSONL file
    - `text_key` (required): key containing text to evaluate
    - `record_id_key` (optional): key for record IDs (`#IDX#` fallback)
2. `metrics`
    - required list of one or more metric specs
    - each spec supports `name`, optional `config`, optional `result_key`

You can provide configuration in two ways:

- through CLI flags, or
- as a config file (`--config`, JSON or YAML)

!!! warning
    If both are used, config file values are loaded first and CLI flags override
    them. **If any `--metric` flags are passed, they replace `config.metrics`!**

!!! info "Supported `--metric` forms"

    === "Registered metric name"
        ```bash
        evallm run -f ./data/dataset.jsonl -t text -i id -m CountsMetric
        ```

    === "Convenience preset"
        ```bash
        evallm run -f ./data/dataset.jsonl -t text -i id -m count_characters
        ```

    === "Preset with argument"
        ```bash
        evallm run -f ./data/dataset.jsonl -t text -i id -m count_tokens:wordpunct
        evallm run -f ./data/dataset.jsonl -t text -i id -m count_tokens:whitespace
        evallm run -f ./data/dataset.jsonl -t text -i id -m count_encodings:bert-base-uncased
        ```

    === "Inline JSON metric spec"
        ```bash
        evallm run \\
          -f ./data/dataset.jsonl -t text -i id \\
          -m '{"name":"CountsMetric","config":{"segments":"tokens"},"result_key":"token_count"}'
        ```

    === "Metric spec from JSON/YAML file"
        ```bash
        evallm run -f ./data/dataset.jsonl -t text -i id -m ./metrics/counts.json
        evallm run -f ./data/dataset.jsonl -t text -i id -m ./metrics/counts.yaml
        ```

#### Configuration file

=== "YAML"
    ```yaml
    # eval_config.yml
    dataset:
        input_file: ./data/dataset.jsonl
        text_key: text
        record_id_key: id
    metrics:
        - name: CountsMetric
            result_key: chars
        - name: CountsMetric
            config:
            segments: tokens
            segmentation: whitespace
            result_key: tokens
        - name: CountsMetric
            config:
            segments: encodings
            segmentation: meta-llama/Llama-3.1-8B
            result_key: llama_3_1
        - name: JsonFormatMetric
            config:
            expected_schema:
                type: object
                properties:
                question:
                    type: string
                    pattern: '\\?$'
                setup:
                    type: string
                joke:
                    type: string
                funniness_score:
                    type: number
                    minimum: 0.0
                    maximum: 1.0
                required:
                - question
                - joke
            check_formats: true
            result_key: structured_output
    ```

=== "JSON"
    ```json
    // eval_config.json
    {
        "dataset": {
            "input_file": "./data/dataset.jsonl",
            "text_key": "text",
            "record_id_key": "id"
        },
        "metrics": [
            {
                "name": "CountsMetric",
                "result_key": "chars"
            },
            {
                "name": "CountsMetric",
                "config": {
                    "segments": "tokens",
                    "segmentation": "whitespace"
                },
                "result_key": "tokens"
            },
            {
                "name": "CountsMetric",
                "config": {
                    "segments": "encodings",
                    "segmentation": "meta-llama/Llama-3.1-8B"
                },
                "result_key": "llama_3_1"
            },
            {
                "name": "JsonFormatMetric",
                "config": {
                    "expected_schema": {
                        "type": "object",
                        "properties": {
                            "question": {
                                "type": "string",
                                "pattern": "\\\\?$"
                            },
                            "setup": {
                                "type": "string"
                            },
                            "joke": {
                                "type": "string"
                            },
                            "funniness_score": {
                                "type": "number",
                                "minimum": 0.0,
                                "maximum": 1.0
                            }
                        },
                        "required": [
                            "question",
                            "joke"
                        ]
                    },
                    "check_formats": true
                },
                "result_key": "structured_output"
            }
        ]
    }
    ```

### Process output

Each record's evaluation result is serialized as a JSON object. By default,
results are streamed to the console, which lets you pipe them however you want.
You can of course also have them written to a results file in JSONL format.

=== "Stream to stdout (default)"
    ```bash
    evallm run -f ./data/dataset.jsonl -t text -i id -m CountsMetric
    ```

=== "Write to a file"
    ```bash
    evallm run \\
      -f ./data/dataset.jsonl -t text -i id \\
      -m CountsMetric \\
      --output ./results.jsonl
    ```

### Help and discovery

!!! info "General help"
    ```bash
    evallm --help
    evallm run --help
    evallm metrics --help
    ```

!!! info "Metric discovery and configuration help"
    ```bash
    evallm metrics list
    evallm metrics info CountsMetric
    evallm metrics info CountsMetric --format json
    evallm metrics info CountsMetric --format yaml
    evallm metrics presets
    ```
"""
