# MEDS Evaluation

Evaluation API for the MEDS Decentralized Extensible Validation
[MEDS-DEV](https://github.com/mmcdermott/MEDS-DEV/) Benchmark.

> \[!NOTE\]
> This is a **work-in-progress** package and currently only supports evaluation of binary classification
> tasks.

# Intended usage

MEDS Evaluation pipeline is intended to be used together with
[MEDS-DEV](https://github.com/mmcdermott/MEDS-DEV/), but can also be adapted to use as a standalone package.

Please refer to the
[MEDS-DEV tutorial](https://github.com/mmcdermott/MEDS-DEV?tab=readme-ov-file#example-workflow) to learn how
to extract and prepare the data in the MEDS format and obtain model predictions ready to be evaluated.

# Prediction schema

Inputs to MEDS Evaluation must follow the *prediction schema*, which by default has five fields:

1. `subject_id`: ID of the subject (patient) associated with the event
2. `prediction_time`: time at which the prediction as being made
3. `boolean_value`: ground truth boolean label for the prediction task
4. `predicted_boolean_value` (optional): predicted boolean label generated by the model
5. `predicted_boolean_probability` (optional): predicted probability logits generated by the model

This is equivalent to the following `polars` schema:

```python
Schema(
    [
        ("subject_id", Int64),
        ("prediction_time", Datetime(time_unit="us")),
        ("boolean_value", Boolean),
        ("predicted_boolean_value", Boolean),
        ("predicted_boolean_probability", Float64),
    ]
)
```

Note that while `predicted_boolean_value` and `predicted_boolean_probability` are optional, at least one of
them must be present and contain non-null values in order to generate the results. In addition, a schema can
contain additional fields but at the moment these will not be used in MEDS Evaluation.
