Metadata-Version: 2.1
Name: meds-evaluation
Version: 0.0.1
Summary: A module for evaluating the predictions of the models trained on MEDS datasets.
Author-email: Kamilė Stankevičiūtė <kamile.stankeviciute@cst.cam.ac.uk>, Vincent Jeanselme <vincent.jeanselme@mrc-bsu.cam.ac.uk>, Matthew McDermott <mattmcdermott8@gmail.com>
Project-URL: Homepage, https://github.com/kamilest/meds-evaluation
Project-URL: Issues, https://github.com/kamilest/meds-evaluation/issues
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: polars>=1.5.0
Requires-Dist: pyarrow
Requires-Dist: numpy
Requires-Dist: scikit-learn
Requires-Dist: hydra-core
Requires-Dist: loguru
Provides-Extra: dev
Requires-Dist: pre-commit; extra == "dev"
Provides-Extra: tests
Requires-Dist: pytest; extra == "tests"

# MEDS Evaluation

Evaluation API for the MEDS Decentralized Extensible Validation
[MEDS-DEV](https://github.com/mmcdermott/MEDS-DEV/) Benchmark.

> \[!NOTE\]
> This is a **work-in-progress** package and currently only supports evaluation of binary classification
> tasks.

# Intended usage

MEDS Evaluation pipeline is intended to be used together with
[MEDS-DEV](https://github.com/mmcdermott/MEDS-DEV/), but can also be adapted to use as a standalone package.

Please refer to the
[MEDS-DEV tutorial](https://github.com/mmcdermott/MEDS-DEV?tab=readme-ov-file#example-workflow) to learn how
to extract and prepare the data in the MEDS format and obtain model predictions ready to be evaluated.

# Prediction schema

Inputs to MEDS Evaluation must follow the *prediction schema*, which by default has five fields:

1. `subject_id`: ID of the subject (patient) associated with the event
2. `prediction_time`: time at which the prediction as being made
3. `boolean_value`: ground truth boolean label for the prediction task
4. `predicted_boolean_value` (optional): predicted boolean label generated by the model
5. `predicted_boolean_probability` (optional): predicted probability logits generated by the model

This is equivalent to the following `polars` schema:

```python
Schema(
    [
        ("subject_id", Int64),
        ("prediction_time", Datetime(time_unit="us")),
        ("boolean_value", Boolean),
        ("predicted_boolean_value", Boolean),
        ("predicted_boolean_probability", Float64),
    ]
)
```

Note that while `predicted_boolean_value` and `predicted_boolean_probability` are optional, at least one of
them must be present and contain non-null values in order to generate the results. In addition, a schema can
contain additional fields but at the moment these will not be used in MEDS Evaluation.
