.. SPDX-FileCopyrightText: Copyright © 2025 Idiap Research Institute <contact@idiap.ch>
..
.. SPDX-License-Identifier: GPL-3.0-or-later

.. _fairical.io:

=============
 I/O Formats
=============

Pydantic_ models for :py:class:`fairical.Scores` and :py:class:`fairical.Solutions` can
be created from or saved into JSON representations that can be stored on disk. When
loaded, JSON representations are validated against the pre-defined schema defined for
those objects.

Scores
------

As indicated in :ref:`fairical.datamodel`, :py:class:`fairical.Scores` correspond to the
primary input to :py:mod:`fairical`. A valid JSON for :py:class:`fairical.Scores`
consists of a dictionary composed of 3 keys:

- **scores:** A list of prediction scores for each Machine Learning (ML) model in
  consideration. Each model (or utility/fairness tradeoff) is represented with a single
  list of floats. A system that has multiple models (utility/fairness tradeoffs) is
  therefore represented by lists of lists of floats. An example of ``scores`` from a
  single-mode model, having predictions for 9 samples is shown below:

  .. code-block:: json

      {
          "scores": [
              [ 0.3970, 0.3434, 0.7074, 0.5787, 0.2451, 0.6383, 0.5937, 0.3629, 0.5526 ]
          ]
      }

  .. caution::

      Even in a single-mode model, prediction scores must be encapsulated within a list
      so that structure is represented as ``"scores": [[.]]``.

  Here is an example of a two-mode model, with predictions for 6 samples:

  .. code-block:: json

      {
          "scores": [
              [ 0.646910, 0.442378, 0.342101, 0.457198, 0.549640, 0.331999 ],
              [ 0.623593, 0.453462, 0.370829, 0.466496, 0.542477, 0.359879 ]
          ]
      }

  In this example, scores for the same underlying sample are paired across the two
  lists. The first entry of each list corresponds to the first sample in a list of test
  samples, the second to the second, and so on.

- **ground-truth:** A list of ground-truth labels paired to each sample in the list or
  lists in ``scores``. Given a binary classification task, an example of
  ``ground-truth`` list with nine test samples is shown below:

  .. code-block:: json

      {
          "ground-truth": [
              1, 0, 0, 0, 0, 0, 0, 1, 0
          ]
      }

- **attributes:** A dictionary of sensitive attributes with each having a list of values
  composed of integers or strings. Similarly to ``ground-truth``, this entry **must be
  paired** with the entries in ``scores`` and ``ground-truth``. An example of
  ``attributes`` dictionary for ``gender`` and ``race`` demographic groups with nine
  test samples is shown below:

  .. code-block:: json

      {
          "attributes": {
              "gender": [
                  "f", "m", "f", "m", "f", "f", "f", "m", "m"
              ],
              "race": [
                  2, 0, 0, 1, 0, 0, 1, 0, 2
              ]
          }
      }

  for a classification task where gender is ``{female, male}`` as two categories ``{"m",
  "f"}`` and race is ``{Asian, Black, White}`` as three categories ``{0, 1, 2}``.

Optionally, an additional ``identifiers`` can be added to give names to each model
defined in the ``scores`` dictionary:

.. code-block:: json

    {
        "identifiers": [
            "my-model-1",
        ]
    }

A complete JSON representation for a single binary classifier analzyed for race with
nine test samples is examplified as below:

.. code-block:: json

    {
        "scores": [
            [ 0.5077, 0.5165, 0.5073, 0.4777, 0.6062, 0.4830, 0.7178, 0.7152, 0.4331 ]
        ],
        "identifiers": [
            "my-model-1"
        ],
        "ground-truth": [
                0, 0, 0, 1, 1, 0, 0, 0, 1
        ],
        "attributes": {
            "race": [
                1, 0, 2, 1, 0, 0, 2, 0, 0
            ]
        }
    }

.. _fairical.usage.io.solutions:

Solutions
---------

As indicated in :ref:`fairical.datamodel`, :py:class:`fairical.Solutions` correspond to
configurable :ref:`utility and fairness performance metrics <fairical.metrics>`
corresponding to operating-points of ML being analyzed, considering **all of its
operating modes** (trade-offs). :py:class:`fairical.Solutions` are computed by this
library from :py:class:`fairical.Scores` and therefore can be saved to a disk
representation, in JSON format.  The user may equally provide a solution JSON file with
pre-calculated OMs for analysis, under-cutting fairical solving.

A valid JSON representation for :py:class:`fairical.Solutions` consists of a dictionary composed of 2 keys:

- **points:** A dictionary of utility and fairness metrics evaluated for all OMs of a model in consideration.
  An example JSON of an ML system with multiple OMs (trade-offs) with one non-dominated solution (NDS) and two
  dominated solutions (DS) is shown below:

.. code-block:: json

    {
        "points": {
            "eod+race": [ 0.104166, 1.0, 1.0 ],
            "eod+gender": [ 0.398741, 1.0, 1.0 ],
            "acc": [ 0.652702, 0.0, 0.0 ]
        }
    }

Here, there one considers two fairness (Equalized Odds Difference for race and gender)
and one utility metric (accuracy).

.. note::

    Dominated solutions in the example are selected to reflect the worst values in
    dimensions of metrics (1.0 for eod and 0.0 for acc).

- **metadata:** Several dictionaries containing information related to the points.
  The expected entries are:

    - **thresholds:** A list specifying the threshold at which each point was computed.

    .. code-block:: json

        {
            "threholds": [0.1, 0.2, 0.3]
        }

    - **identifier-names:** A list of unique model names.
      If ``identifiers`` was not given in the score file, these are generated automatically as
      ``model-1``, .., ``model-n``.

    .. code-block:: json

        {
            "identifier-names": [ "model-1" ]
        }

    - **identifiers:** A list of indexes into identifier-names indicating which model the points are taken from.

    .. code-block:: json

        {
            "identifiers": [ 0, 0, 0 ]
        }

    - **nds-from:** If the file contains a-priori solutions, holds the name of the system the thresholds were taken from.

    .. code-block:: json

        {
            "nds-from": null
        }

A complete JSON representation for a solutions file containing three points composed of two fairness metrics and one utiliy metric is shown below:

.. code-block:: json

    {
        "points": {
            "eod+race": [ 0.104166, 1.0, 1.0 ],
            "eod+gender": [ 0.398741, 1.0, 1.0 ],
            "acc": [ 0.652702, 0.0, 0.0 ]
        }
        "metadata": {
        {
            "threholds": [ 0.1, 0.2, 0.3 ],
            "identifier-names": [ "model-1" ],
            "identifiers": [ 0, 0, 0 ],
            "nds-from": null
        }
    }


.. include:: ../links.rst
