Detailed overview of MetaSynth
==============================

``Metasynth`` is a python package for generating synthetic data with a focus on maintaining privacy. It is aimed at owners of sensitive datasets such as public organisations, research groups, and individual researchers who want to improve the accessibility, reproducibility and reusability of their data. The goal of ``metasynth`` is to make it easy for data owners to share the structure and approximation of the content of their data with others without any privacy concerns.

With this goal in mind, ``metasynth`` restricts itself to the `\"synthetically-augmented plausible" <https://www.ons.gov.uk/methodology/methodologicalpublications/generalmethodology/onsworkingpaperseries/onsmethodologyworkingpaperseriesnumber16syntheticdatapilot>`__ category of synthetic data, as categorized by the Office for National Statistics (ONS).

.. admonition:: ONS criteria for a Synthetically-augmented plausible dataset:
   
  * Preserve the format and record-level plausibility as detailed previously and replicate marginal (univariate) distributions where possible.
  * Constructed based on the real dataset, values are generated based on observed distributions (with added fuzziness and smoothing) but no attempt made to preserve relationships.
  * Missing value codes and their frequency is to be preserved.
  * Disclosure control evaluation is necessary case by case, special care to be taken with names and so on.
  * To be used for extended code testing, minimal analytical value, non-negligible disclosure risk.


This choice enables the software to generate synthetic data with **privacy and disclosure guarantees** through a plug-in system. Moreover, our system provides an **auditable and editable intermediate representation** in the form of a human- and machine-readable ``.json`` metadata file from which new data can be synthesized.

Through our focus on privacy and transparency, ``metasynth`` explicitly avoids generating synthetic data with high analytical validity. The data generated by our system is realistic in terms of data structure and plausible in terms of values for each variable, but any multivariate relations or conditional patterns are excluded. This has implications for how this synthetic data can be used: not for statistical analysis and inference, but rather for initial exploration, analysis script development, and communication outside the data owner’s institution. In the intended use case, external researchers can make use of the synthetic data to assess the feasibility of their intended research before making the (often time-consuming) step of requesting access to the sensitive source data for the final analysis.

The MetaSynth Pipeline
----------------------
The MetaSynth package offers a seamless and efficient pipeline for synthetic data generation. It is meticulously designed to ensure the privacy-preserving and reproducible generation of realistic tabular data. The three key stages of this pipeline include the **estimation** of the MetaFrame from the original data, the **serialization** of the MetaFrame into an auditable and editable intermediate representation, and the **generation** of the synthetic data from the model represented by the MetaFrame or its serialized representation. This section provides a walkthrough of these steps.

.. image:: /images/pipeline_basic.png
   :alt: MetaSynth Pipeline
   :align: center

.. admonition:: An example use case

  The following illustrates a simple example of MetaSynth being used, and where the steps in the pipeline (estimation, serialization/deserialization and generation) occur. 
  
  A public health, aiming to conduct research on a sensitive dataset could:

    #. Conduct statistical research on a sensitive dataset of medical records.
    #. Fit a MetaFrame to the sensitive dataset. (Estimation)
    #. Export the MetaFrame to a JSON file following the GMF standard. (Serialization)
    #. Share the research report alongside the GMF file and their script(s) used for the analysis. 

  Other researchers can then:

    #. Load the MetaFrame by importing the GMF file. (Deserialization) 
    #. Use the MetaFrame to generate a synthetic dataset. (Generation)
    #. Rerun the scripts on the synthetic data to validate that the *code* used to analyse the results is sound, and behaves as expected. (Note that the *results* themselves can not be reproduced, as the synthetic data generated by MetaSynth is not identical on a statistical level)

  This builds confidence that when the same modeling approach is applied to the real sensitive data, the results and conclusions will be accurate. Sharing MetaFrames, analysis scripts and results in this way improves transparency and trust in data-based research without requiring the release of irreplicable sensitive data. 

  Note that this is just one of many possible ways in which MetaSynth can be used.


Estimation
^^^^^^^^^^^^^
.. image:: /images/pipeline_estimation_simple.png
   :alt: MetaSynth Estimation Step in Pipeline
   :align: center

The generative model for multivariate datasets in ``metasynth`` makes the simplifying assumption of marginal independence: each column is considered separately, just as is done in e.g., `naïve Bayes classifiers <https://springer.com/book/10.1007/978-0-387-84858-7>`_. Formally, this leads to the following generative model for the :math:`K`-variate data :math:`\mathbf{x}`:

.. math::

    p(x) = \prod_{k=1}^K p(x_k)

There are many advantages to this naïve approach when compared to more advanced generative models: it is transparent and explainable, it is able to flexibly handle data of mixed types, and it is computationally scalable to high-dimensional datasets. As mentioned before, the tradeoff is the limited analytical validity when the independence assumption does
not hold: in the synthetic data, the expected value of correlations, regression parameters, and other measures of association is 0.

Model estimation starts with an appropriately pre-processed data frame. For ``metasynth``, this means the data frame is `tidy <https://www.jstatsoft.org/article/view/v059i10>`_, each column has the correct data type, and missing data are represented by a missing value. Internally, our software uses the `Polars <https://www.pola.rs/>`_ data frame library, as it is performant, has consistent data types, and native support for missing data (``null``). A simple example source  table could look like this (note that categorical data has the appropriate ``cat`` data type, not ``str``):

.. list-table::
   :widths: 10 20 10 20 20
   :header-rows: 1

   * - ID (i64)
     - fruits (cat)
     - B (i64)
     - cars (cat)
     - optional (i64)
   * - 1
     - banana
     - 5
     - beetle
     - 28
   * - 2
     - banana
     - 4
     - audi
     - 300
   * - 3
     - apple
     - 3
     - beetle
     - null
   * - 4
     - apple
     - 2
     - beetle
     - 2
   * - 5
     - banana
     - 1
     - beetle
     - -30


For each data type supported by ``metasynth``, there is a set of candidate distributions that can be fitted to that data type (see Table below). To estimate the generative model of Equation, for each variable the software fits all compatible candidate distributions — by default with maximum likelihood estimation — and then selects the one with the lowest `AIC <https://springer.com/chapter/10.1007/978-1-4612-1694-0_15>`_.

.. list-table::
   :header-rows: 1

   * - Variable type
     - Data type
     - example
     - candidate distributions
   * - continuous
     - float
     - 1.0, 2.1, ...
     - UniformDistribution, NormalDistribution, ...
   * - discrete
     - int
     - 1, 2, ...
     - DiscreteUniformDistribution
   * - categorical
     - pl.Categorical
     - gender, country
     - MultinoulliDistribution
   * - structured string
     - str
     - Room number A108, C122
     - RegexDistribution
   * - unstructured string
     - str
     - Names, open answers
     - FakerDistribution, LLMDistribution
   * - temporal
     - Date, Datetime
     - 2021-01-13, 01:40:12
     - DateUniformDistribution

.. note:: 
  See the :doc:`/usage/generating_metaframes` page for information on *how* to generate a MetaFrame.

Serialization and deserialization
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. image:: /images/pipeline_serialization_simple.png
   :alt: MetaSynth Serialization Step in Pipeline
   :align: center

After a ``MetaFrame`` object is created, ``metasynth`` allows it to be stored in a human- and machine-readable ``.json`` file. This file can be considered as metadata.
Exported :obj:`MetaFrames <metasynth.dataset.MetaFrame>` follow the  `Generative Metadata Format (GMF) <https://github.com/sodascience/generative_metadata_format>`__, a standard designed to be easy to read and understand. 
This allows for manual and automatic editing, as well as easy sharing.

.. raw:: html

   <details> 
   <summary> An example of an exported MetaFrame: </summary>

.. code-block:: json

    {
        "n_rows": 5,
        "n_columns": 5,
        "provenance": {
            "created by": {
                "name": "MetaSynth",
                "version": "0.4.0"
            },
            "creation time": "2023-08-07T12:14:06.232957"
        },
        "vars": [
            {
                "name": "ID",
                "type": "discrete",
                "dtype": "Int64",
                "prop_missing": 0.0,
                "distribution": {
                    "implements": "core.unique_key",
                    "provenance": "builtin",
                    "class_name": "UniqueKeyDistribution",
                    "parameters": {
                        "low": 1,
                        "consecutive": 1
                    }
                }
            },
            {
                "name": "fruits",
                "type": "categorical",
                "dtype": "Categorical",
                "prop_missing": 0.0,
                "distribution": {
                    "implements": "core.multinoulli",
                    "provenance": "builtin",
                    "class_name": "MultinoulliDistribution",
                    "parameters": {
                        "labels": [
                            "apple",
                            "banana"
                        ],
                        "probs": [
                            0.4,
                            0.6
                        ]
                    }
                }
            },
            {
                "name": "B",
                "type": "discrete",
                "dtype": "Int64",
                "prop_missing": 0.0,
                "distribution": {
                    "implements": "core.poisson",
                    "provenance": "builtin",
                    "class_name": "PoissonDistribution",
                    "parameters": {
                        "mu": 3.0
                    }
                }
            },
            {
                "name": "cars",
                "type": "categorical",
                "dtype": "Categorical",
                "prop_missing": 0.0,
                "distribution": {
                    "implements": "core.multinoulli",
                    "provenance": "builtin",
                    "class_name": "MultinoulliDistribution",
                    "parameters": {
                        "labels": [
                            "audi",
                            "beetle"
                        ],
                        "probs": [
                            0.2,
                            0.8
                        ]
                    }
                }
            },
            {
                "name": "optional",
                "type": "discrete",
                "dtype": "Int64",
                "prop_missing": 0.2,
                "distribution": {
                    "implements": "core.discrete_uniform",
                    "provenance": "builtin",
                    "class_name": "DiscreteUniformDistribution",
                    "parameters": {
                        "low": -30,
                        "high": 301
                    }
                }
            }
        ]
  }

.. raw:: html

   </details> 
   <br>

.. note:: 
  See the :doc:`/usage/exporting_metaframes` page for information on *how* to export and load MetaFrame to and from JSON files.
  
Data generation
^^^^^^^^^^^^^^^^

.. image:: /images/pipeline_generation_simple.png
   :alt: MetaSynth Estimation Step in Pipeline
   :align: center

Once a MetaFrame model has been created or loaded from a JSON file, new synthetic datasets can be generated from it.

This process involves repeatedly sampling values from the statistical distributions specified in the MetaFrame. For each variable, values are drawn randomly based on the modeled distribution for that variable. The software handles missing data by occasionally generating null values based on the missing data percentage.

These per-variable synthetic value samples are collected into a dictionary, with the variable names as keys. This dictionary is then converted into a `Polars <https://www.pola.rs/>`_  DataFrame to create the full synthetic dataset.


.. note:: 
  See the :doc:`/usage/generating_synthetic_data` page for information on *how* to generate synthetic data based on a MetaFrame.


MetaSynth Extensions
---------------------
The privacy capacities of ``metasynth`` are extensible through a plug-in system, recognizing that different data owners have different needs and definitions of privacy. A data owner can define under which conditions they would accept open distribution of their synthetic data — be it based on `differential privacy <https://springer.com/chapter/10.1007/11787006_1>`_, `statistical disclosure control <https://www.wiley.com/en-us/Statistical+Disclosure+Control-p-9781118348215>`_, `k-anonymity <https://epic.org/wp-content/uploads/privacy/reidentification/Sweeney_Article.pdf>`_, or another specific definition of privacy. 

Currently released extensions and more information on them is available on the :doc:`/usage/extensions` page.