.. highlight:: text

=====
Usage
=====

``mihifepe`` is invoked as follows::

    python -m mihifepe -data_filename <data_filename.hdf5>
                       -model_generator_filename <model_generator_filename.py>
                       -hierarchy_filename <hierarchy_filename.csv>
                       -output_dir <output_dir>

Performance can be greatly improved by running on a distributed system. Presently ``mihifepe`` only supports HTCondor_ running on a shared filesystem, invoked as follows::

    python -m mihifepe -condor ...

.. _HTCondor: https://research.cs.wisc.edu/htcondor/

To see a complete list of options, run::

    python -m mihifepe -h

------
Inputs
------

* **Test data**: *<data_filename.hdf5>* Test data in HDF5 format
* **Trained model**: *<model_generator_filename.py>* Python script that generates model object for subsequent callbacks to model.predict
* **Hierarchy over features**: *<hierarchy_filename.csv>* CSV specifying hierarchy over features

See `Input specification`_ for a detailed descriptions of the input data.

-------
Outputs
-------

* *<output_dir>/p_values.csv*: CSV, listing for all nodes in hierarchy:

    * Accuracy of model with given node perturbed
    * p-values of paired statistical test comparing perturbed model loss to baseline (unperturbed) model loss

* *<output_dir>/hierarchical_fdr_control/tree.png*: PNG showing subtree of hierarchy corresponding to rejected nodes, subsequent to hierarchical FDR control

.. _`Input specification`:

-------------------
Input specification
-------------------

Data types
~~~~~~~~~~
The library deals with models that accept one or both of the following types of input:

* **Static input**: Represented as a single vector per instance of length *L*, cumulatively represented as a data matrix
* **Temporal input**: Represented as an input sequence of variable length *V*, each element of which is a vector of fixed length *W*.

Models commonly take only static input, but models such as Recurrent Neural Networks (RNNs) work with input sequences. Models comprising bigger networks with RNN sub-networks may take both kinds of inputs.

Data representation
~~~~~~~~~~~~~~~~~~~
The data must be in HDF5_ format, which (among other things) allows easy and scalable storage and access of a combination of static and variable-length temporal data.
The recommended method of generating HDF5 inputs is via h5py_.

HDF5 data is organized into a hierarchy comprising groups (containers) and datasets (data collections).
The input data for ``mihifepe`` must be organized as follows (see https://mihifepe.readthedocs.io/en/latest/examples.html for examples).

Groups::

    /temporal               (Group containing temporal data)

Datasets::

    /record_ids             (List of record identifiers (strings) of length M = number of records/instances)
    /targets                (vector of target values (regression/classification outputs) of length M)
    /static_data            (matrix of static data of size M x L)
    /temporal/<record_id>   (One dataset per record_id) (List (of variable length V) of vectors (of fixed length W))

**[TODO]: Sparse representations may be used for both temporal and static data, in which case the attribute 'sparse' must be specified.**

.. _HDF5: https://support.hdfgroup.org/HDF5/
.. _h5py: http://docs.h5py.org/en/latest/index.html

Trained model
~~~~~~~~~~~~~
The caller must create a ``model`` object, corresponding to the trained model, that implements the following method:

.. code-block:: python

    model.predict(target, static data, temporal_data)
    """
    Predicts the model's output (loss, prediction) for the given target and instance.

    Args:
        target:         classification label or regression output (scalar value)
        static_data:    static data (vector)
        temporal_data:  temporal data (matrix, where number of rows are variable across instances)

    Returns:
        loss:           model's output loss
        prediction:     model's output prediction, only used for classifiers
    """

This object must be generated by a standalone Python script that is passed to ``mihifepe``. This allows ``mihifepe`` to distribute the feature perturbations across multiple worker nodes, each with its own copy of ``model``.
For instance, if the script path is */a/b/c/d/gen_model.py*, then ``mihifepe`` will access ``model`` as follows:

.. code-block:: python

    sys.path.insert(0, "/a/b/c/d/") # Makes python search this folder for modules
    from gen_model import model

The test data type must match the data type of the *predict* function (e.g. if the model requires both static and temporal input, the input test data must provide both for every instance).

Hierarchy over features
~~~~~~~~~~~~~~~~~~~~~~~
The caller must provide a hierarchy over features as a CSV file. Each node (including leaf nodes) may correspond to a single feature or a group of features.
Two sets of indices must be specified for each leaf node, at least one of which must be non-empty. Indices of the same data type must be mutually exclusive across leaf nodes.
The CSV must contain the following columns::

    name:             feature name, unique across features
    parent_name:      name of parent if it exists, else '' (root node)
    description:      node description
    static_indices:   [only required for leaf nodes] list of tab-separated indices corresponding to the indices
                        of these features in the static data
    temporal_indices: [only required for leaf nodes] list of tab-separated indices corresponding to the indices
                        of these features in the temporal data
