==================
Create new example
==================

👋 Everyone is welcomed to contribute with their own forward code. We aim to reduce the
barrier of contributing so don't worry if you are not familiar with some technical
stuff - we are here to help.


Add your own Espresso problem
-----------------------------

#. A new contribution to Espresso has to conform to a consistent
   file structure. The simplest way to ensure that a new contribution includes
   all files is to start the creation of a new problem from the template. Run the 
   following command from the the path to Espresso:

   .. code-block:: bash

        python espresso_machine/new_contribution/create_new_contrib.py <problem-name>

   Replacing :code:`problem-name` with your Espresso problem name in snake case 
   (e.g. :code:`gravity_density`, :code:`polynomial_regression`). Remember to choose a 
   sensible, unique name for your contribution that makes it easy to understand what 
   the example is about.

#. Navigate to folder :code:`<path-to-espresso>/contrib/<problem-name>`, and you'll see template 
   files needed for a new contribution. Each Espresso example is organised around a
   central class object that contains, at minimum, a set of functions with names
   that are shared by all examples.

   .. figure:: ../_static/contrib_edit1.png
    :align: center

#. Now the folder is all yours, get started with the following checklist.

   .. dropdown:: Checklist and tips
      :icon: tasklist
      :open:

      - ``contrib/<problem-name>/README.md``

        - Document anything you'd like to add for this problem, such as the what the 
          problem is about and some brief intro of the theory behind.
        - This will be automatically rendered into :doc:`../user_guide/contrib/index`.

      - ``contrib/<problem-name>/LICENCE``
      
        - We use a 2-clause BSD licence as the default one. Feel free to replace it
          with a licence that suits you best.

      - ``contrib/<problem-name>/<problem-name>.py``

        - The development will be centered around the autogenerated class 
          ``ProblemName``. It is a subclass of 
          :doc:`EspressoProblem <../user_guide/api/generated/espresso.EspressoProblem>`
          so you might find the API reference helpful.
        - All standard attributes have been declared in the template but are left for 
          you to implement. You'll see that some are required (with ``TODO``) and 
          others are optional.
        - If you would like to load data from files, please use our 
          :doc:`utility functions <../user_guide/api/generated/espresso.utils>`
          to get the absolute path before calling your loading function.
        - Apart from the standard attributes, there are many more functions and values 
          that a new contribution can contain, for examples:

          - inversion_suggestion: a string containing inversion suggestions
          - reg_param_suggestion: a sensible value for regularization parameter
          - dx: spatial resolution in x-direction
          - dt: temporal resolution
          - nt: number of time steps
          - The possibilities are endless! Whatever information you find helpful is
            probably also helpful for the users. Simply attach them to the problem 
            class as additional attributes.
         
        - We aim to follow `Python PEP8 style conventions <https://peps.python.org/pep-0008/>`_
          to make source code readable. Though not strictly enforced, we recommend
          `Black formatter <https://github.com/psf/black#installation-and-usage>`_ and
          `PyLint <https://github.com/pylint-dev/pylint#install>`_ to maintain a good 
          coding style.
         
      - ``setup.py``, in particular the ``INSTALL_REQUIRES`` variable, if needed.
   
#. Test running your code. 

   .. dropdown:: Troubleshooting: issue with relative import
      :icon: tools

      Note that you might see an error if you have any relative import in the main Python 
      file:

      .. code-block:: python

         # file: <problem-name>.py

         from .lib import *

      In this case, use ``contrib`` as your working directory and import your contribution
      in the following example way:

      .. code-block:: pycon

         $ pwd                                        # check you are in the right folder
         <path-to-espresso>/contrib
         $ python
         >>> from example_name import ExampleName     # ...and import this way
      
      Or the following example if you are running a file:

      .. code-block:: python

         # file: contrib/tmp.py                       # create tmp file in the right folder
         from example_name import ExampleName         # ...and import this way

#. Validate your code with Espresso by running:

   .. code-block:: console

      $ python espresso_machine/build_package/build.py --pre --post -c <example_name>

   which includes the following steps:

   - Validate all required and standard attributes that you've implemented
   - Build your contribution into a temporary source code folder ``_esp_build``
   - Install your local development Espresso version
   - Validate all required and standard attributes from the installed package
   - Check that packages needed to run your code are listed by Espresso

   Read on the :ref:`appendix sections <appendix>` in this page for how an Espresso 
   example is validated and how Espresso is built, and continue with the 
   :doc:`../developer_guide/index` further details the whole infrastructure (i.e. the 
   ``espresso_machine``).


Jupyter Notebook
----------------

Additionally, we encourage you to add a Jupyter Notebook with an identical name
into the folder Jupyter Notebooks that contains the following:

1. An extensive description of the new Espresso Problem, containing
   information about (but not limited to):

   - the forward calculation (ie. the underlying physics) and how it was implemented.
   - which inversion method is used (and regularisation) and how it was implemented.
   - the physical unit of relevant variables, but at least of ``model`` and ``data``.
   - all changeable parameters, possibly in a list.


2. An example of the new problem being used, with a reasonable output.


.. _appendix:

Appendix I: build steps
-----------------------

Usage:

.. code-block:: console

   $ python build.py [--pre] [--post] [--no-install] [-c <example_name>] [--file <file_name>]

For instance, to install your development version locally, run the following in your 
terminal:

.. code-block:: console

   $ python espresso_machine/build_package/build.py

Run the following for detailed usage information:

.. code-block:: console

   $ python espresso_machine/build_package.build.py --help

The following table describes what happens when we package Espresso:

.. list-table:: How Espresso is packaged
   :widths: 10 45 45
   :header-rows: 1

   * - Step
     - What's done
     - How it's done
   * - 1
     - Clean ``_esp_build/``
     - ``shutil.rmtree``
   * - 2
     - Move meta data files to ``_esp_build/``
     - ``shutil.copy``
   * - 3
     - Move ``src/`` content to ``_esp_build/src/espresso``
     - ``shutil.copytree``
   * - 4
     - Move ``contrib/`` content to ``_esp_build/src/espresso/`` + ``_esp_build/src/espresso/__init__.py`` + ``_esp_build/src/espresso/list_problems.py``
     - ``shutil.copytree``, a series of file opening and string manipulation
   * - 5
     - Write dynamic version and extra versioningit configs into ``_esp_build/pyproject.toml``
     - ``versioningit``
   * - 6
     - Install package from ``_esp_build/``
     - ``pip install _esp_build``


Appendix II: validation steps
-----------------------------

Usage:

.. code-block:: console

   $ python validate.py [-h] [--all] [--pre] [--post] [--contrib CONTRIBS] [--file <file_name>]

For instance, to test whether your new contribution aligns with the Espresso standard, 
run the following in your terminal:

.. code-block:: console

   $ python espresso_machine/build_package/validate.py -c <contrib-name>

Or the following for a complete check on all examples (including yours and existing 
ones for regression test), both before and after Espresso installed:

.. code-block:: console

   $ python espresso_machine/build_package/validate.py

Anyway, run the following for a detailed usage of this script:

.. code-block:: console

   $ python espresso_machine/build_package/validate.py --help


The following table describes what happens when we validate a certain version
of Espresso:

.. list-table:: How an Espresso contribution is validated
   :widths: 10 45 45
   :header-rows: 1

   * - Step
     - What's done
     - How it's done
   * - 1
     - Check the contribution folder name matches the main Python file name (``contrib/<contrib_name>/<contrib_name>.py``)
     - ``assert f"{contrib_name}.py" in file_names``
   * - 2
     - Check ``README.md``, ``LICENCE`` and ``__init__.py`` exist
     - ``assert required_file in file_names``
   * - 3
     - Check the class name is listed in ``__all__`` in file ``__init__.py``
     - ``assert contrib_name_class in parent_module.__all__``
   * - 4
     - Check the contribution provides access to the required metadata
     - Pull out the ``metadata`` field of the contribution class and check those attributes are of correct types
   * - 5
     - Check required methods / properties are implemented and a complete workflow can run for each example number
     - Run from ``example_number=1`` up until an exception is raised or reached 100. For each example, try to get ``model_size``, ``data_size``, ``good_model`` (flat array like, length = ``model_size``), ``starting_model`` (flat array like, length = ``model_size``), ``data`` (flat array like, length = ``data_size``); Run ``forward(model)`` (output to be flat array like, length = ``data_size``). Where "flat array like" is checked via ``np.ndim(obj) == 1``
   * - 6
     - Check optional methods / properties, if implemented, have the correct type signatures
     - For each example, check that the outputs of ``forward(model, with_jacobian=True)``, ``jacobian(model)`` (if implemented) have flat array like synthetics and 2D array like jacobian; Check ``description`` (if exists) is string; Check ``covariance_matrix`` and ``inverse_covariance_matrix`` are in shape ``(data_size, data_size)`` and one is the inverse of the other (if implemented); Check ``plot_model`` and ``plot_data`` (if implemented) return an instance of ``matplotlib.figure.Figure``; Check ``misfit``, ``log_likelihood`` and ``log_prior`` (if implemented) return float
   * - 7
     - Check ``LICENCE`` file is not empty
     - ``assert os.stat("LICENCE").st_size != 0``
