<<<<<<< HEAD
.. _pipeline-dev:

=======
>>>>>>> infrastructure-updates
Developing new pipelines
========================

Asimov only supports a small number of pipelines "out of the box", but allows for new pipelines to be added as plugins.

There are two broad approaches to writing a plugin for asimov.
Either you can incorporate it directly into your codebase, which is especially suitable if your pipeline is written in python, or you can write an interface plugin which allows interaction between asimov and the pipeline.

Asimov uses a feature of python packages called an "entrypoint" in order to identify pipelines which are installed with an asimov interface.

How asimov interacts with pipelines
-----------------------------------

Asimov provides information to the pipeline interface from its ledger, which stores the configuration settings for pipelines, events, and analyses in one single source of truth.

Getting an analysis started is effectively a two-part process, with a "build" phase and a "submit" phase.

During the "build" phase asimov collects the appropriate configuration settings for the pipeline, and creates the configuration file which will be passed to the pipeline's builder script.

This stage is normally run via the command line as ``asimov manage build``.

This stage runs the ``Production.make_config`` method to produce the templated configuration file.

During the "submit" phase asimov runs the pipeline builder script, and then submits the jobs to the scheduler.

This stage is normally run via the command line as ``asimov manage submit``.

The ``Pipeline.build_dag`` method is used to run the pipeline builder, and the ``Pipeline.submit_dag`` method is used to handle the scheduler submission process.

The ``asimov monitor`` command line tool can then be used to check the status of 

Creating an interface
---------------------

An asimov pipeline interface is simply a python class which subclasses ``asimov.pipeline.Pipeline``, and provides the pipeline specific logic to allow asimov to interact with it.

You should implement the new logic for your pipeline by overloading the methods defined in ``asimov.pipeline.Pipeline``, though you can add pipeline-specific methods to the interface as well.


Required methods
~~~~~~~~~~~~~~~~

These methods are required for the interface to function properly.

``Pipeline.detect_completion``
    This contains the logic which allows asimov to determine if the job has completed sucessfully.
    While the performance of the job is monitored via the job scheduling system, the pipeline interface should check for final products of the analysis to determine completion.
    For example, the bilby pipeline interface checks if a posterior samples file has been produced (by checking that the files returned by `Pipeline.collect_assets` exist).

``Pipeline.collect_assets``
    This method contains logic to find results files from the analysis run directory.
    These should be returned in a dictionary, and can be used by other pipelines to find analysis assets.
    For example, the Bayeswave interface returns a dictionary in the format

    ::
       
       {"psds": {"L1": /path/to/L1/psd, "H1": /path/to/H1/psd}}

``Pipeline.collect_logs``
    This method should find all of the log files for the running analysis, and return them as a dictionary.


The most important of these is the ``build_dag`` method, which is used by the asimov framework to construct the DAG file to be submitted to the condor scheduler.

An example of a complete pipeline interface can be seen in the code for :class:``asimov.pipelines.bilby.BilbyPipeline``.


Pipeline hooks
--------------

It is possible to customise the run process of the asimov pipeline runner using hooks.
By overloading the hook methods (listed below) inherited from the ``asimov.pipeline.Pipeline`` class additional operations can
be conducted during the processing workflow.
Hooks should take no arguments.

``Pipeline.build_dag``
    This method should call the pipeline script which will take the configuration file for the analysis, and use them to generate submission files for the scheduler.

``Pipeline.submit_dag``
    This method should submit the submission files generated by ``Pipeline.build_dag`` and submit them to the scheduler.

``Pipeline.before_submit``
    This method is called by asimov before the ``submit_dag`` method is run, and can be used to perform any pre-processing which is required before the job is submitted to the scheduller.

``Pipeline.after_completion``
    This method is called by asimov after completion of a job has been detected, and can be used to begin post-processing.
    For example, the bilby interface uses this hook to start PESummary post-processing.

``Pipeline.after_processing``
    This method is run by asimov after the completion of the post-processing for a job has been completed.
    The default version of this hook runs the ``Pipeline.store_results`` method in order to place the final post-processed results into storage.
    
Optional methods
----------------

``Pipeline.run_pesummary``
    This method will run PE summary on the samples created by an analysis pipeline.
    You may overload this method if you need to run PESummary in a non-standard way.
    You may also need to overload the ``Pipeline.detect_completion_processing`` method if you change this method.

``Pipeline.store_results``
    This method will store the results of the analysis in the asimov results store for the project.
    The default method will collect the results files from PESummary, but it can be overloaded in order to store a different set of files or perform additional tasks prior to storage.

``Pipeline.detect_completion_processing``
    This method provides the logic for determining if PESummary, or whichever post-processing commands are run, have completed successfully, and produced outputs.
    You should only need to overload this method if you have altered ``Pipeline.run_pesummary``.

``Pipeline.eject_job``
    This method is run by asimov to remove the analysis job from the scheduler.
    For example, it will be run if the status of an analysis is set to ``stop``.

``Pipeline.clean``
    This method should remove all of the artefacts of a job from the working directory.

``Pipeline.resurrect``
    This method will be called on jobs which are marked as ``stuck`` in the asimov ledger, and can be used to, for example, submit a rescue DAG for a job on the condor scheduler.

``Pipeline.read_ini``
    This should be implemented as a class method, and should parse the configuration file for the pipeline into a dictionary.

``Pipeline.check_progress``
    This method will be run by asimov to gather information about the current status of the analysis.
    
Adding an entrypoint
--------------------

Asimov uses an "entrypoint" to make pipelines discoverable.

In these examples we assume that your pipeline interface is a class called ``MyPipeline`` (which subclasses ``asimov.pipeline.Pipeline``), and is located in a file called ``asimov.py`` in the main package, i.e.

.. code-block::

   |- setup.py
   |- mypipeline
      |- __init__.py
      |- ...
      |- asimov.py
      |- ...
   |- ...
   

There are a number of different python packaging technologies, so we will provide examples for just a few here.
   
``setup.cfg``
~~~~~~~~~~~~~~

.. code-block:: toml

   [options]
   install_requires =
		    asimov
		    mypipeline

   [metadata]
   name = mypipeline
   version = attr: mypipeline.__version__
   description = A pipeline integration between asimov and mypipeline

   [options.entry_points]
   asimov.pipelines =
		    mypipeline = mypipeline.asimov:MyPipeline

``setup.py``
~~~~~~~~~~~~

.. code-block:: python

   import setuptools

   setuptools.setup(
      name = "MyPipeline",
      ...,
      entry_points = {
        "asimov.pipelines": ["mypipeline = mypipeline.asimov:MyPipeline"]
    },


.. _templates:
    
Pipeline Configuration Templates
--------------------------------

All of the pipelines which ``asimov`` is designed to work with use some manner of configuration file to define their operation.
Previously, creating these configuration files could be a tedious manual process, but ``asimov`` allows these files to be *templated*, combining various pieces of data and metadata from the production ledger with a template configuration file to produce the configuration which is then used to generate the DAG files which run the analysis.

Details of the metadata stored in the ledger can be found in the :ref:`documentation for the ledger format<ledger>`.

In order to run a ``bilby`` job a config file is required for ``bilby_pipe``.
``Asimov`` can produce this from a template.

Config file templates can be written using the liquid templating language, and should be kept in a directory which is specified in the `asimov` configuration file under the ``templating>directory`` configuration value e.g.

.. code-block:: ini

   [templating]
   directory = config-templates


The liquid language allows some logic to be included in the template.
This can be used to only include a given value if an interferometer is included in the analysis.
For example:

.. code-block:: ini

    spline-calibration-envelope-dict={
      {% if production.meta['interferometers'] contains "H1" %}
	H1:{{ production.event.repository.directory }}/{{ production.meta['calibration']['H1'] }},
      {% endif %}
      {% if production.meta['interferometers'] contains "L1" %}
	L1:{{ production.event.repository.directory }}/{{ production.meta['calibration']['L1'] }},
      {% endif %}
      {% if production.meta['interferometers'] contains "V1" %}
	V1:{{ production.event.repository.directory }}/{{ production.meta['calibration']['V1'] }}
      {% endif %}
    }

Adds only the calibration files for the required detectors to the configuration file.


The majority of the data passed to the template can be found in the ``production.meta`` dictionary.
These are stored in the same nested format as the production ledger; evbent-wide values are inherited by the production, so in the example ledger below the sample rate can be retrieved from ``production.meta['quality']['sample-rate']``, for example.

There are also a number of additional variables are available for convenience:

+ ``production.quality`` is an alias for ``production.meta['quality']``
+ ``production.psds`` provides the dictionary of PSDs for this event's specified sample rate.
+ ``production.event`` provides access to the data from the event (e.g. for the repository directory path, located at ``production.event.repository.directory``)

A full example ``bilby`` template is available below:

.. code-block:: ini

    ################################################################################
    ## Calibration arguments
    ################################################################################

    calibration-model=CubicSpline
    spline-calibration-envelope-dict={ {% if production.meta['interferometers'] contains "H1" %}H1:{{ production.event.repository.directory }}/{{ production.meta['calibration']['H1'] }},{% endif %}{% if production.meta['interferometers'] contains "L1" %}L1:{{ production.event.repository.directory }}/{{ production.meta['calibration']['L1'] }},{% endif %}{% if production.meta['interferometers'] contains "V1" %}V1:{{ production.event.repository.directory }}/{{ production.meta['calibration']['V1'] }}{% endif %} }
    spline-calibration-nodes=10
    spline-calibration-amplitude-uncertainty-dict=None
    spline-calibration-phase-uncertainty-dict=None

    ################################################################################
    ## Data generation arguments
    ################################################################################

    ignore-gwpy-data-quality-check=True
    gps-tuple=None
    gps-file=None
    timeslide-file=None
    timeslide-dict=None
    trigger-time={{ production.meta['event time'] }}
    gaussian-noise=False
    n-simulation=0
    data-dict=None
    data-format=None
    channel-dict={ {% if production.meta['interferometers'] contains "H1" %}{{ production.meta['data']['channels']['H1'] }},{% endif %} {% if production.meta['interferometers'] contains "L1" %}{{ production.meta['data']['channels']['L1'] }},{% endif %}{% if production.meta['interferometers'] contains "V1" %}{{ production.meta['data']['channels']['V1'] }}{% endif %} }

    ################################################################################
    ## Detector arguments
    ################################################################################

    coherence-test=False
    detectors={{ production.meta['interferometers'] }}
    duration={{ production.meta['quality']['segment-length'] }}
    generation-seed=None
    psd-dict={ {% if production.meta['interferometers'] contains "H1" %}H1:{{ production.psds['H1'] }},{% endif %} {% if production.meta['interferometers'] contains "L1" %}L1:{{ production.psds['L1'] }},{% endif %} {% if production.meta['interferometers'] contains "V1" %}V1:{{ production.psds['V1'] }}{% endif %} }
    psd-fractional-overlap=0.5
    post-trigger-duration=2.0
    sampling-frequency={{ production.meta['quality']['sample-rate'] }}
    psd-length={{ production.meta['quality']['psd-length'] }}
    psd-maximum-duration=1024
    psd-method=median
    psd-start-time=None
    maximum-frequency=1024
    minimum-frequency={{ production.meta['quality']['reference-frequency'] }}
    zero-noise=False
    tukey-roll-off=0.4
    resampling-method=lal

    ################################################################################
    ## Injection arguments
    ################################################################################

    injection=False
    injection-dict=None
    injection-file=None
    injection-numbers=None
    injection-waveform-approximant=None

    ################################################################################
    ## Job submission arguments
    ################################################################################

    accounting=ligo.dev.o3.cbc.pe.lalinference
    label={{ production.name }}
    local=False
    local-generation=False
    local-plot=False
    outdir={{ production.rundir }}
    periodic-restart-time=28800
    request-memory=4.0
    request-memory-generation=None
    request-cpus=4
    singularity-image=None
    scheduler=condor
    scheduler-args=None
    scheduler-module=None
    scheduler-env=None
    transfer-files=False
    log-directory=None
    online-pe=False
    osg=False

    ################################################################################
    ## Likelihood arguments
    ################################################################################

    distance-marginalization=True
    distance-marginalization-lookup-table=None
    phase-marginalization=True
    time-marginalization=True
    jitter-time=True
    reference-frame={% if production.meta['interferometers'] contains "H1" %}H1{% endif %}{% if production.meta['interferometers'] contains "L1" %}L1{% endif %}{% if production.meta['interferometers'] contains "V1" %}V1{% endif %}
    time-reference={% if production.meta['interferometers'] contains "H1" %}H1{% elsif production.meta['interferometers'] contains "L1" %}L1{% elsif production.meta['interferometers'] contains "V1" %}V1{% endif %}
    likelihood-type=GravitationalWaveTransient
    roq-folder=None
    roq-scale-factor=1
    extra-likelihood-kwargs=None

    ################################################################################
    ## Output arguments
    ################################################################################

    create-plots=True
    plot-calibration=False
    plot-corner=False
    plot-marginal=False
    plot-skymap=False
    plot-waveform=False
    plot-format=png
    create-summary=False
    email=None
    existing-dir=None
    webdir=/home/pe.o3/public_html/LVC/o3b-catalog/{{ production.event.name }}/{{ production.name }}
    summarypages-arguments=None

    ################################################################################
    ## Prior arguments
    ################################################################################

    default-prior=BBHPriorDict
    deltaT=0.2
    prior-file=4s
    prior-dict=None
    convert-to-flat-in-component-mass=False

    ################################################################################
    ## Post processing arguments
    ################################################################################

    postprocessing-executable=None
    postprocessing-arguments=None
    single-postprocessing-executable=None
    single-postprocessing-arguments=None

    ################################################################################
    ## Sampler arguments
    ################################################################################

    sampler=dynesty
    sampling-seed=None
    n-parallel=5
    sampler-kwargs={'queue_size': 4, 'nlive': 2000, 'sample': 'rwalk', 'walks': 100, 'n_check_point': 2000, 'nact': 10, 'npool': 4}

    ################################################################################
    ## Waveform arguments
    ################################################################################

    waveform-generator=bilby.gw.waveform_generator.WaveformGenerator
    reference-frequency={{ production.meta['quality']['reference-frequency'] }}
    waveform-approximant={{ production.meta['approximant'] }}
    catch-waveform-errors=False
    pn-spin-order=-1
    pn-tidal-order=-1
    pn-phase-order=-1
    pn-amplitude-order=0
    mode-array=None
    frequency-domain-source-model=lal_binary_black_hole
