Pipeline YAML Files
===================

Two YAML-format configuration files are needed to run a pipeline.

The first describes which steps to run in a pipeline, the overall inputs for it, execution information, and directories for outputs.  It is described on this page.  It includes the path to the second file, (see `Config`_ below); that file is described in more depth on the page :ref:`config2`.

Here is an example, from ``test/test.yml``.  The different pieces are described below.


.. code-block:: yaml

  # There are currently three defined launchers
  # mini, parsl, and cwl
  launcher:
      name: mini
      interval: 0.5
  # and three sites:
  # local, cori, and cori-interactive
  site:
      name: local
      max_threads: 2


  # The list of stages to run and the number of processors
  # to use for each.
  stages:
      - name: WLGCSummaryStatistic
        nprocess: 1
      - name: SysMapMaker
        nprocess: 1
      - name: shearMeasurementPipe
        nprocess: 1
      - name: PZEstimationPipe
        nprocess: 1
      - name: WLGCRandoms
        nprocess: 1
      - name: WLGCSelector
        nprocess: 1
      - name: SourceSummarizer
        nprocess: 1
      - name: WLGCTwoPoint
        nprocess: 1
      - name: WLGCCov
        nprocess: 1

  # Definitions of where to find inputs for the overall pipeline.
  # Any input required by a pipeline stage that is not generated by
  # a previous stage must be defined here.  They are listed by tag.
  inputs:
      DM: ./test/inputs/dm.txt
      fiducial_cosmology: ./test/inputs/fiducial_cosmology.txt

  # Overall configuration file 
  config: ./test/config.yml

  # If all the outputs for a stage already exist then do not re-run that stage
  resume: False

  # Put all the output files in this directory:
  output_dir: ./test/outputs

  # Put the logs from the individual stages in this directory:
  log_dir: ./test/logs

  # These will be run before and after the pipeline respectively
  pre_script: ""
  post_script: ""

Modules
-------

The ``modules`` option, which is a string, consists of the names of python modules to import and search for pipeline stages (with spaces between each).

Each module is imported at the start of the pipeline.  For a stage to be found, it should be imported somewhere in the chain of imports under ``__init__.py`` in one of the packages listed here.  You can specify subpackages, like ``module.submodule`` in this list after ``module`` if you need to.

The ``python_paths`` option can be set to a single string or list of strings, and gives paths to add to python's ``sys.path`` before attempting the import above.

Stages
------

The ``stages`` parameter should be a list of dictionaries.  Each element in the list is one pipeline stage to be run.  You don't have to put the stages in order - ceci will figure that out for you.

Each dictionary represents one stage, and has these options, with the defaults as shown:


.. code-block:: yaml

  - name: NameOfClass       # required
    nprocess: 1             # optional
    threads_per_process: 1  # optional
    nodes: 1                # optional


``threads_per_process`` is the number of threads, and therefore also the number of cores to assign to each process.  OpenMP is the usual threading method used for our jobs, so ``OMP_NUM_THREADS`` is set to this value for the job.

``nodes`` is the number of nodes to assign to the job.  The processes are spread evenly across nodes.

``nprocess`` is the total number of processes, (across all nodes, not per-node).  Process-level parallelism is currently implemented only using MPI, but if you need other approaches please open an issue.


Launcher
--------

The ``launcher`` parameter should be a dictionary that configures the workflow manager used to launch the jobs.

The ``name`` item in the dictionary sets which launcher is used.  These options are currently allowed: ``mini``, ``parsl``, and ``cwl``.

See the :ref:`launchers` page for information on these launchers, and the other options they take.


Site
----

The ``site`` parameter should be a dictionary that configures the machine on which you are running the pipeline.

The ``name`` item in the dictionary sets which site is used.  These options are currently allowed: ``local``, ``cori-batch``, and ``cori-interactive``.

See the :ref:`sites` page for information on these sites, and the other options they take.


Inputs
------

The ``inputs`` parameter is required, and should be set to a dictionary.  It must describe any files that are overall inputs to the pipeline, and are not generated internally by it.  Files that are made inside the pipeline must not be listed.

The keys are tags, strings from the ``inputs`` attribute on the classes that represent the pipeline stage.  They should map to values which are the paths to find those inputs.

Config
------

The parameter ``config`` is required, and should be set to a path to another input YAML config file.

See the :ref:`config2` page for what that file should contain.

Resume
------

The parameter ``resume`` is required, and should be set to ``True`` or ``False``.

If the parameter is ``True``, then any pipeline stages whose outputs all exist already will be skipped and not run.

In the current implementation, a pipeline stage with missing input will not cause "downstream" stages to be run as well - e.g. if the final stage in your pipeline has all its outputs present it will *not* be re-run, even if earlier stages *are* re-run because their outputs had been removed.

Directories
-----------

The parameter ``output_dir`` is required, and should be set to a directory where all the outputs from the pipeline will be saved.  If the directory does not exist it will be created.

If the resume parameter is set to True, then this is the directory that will be checked for existing outputs.

The parameter ``log_dir`` is required, and should be set to a directory where the printed output of the stages will be saved, in one file per stage.

Scripts
-------

Two parameters can be set to run additional scripts before or after a pipeline stage.  You can use them to perform checks or process results.

Any executable specified by ``pre_script`` will be run before the pipeline.  If it returns a non-zero status then the pipeline will not be run and an exception will be raised.

Any executable specified by ``post_script`` will be run after the pipeline, but only if the pipeline completes successfully.  If the post_script returns a non-zero status then it will be returned as the ceci exit code, but no exception will be raised.

Both scripts are called with the same arguments as the original executable was called with.
