==================
Workflow Templates
==================


**Workflow Templates** are parameterized workflow specifications for the *Reproducible Open Benchmarks for Data Analysis Platform (ROB)*. Workflow templates are motivated by the goal to allow users to run pre-defined data analytics workflows while providing their own input data, parameters, as well as their own code modules. Workflow templates are inspired by, but not limited to, workflow specifications for the `Reproducible Research Data Analysis Platform (REANA)`.



Motivation for Parameterized Workflow Templates
===============================================

Consider the `REANA Hello World Demo <https://github.com/reanahub/reana-demo-helloworld>`_. The demo workflow takes as input a file ``data/names.txt`` containing a list of person names and a timeout parameter ``sleeptime``. For each line in ``data/names.txt`` the workflow writes a line "Hello *name*!" to an output file ``results/greetings.txt``. For each line that is written to the output file program execution is delayed by a number of seconds defined by the `sleeptime` parameter.

Workflow specifications in REANA are serialized in YAML or JSON format. The names of the input and output files as well as the value for the sleep period are currently hard-coded in the workflow specification file ( e.g.  `reana.yaml <https://raw.githubusercontent.com/reanahub/reana-demo-helloworld/master/reana.yaml>`_ ).

.. code-block:: yaml

    inputs:
      files:
        - code/helloworld.py
        - data/names.txt
      parameters:
        helloworld: code/helloworld.py
        inputfile: data/names.txt
        outputfile: results/greetings.txt
        sleeptime: 0
    workflow:
      type: serial
      specification:
        steps:
          - environment: 'python:2.7'
            commands:
              - python "${helloworld}"
                  --inputfile "${inputfile}"
                  --outputfile "${outputfile}"
                  --sleeptime ${sleeptime}
    outputs:
      files:
       - results/greetings.txt

Assume we want to build a system that allows users to run the Hello world demo via a (web-based) interface where they provide a text file with person names and a sleep period value. There are three main parts to such a system. First, we need to display a form where the user can select (upload) a text file and enter a sleep time value. Second, after the user submits their input data, we need to create an update version of the workflow specification as shown above where we replace the value of ``inputfile`` and ``sleeptime`` with the user-provided values. We then pass the modified workflow specification to a REANA instance for execution. There are several way for implementing such a system. Parameterized workflow templates are part of one possible solution.



What are Parameterized Workflow Templates?
==========================================

Similar to REANA workflow specifications, parameterized workflow templates are serialized in YAML or JSON format. Each template has four main sections: ``workflow``, ``engine``, ``parameters``, and ``resources``. The ``workflow`` element contains a workflow specification. The structure and syntax of this specification is dependent on the backend (engine) that is used to execute workflows. Each backend has a unique identifier which is specified in the ``engine`` element.  The ``parameters`` section defines those parts of a workflow that are variable with respect to user inputs. We refer to these are *template parameters*. Template parameters can for example be used to define input and output values for workflow steps or identify Docker container images that contain the code for individual workflow steps. The ``resources`` element specifies the list of resources that are generated by each workflow run. The ost common type of resource are files that are generated during workflow execution.

An example template for the **Hello World example** is shown below.

.. code-block:: yaml

    workflow:
        inputs:
          files:
            - code/helloworld.py
            - $[[names]]
          parameters:
            helloworld: code/helloworld.py
            inputfile: $[[names]]
            outputfile: results/greetings.txt
            sleeptime: $[[sleeptime]]
        workflow:
          type: serial
          specification:
            steps:
              - environment: 'python:2.7'
                commands:
                  - python "${helloworld}"
                      --inputfile "${inputfile}"
                      --outputfile "${outputfile}"
                      --sleeptime ${sleeptime}
        outputs:
          files:
           - results/greetings.txt
    engine: REANA
    parameters:
        - id: names
          name: Person names
          description: Text file containing person names
          datatype: file
        - id: sleeptime
          name: Sleep period
          description: Sleep period in seconds
          datatype: int
    resources:
        - id: greetings.txt
          type: file


The template is divided into the three top-level elements ``workflow``, ``engine``, ``parameters``, and ``resources``. The workflow section is a REANA workflow specification. The main modification to the workflow specification is a simple addition to the syntax in order to allow references to template parameters. Such references are always enclosed in ``$[[...]]``. The parameters section is a list of template parameter declarations. Each parameter declaration has a unique identifier. The identifier is used to reference the parameter from within the workflow specification (e.g., ``$[[sleeptime]]`` to reference the user-provided value for the sleep period). Other elements of the parameter declaration are a human readable short name, a parameter description, and a specification of the data type. Refer to [here]() for a full description of the template parameter declaration syntax. The ``resources`` section is a list of resource definitions. Each resource has a unique identifier and a resource type. The identifier is unique among the resources that are generated by a workflow run.



Usage of Parameterized Workflow Templates
=========================================

The detailed parameter declarations are intended to be used by front-end tools to render forms that collect user input. Given a set of user-provided values for the template parameters, the references to parameters are replaced withing the workflow specification with the given values to generate a valid workflow specification that can be executed by the respective workflow engine.
