Variables
=========

There are a number of variables which can be placed in a merlin input .yaml
file that can control workflow execution, such as via string expansion and
control flow.

.. note:: Only user variables and ``OUTPUT_PATH`` may be reassigned or overridden from the command line.

Directory structure context
---------------------------
The directory structure of merlin output looks like this:

.. code::

    SPECROOT
        <spec.yaml>

    ...

    OUTPUT_PATH
        MERLIN_WORKSPACE
            MERLIN_INFO
                <provenance_spec.yaml>
            <other_step_name>.workspace
            WORKSPACE


Reserved variables
------------------
.. list-table:: Study variables that Merlin uses. May be referenced within a specification file, but not reassigned or overridden.

  * - Variable
    - Description
    - Example Expansion
  * - ``$(SPECROOT)``
    -  Directory path of the specification file.
    - ``/globalfs/user/merlin_workflows``
  * - ``$(OUTPUT_PATH)``
    - Directory path the study output will be written to. If not defined
      will default to the current working directory. May be reassigned or
      overridden.
    - ``./studies``
  * - ``$(MERLIN_TIMESTAMP)``
    - The time a study began. May be used as a unique identifier.
    - ``"YYYYMMDD-HHMMSS"``
  * - ``$(MERLIN_WORKSPACE)``
    - Output directory generated by a study at ``OUTPUT_PATH``. Ends with
      ``MERLIN_TIMESTAMP``.
    - ``$(OUTPUT_PATH)/ensemble_name_$(MERLIN_TIMESTAMP)``
  * - ``$(WORKSPACE)``
    - The workspace directory for a single step.
    - ``$(OUTPUT_PATH)/ensemble_name_$(MERLIN_TIMESTAMP)/step_name/``
  * - ``$(MERLIN_INFO)``
    - Directory within ``MERLIN_WORKSPACE`` that holds a provenance spec.
      Commonly used to hold ``samples.npy``.
    - ``$(MERLIN_WORKSPACE)/merlin_info/``
  * - ``$(MERLIN_SAMPLE_ID)``
    - Sample index in an ensemble
    - ``0`` ``1`` ``2`` ``3``
  * - ``$(MERLIN_SAMPLE_PATH)``
    - Path in the sample directory tree to a sample's directory, i.e. where the
      task is actually run.
    - ``/0/0/0/`` ``/0/0/1/`` ``/0/0/2/`` ``/0/0/3/``
  * - ``$(MERLIN_GLOB_PATH)``
    - All of the directories in a simulation tree as a glob (*) string
    - ``/\*/\*/\*/\*``
  * - ``$(MERLIN_PATHS_ALL)``
    - A space delimited string of all of the paths;
      can be used as is in bash for loop for instance with:

      .. code-block:: bash

         for path in $(MERLIN_PATHS_ALL)
           do
             ls $path
           done

         for path in $(MERLIN_PATHS_ALL)
         do
           ls $path
         done
    - ``0/0/0 0/0/1 0/0/2 0/0/3``


User variables
-------------------
Variables defined by a specification file in the ``env`` section, as in this example:

.. code-block:: yaml

    env:
        variables:
            EXAMPLE_VAR:    hello


Like all other Merlin variables, these may be used within specification steps as below:

.. code-block:: yaml

    cmd: echo "$(EXAMPLE_VAR), world!"




Step return variables
-----------------------------------
.. list-table:: Special return code variables for task steps.
   :widths: 25 50 25
   :header-rows: 1

   * - Variable
     - Description
     - Example Usage
   * - ``$(MERLIN_SUCCESS)``
     - This step was successful. Keep going to the next task. Default step
       behavior if no exit code given.
     -
       ::

           echo "hello, world!"
           exit $(MERLIN_SUCCESS)

   * - ``$(MERLIN_RESTART)``
     - Run this step's ``restart`` command, or re-run ``cmd`` if ``restart``
       is absent.
     -
       ::

          run:
            cmd: |
               touch my_file.txt
               echo "hi mom!" >> my_file.txt
               exit $(MERLIN_RESTART)
            restart: |
               echo "bye, mom!" >> my_file.txt

   * - ``$(MERLIN_RETRY)``
     - Retry this step's ``cmd`` command. The default maximum number of retries for any given step
       is 30. You can override this by adding a ``max_retries`` field under the run
       field in the specification. Issues a warning.
     - ::

          run:
            cmd: |
               touch my_file.txt
               echo "hi mom!" >> my_file.txt
               exit $(MERLIN_RETRY)
            max_retries: 23

   * - ``$(MERLIN_SOFT_FAIL)``
     - Mark this step as a failure, note in the warning log but keep going.
       Unknown return codes get translated to soft fails, so that they can
       be logged.
     -
       ::

           echo "Uh-oh, this sample didn't work"
           exit $(MERLIN_SOFT_FAIL)

   * - ``$(MERLIN_HARD_FAIL)``
     - Something went terribly wrong and I need to stop the whole workflow.
       Raises a ``HardFailException`` and stops all workers connected to that
       step. Workers will stop after a 60 second delay to allow the step to
       be acknowledged by the server.

       .. note::
          Workers in isolated parts of the
          workflow not consuming from the bad step will continue. You can stop
          all workers with ``$(MERLIN_STOP_WORKERS)``.

     -
       ::

           echo "Oh no, we've created skynet! Abort!"
           exit $(MERLIN_HARD_FAIL)

   * - ``$(MERLIN_STOP_WORKERS)``
     - Launch a task to stop all active workers. To allow the current task to
       finish and acknowledge the results to the server, will happen in 60
       seconds.
     -
       ::

          # send a signal to all workers to stop
          exit $(MERLIN_STOP_WORKERS)
