Merlin Commands
===============

Commands
--------

The merlin executable defines a number of commands to create tasks,
launch workers to run the tasks and remove tasks from the task server.
The tasks are communicated to a task server, or broker, that are then
requested by workers on an allocation to run. The celery python module
is used to implement the tasks and worker functionality.


Help
----

Descriptions of the Merlin commands will be output when the ``-h`` or
``--help`` commands are used.

.. code:: bash

    (merlin3_7) $ merlin -h


Version info
------------

The Merlin can be output using the ``-v`` argument.

.. code:: bash

    (merlin3_7) $ merlin -v


Information
-----------

Information about the run environment can be printed out using the
``info`` command.

.. code:: bash

    (merlin3_7) $ merlin info

Config
------

Create a default config file in the ${HOME}/.merlin directory using the ``config`` command. This file
can then be edited for your system configuration.

.. code:: bash

    (merlin3_7) $ merlin config [--task_server]  [--output_dir <dir>]

The ``--task_server`` option will select the appropriate configuration for the
given task server. Currently only celery is implemented.

The ``--output_dir`` or ``-o`` will output the configuration in the given directory.
This file can then be edited and copied into ${HOME}/.merlin.

Run the workflow
----------------

To run the merlin workflow use the  ``run`` command and the path to the
input yaml file ``<input.yaml>``. This will define the tasks and queue
them on the task server also called the broker.

.. code:: bash

    (merlin3_7) $ merlin run [--local] <input.yaml>

The ``--local`` option will run tasks sequentially in your current shell.

Dry Run
-------
'Dry run' means telling workers to create a study's workspace and all of its necessary
subdirectories and scripts (with variables expanded) without actually executing
the scripts.

To dry-run a workflow, use:

.. code:: bash

    (merlin3_7) $ merlin run --local --dry <input.yaml>

In a distributed fashion:

.. code:: bash

    (merlin3_7) $ merlin run --dry <input.yaml> ; merlin run-workers <input.yaml>

You can also specify dry runs from the workflow specification file:

.. code:: yaml

    batch:
        dry_run: True

If you wish to execute a workflow after dry-running it, simply use ``restart``.

Restart the workflow
--------------------

To restart a previously started merlin workflow, use the  ``restart`` command
and the path to root of the merlin workspace that was generated during the
previously run workflow. This will define the tasks and queue
them on the task server also called the broker.

.. code:: bash

    (merlin3_7) $ merlin restart [--local] <path/to/workspace_timestamp>

Merlin currently writes file called ``MERLIN_FINISHED`` to the directory of each
step that was finished successfully. It uses this to determine which steps to
skip during execution of a workflow.

The ``--local`` option will run tasks sequentially in your current shell.

Launching Workers
-----------------

The tasks queued on the broker are run by a collection of workers. These
workers can be run local in the current shell or in parallel on a batch
allocation.
The workers are launched using the
``run-workers`` command which reads the configuration for the worker launch
from the ``<input.yaml>`` file.
The batch and merlin resources section are both used to configure the
worker launch.
The top level batch section can be overridden in the merlin
workers resource section.
Parallel workers should be scheduled using the system's batch scheduler.
Once the workers are running, tasks from the broker will be processed.

To launch workers for your workflow:

.. code:: bash

    (merlin3_7) $ merlin run-workers <input.yaml>

An example of launching a simple celery worker using srun:

.. code:: bash

    (merlin3_7) $ srun -n 1 celery worker -A merlin -l INFO

A parallel batch allocation launch is configured to run a single worker
process per node. This worker process will then launch a number of worker
threads to process the tasks. The number of threads can be configured by
the users and will be the number of parallel jobs that can be run at once
on the allocation plus threads for any non-parallel tasks.
If there are 36 cores on a
node and all the tasks are single core, the user may want to start 36
threads per node. If the parallel jobs uses 8 tasks, then the user should run
4 or 5 threads. For the celery workers the number of threads is set using
the ``--concurrency`` argument, see the :ref:`celery-config` section.

A full SLURM batch submission script to run the workflow on 4 nodes is
shown below.

.. code:: bash

  #!/bin/bash
  #SBATCH -N 4
  #SBATCH -J Merlin
  #SBATCH -t 30:00
  #SBATCH -p pdebug
  #SBATCH --mail-type=ALL
  #SBATCH -o merlin_workers_%j.out

  # Assumes you are run this in the same dir as the yaml file.
  YAML_FILE=input.yaml

  # Source the merlin virtualenv
  source <path to merlin venv>/bin/activate

  # Remove all tasks from the queues for this run.
  #merlin purge -f ${YAML_FILE}

  # Submit the tasks to the task server
  merlin run  ${YAML_FILE}

  # Print out the workers command
  merlin run-workers  ${YAML_FILE} --echo

  # Run the workers on the allocation
  merlin run-workers  ${YAML_FILE}

  # Delay until the allocation is complete to keep the workers running
  sleep inf

Searching for any workers
-------------------------

If you want to see all workers that are currently connected to
the task server you can use:

.. code:: bash

    (merlin3_7) $ merlin query-workers

This will broadcast a command to all connected workers and print
the names of any that respond. This is useful for interacting
with workers, such as via ``merlin stop-workers --workers``.

Stopping workers
----------------

To send out a stop signal to some or all connected workers, use:

.. code:: bash

    (merlin3_7) $ merlin stop-workers

The default behavior will send a stop to all connected workers,
having them shutdown softly.

You can also filter by those connected to certain queues and/or
whose name matches a regular expression:

.. code:: bash

    # Stop all workers connected to queues 1 and 2 whose name matches the pattern
    (merlin3_7) $ merlin stop-workers --queues queue1 queue2 --workers "celery@my_host*"

    # Stop all workers whose name matches this pattern, no matter the queue
    # Note the ".*" convention at the start, per regex
    (merlin3_7) $ merlin stop-workers --workers ".*@my_other_host*"

    # Stop all workers on these queues, no matter their name
    (merlin3_7) $ merlin stop-workers --queues queue1 queue2

.. attention::

   If you've named workers identically (you shouldn't)
   only one might get the signal. In this case, you can send it
   again.

Generate working examples
-------------------------

If you want to run an example workflow, use Merlin's ``merlin example``:

.. code:: bash

    (merlin3_7) $ merlin example --help

This will list the available example workflows and a description for each one. To
select one:

.. code:: bash

    (merlin3_7) $ merlin example <example_name>

This will copy the example workflow to the current working directory. It is
possible to specify another path to copy to.

.. code:: bash

    (merlin3_7) $ merlin example <example_name> -p path/to/dir

If the specified directory does not exist Merlin will automatically create it.

This will generate the example workflow at the specified location, ready to be run.


Purging Tasks
-------------

Once the merlin run command succeeds, the tasks are now on the task server
waiting to be run by the workers. If you would like to remove the tasks from
the server, then use the purge command.

.. attention::

    Any tasks reserved by workers will not be purged from the queues. All
    workers must be first stopped so the tasks can be returned to the task
    server and then they can be purged.

    You probably want to use ``merlin stop-workers`` first.

To purge all tasks in all queues defined by the workflow yaml file from the
task server, run:

.. code:: bash

    (merlin3_7) $ merlin purge <input.yaml>

This will ask you if you would like to remove the tasks, you can use the
``-f`` option if you want to skip this.

.. code:: bash

    (merlin3_7) $ merlin purge -f <input.yaml>

If you have different queues in your workflow yaml file, you can
choose which queues are purged by using the ``--steps`` argument and
giving a space separated list of steps.

.. code:: bash

    (merlin3_7) $ merlin purge <input.yaml> --steps step1 step2


Debug Info
----------
More information can be output by increasing the logging level
using the ``-lvl`` or ``--level`` argument.

Options for the level argument are: DEBUG, INFO, WARNING, ERROR.

.. code:: bash

    (merlin3_7) $ merlin -lvl DEBUG run <input.yaml>


