.. _htc-plugin-overview:

Overview
--------

LSST Batch Processing Service (BPS) allows large-scale workflows to execute in
well-managed fashion, potentially in multiple environments.  The service is
provided by the `ctrl_bps`_ package.  ``ctrl_bps_htcondor`` is a plugin
allowing `ctrl_bps` to execute workflows on computational resources managed by
`HTCondor`_.

.. _htc-plugin-preqs:

Prerequisites
-------------

#. `ctrl_bps`_, the package providing BPS.
#. `HTCondor`_ cluster.
#. HTCondor's Python `bindings`__.

.. __: https://htcondor.readthedocs.io/en/latest/apis/python-bindings/index.html

.. _htc-plugin-installing:

Installing the plugin
---------------------

Starting from LSST Stack version ``w_2022_18``, the HTCondor plugin package for
Batch Processing Service, ``ctrl_bps_htcondor``, comes with ``lsst_distrib``.
However, if you'd like to  try out its latest features, you may install a
bleeding edge version similarly to any other LSST package:

.. code-block:: bash

   git clone https://github.com/lsst/ctrl_bps_htcondor
   cd ctrl_bps_htcondor
   setup -k -r .
   scons

.. _htc-plugin-wmsclass:

Specifying the plugin
---------------------

The class providing `HTCondor`_ support for `ctrl_bps`_ is ::

    lsst.ctrl.bps.htcondor.HTCondorService

Inform `ctrl_bps`_ about its location using one of the methods described in its
`documentation`__.

.. __: https://pipelines.lsst.io/v/weekly/modules/lsst.ctrl.bps/index.html

.. _htc-plugin-defining-submission:

Defining a submission
---------------------

BPS configuration files are YAML files with some reserved keywords and some
special features. See `BPS configuration file`__ for details.

The plugin supports all settings described in `ctrl_bps documentation`__
*except* **preemptible**.

.. Describe any plugin specific aspects of defining a submission below if any.

`HTCondor`_ is able to to send jobs to run on a remote compute site, even when
that compute site is running a non-HTCondor system, by sending "pilot jobs", or
**gliedins**, to remote batch systems.

Nodes for HTCondor's glideins can be allocated with help of `ctrl_execute`_.
Once you allocated the nodes, you can specify the site where there are
available in your BPS configuration file. For example:

.. code-block:: YAML

   site:
     acsws02:
       profile:
         condor:
           requirements: '(ALLOCATED_NODE_SET == &quot;${NODESET}&quot;)'
           +JOB_NODE_SET: '&quot;${NODESET}&quot;'

.. __: https://pipelines.lsst.io/v/weekly/modules/lsst.ctrl.bps/quickstart.html#bps-configuration-file
.. __: https://pipelines.lsst.io/v/weekly/modules/lsst.ctrl.bps/quickstart.html#supported-settings

.. .. _htc-plugin-authenticating:

.. Authenticating
.. --------------

.. Describe any plugin specific aspects of an authentication below if any.

.. _htc-plugin-submit:

Submitting a run
----------------

See `bps submit`_.

.. Describe any plugin specific aspects of a submission below if any.

.. _htc-plugin-report:

Checking status
---------------

See `bps report`_.

.. Describe any plugin specific aspects of checking a submission status below
   if any.

In order to make the summary report (``bps report``) faster, the plugin
uses summary information available with the DAGMan job.  For a running
DAG, this status can lag behind by a few minutes.  Also, DAGMan tracks
deletion of individual jobs as failures (no separate counts for
deleted jobs).  So the summary report flag column will show ``F`` when
there are either failed or deleted jobs.  If getting a detailed report
(``bps report --id <id>``), the plugin reads detailed job information
from files.  So, the detailed report can distinguish between failed and
deleted jobs, and thus will show ``D`` in the flag column for a running
workflow if there is a deleted job.

Occasionally, some jobs are put on hold by HTCondor.  To see the reason why
jobs are being held, use

.. code-block:: bash

   condor_q -hold <id>    # to see a specific job being held
   condor-q -hold <user>  # to see all held jobs owned by the user

.. _htc-plugin-cancel:

Canceling submitted jobs
------------------------

See `bps cancel`_.

.. Describe any plugin specific aspects of canceling submitted jobs below
   if any.

If jobs are hanging around in the queue with an ``X`` status in the report
displayed by ``bps report``, you can add the following to force delete those
jobs from the queue ::

    --pass-thru "-forcex"

.. _htc-plugin-restart:

Restarting a failed run
-----------------------

See `bps restart`_.

.. Describe any plugin specific aspects of restarting failed jobs below
   if any.

A valid run id is one of the following:

* job id, e.g., ``1234.0`` (using just the cluster id, ``1234``, will also
  work),
* global job id (e.g.,
  ``sdfrome002.sdf.slac.stanford.edu#165725.0#1699393748``),
* run's submit directory (e.g.,
  ``/sdf/home/m/mxk/lsst/bps/submit/u/mxk/pipelines_check/20230713T135346Z``).

.. note::

   If you don't remember any of the run's id you may try running

   .. code::

      bps report --username <username> --hist <n>

   where ``<username>`` and ``<n>`` are respectively your user account and the
   number of past days you would like to include in your search.  Keep in mind
   though that availability of the historical records depends on the HTCondor
   configuration and the load of the computational resource in use.
   Consequently, you may still get no results and using the submit directory
   remains your only option.

When execution of a workflow is managed by `HTCondor`_, the BPS is able to
instruct it to automatically retry jobs which failed due to exceeding their
memory allocation with increased memory requirements (see the documentation of
``memoryMultiplier`` option for more details).  However, these increased memory
requirements are not preserved between restarts.  For example, if a job
initially run with 2 GB of memory and failed because of exceeding the limit,
`HTCondor`_ will retry it with 4 GB of memory.  However, if the job and as a
result the entire workflow fails again due to other reasons, the job will ask
for 2 GB of memory during the first execution after the workflow is restarted.

.. _htc-plugin-provisioning:

Provisioning resources automatically
------------------------------------

Computational resources required to execute a workflow may not always be
managed directly by HTCondor and may need to be provisioned first by a
different workload manager, for example, `Slurm`_.  In such a case
**ctrl_bps_htcondor** can be instructed to run a provisioning job alongside of
the workflow which will firstly create and then maintain `glideins`__ necessary
for the execution of the workflow.

This provisioning job is called ``provisioning_job.bash`` and is managed by
HTCondor.  Be careful not to remove it by accident when using ``condor_rm`` or
``kill`` command.  The job is run on a best-effort basis and will not be
automatically restarted once deleted.

To enable automatic provisioning of the resources, add the following settings to
your BPS configuration:

.. code-block:: yaml

   provisionResources: true
   provisioning:
     provisioningMaxWallTime: <value>

where ``<value>`` is the approximate time your workflow needs to complete,
e.g., 3600, 10:00:00.

This will instruct **ctrl_bps_htcondor** to include a service job that will run
alongside the other payload jobs in the workflow that should automatically
create and maintain glideins required for the payload jobs to run.

If you enable automatic provisioning of resources, you will see the status of
the provisioning job in the output of the ``bps report --id <id>`` command.
Look for the line starting with "Provisioning job status".  For example

.. code-block:: bash
   :emphasize-lines: 8

    X   STATE   %S   ID  OPERATOR PROJECT CAMPAIGN PAYLOAD                  RUN
   --- ------- --- ----- -------- ------- -------- ------- ---------------------------------------
       RUNNING   0   1.0     jdoe     dev    quick  pcheck u_jdoe_pipelines_check_20240924T201447Z


   Path: /home/jdoe/submit/u/jdoe/pipelines_check/20240924T201447Z
   Global job id: node001#1.0#1727208891
   Provisioning job status: RUNNING


                     UNKNOWN MISFIT UNREADY READY PENDING RUNNING DELETED HELD SUCCEEDED FAILED PRUNED EXPECTED
   ----------------- ------- ------ ------- ----- ------- ------- ------- ---- --------- ------ ------ --------
   TOTAL                   0      0       4     0       1       0       0    0         0      0      0        5
   ----------------- ------- ------ ------- ----- ------- ------- ------- ---- --------- ------ ------ --------
   pipetaskInit            0      0       0     0       1       0       0    0         0      0      0        1
   isr                     0      0       1     0       0       0       0    0         0      0      0        1
   characterizeImage       0      0       1     0       0       0       0    0         0      0      0        1
   calibrate               0      0       1     0       0       0       0    0         0      0      0        1
   finalJob                0      0       1     0       0       0       0    0         0      0      0        1

The service job managing the glideins will be automatically canceled once the
workflow is completed.  However, the existing glideins will be left for
HTCondor to shut them down once they remain inactive for the period specified
by ``provisioningMaxIdleTime`` (default value: 10 min., see below) or maximum
wall time is reached.

If the automatic provisioning of the resources is enabled, the script that the
service job is supposed to run in order to provide the required resources *must
be* defined by the ``provisioningScript`` setting in the ``provisioning``
section of your BPS configuration file.  By default, **ctrl_bps_htcondor** will
use ``allocateNodes.py`` from `ctrl_execute`_ package with the following
settings:

.. code-block:: yaml

   provisioning:
     provisioningNodeCount: 10
     provisioningMaxIdleTime: 900
     provisioningCheckInterval: 600
     provisioningQueue: "milano"
     provisioningAccountingUser: "rubin:developers"
     provisioningExtraOptions: ""
     provisioningPlatform: "s3df"
     provisioningScript: |
       #!/bin/bash
       set -e
       set -x
       while true; do
           ${CTRL_EXECUTE_DIR}/bin/allocateNodes.py \
               --account {provisioningAccountingUser} \
               --auto \
               --node-count {provisioningNodeCount} \
               --maximum-wall-clock {provisioningMaxWallTime} \
               --glidein-shutdown {provisioningMaxIdleTime} \
               --queue {provisioningQueue} \
               {provisioningExtraOptions} \
               {provisioningPlatform}
           sleep {provisioningCheckInterval}
       done
       exit 0

``allocateNodes.py`` requires a small configuration file located in the user's
directory to work. With automatic provisioning enabled **ctrl_bps_htcondor**
will create a new file if it does not exist at the location defined by
``provisioningScriptConfigPath`` using the template defined by
``provisioningScriptConfig`` settings in the ``provisioning`` section:

.. code-block:: yaml

   provisioning:
     provisioningScriptConfig: |
       config.platform["{provisioningPlatform}"].user.name="${USER}"
       config.platform["{provisioningPlatform}"].user.home="${HOME}"
     provisioningScriptConfigPath: "${HOME}/.lsst/condor-info.py"

If you're using a custom provisioning script that does not require any
external configuration, set ``provisioningScriptConfig`` to an empty string.

If the file already exists, it will be used as is (BPS will not update it with
config settings). If you wish BPS to overwrite the file with the
``provisioningScriptConfig`` values, you need to manually remove or rename the
existing file.

.. note::

   ``${CTRL_BPS_HTCONDOR_DIR}/python/lsst/ctrl/bps/htcondor/etc/htcondor_defaults.yaml``
   contains default values used by every bps submission when using
   ``ctrl_bps_htcondor`` plugin that are automatically included in your
   submission configuration.

.. __: https://htcondor.readthedocs.io/en/latest/codes-other-values/glossary.html#term-Glidein

.. _htc-plugin-troubleshooting:

Troubleshooting
---------------

Where is stdout/stderr from pipeline tasks?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

For now, stdout/stderr can be found in files in the run submit directory.

Why did my submission fail?
^^^^^^^^^^^^^^^^^^^^^^^^^^^

Check the ``*.dag.dagman.out`` in run submit directory for errors, in
particular for ``ERROR: submit attempt failed``.

I enabled automatic provisioning, but my jobs still sit idle in the queue!
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The service node responsible for executing the provisioning script runs on a
best-effort basis.  If this node fails to submit correctly or crashes during
the workflow execution, this will not register as an error and the workflow
will continue normally until the existing gliedins expire.  As a result,
payload jobs may get stuck in the job queue if the glideins were not created
or expired before the execution of the workflow could be completed.

Firstly, use ``bps report --id <run id>`` to display the run report and look
for the line

.. code-block::

   Provisioning job status: <status>

If the ``<status>`` is different from RUNNING, it means that the automatic
provisioning is not working.  In such a case, create `glideins manually`__ to
complete your run.

.. __: https://developer.lsst.io/usdf/batch.html#ctrl-bps-htcondor

.. _HTCondor: https://htcondor.readthedocs.io/en/latest/
.. _Slurm: https://slurm.schedmd.com/overview.html
.. _bps cancel: https://pipelines.lsst.io/v/weekly/modules/lsst.ctrl.bps/quickstart.html#canceling-submitted-jobs
.. _bps report: https://pipelines.lsst.io/v/weekly/modules/lsst.ctrl.bps/quickstart.html#checking-status
.. _bps restart: https://pipelines.lsst.io/v/weekly/modules/lsst.ctrl.bps/quickstart.html#restarting-a-failed-run
.. _bps submit: https://pipelines.lsst.io/v/weekly/modules/lsst.ctrl.bps/quickstart.html#submitting-a-run
.. _ctrl_bps: https://github.com/lsst/ctrl_bps
.. _ctrl_execute: https://github.com/lsst/ctrl_execute
.. _condor_q: https://htcondor.readthedocs.io/en/latest/man-pages/condor_q.html
.. _condor_rm: https://htcondor.readthedocs.io/en/latest/man-pages/condor_rm.html
.. _lsst_distrib: https://github.com/lsst/lsst_distrib.git
