Software Compatibility
=======================

**e4s-cl** wraps itself around a MPI command and inserts a container launch before the command is run. This implies an understanding of both the process launcher and the container technology to use. This section details the software **e4s-cl** was developped for and tested with.

MPI implementations
--------------------

.. _mpi_compatibility:

**e4s-cl** is able to launch executables compiled with a different version than the one available on the system at runtime.

MPI families
+++++++++++++

With the introduction of the MPICH ABI initiative and the MPI implementations that followed it, we see the emergence of multiple 'families' of MPI implementations. Members within a family can usually swap their libraries at runtime, while this is impossible accross families.

If you are unsure about the implementation used to compile a binary, a simple check is to run :code:`ldd` on the executable and looking at the MPI library listed as a dependency. The below table lists the usual libray name in the :code:`soname` column.

.. csv-table::
   :file: mpi.csv

Enabling translation
+++++++++++++++++++++

**e4s-cl** is able to detect the MPI implementation used on the host from the input parameters; however the implementation used in the container cannot be ascertained easily. MPI translation is thus not needed when running executables with an implementation from the same family as the host's; when this is not the case, one can enable it from the command line using the :code:`--from` flag:

.. code::

   # Enable translation from an OpenMPI binary to a MPICH-based profile
   e4s-cl --from openmpi srun -n 4 [...] path/to/executable

This will install and leverage :code:`WI4MPI` (git repos `here <https://github.com/cea-hpc/wi4mpi>`_), which is a wrapper interface that performs translations from one MPI implementation to another. This allows :code:`e4s-cl` to function even when substituting two MPI libraries that are ABI-incompatible.

Container backends
-------------------

During a :code:`launch` command, a list of required files will be computed on each target node, and bound to a container spawned on those nodes. Container backends are the container technologies **e4s-cl** is able to use.

:code:`apptainer`, :code:`podman`, :code:`shifter` and :code:`singularity` are supported. The :code:`docker` backend can be enabled by installing **e4s-cl** with the :code:`docker` extra feature.

Most of the container backends have some degree of configuration available to them.
Configuration is done by setting the corresponding values in the :code:`backends` section of the configuration file, or by passing a given environment variable.

An option passed in the environment will override the same option set in the configuration file.

:code:`apptainer`
++++++++++++++++++

:code:`apptainer` support is achieved using its command line interface. :code:`apptainer` has to be installed on the system and available for it to be used by **e4s-cl**.

The configuration file section for :code:`apptainer` is :code:`backends/apptainer`.
The available options are:

.. list-table::
   :widths: 10 10 5 5 20
   :header-rows: 1

   * - Configuration variable
     - Environment variable
     - Type
     - Default
     - Description

   * - :code:`executable`
     - :code:`E4S_CL_APPTAINER_EXECUTABLE`
     - String
     - :code:`""`
     - Path to the :code:`apptainer` executable to use.

   * - :code:`options`
     - :code:`E4S_CL_APPTAINER_OPTIONS`
     - List
     - :code:`[]`
     - Options to pass to the spawned :code:`apptainer` process.

   * - :code:`run_options`
     - :code:`E4S_CL_APPTAINER_EXEC_OPTIONS`
     - List
     - :code:`[]`
     - Options to pass to the :code:`exec` command of the spawned :code:`apptainer` process.

:code:`docker`
++++++++++++++

.. admonition:: :code:`docker` support

   :code:`docker` support in HPC is limited at best. Due to its architecture, it will most likely not work with your MPI implementation. To enable :code:`docker` support from **e4s-cl**, :ref:`install<install>` with the :code:`docker` extra enabled.

:code:`docker` support is achieved using the :code:`docker-py` `module <https://github.com/docker/docker-py>`_. The :code:`docker` daemon has to be installed and running on the system to be accessed.

There are no configuration options for the :code:`docker` backend.

.. warning:: Using **docker** with MPI

   Several MPI implementations expect their processes to inherit opened file descriptors; because of docker's client-daemon architecture, this is not possible. To use docker images with MPI, it is encouraged to used :ref:`podman`.

.. _podman:

:code:`podman`
+++++++++++++++

:code:`podman` support is achieved using its command line interface. :code:`podman` has to be installed on the system and available for it to be used by **e4s-cl**.

The configuration file section for :code:`podman` is :code:`backends/podman`.
The available options are:

.. list-table::
   :widths: 10 10 5 5 20
   :header-rows: 1

   * - Configuration variable
     - Environment variable
     - Type
     - Default
     - Description

   * - :code:`executable`
     - :code:`E4S_CL_PODMAN_EXECUTABLE`
     - String
     - :code:`""`
     - Path to the :code:`podman` executable to use.

   * - :code:`options`
     - :code:`E4S_CL_PODMAN_OPTIONS`
     - List
     - :code:`[]`
     - Options to pass to the spawned :code:`podman` process.

   * - :code:`run_options`
     - :code:`E4S_CL_PODMAN_RUN_OPTIONS`
     - List
     - :code:`[]`
     - Options to pass to the :code:`run` command of the spawned :code:`podman` process.

:code:`shifter`
++++++++++++++++

The :code:`shifter` container technology is an interesting case, as it offers much of what **e4s-cl** is trying to propose, but in a much more restrictive way. 

.. warning:: :code:`shifter` MPI modules

    Some MPI libraries will be configured as modules for :code:`shifter` and imported automatically. You might not need **e4s-cl** ! Check the configuration file in :code:`/etc/shifter/udiRoot.conf` to see if the MPI library you are trying to use is imported.

.. warning:: Binding files with :code:`shifter`

    :code:`shifter` is the only container backend that does not support binding files to the container.
    :code:`shifter` also forbids binding directories to certain locations, like :code:`/etc`.
    This is circumvented by **e4s-cl** by copying all required libraries in a temporary directory, then binding it to the container. Files are ignored.

The configuration file section for :code:`shifter` is :code:`backends/shifter`.

.. list-table::
   :widths: 10 10 5 5 20
   :header-rows: 1

   * - Configuration variable
     - Environment variable
     - Type
     - Default
     - Description

   * - :code:`executable`
     - :code:`E4S_CL_SHIFTER_EXECUTABLE`
     - String
     - :code:`""`
     - Path to the :code:`shifter` executable to use.

   * - :code:`options`
     - :code:`E4S_CL_SHIFTER_OPTIONS`
     - List
     - :code:`[]`
     - Options to pass to the spawned :code:`shifter` process.

:code:`singularity`
++++++++++++++++++++

:code:`singularity` support is achieved using its command line interface. :code:`singularity` has to be installed on the system and available for it to be used by **e4s-cl**.

The configuration file section for :code:`singularity` is :code:`backends/singularity`.
The available options are:

.. list-table::
   :widths: 10 10 5 5 20
   :header-rows: 1

   * - Configuration variable
     - Environment variable
     - Type
     - Default
     - Description

   * - :code:`executable`
     - :code:`E4S_CL_SINGULARITY_EXECUTABLE`
     - String
     - :code:`""`
     - Path to the :code:`singularity` executable to use.

   * - :code:`options`
     - :code:`E4S_CL_SINGULARITY_OPTIONS`
     - List
     - :code:`[]`
     - Options to pass to the spawned :code:`singularity` process.

   * - :code:`run_options`
     - :code:`E4S_CL_SINGULARITY_EXEC_OPTIONS`
     - List
     - :code:`[]`
     - Options to pass to the :code:`exec` command of the spawned :code:`singularity` process.

:code:`barebones`
++++++++++++++++++++

:code:`barebones` support is achieved by bypassing the use of a container technology.

The configuration file section for :code:`barebones` is :code:`backends/barebones`.
The available options are:

.. list-table::
   :widths: 10 10 5 5 20
   :header-rows: 1

   * - Configuration variable
     - Environment variable
     - Type
     - Default
     - Description

   * - :code:`options`
     - :code:`E4S_CL_BAREBONES_OPTIONS`
     - List
     - :code:`[]`
     - Options to pass to the spawned process.

   * - :code:`run_options`
     - :code:`E4S_CL_BAREBONES_EXEC_OPTIONS`
     - List
     - :code:`[]`
     - Options to pass to the script of the spawned process.

Other container backends
++++++++++++++++++++++++++

More container technologies can be supported. Create an issue on github or write a dedicated module in :code:`e4s_cl/cf/containers`. Refer to :code:`e4s_cl/cf/containers/__init__.py` for details.

Process launchers
------------------

The following process managers were successfully tested with **e4s-cl**:

- The stock :code:`mpirun` of multiple MPI distributions;
- LLNL's SLURM using :code:`srun`;
- CRAY's ALPS using :code:`aprun`;
- IBM's JSM using :code:`jsrun`.

Support implies the automatic detection of parameters. If a launcher is not
supported, use the ':code:`--`' syntax to separate launcher and process arguments.

Additional options can be configured through the configuration file or the environment:

.. list-table::
   :widths: 10 10 5 5 20
   :header-rows: 1

   * - Configuration variable
     - Environment variable
     - Type
     - Default
     - Description

   * - :code:`launcher_options`
     - :code:`E4S_CL_LAUNCHER_OPTIONS`
     - List
     - :code:`[]`
     - List of options to pass to the launcher
