# Getting started with hpc-tools

__version__: `0.2.1dev0`

[hpc-tools](https://gitlab.com/cfinan/cluster) is a Python package with some additional bash scripts/configuration files designed to simplify submitting array jobs on a cluster. Whilst it is written with Grid Engine in mind it can probably be re-purposed for other systems (such as LSF) - although this has not been tested in any way. Also, this has only been tested on the University College London (UCL) systems, Myriad and the CS cluster.

There is [online](https://cfinan.gitlab.io/cluster/index.html) documentation for hpc-tools and offline PDF documentation can be downloaded [here](https://gitlab.com/cfinan/cluster/-/blob/master/resources/pdf/hpc-tools.pdf).

## Installation instructions
At present, hpc-tools is undergoing development and no packages exist yet on PyPi. Therefore it is recommended that it is installed in either of the two ways listed below. First, clone this repository and then `cd` to the root of the repository.

```
git clone git@gitlab.com:cfinan/cluster.git
cd cluster
```

### Installation not using any conda dependencies
If you are not using conda in any way then install the dependencies via `pip` and install hpc-tools as an editable install also via pip:

Install dependencies:
```
python -m pip install --upgrade -r requirements.txt
```

For an editable (developer) install run the command below from the root of the bio-misc repository (or drop `-e` for static install):
```
python -m pip install -e .
```

### Installation using conda
A conda build is also available for Python v3.7, v3.8 and v3.9 on linux-64.
```
conda install -c cfin hpc-tools
```

If you are using conda and require anything different then please install with pip and install the dependencies with the environments specified in `resources/conda_env`. There is also a build recipe in `./resourses/build/conda` that can be used to create new packages.

This will also install `pyjasub` and `make-jaf` as console script endpoints that will be executable from the command line.

### Post Python install
After installation of hpc-tools, there are some manual setup steps.

#### Setup a mapping file
The main submission script, `pyjasub` works by building job submission commands from user arguments. Therefore there is a mapping step to map between the user aguments and the cluster arguments. This happens internally but how the mapping is performed is controlled by a mapping config file. Examples, can be seen in the `./resources/mappings` directory of the repository and is also shown below:

```
[mappings]
array=-t {0}-{1}:{2}

# Multiple arguments are separated by commas, such as two memory arguments
mem=-l hmem={0}, -l vmem={0}
time=-l h_rt={0}
nodes=-l hostname={0}
shell=-S {0}
out=-o {0}
err=-e {0}
batch=-tc {0}
tmp_size=-l tmpfs={0}
scratch_size=-l tscratch={0}
```

This follows a `.ini` format and and the Python formatting characters `{0}`, `{1}` etc... will be substituted with arguments. The mappings should be under the section header `[mappings]`.

As a single the mappings file is expected per system, the location of the mappings file should be set in your `~/.bashrc`:

```
export QSUB_PYTHON_MAPPING="/path/to/mappings.conf"
```

As the mappings file is a potential security risk, is should only be readable by you. This is checked by `pyjasub` and if any other users have any of `rwx` on the mapping file then `pyjasub` will error out with a `PermissionError`.

The abstraction of the cluster arguments has some plusses and minuses, it allows similar commands and job files can be used in different setups. However, this is at the expense of granular control. Having said that `pyjasub` can handle most everyday cluster uses and can probably be extended easily. Currently, `pyjasub` has only really been tested using `GridEngine`, however, it should be able to be adapted for `LSF`.

#### Add the `bin` directory to your `PATH`
There are a number of bash scripts used by `pyjasub`. Whilst, they are not obligatory and the user has full control of the task script, they can be used to perform some setup and cleanup operations. If you make sure that the directory `.resources/bin` is added to your `PATH`, for example, in your `~/.bashrc` file:

```
export "${PATH}:/path/to/cluster/resources/bin"
```
