===========
``pyjasub``
===========

``pyjasub`` is the main job array submission script and should be available after installation of hpc-tools.

Setup
-----
If you have already followed the post Python installation tasks in getting started, then you can skip this section. If not, then please complete the steps below. ``pyjasub`` works by building job submission commands from user arguments. Therefore there is a mapping step to map between the user aguments and the cluster arguments. This happens internally but how the mapping is performed is controlled by a mapping config file. Examples, can be seen in the ``./resources/mappings`` directory and is also shown below:

.. code-block::

   [mappings]
   array=-t {0}-{1}:{2}
   
   # Multiple arguments are separated by commas, such as two memory arguments
   mem=-l hmem={0}, -l vmem={0}
   time=-l h_rt={0}
   nodes=-l hostname={0}
   shell=-S {0}
   out=-o {0}
   err=-e {0}
   batch=-tc {0}
   tmp_size=-l tmpfs={0}
   scratch_size=-l tscratch={0}
 

This follows a ``.ini`` format and and the python formatting characters ``{0}``, ``{1}`` etc... will be substituted with arguments. The mappings should be under the section header ``[mappings]``.

As a single the mappings file is expected per system, the location of the mappings file should be set in your ``~/.bashrc``:

.. code-block::

   export QSUB_PYTHON_MAPPING="/path/to/mappings.conf


As the mappings file is a potential security risk, is should only be readable by you. This is checked by ``pyjasub`` and if any other users have any of ``rwx`` on the mapping file then ``pyjasub`` will error out with a ``PermissionError``.

The abstraction of the cluster arguments has some plusses and minuses, it allows similar commands and job files can be used in different setups. However, this is at the expense of granular control. Having said that ``pyjasub`` can handle most everyday cluster uses and can probably be extended easily. Currently, ``pyjasub`` has only really been tested using ``GridEngine``, however, it should be able to be adapted for ``LSF``.

There are a number of bash scripts used by ``pyjasub``. Whilst, they are not obligatory and the user has full control of the task script, they can be used to perform some setup and cleanup operations. If you make sure that the directory ``.resources/bin`` is added to your ``PATH``, for example, in your ``~/.bashrc`` file:

.. code-block::

    export "${PATH}:/path/to/cluster/resources/bin"

Usage
-----

The options from the command line are as follows, these take priority over options defined in the config file.

.. argparse::
   :module: hpc_tools.pyjasub
   :func: _init_cmd_args
   :prog: pyjasub


Job submission workflow
-----------------------

``pyjasub`` is designed to make it easier to manage array job submissions and reruns of failed tasks. It can be run on the command line or be imported and the main submission class can be inherited from to taylor a job submission.

The general workflow for using ``pyjasub`` is as follows:

1. Create a job array file  using a script such as ``make_general_job_array_file``, or deriving from it. A job array file is a standard flat file that should have a header and be tab delimited. Its role is to supply task specific arguments to each task in the job array. The first column should be a column called ``rowidx`` and  should contain an incremental counter with the row after the header being 1.
2. Create a job config file that will supply the parameters for the job. Whilst this is optional and the parameters can be supplied on the command line it is recommended to use a config file to avoid errors, arguments given on the command line can be used to override config file parameters. An example of a config file is shown below:

.. code-block::

   # A job array config file for extracting the top hits from the interval data
   [qsub]
   # The general parameters for the job. The script has defaults for many of these
   step=1
   batch=50
   mem=2G
   shell=/bin/bash
   time=00:20:00
   nodes=None
   tmp_size=10G
   tmp_dir=~/Scratch

   [logs]
   # The root for all the job/run logs. If it does not exist it will be created
   log_dir=~/Scratch/somalogic_interval_top_hits_jobs

   [job_array]
   # The location of the job array file
   ja_file=~/somalogic_interval.jobs

   # The location of the task script that runs the job. Task scripts should
   # accept 4 arguments.
   # 1. the job array file
   # 2. the location of the tmp folder i.e. scratch
   # 3. the step size for the job array
   # 4. the job ID i.e. SGE_TASK_ID
   task_script=~/code/gwas_import_pipeline/scripts/extract_pvalue_job_array_task.sh

   # A comma separated list of column names in the jobarray files
   # where the input files are located and the output files are located
   # only list them if you want the presence of the input files/output
   # files checked by the submission script. if the column name contains
   # a comma, protenct with quotes ""
   infiles=infile
   outfiles=outfile

   [script]
   # Arguments to the script, at the moment it is just verbose that can be
   # true or false
   verbose=true

3. Keep on running ``pyjasub`` until all the tasks have been completed. If you have some stubbort tasks the keep failing the check your ``error`` and ``out`` files. These will be located in the log directory under the different run directories. 
4. Once completed, you will asked iff you want to archive your run files. Archiving creates a compressed tar archive and will delete your run directories. An example from a job array that took 4 runs to complete is shown below:

.. code-block::

   $ pyjasub interval.ja.conf
   === pyjasub (hpc_tools v0.1.0a1) ===
   [info] step: 1
   [info] batch: 50
   [info] mem: 2G
   [info] shell: /bin/bash
   [info] time: 00:20:00
   [info] nodes: None
   [info] tmp_dir: ~/Scratch
   [info] scratch_size: None
   [info] tmp_size: 10G
   [info] log_dir: ~/Scratch/somalogic_interval_top_hits_jobs
   [info] ja_file: ~/somalogic_interval.jobs
   [info] task_script: ~/code/gwas_import_pipeline/scripts/extract_pvalue_job_array_task.sh
   [info] infiles: ['infile']
   [info] outfiles: ['outfile']
   [info] full job log directory '/lustre/scratch/scratch/rmjdcfi/somalogic_interval_top_hits_jobs'
   [info] run started at: '2020-02-04 18:07:12.963490'
   [info] current run number: '5'
   [info] run log path: '/lustre/scratch/scratch/rmjdcfi/somalogic_interval_top_hits_jobs/run.log'
   [info] run directory: '/lustre/scratch/scratch/rmjdcfi/somalogic_interval_top_hits_jobs/5_1580839632'
   [info] run out files: '/lustre/scratch/scratch/rmjdcfi/somalogic_interval_top_hits_jobs/5_1580839632/out/\$TASK_ID_5_1580839632.out'
   [info] run err files: '/lustre/scratch/scratch/rmjdcfi/somalogic_interval_top_hits_jobs/5_1580839632/err/\$TASK_ID_5_1580839632.err'
   [info] full job array path: '/lustre/home/rmjdcfi/somalogic_interval.jobs'
   [info] job array md5: '6b637ce3197d9fcd1ca9bfa875b88a10'
   [info] full task script path: '/lustre/home/rmjdcfi/code/gwas_import_pipeline/scripts/extract_pvalue_job_array_task.sh'
   [info] task script md5: '9094df2fa17e6ff390493094b2c32c9c'
   [info] previous task script md5: '9094df2fa17e6ff390493094b2c32c9c'
   [info] copying task script: '/lustre/scratch/scratch/rmjdcfi/somalogic_interval_top_hits_jobs/5_1580839632/9094df2fa17e6ff390493094b2c32c9c.task'
   [info] run job array path: '/lustre/scratch/scratch/rmjdcfi/somalogic_interval_top_hits_jobs/5_1580839632/run_job_array.job'
   [info] check results below:
   [info] input file present: '72226'
   [info] input file absent: '0'
   [info] output file present: '3283'
   [info] output file absent: '0'
   [info] total tasks: '3283'
   [info] tasks to be submitted: '0'
   [complete] no tasks to submit - job is complete
   [info] run log details below...
   +----------------------------+--------------+------+-------+-----+----------+-------------+-----------------------+--------------+----------------+---------------+-----------------+---------------+
   | run_time                   | run_dir      | step | batch | mem | time     | total_tasks | no_of_tasks_submitted | infile_exist | infile_missing | outfile_exist | outfile_missing | job_id        |
   +----------------------------+--------------+------+-------+-----+----------+-------------+-----------------------+--------------+----------------+---------------+-----------------+---------------+
   | 2020-01-22 17:03:34.802277 | 1_1579712614 | 1    | 50    | 2G  | 00:20:00 | 3283        | 3283                  | 72226        | 0              | 0             | 3283            | 2582791       |
   | 2020-02-03 17:38:19.356899 | 2_1580751499 | 1    | 50    | 2G  | 00:20:00 | 3283        | 246                   | 72226        | 0              | 3037          | 246             | 2751956       |
   | 2020-02-04 05:56:59.818196 | 3_1580795819 | 1    | 50    | 2G  | 00:20:00 | 3283        | 246                   | 72226        | 0              | 3037          | 246             | 2755344       |
   | 2020-02-04 06:34:21.751488 | 4_1580798061 | 1    | 50    | 2G  | 00:20:00 | 3283        | 245                   | 72226        | 0              | 3038          | 245             | 2755345       |
   | 2020-02-04 18:07:12.963490 | 5_1580839632 | 1    | 50    | 2G  | 00:20:00 | 3283        | 0                     | 72226        | 0              | 3283          | 0               | NOT_SUBMITTED |
   +----------------------------+--------------+------+-------+-----+----------+-------------+-----------------------+--------------+----------------+---------------+-----------------+---------------+
   [info] job is complete
   [info] *** job submission finished ***
   [archive] do you want to archive the run files, press [ENTER] to archive or [CTRL-C] to quit > 

   [archive] archiving to '/lustre/scratch/scratch/rmjdcfi/somalogic_interval_top_hits_jobs.tar.gz'
   *** END ***

Checks on the job array file
----------------------------
Each run of ``pyjasub`` will start with a check or the job array file. The following are checked, if they to not pass then we get a fatal error

1. Has the job array file changed? The job array file is crucial to the integrety of all the jobs, therefore it must stay constant for the lifetime off all runs of the job.
2. Is the first column of the job array ``rowidx``, if not it is an error.
3. For each each row in the job array file, is it the same length (no of columns) as the header
4. For each each row in the job array file, is the first column of the row an expected incremental counter i.e. first column of the first row after the header should have the value ``1``.

Defining a completed job
------------------------
So here it can get tricky. We have to take into account the step size when evaluating the output files. So, we have to have all output files present for all steps in the task.
