================
General Workflow
================

Here the general idea behind hpc-tools is outlined along with some of the things that make your life on the cluster a bit easier. Once again, your experience on various HPCs may vary and this is geared towards GridEngine on UCL systems.

Array Jobs
----------

An array job is a single cluster job that is split into multiple tasks. for example each task might do the same thing do different input files and produce a corresponding output file. So, you might want to perform the same actions on 10 different input files producing 10 different output files. So you would submit an array job that is 10 tasks. Obviously, you also have the option of submitting 10 separate jobs in a for loop. However, each of these will have to be scheduled and will have it's own job ID. So if you notice a problem with the jobs and they need to be killed, then you will have to kill 10 separate jobs. With an array job, you get a single job ID, so it is simple to kill with ``qdel`` (on GridEngine). Additionally, submitting 10 separate jobs puts unnecessary strain on the scheduler. So, if you have multiple tasks, it is a good idea to use array jobs.

So, how to you supply all the parameters required for each task in a job array? The easiest approach is to use a flat file, which has each task on a separate line and each parameter for a task in a separate column. The line in the file that corresponds to the current task can then be interpreted by the script running the task (called the task script) and the values in that line can be used as parameters to run the task. In this readme and in the code, this is referred to as a job array file and I would typically give it a ``.job`` extension although there is no requirement for this. However, there is no requirement for a job array file to be a flat file. It could be something more exotic such as an XML file, as long as the task script knows how to parse it to interpret what is needed to run the current task then that is all that is required.

So, how does the current task know what line it is supposed to read from the job array file? This is the job of a specially created environment variable called the ``SGE_TASK_ID`` (on GridEngine). This is available as ``$SGE_TASK_ID`` in a bash script. This will contain an integer, corresponding to the task that is currently be run. So, if it is task numer 5 in the job array then ``$SGE_TASK_ID`` will equal ``5``. The task script will then know, to extract the line number 5 from the job array file and use it's contents as parameters to run the task.

The most tricky part of managing an array job is working out what tasks have failed and what tasks have been successful. Tasks can fail for a number of reasons, it could be they have exceeded their memory or time requirements, input/output files are missing or simply a bug in the code. Figuring out what has been successful and what has failed maybe easy if the array job is only 10 tasks but can be tricky/time consuming if there are 100,000s of tasks. There are several good practices that you can use to make your life a lot easier, this package implements some of them directly and assumes that you are using others:

1. Always make your task script a bash script (or whatever your favourite shells is): Whilst not an absolute requirement You can make sure that all your environment is setup (for example conda). The bash task script can then call whatever downstream scripts you need (for example python)
2. Use ``err`` and ``log`` files and name them appropriately. ``err`` files capture output to ``STDERR`` and outfiles capture output to ``STDOUT``. For example make sure the variable ``$TASK_ID`` is in the file name specification. Also, remember that these files will be appended to not overwritten. So if you re-run a job several times and point to the same files, then the output of all of your runs will be in the same files, this can get quite confusing. 
3. Make sure that any normal output (progress) is output to ``STDOUT`` and any error output is redirected to ``STDERR``. In ``Python``, this will happen implicitly for ``print`` and ``raises``, in ``Python`` a print statement can also be redirected to ``STDERR``, using the ``file=sys.stderr`` argument. but in bash you will have to redirect ``echo`` to ``1>&2`` for ``STDERR``.
4. Make sure that you can test your task script outside of the job array submission but in exactly the same way as it will be run when inside the job array. So, for example, you can submit an interactive job via ``qrsh`` and profile the time and memory requirements, i.e. run via ``/usr/bin/time -v``. It will also, allow you to efficiently debug your task without submitting the job. It is very difficult to debug a task by continuously submitting jobs and watching them fail.
5. A common cause of job failures are inaccessible input files and output file locations. These could be caused by a mistake in the file location or a permission error. It is good practice to make sure that all file/directory locations are present and accessible __before__ the job has been submitted. This way, you do not waste time submitting loads of jobs and the deleting them when they all start to fail.
6. Many scripts will create temporary output or intermediate files. These way be created in a working output directory or in some sort of job runtime ``tmp`` space. What happens to these files when your jobs finishes or if your job errors out? There could be temporary files littered all over the file system and filling up precious ``tmp`` space on a node. It is good practice to make sure that your task script and any other processes it uses clean up after themselves both when they execute cleanly and when they fail. ``bash`` ``traps`` are very useful for this purpose, trapping ``EXIT`` and ``ERR`` signals and executing clean up functions in response. Similarly in ``Python``, ``try``, ``except`` and ``finally`` blocks or ``context managers`` can perform these tasks.
7. If you are submitting thousands of jobs, then sometimes you will get failures for unspecified reasons, it could be other users overloading nodes of filling up ``/tmp``, sometimes, a job may run out of time or memory, even when you think it will be ok. Either way, the question is, how can you identify what tasks have failed and ones that have run to completion? If you have ``err`` and ``out`` files, these can be parsed to look for telltale signs of failure or completion, for example a ``"*** JOB COMPLETED ***"`` message as the last thing to ``echo`` out as the script exits, perhaps on a clean exit ``trap``. However, parsing all these files can be error prone and cumbersome. A far better approach, is to look for the presence of expected output files. These paths will often be present in your job array file, so they could be checked against that to determine and flag tasks where the expected output is absent. However, what if your job runs out of memory in the middle of writing an output file? Then you will have an expected output file present but it will be half written. If you have set up an ERROR trap, you could have it fire in response to ERR exit to remove output files. However, it is unclear if these will get fired in an out of memory situation. A good practice to get into is to write output files to a temporary location and when written one of the final commands in the task script is to move the output to it's final location. That way, any during any mid run failures the output file will not be in it's final location. An additional safety check is to take the ``MD5`` hash of the file before and after moving. If they do not match then the output file can be deleted before the script exists. This gets around the possibility of something going wrong when moving the output file. Either way, the end goal is to make sure that the output files are not present in their final location until you are 100% sure that they are the final output that you expect. Then you can safely say the job has completed by looking at the expect output locations in your job array file.

With the points above in mind, the general workflow for submitting an array job would be:

1. Create a job array file. There is a script/module ``make-jaf``  that can aid in doing this. See the documentation below for full details.
2. Write a task script that will perform each task of the job array. Whist, this is probably very specific for the task, there are some general skeleton task files in the ``resources`` directory that can be used as a base for this and adhere to many of the good practices listed above.
3. Write a job config (ini) file that details the parameters of the job.
4. Submit the job using ``pyjasub`` (`py` thon `j` ob `a` rray `sub` mission) 
5. After the submission has finished - re-run ``pyjasub`` to determine if all the tasks were processed successfully and re-submit any failed tasks.

