# Example job array
Here we will run through a simple toy example of submitting a job with `pyjasub`. This assumes that you have installed the package and have your mapping file in place and the path to it set in your `~/.bashrc`, if not please see [here](https://gitlab.com/cfinan/cluster). Also, to run the scripts in the example you will need to clone the repository and have assess to `./cluster/resources/example` where `.` is the root of the repository.

## Creating example files
For this there is a small script that will create a directory called `~/hpc_tools_examples` in the root of your home directory. If it already exists, it will error out. This directory, will have everything in it to run the examples. Please make sure you are in the `./cluster/resources/example` directory when running the script:

```
./create_example_files.sh
```

After running the test input files are available in `~/hpc_tools_examples/test_input` and the scripts etc are available in `~/hpc_tools_examples/example`

## Creating a job array file
Now we will use the file and directory structure just created to build a job array file that has the following columns:

1. `rowidx` - A counter starting at 1 for the file non-header row. This always has to be present in this position in job array files used by `pyjasub`
2. `infile` - The location of the input file (what we just created with `./create_example_files.sh`
3. `outfile` - The location of the output file that will be created by our task script

We will use the helper script `make-jaf` to create the basic file structure above. Now we create our job array file using the command below:

```
make-jaf -v \
         -i ~/hpc_tools_examples/test_input \
         -g '*/*.txt' \
         -o ~/hpc_tools_examples/test_output \
         --regexp 'i_am_a_big_long_file_name_with_(?P<B>\d+)_and_all_this_as_well_(?P<A>\w+)\.txt' \
         --levels 1 \
         --strip_ext \
         --suffix ".out" \
         --outfile "./example_job_array.job"
```

The command above can also be run via the script `create_job_array.sh`

```
./create_job_array.sh
```

Running should produce an output similar to below:
```
=== make_general_job_array (hpc_tools v0.1.0a1) ===
[info] indir value: /home/rmjdcfi/hpc_tools_examples/test_input
[info] infiles length: 0
[info] inglob value: */*.txt
[info] levels value: 1
[info] outdir value: /home/rmjdcfi/hpc_tools_examples/test_output
[info] outfile value: ./example_job_array.job
[info] outfile_errors value: error
[info] prefix value: None
[info] regexp value: i_am_a_big_long_file_name_with_(?P<B>\d+)_and_all_this_as_well_(?P<A>\w+)\.txt
[info] strip_ext value: True
[info] suffix value: .out
[info] verbose value: True
[info] 10 input found
[info] *** END ***
```

The job array file should look similar to this, although the home directory paths will differ:
```
$ cat example_job_array.job 
rowidx  infile  outfile
1       /home/rmjdcfi/hpc_tools_examples/test_input/india/i_am_a_big_long_file_name_with_9_and_all_this_as_well_nine.txt /home/rmjdcfi/hpc_tools_examples/test_output/india/nine_9.out
2       /home/rmjdcfi/hpc_tools_examples/test_input/foxtrot/i_am_a_big_long_file_name_with_6_and_all_this_as_well_six.txt        /home/rmjdcfi/hpc_tools_examples/test_output/foxtrot/six_6.out
3       /home/rmjdcfi/hpc_tools_examples/test_input/bravo/i_am_a_big_long_file_name_with_2_and_all_this_as_well_two.txt  /home/rmjdcfi/hpc_tools_examples/test_output/bravo/two_2.out
4       /home/rmjdcfi/hpc_tools_examples/test_input/alfa/i_am_a_big_long_file_name_with_1_and_all_this_as_well_one.txt   /home/rmjdcfi/hpc_tools_examples/test_output/alfa/one_1.out
5       /home/rmjdcfi/hpc_tools_examples/test_input/golf/i_am_a_big_long_file_name_with_7_and_all_this_as_well_seven.txt /home/rmjdcfi/hpc_tools_examples/test_output/golf/seven_7.out
6       /home/rmjdcfi/hpc_tools_examples/test_input/hotel/i_am_a_big_long_file_name_with_8_and_all_this_as_well_eight.txt        /home/rmjdcfi/hpc_tools_examples/test_output/hotel/eight_8.out
7       /home/rmjdcfi/hpc_tools_examples/test_input/delta/i_am_a_big_long_file_name_with_4_and_all_this_as_well_four.txt /home/rmjdcfi/hpc_tools_examples/test_output/delta/four_4.out
8       /home/rmjdcfi/hpc_tools_examples/test_input/echo/i_am_a_big_long_file_name_with_5_and_all_this_as_well_five.txt  /home/rmjdcfi/hpc_tools_examples/test_output/echo/five_5.out
9       /home/rmjdcfi/hpc_tools_examples/test_input/charlie/i_am_a_big_long_file_name_with_3_and_all_this_as_well_three.txt      /home/rmjdcfi/hpc_tools_examples/test_output/charlie/three_3.out
10      /home/rmjdcfi/hpc_tools_examples/test_input/juliett/i_am_a_big_long_file_name_with_10_and_all_this_as_well_ten.txt       /home/rmjdcfi/hpc_tools_examples/test_output/juliett/ten_10.out
```

The command above illustrates several useful features of `make_general_job_array_file`, in fact, `make_general_job_array_file` can be imported and extended to make more specific job arrays with other parameters in the file.

So the first thing to point out is that we supplied an input directory `-i` and an input glob `-g`. These are placed together and any files matching that glob will be used. We could have also used the unix globing and placed `/home/rmjdcfi/hpc_tools_examples/test_input/*/*.txt` after the `--outfile` argument. Also, if suppling a glob via `-g` then make sure it is single quoted so unix path expansion does not take place.

We also specified an output directory argument. If we had not, then the input directory would have been used as the output directory. The output directory is also created if it does not exist.

You will also notice that the there are a few differences between the structure of the input file name and that if the output file. Firstly, they have different extensions, the input files have `.txt` and the output files have `.out`, this is achieved by a combination of `--strip_ext` and `--suffix` arguments. Also, our long unweidly file name has been reduced considerably. This is a result of the `--regexp` argument. Here we supplied a regular expression to match against the file name and capture the parts we want to keep using python named capture groups, labelled alphabetically in the ourder we want them in the output file name. This allows us to extract the relevent parts of the file name. If this option is used, each file name must match the regexp and the output files produced must be unique in the whole job array file (see the `--outfile_errors` argument if not). Also, another important point about the `--regexp` argument is that it must also be single quoted when supplied on the command line.

Finally you will also noticed that we retained the "phonetic" sub-directory structure of the input files, with `alfa`, `bravo`, `charlie`, `delta` etc... . This was accomlished using the `--levels` argument. So if `--levels 0` was supplied (the default) then your output files wouldnot have the phonetic sub directory attached, so instead of this `/home/rmjdcfi/hpc_tools_examples/test_output/juliett/ten_10.out`, we would have has this: `/home/rmjdcfi/hpc_tools_examples/test_output/ten_10.out`. Also, if we had supplied `--levels 2`, then we would have had this: `/home/rmjdcfi/hpc_tools_examples/test_output/test_input/juliett/ten_10.out`. So `--levels N` will take N many directory levels above the input file and add it to below the output directory. All the sub-directories defined with `--levels` are created automatically so do not have to be created by the task script.

## Testing the task script
You will want/need to be able to test your task script from outside of your job array and that is what we will do here. The example task script is very simple. It accepts 3 required positional arguments and an optional argument

1. The job array file
2. Temp location
3. Step size
4. Task ID (optional) - if not supplied then it will default to the `$SGE_TASK_ID` environment variable

When running as a job array, the first 3 arguments are supplied by `pyjasub` and the forth one defaults to the `$SGE_TASK_ID` supplied by `GridEngine`. However, when testing on the command line, we supply all 4 with the forth one being the job row in the job array file we want to test.

Now, ensure that the task script is executable with `chmod +x example_task.sh` and assuming you are in the `example` directory. Running the first task from the job array should looklike this:
```
~/hpc_tools_examples/example/example_task.sh ~/hpc_tools_examples/example/example_job_array.job ~/Scratch/ 1 1
=== ./example_task.sh ===
[info] started at Wed  5 Feb 11:39:11 GMT 2020
[info] hostname=login12.myriad.ucl.ac.uk
[info] job array file=example_job_array.job
[info] temp location=/home/rmjdcfi/Scratch/
[info] step=1
[info] job idx line (SGE_TASK_ID)=1
[info] pull line (accounting for header)=2
[info] quit line=3
[info] # infiles=1
[info] outfile=/home/rmjdcfi/hpc_tools_examples/test_output/india/nine_9.out
[info] temp file=/home/rmjdcfi/Scratch/.RUN_rmjdcfi_1_mipaXs6bp8
[info] ended at Wed  5 Feb 11:39:11 GMT 2020
*** END ***
```

And will produce an output file like this:
```
$ cat /home/rmjdcfi/hpc_tools_examples/test_output/india/nine_9.out
output file: 
This is the file: /home/rmjdcfi/hpc_tools_examples/test_input/india/i_am_a_big_long_file_name_with_9_and_all_this_aswell_nine.txt
```

Now we attempt to run the second task in the job array. Remember that we have set this to fail before the output is written
```
~/hpc_tools_examples/example/example_task.sh ~/hpc_tools_examples/example/example_job_array.job ~/Scratch/ 1 2
=== ./example_task.sh ===
[info] started at Wed  5 Feb 11:41:45 GMT 2020
[info] hostname=login12.myriad.ucl.ac.uk
[info] job array file=example_job_array.job
[info] temp location=/home/rmjdcfi/Scratch/
[info] step=1
[error] simulated failure job index: 2
```

Whilst not indicated in the output, our second task should have output a file called `hpc_tools_examples/test_output/foxtrot/six_6.out`, it will not exist.

So we know that our task script is working ok, we can now prepare to run our job array

## Job array arguments
When supplying arguments to `pyjasub`, they can either be supplied via the command line or a config file (in `ini` format). A config file is easier as it is less error prone. However, both can be used with command like arguments overiding those in the config file. An example config file is given to run the job. It will output the logs of the job into the root of the `~/hpc_tools_examples` directory. It is also shown below: 

```
# A job array config file for extracting the top hits from the interval data
[qsub]
# The general parameters for the job. The script has defaults for many of these
step=1
batch=50
mem=1M
shell=/bin/bash
time=00:01:00
nodes=None
tmp_size=10M
tmp_dir=~/Scratch

[logs]
# The root for all the job/run logs. If it does not exist it will be created
log_dir=~/hpc_tools_examples/example_job_logs

[job_array]
# The location of the job array file, I have given a full path
# so there are no issues, relative paths are ok, but remember they will
# be relative to where the script is run not where the config file is
# also ~/ can be used
ja_file=~/hpc_tools_examples/example/example_job_array.job

# The location of the task script that runs the job. Task scripts should
# accept 4 arguments.
# 1. the job array file
# 2. the location of the tmp folder i.e. scratch
# 3. the step size for the job array
# 4. the job ID i.e. SGE_TASK_ID
task_script=~/hpc_tools_examples/example/example_task.sh

# A comma separated list of column names in the jobarray files
# where the input files are located and the output files are located
# only list them if you want the presence of the input files/output
# files checked by the submission script. if the column name contains
# a comma, protenct with quotes ""
infiles=infile
outfiles=outfile

[script]
# Arguments to the script, at the moment it is just verbose that can be
# true or false
verbose=true
```

## Running the jobs
Now with everything in place the jobs can be submitted with the command:

```
pyjasub ~/hpc_tools_examples/example/example_job.cnf
```

Which should produce a report like this if successfully submitted. Note that `pyjasub` detects that one of the tasks has already been run (it was run during testing) so it will only submit none tasks. We expect all the evenly numbered tasks to fail as we have engineered it that way, so we expect at least 4 failues on this run, then 2 failures on the second run, 1 failure onn the 3rd run and 0 failures on the forth run. So when we run the fifth time we should get a message saying that the job is complete. Of course, the numbers above assume that no failures occur for other reasons that we can't control.
```
=== pyjasub (hpc_tools v0.1.0a1) ===
[info] step: 1
[info] batch: 50
[info] mem: 1M
[info] shell: /bin/bash
[info] time: 00:01:00
[info] nodes: None
[info] tmp_dir: ~/Scratch
[info] scratch_size: None
[info] tmp_size: 10M
[info] log_dir: ~/hpc_tools_examples/example_job_logs
[info] ja_file: ~/hpc_tools_examples/example/example_job_array.job
[info] task_script: ~/hpc_tools_examples/example/example_task.sh
[info] infiles: ['infile']
[info] outfiles: ['outfile']
[info] full job log directory '/lustre/home/rmjdcfi/hpc_tools_examples/example_job_logs'
[info] creating log directory '/lustre/home/rmjdcfi/hpc_tools_examples/example_job_logs'
[info] run started at: '2020-02-05 12:29:31.538651'
[info] current run number: '1'
[info] run log path: '/lustre/home/rmjdcfi/hpc_tools_examples/example_job_logs/run.log'
[info] run directory: '/lustre/home/rmjdcfi/hpc_tools_examples/example_job_logs/1_1580905771'
[info] run out files: '/lustre/home/rmjdcfi/hpc_tools_examples/example_job_logs/1_1580905771/out/\$TASK_ID_1_1580905771.out'
[info] run err files: '/lustre/home/rmjdcfi/hpc_tools_examples/example_job_logs/1_1580905771/err/\$TASK_ID_1_1580905771.err'
[info] full job array path: '/lustre/home/rmjdcfi/hpc_tools_examples/example/example_job_array.job'
[info] job array md5: 'db812ee5a433b04843353f4ef813796b'
[info] full task script path: '/lustre/home/rmjdcfi/hpc_tools_examples/example/example_task.sh'
[info] task script md5: '1278fb188fb52d4c6615325db565fa8a'
[info] copying task script: '/lustre/home/rmjdcfi/hpc_tools_examples/example_job_logs/1_1580905771/1278fb188fb52d4c6615325db565fa8a.task'
[info] run job array path: '/lustre/home/rmjdcfi/hpc_tools_examples/example_job_logs/1_1580905771/run_job_array.job'
[info] check results below:
[info] input file present: '10'
[info] input file absent: '0'
[info] output file present: '1'
[info] output file absent: '9'
[info] total tasks: '10'
[info] tasks to be submitted: '9'
[run] You are about to run '9' tasks over '9' rows in a job array with a step of '1', using the following command:
[run] qsub -t 1-9:1 -e /lustre/home/rmjdcfi/hpc_tools_examples/example_job_logs/1_1580905771/err/\$TASK_ID_1_1580905771.err -o /lustre/home/rmjdcfi/hpc_tools_examples/example_job_logs/1_1580905771/out/\$TASK_ID_1_1580905771.out -l mem=1M -S /bin/bash -l h_rt=00:01:00 -l tmpfs=10M /lustre/home/rmjdcfi/hpc_tools_examples/example/example_task.sh /lustre/home/rmjdcfi/hpc_tools_examples/example_job_logs/1_1580905771/run_job_array.job /lustre/scratch/scratch/rmjdcfi 1
[run] if all this looks correct, press [ENTER] to submit or [CTRL-C] to quit > 

[run] submitting job...
[run] submitted array job with ID 2768482
[info] run log details below...
+----------------------------+--------------+------+-------+-----+----------+-------------+-----------------------+--------------+----------------+---------------+-----------------+---------+
| run_time                   | run_dir      | step | batch | mem | time     | total_tasks | no_of_tasks_submitted | infile_exist | infile_missing | outfile_exist | outfile_missing | job_id  |
+----------------------------+--------------+------+-------+-----+----------+-------------+-----------------------+--------------+----------------+---------------+-----------------+---------+
| 2020-02-05 12:29:31.538651 | 1_1580905771 | 1    | 50    | 1M  | 00:01:00 | 10          | 9                     | 10           | 0              | 1             | 9               | 2768482 |
+----------------------------+--------------+------+-------+-----+----------+-------------+-----------------------+--------------+----------------+---------------+-----------------+---------+
[info] *** job submission finished ***
*** END ***
```

After the 5th run you should see something like this, and you will get asked if you want to archive all the log files. Pressing `ENTER` will write a tar file on the run directory contents and delete the run directory:
```
$ pyjasub example_job.cnf
=== pyjasub (hpc_tools v0.1.0a1) ===
[info] step: 1
[info] batch: 50
[info] mem: 1G
[info] shell: /bin/bash
[info] time: 00:10:00
[info] nodes: None
[info] tmp_dir: ~/Scratch
[info] scratch_size: None
[info] tmp_size: 1G
[info] log_dir: ~/hpc_tools_examples/example_job_logs
[info] ja_file: ~/hpc_tools_examples/example/example_job_array.job
[info] task_script: ~/hpc_tools_examples/example/example_task.sh
[info] infiles: ['infile']
[info] outfiles: ['outfile']
[info] full job log directory '/lustre/home/rmjdcfi/hpc_tools_examples/example_job_logs'
[info] run started at: '2020-02-05 14:15:24.519655'
[info] current run number: '5'
[info] run log path: '/lustre/home/rmjdcfi/hpc_tools_examples/example_job_logs/run.log'
[info] run directory: '/lustre/home/rmjdcfi/hpc_tools_examples/example_job_logs/5_1580912124'
[info] run out files: '/lustre/home/rmjdcfi/hpc_tools_examples/example_job_logs/5_1580912124/out/\$TASK_ID_5_1580912124.out'
[info] run err files: '/lustre/home/rmjdcfi/hpc_tools_examples/example_job_logs/5_1580912124/err/\$TASK_ID_5_1580912124.err'
[info] full job array path: '/lustre/home/rmjdcfi/hpc_tools_examples/example/example_job_array.job'
[info] job array md5: 'db812ee5a433b04843353f4ef813796b'
[info] full task script path: '/lustre/home/rmjdcfi/hpc_tools_examples/example/example_task.sh'
[info] task script md5: '1278fb188fb52d4c6615325db565fa8a'
[info] previous task script md5: '1278fb188fb52d4c6615325db565fa8a'
[info] copying task script: '/lustre/home/rmjdcfi/hpc_tools_examples/example_job_logs/5_1580912124/1278fb188fb52d4c6615325db565fa8a.task'
[info] run job array path: '/lustre/home/rmjdcfi/hpc_tools_examples/example_job_logs/5_1580912124/run_job_array.job'
[info] check results below:
[info] input file present: '10'
[info] input file absent: '0'
[info] output file present: '10'
[info] output file absent: '0'
[info] total tasks: '10'
[info] tasks to be submitted: '0'
[complete] no tasks to submit - job is complete
[info] run log details below...
+----------------------------+--------------+------+-------+-----+----------+-------------+-----------------------+--------------+----------------+---------------+-----------------+---------------+
| run_time                   | run_dir      | step | batch | mem | time     | total_tasks | no_of_tasks_submitted | infile_exist | infile_missing | outfile_exist | outfile_missing | job_id        |
+----------------------------+--------------+------+-------+-----+----------+-------------+-----------------------+--------------+----------------+---------------+-----------------+---------------+
| 2020-02-05 13:17:21.707045 | 1_1580908641 | 1    | 50    | 1G  | 00:10:00 | 10          | 9                     | 10           | 0              | 1             | 9               | 2768778       |
| 2020-02-05 13:43:39.033742 | 2_1580910219 | 1    | 50    | 1G  | 00:10:00 | 10          | 4                     | 10           | 0              | 6             | 4               | 2768812       |
| 2020-02-05 14:01:20.850012 | 3_1580911280 | 1    | 50    | 1G  | 00:10:00 | 10          | 2                     | 10           | 0              | 8             | 2               | 2768958       |
| 2020-02-05 14:09:40.718982 | 4_1580911780 | 1    | 50    | 1G  | 00:10:00 | 10          | 1                     | 10           | 0              | 9             | 1               | 2768969       |
| 2020-02-05 14:15:24.519655 | 5_1580912124 | 1    | 50    | 1G  | 00:10:00 | 10          | 0                     | 10           | 0              | 10            | 0               | NOT_SUBMITTED |
+----------------------------+--------------+------+-------+-----+----------+-------------+-----------------------+--------------+----------------+---------------+-----------------+---------------+
[info] job is complete
[info] *** job submission finished ***
[archive] do you want to archive the run files, press [ENTER] to archive or [CTRL-C] to quit > 

[archive] archiving to '/lustre/home/rmjdcfi/hpc_tools_examples/example_job_logs.tar.gz'
*** END ***
```

**TODO: Add some notes about the run directories**
