============
``make-jaf``
============

Create a general job array file that has 3 columns. The file also has a header and is tab separated. This can me executed as a script or imported as a module to aid building of more specific job array files.

Usage
-----

The options from the command line are as follows, these take priority over options defined in the config file.

.. argparse::
   :module: hpc_tools.make_general_job_array_file
   :func: _init_cmd_args
   :prog: make-jaf


Output file
-----------

When executed as a script it will generate a three column TAB delimited job array file with the columns:

1. ``rowidx`` An integer counter of the job number. Note that ``pyjasub`` always expects the first column to be a ``rowidx`` column.
2. ``infile`` An input file column, this is not mandatory but most job tasks will probably require at least a single input file. Note, that is is valid for this column to contain a pipe '|' separated set on input files in this column also. These are generated if the ``--flatten`` option is active and the output file names are not unique.
3. ``outfile`` An output file column. Note it is also valid to have a pipe ``|`` separated set of output files in this column. Although, ``make-jaf`` does not do this.

The ``make-jaf`` script firsts gathers all the available input files, depending on the user arguments. The user can either supply input files as positional arguments to the script or additionally an input directory (``--indir``) and/or a file globing option to match unix wild cards (``--inglob``). The ``inglob`` is applied to the input directory and may actually be made into a single option in future. If ``indir`` is supplied and ``inglob`` is not supplied then the entire contents of ``indir`` are used as input files. All
 input files are checked to make sure that they are actually files and not directories, any input files that do end up being directories are ignored. If the ``infiles`` positional argument, ``--indir`` and ``--inglob`` are all used then the full complement of files gathered by all of them will be used. All input files are checked to make sure they are readable by opening and closing the file. If this fails then it is a fatal error.

Output file names are generated from the input file names. The ``--outdir``, ``--levels``, ``--regexp``, ``--strip_ext``, ``--prefix`` and ``--suffix`` options control how output file names are generated. If ``--outdir`` is not specified then the output file for each input file in the job array is the same as the input file directory and ``--levels`` has no effect. Also, in this instance, if no other output options are used then the output file name will
 be the same as the input file name and this will cause an error. If ``--outdir`` is supplied then the basename of the input file is added to the output file name. If ``--levels`` is also supplied, then the ``--levels`` number of directories up from the input files are created below the ``--outdir``. The default is 0, i.e. no levels down from the input directory.

 The ``--prefix`` and ``--suffix`` generates an output file name with strings added to the start or the end of the input file basename. ``--suffix`` is added before the file extension. ``--strip_ext`` will generate an output file name from the input file name with the extension removed. So a new output file extension can be generated with a combination of ``--strip_ext`` and ``--suffix``.

The ``--regexp`` argument allows the user to supply a regexp on the command line that will be applied to the full path of the input file and generate an output file name. Even if ``--regexp`` is included, all the other options still apply. This regexp uses named capture groups to extract the parts that should be used in the output file name. the capture groups should be named in uppercase alphabet characters and the order in the alphabet is the order they appear in the output file name. i.e. ``r'(?P<A>first_bit)'``.

By default, if all the output file names are not unique, then an error is generated (``--outfile_errors 'error'``). This can be set to ``'ignore'`` if this is what is intended, i.e. it could be that the job array has a different step size. Also, this can be set to ``'flatten'`` and in this case the input files are flattened into a pipe ``|`` separated list to give a unique number of output files.

Basic API usage
---------------

This can also be imported as a module. See the reference documentation for the functions within.

.. code-block:: python

   from cluster import make_general_job_array_file
