## Overview of `file_info`
A program (`finfo`) for determining basic information on files. This is a main commandline entry point for running the `file_helper.FileInfo` class. It can write a delimited table of the file attibutes to a file or produce a terminal table to `STDOUT`. The help options are shown below:

```
$ finfo --help
usage: finfo [-h] [-i INDIR] [-g INGLOB] [-o OUTFILE]
             [-c COLUMNS [COLUMNS ...]] [-r REGEXP] [-d DELIMITER]
             [-t OUT_DELIMITER] [-m] [-n] [-v] [-b]
             [infiles [infiles ...]]

A helper script to merge a bunch of files to form asingle file. This acts like
`cat` but is slower and also performs more checks. So the files should be
tabular with all the same delimiter and header. However, there is an option to
supply groups of column names that will act as a proxy for a single column
type. It will also produce a merge log file and there are options to leave the
files being merged intact or to have them removed during merging, either as
the merge is being done (--unsafe) or after the merge has been completed

positional arguments:
  infiles               The input files to extract from, It is expectedthat
                        all input files have a header. The files can
                        becompressed (gzip) or uncompressed (default=None)

optional arguments:
  -h, --help            show this help message and exit
  -i INDIR, --indir INDIR
                        The input directory, this should be set if thereare no
                        input files, it can also be used in conjunction with
                        input files (default=None)
  -g INGLOB, --inglob INGLOB
                        An input directory glob that is used in conjunction
                        with the --indir, the default is no glob pattern
  -o OUTFILE, --outfile OUTFILE
                        The output file, if not provided, output is to STDOUT,
                        default is to STDOUT
  -c COLUMNS [COLUMNS ...], --columns COLUMNS [COLUMNS ...]
                        Specific column names to use from the files (if they
                        all have the same header) or groups of column names
                        that contain the same data, groups should be given in
                        the format: 'input_col1,input_col2,input_colX...' with
                        no sapces and may need quoting. The idea is that the
                        presence of any columns or column groups will be
                        checked in each file. If single column names are
                        given, then they must be in each fil. If groups of
                        columns are set then a single column from the group
                        must be in each file. If no column names are passed
                        (the default) then the files are tested to make sure
                        they have exactly the same header. If there are any
                        violations of these then the 'has_errors' column is
                        yes to 'Y' for the file.
  -r REGEXP, --regexp REGEXP
                        A regular expression used to generate the ID column
                        from the full input file path. The regexp will be
                        applied using re.search. All groups will be captured,
                        although the groups required in the ID should be
                        labeled 'A', 'B', 'C' using the named capure groups
                        `(?P<C>)` etc... . Remember to also use non-capturing
                        groups `(?:)` where necessary. (default=no regexp)
  -d DELIMITER, --delimiter DELIMITER
                        The delimiter of the input files. Only use if all
                        input files have the same delimiter. If not, the
                        delimiter is sniffed using csv.Sniffer (the default)
  -t OUT_DELIMITER, --out-delimiter OUT_DELIMITER
                        The delimiter of the output file (or STDOUT).if it is
                        set to - (hyphen) then a fixed width terminal table
                        will be output instead of a delimited file (default='
                        ')
  -m, --md5             Take an MD5 of each input file (default=False)
  -n, --ncols           calculate the min/max number of columns for each file
                        and also the number of rows (including the header)
                        (default=False)
  -v, --verbose         Shall I print progress to STDERR? (default=False)
  -b, --basename        Shall I output the file file basename instead of the
                        full filename. Can be useful if the file names are
                        very long and you are outputting to
                        STDOUT(default=Full file path)
```

### Checks and Output
The default checks and information that is gathered by `finfo` is the:

* Full path of the file
* Does the file exist?
* Is the file gzipped?
* The delimiter of the file
* Does the file have a header
* What is the header length
* If specific columns were supplied, are they present, if not do all the headers match?

If any of these fails, then `finfo` is designed not to error out (but that does not mean that it won't), rather it will indicate if errors have been found and the errors should be evident from the values in the report. 

The optional flags update the report with additonal data:

* The minimum number of columns in the file (`--ncols`)
* The maximum number of columns in the file (`--ncols`)
* The number of rows in the file (`--ncols`)
* A `regexp_id` - a string that is derived from the input file name (`--regexp '<REGEXP>'`)

### The report
By default `finfo` will output a tab (`'\t'`) delimited report. The delimiter can be changed with `-t` or `--out-delimiter` options. If set to `-` it will output a terminal table report (as shown in the example below). The report is output to `STDOUT` inless a file name is supplied to `-o` or `--outfile`. Amongst the columns in the report is the `has_errors` column. This will be `Y` (`yes` or `True`) under the following conditions:

* The file does not exist
* Any column that has been supplied (or group of columns) does not exist in the file (or more than 1 column from a group of columns exists).
* If no column have been supplied and the header of the file does not match the header of the previous file
* if `--ncols` is active and `min_col` does not equal `max_col`
* if `--regexp` is active and no `regexp_id` could be derived from the filename using the `--regexp`

The full list of columns in the report is shown below:

1. **`filename`** - The full path to the file or the basename of the file if `-b` or `--basename` is active
2. **`file_exists`** - Can take the values `Y`/`N` and indicates if the file exists and can be openned. It will be `N` if the file does not exist or you do not have permissions to open it
3. **`is_gzip`** - Is the file `gzipped`? Can take the values `Y`/`N`. This is determined by looking at the first few bytes of the file. Note, I am not 100% sure if this is accurate with `bgzipped` files.
4. **`delimiter`** - The delimiter of the file. Note that tabs `'\t'` are converted to the word `TAB` in the report.
5. **`has_header`** - Does the file have a header? Can take the values `Y`/`N`. This is as determined by [`csv.Sniffer().has_header()`](https://docs.python.org/3.7/library/csv.html#csv.Sniffer). I am not 100% sure how good this is. Nevertheless, even if this comes back with `N` (no header) the various header checks are still carried out as if the file does have a header.
6. **`header_len`** - The length (in number of columns) of the first line in the file, this may or may not be the header
7. **`columns_ok`** - Have any columns or groups of columns that have been supplied been found in the header - or does the header of the file match the header of the previous file? Can take the values `Y`/`N`
8. **`nrows`** - The number of rows in the file will be `NA` unless `--ncols` is active.
9. **`min_cols`** - The minimum number of columns in the file will be `NA` unless `--ncols` is active.
10. **`min_cols`** - The maximum number of columns in the file will be `NA` unless `--ncols` is active.
11. **`md5sum`** - The md5sum of the file will be `NA` unless `--ncols` is active.
11. **`regexp_id`** - The identifier extracted when a regular expression is supplied to the full path of the input file. Will be `NA` if the `-r` or `--regexp` argument is not used or if it is and the regular expression can't extract an ID.
11. **`has_errors`** - Have any error been seen when checking the file and extracting information, see the conditions in which this is `Y` above.

In addition to the report, `finfo` can generate some warning messages. These are generated if the `--regexp` is used but the IDs that are created are not unique for all the files supplied. Also, if for any of the files `has_errors` is True a further warning is issues to remind the user to look at the report.

### Examples
Below is example usage of `finfo`:

Output to a terminal table and also requesting MD5 hashes (the `--md5` argument) of the files and calculation of the number of minimum number of columns in the file, maximum number of columns in the file and number of rows (the `--ncols` argument). A regular expression is also supplied for gathering a `regexp_id` for the file.
```
$ finfo -b -v \
        -r'MetaAnalysis_MAF0.001_Rsq0.3_INT_(?P<A>.+)_ALL_CHR_imputed_unadj_METAL_top_hits.txt.gz' \
		--ncol --md5 --out-delimiter - \
		~/Scratch/ucleb_kett_alspac_metabolomics/MetaAnalysis_MAF0.001_Rsq0.3_INT_A*
```

Output:
```
=== finfo (pyaddons v0.1.0a4) ===
[info] basename value: True
[info] columns value: None
[info] delimiter value: None
[info] indir value: None
[info] infiles length: 7
[info] inglob value: None
[info] md5 value: True
[info] ncols value: True
[info] out_delimiter value: -
[info] outfile value: None
[info] regexp value: re.compile('MetaAnalysis_MAF0.001_Rsq0.3_INT_(?P<A>.+)_ALL_CHR_imputed_unadj_METAL_top_hits.txt.gz')
[info] verbose value: True
[start] 2020-01-30 08:05:04
[finish] 2020-01-30 08:05:41 [runtime] 0:00:36 
+-----------------------------------------------------------------------------------------+-------------+---------+-----------+------------+------------+------------+--------+----------+----------+----------------------------------+------------+------------+
| filename                                                                                | file_exists | is_gzip | delimiter | has_header | header_len | columns_ok | nrows  | min_cols | max_cols | md5sum                           | regexp_id  | has_errors |
+-----------------------------------------------------------------------------------------+-------------+---------+-----------+------------+------------+------------+--------+----------+----------+----------------------------------+------------+------------+
| MetaAnalysis_MAF0.001_Rsq0.3_INT_ACACE_ALL_CHR_imputed_unadj_METAL_top_hits.txt.gz      | Y           | Y       | TAB       | Y          | 23         | Y          | 657559 | 23       | 23       | 157253841d2c66f57842d6943314d5e1 | ACACE      | N          |
| MetaAnalysis_MAF0.001_Rsq0.3_INT_ACE_ALL_CHR_imputed_unadj_METAL_top_hits.txt.gz        | Y           | Y       | TAB       | Y          | 23         | Y          | 673437 | 23       | 23       | fc5bb4000e42bd00797714ec62159d59 | ACE        | N          |
| MetaAnalysis_MAF0.001_Rsq0.3_INT_ALA_ALL_CHR_imputed_unadj_METAL_top_hits.txt.gz        | Y           | Y       | TAB       | Y          | 23         | Y          | 687742 | 23       | 23       | 9845d9d980531b448f528dcf307d49f2 | ALA        | N          |
| MetaAnalysis_MAF0.001_Rsq0.3_INT_ALB_ALL_CHR_imputed_unadj_METAL_top_hits.txt.gz        | Y           | Y       | TAB       | Y          | 23         | Y          | 678174 | 23       | 23       | 09129ae961a810127e12f86c87348ddb | ALB        | N          |
| MetaAnalysis_MAF0.001_Rsq0.3_INT_APOA1_ALL_CHR_imputed_unadj_METAL_top_hits.txt.gz      | Y           | Y       | TAB       | Y          | 23         | Y          | 741956 | 23       | 23       | ff9a348d9ad66955da5af972db26892a | APOA1      | N          |
| MetaAnalysis_MAF0.001_Rsq0.3_INT_APOB_ALL_CHR_imputed_unadj_METAL_top_hits.txt.gz       | Y           | Y       | TAB       | Y          | 23         | Y          | 738716 | 23       | 23       | 6a19b6eaedbf6fec5b82fe519e3f5887 | APOB       | N          |
| MetaAnalysis_MAF0.001_Rsq0.3_INT_APOB_APOA1_ALL_CHR_imputed_unadj_METAL_top_hits.txt.gz | Y           | Y       | TAB       | Y          | 23         | Y          | 683475 | 23       | 23       | ed70c571ef5cf5726ddb11ebab4e2d04 | APOB_APOA1 | N          |
+-----------------------------------------------------------------------------------------+-------------+---------+-----------+------------+------------+------------+--------+----------+----------+----------------------------------+------------+------------+

```
