# Overview of `file_index`
This module contains both command line and API tools for creating indexes of tabular files based on the data in a single column.

## Command line usage
Create an index with:

```
$ make_file_index --help
usage: make_file_index [-h] [-i INDEX_FILE] [-c INDEX_COL] [-v] [-l]
                       [-s SKIP_LINES] [-k KEY_SIZE] [-d DELIMITER]
                       [-e ENCODING] [-t TMP_DIR] [-b SNIFF_BYTES]
                       file_path outfile

build a compressed indexed file for random access. The file must be presorted
on the index column

positional arguments:
  file_path             The path to the file file, input from STDIN use - for
                        input file
  outfile               An output indexed file path. The output file will be
                        bgzipped, as part of the indexing process

optional arguments:
  -h, --help            show this help message and exit
  -i INDEX_FILE, --index-file INDEX_FILE
                        An alternative path to the index file name. By default
                        the index file will be the same name as the outfile
                        but with an additional .idx extension
  -c INDEX_COL, --index-col INDEX_COL
                        The column number (1-based) that should be indexed.
                        Note that the file should be pre-sorted on this column
                        as well (default=1)
  -v, --verbose         Give more output
  -l, --clobber         Delete any existing indexes
  -s SKIP_LINES, --skip-lines SKIP_LINES
                        number of header lines to skip (default=1)
  -k KEY_SIZE, --key-size KEY_SIZE
                        The max number of characters a key entry can be. Index
                        values above this length will generate errors,
                        however, larger key-size will generate bigger index
                        files (default=2000)
  -d DELIMITER, --delimiter DELIMITER
                        the file delimiter (default=TAB)
  -e ENCODING, --encoding ENCODING
                        the file encoding (default=utf-8)
  -t TMP_DIR, --tmp-dir TMP_DIR
                        an alternative tmp-dir location (default=system tmp)
  -b SNIFF_BYTES, --sniff-bytes SNIFF_BYTES
                        number of bytes at the start of the file to sniff to
                        detect the dialect of the file. If auto detection
                        fails then increase this value, if it still fails use
                        the API to index the file (default=2000000 - (2MB))
```

An example with input from STDIN
```
cat gp_scripts.txt | body sort -t$'\t' -k1,1 | make_file_index -e'latin1' -k7 -s1 -v -c1 - gp_prescriptions.txt.gz
```

Getting rows from the indexed file
```
$ fetch --help
usage: fetch [-h] [-i INDEX_FILE] [-v] [-d DELIMITER] [-e ENCODING]
             [-b SNIFF_BYTES]
             infile values [values ...]

fetch rows matching index values from an indexed file

positional arguments:
  infile                The path to the indexed file
  values                index values to extract

optional arguments:
  -h, --help            show this help message and exit
  -i INDEX_FILE, --index-file INDEX_FILE
                        An alternative path to the index file name. By default
                        the index file will be the same name as the outfile
                        but with an additional .idx extension
  -v, --verbose         Give more output
  -d DELIMITER, --delimiter DELIMITER
                        the file delimiter (default=detect)
  -e ENCODING, --encoding ENCODING
                        the file encoding (default=utf-8)
  -b SNIFF_BYTES, --sniff-bytes SNIFF_BYTES
                        number of bytes at the start of the file to sniff to
                        detect the dialect of the file. If auto detection
                        fails then increase this value, if it still fails use
                        the API to index the file (default=2000000 - (2MB))
```

An example
```

```
