# Overview of `grpcc`
This performs a function similar to a group concatinate function in a relational database. i.e. collapse a field or fields. The user can supply one or more keys as column numbers (or optionally column names if the **--header** flag is active). The keys are then made unique and all other fields are collapsed into lists that are delimited by commas (the default, this can be changed with **-g**/**--grp-del**). **grpcc** _should_ be memory efficient under _normal_ conditions as very little data is stored in memory. It first sorts the data, for this **-b**/**--buffer** lines (default 100000) are read into memory, sorted on the keys and then written to a temp file. This process continues until the whole file is processed. All the temp files are then opened and there is a generator function that takes the top line from each in the correct sort order. Now the keys are in the correct order we can store the data in sets until the keys change, when they do the sets are joined and all the data is written to STDOUT. Using sets guarantees that the collapsed fields only contain unique entries. The temp files are deleted if **grpcc** exits cleanly. The idea for the generator and the external sort is from [here](http://stackoverflow.com/questions/14465154/sorting-text-file-by-using-python/16954837#16954837)

There are probably some edge cases where grpcc could end up using a lot of memory. For example, if it has a key column in a large file with only one type of value in it, then this will all be slurped into memory during the data collapsing phase. Having said that this is pretty extreme and not really what **grpcc** was designed for. I could potentially improve the the collapsing algorithm.

## Usage
Command line usage is shown below
```
$ grpcc --help
usage: grpcc [-h] [-d DELIMITER] [-g GRP_DEL] [-q QUOTE] [-e ENCODING]
             [-b BUFFER] [-k KEY] [--header]

Takes tabular STDIN and makes the keys unique and collapses all other values

optional arguments:
  -h, --help            show this help message and exit
  -d DELIMITER, --delimiter DELIMITER
                        The delimiter of STDIN
  -g GRP_DEL, --grp-del GRP_DEL
                        The delimiter of the grouped columns
  -q QUOTE, --quote QUOTE
                        The Quoting character (to protect dlimiting characters
                        that are not real delimiters)
  -e ENCODING, --encoding ENCODING
                        The unicode encoding in the file
  -b BUFFER, --buffer BUFFER
                        The number of lines to sub sort in memory
  -k KEY, --key KEY     The column(s) number to group on, can be column name
                        if --header works similar to unix sort -k1 -k4 will
                        unique on columns 1 and 4. Note this is 1-based
  --header              Does the file have a header
```

## grpcc TODO
Some things that need improving:

+ More testing, for example collapse on field 2 doesn't give the expected sort order, this shouldn't change the program result but I need to understand what is going on (does python sort treat uppercase and lower case differently).

+ Delete all temp files on destry. So under normal operation grpcc will delete temp files at the end of execution but if the script exits early, for example it could be piped into less, looked at and then CTRL-C so grpcc will not get a chance to delete the temp files. So if you do this a lot remember to clean out **/tmp**

+ Fix the proken pipe issue when terminating early for example for example it could be piped into less, looked at and then CTRL-C. This results in a IOError of a broken pipe.

For the two points above these URLs will be useful:

+ http://stackoverflow.com/questions/12686160/clean-up-code-when-ctrlc-is-caught-in-python
+ http://stackoverflow.com/questions/1112343/how-do-i-capture-sigint-in-python
+ https://docs.python.org/2/library/signal.html
+ http://stackoverflow.com/questions/3347775/csv-writer-not-closing-file
