# DREEM Vectoring Module
Contributor: Matty Allan, Yves Martin

## Purpose
- Convert mapped reads into mutation vectors
- Filter out the reads with abnormaly high mutation rate (3 sigma interval)

## Interface

### Command-line usage
```dreem-vectoring [options] -o path/to/out_dir -r {reference}.fasta -a {alignment_1}.bam [{alignment_2}.bam ... {alignment_n}.bam]```

#### Positional arguments
- [=1] ```-o / --out_dir```: output directory.
- [=1] ```-r / --ref```: Input fasta file path.
- [≥1] ```-a / --alignment```: Binary alignment map (BAM) file(s) generated by ```alignment```.

#### Optional arguments
- [≥0] ```-c / --coords [ref | first | last]```: First and last coordinates for reference ref (creates one mutational profile); using 0 for last is shorthand for the last coordinate of the sequnce, -1 for the second-to-last, and so on.
- [≥0] ```-p / --primers [ref | fwd | rev]```: Forward and reverse primers for reference ref (creates one mutational profile)
- [≤1] ```-L / --library {library}.csv```: A CSV file specifying the coordinates and/or primers.

#### Boolean Flags
- [≤1] ```-f / --fill```: For every reference in ```reference.fasta``` that was not explicitly specified using a ```-c``` or ```-p``` option, create a mutational profile for the entire sequence. Note: if none of ```-c```, ```-p``` or ```--fill``` are given, then no mutational profiles will be generated.
- [≤1] ```-P / --parallel```: Parallelize the processing of mutational ```profiles``` or process each profile in series and parallelize processing ```reads``` within each profile, turn all paralleization ```off```, or (default) ```auto```matically choose "reads" if processing 1 profile, otherwise "profiles".
- [≤1] ```-n / --num_vectors```: Limit the number of vectors in each batch to ```n``` (positive integer). Default is no limit. If ```b``` is also given, the batch will be capped by whichever limit is reached first.
- [≤1] ```-b / --num_bytes```: Limit the number of bytes in each batch to ```b``` (positive integer). Default is no limit. If ```n``` is also given, the batch will be capped by whichever limit is reached first.

### Input Files
- [=1] ```path/to/{reference}.fasta``` Sequence record file that contains the names and sequences of all references.
- [≥1] ```path/to/{alignment}.bam``` Binary alignment map (BAM) file(s) generated by ```alignment```. Multiple files can be specified, and they do need to be in the same directory. Note that for every ```{alignment_i}.bam``` file, a BAM index file with the same name (i.e. ```{alignment_i}.bam.bai```) must be present in the same directory; this file is generated automatically near the end of ```alignment```. If the BAM index file is deleted or corrupted, it can be regenerated via the command ```samtools index {alignment}.bam```.

### Output Files
- [≥1] ```{out_dir}/{ref}/{first}-{last}/{sample}_report.txt``` Report in plain text format containing information about the vectoring of sample ```sample``` aligned to reference ```ref``` over the region from ```first``` to ```last```.
- [≥1] ```{out_dir}/{ref}/{first}-{last}/{sample}/vectors_{num}.orc``` Mutation vectors in Apache ORC format. Each row is one mutation vector; each column is one position in the region from ```first``` to ```last```, inclusive (thus there are ```last - first + 1``` columns).

## Mutation vector format

### Encoding scheme for mutation vectors

Each element in a mutation vector indicates the possible mutation states at that position in the region. Each element is one byte that is the logical union of all possible states for that position in the mutation vector:
  - ```00000001```: match
  - ```00000010```: deletion
  - ```00000100```: insertion of ≥1 base(s) immediately 3' of this position
  - ```00001000```: insertion of ≥1 base(s) immediately 5' of this position
  - ```00010000```: substitution to ```A```
  - ```00100000```: substitution to ```C```
  - ```01000000```: substitution to ```G```
  - ```10000000```: substitution to ```T```

Additionally, ```00000000``` indicates that the mutation vector does not cover the position. For example, ```00000011``` means that the element of the mutation vector could be a match or deletion, and ```00100001``` means a match or a substitution to ```C```. Note several important differences between the encoding of insertions and deletions:
- Every base deleted from the read results in one byte with a deletion bit (unless the deletion is ambiguous). Every insertion results in two bytes with an insertion bit: a 3' insertion bit set immediately upstream of the insertion (unless it occurs at the first position in the region) and a 5' insertion bit immediately downstream (unless the insertion occurs the last position in the region).
- A deletion bit indicates that the position is deleted from the read. However, insertion bits indicate that one or more bases are inserted 5' or 3' of the position but do not indicate anything about the position itself. Thus, every position with an insertion bit also has one or more bits indicating the state of the position itself. For example, if the reference is ```GACTA``` and the read is ```GACAATA``` (i.e. ```AA``` is inserted between the ```C``` and the ```T```), then the ```C``` will be ```00000101``` (a match with an insertion on its 3' side) and the ```T``` will be ```00001001``` (a match with an insertion on its 5' side). No information about the insertion (```AA```) is stored.
- For deletions, every base deleted from the read corresponds to one base in the reference, so the deleted bases (and read sequence) can be determined exactly given only the reference sequence and the mutation vector (unless the deletion is ambiguous). However, mutation vector files store only the location of an insertion, but not its length or sequence; not only is this information not needed for DREEM, but also storing it would require adding a special field of arbitrary size to the mutation vector, which would vastly complicate the code. Thus, the inserted bases (and read sequence) cannot be determined given only the reference sequence and the mutation vector.

### Ambiguous positions in mutation vectors

If, at a position in a mutation vector, one of several alternative mutations could have occured that would have generated the same read sequence, then an ambiguous encoding is used for the position. This encoding is the bitwise union of all possible states in which the position could exist. Several situations can cause ambiguous positions:

#### Low-quality base calls
Every base whose phred score is below the quality threshold is masked as an ```N``` before alignment. Since the reference cannot contain ```N```s, Bowtie2 calls each of these bases a substitution (i.e. ```X``` in the CIGAR string). When the vectoring module encodes each substitution, ```N``` positions could be either a match or substitution (since the base identity is unknown), so they are encoded as follows:
- If the reference base is ```A```: ```11100001``` (match or any substitution except A)
- If the reference base is ```C```: ```11010001``` (match or any substitution except C)
- If the reference base is ```G```: ```10110001``` (match or any substitution except G)
- If the reference base is ```T```: ```01110001``` (match or any substitution except T)

#### Deletions
A deletion that occurs amid bases of the same type is ambiguous. For example, if the reference is ```AGCTTG``` and the read is ```AGCTG```, it is clear that one ```T``` has been deleted from the read, but impossible to tell which. The one that is not deleted is a match (i.e. ```T``` = ```T```). Thus both positions 4 and 5 in the reference should be encoded ambiguously as a match or deletion (```00000011```).

If a deletion spans two or more consecutive bases, Bowtie2 will encode them all in the same deletion operation, e.g. mutating ```AGCTTG``` to ```ATTG``` would yield the CIGAR string ```1=2D3=```, corresponding to the read alignment ```AG--TG```. It is possible for one or more of these deletions to be ambiguous, for example ```AGCTTG``` could mutate to ```AGTG``` via either ```AG--TG``` or ```AG-T-G```. In a sequence alignment designed for phylogenetic analysis, where mutations are relatively rare, it is reasonable to assume that deleting two or more consecutive bases simultaneously is much more likely than deleting each separately, favoring the interpretation ```AG--TG```. However, when mutations are deliberately introduced at a relatively high frequency, as in DMS-MaPseq, getting the mutation ```AG--TG``` is not necessarily more likely than ```AG-T-G```. This ambiguity is more obvious if two consecutive homopolymers each contained one deletion, such as ```AAAAACCCCC``` mutating to ```AAAACCCC```. Any of the 5 ```A```s could be deleted, as well as any of the 5 ```C```s, and only one of the 25 possibile combinations of an ```A``` and a ```C``` deletion is the consecutive deletion ```AAAA--CCCC``` (the one that Bowtie2 would likely output).

Additionally, a deletion that spans two or more consecutive bases and is flanked by the same sequence that was deleted is also ambiguous. For example, if the reference is ```CAGCAG``` and the read is ```CAG```, then the deletion could have been any three consecutive nucleotides (```---CAG```, ```C---AG```, ```CA---G```, or ```---CAG```).

A deletion next to any position other than an unambiguous match (```00000001```) is always ambiguous. For example, if the reference is ```AGCTTG``` and the read is ```AATTG```, then there are two possible alignments of the read (positions with a substitution relative to the reference are in lowercase): ```Aa-TTG``` and ```A-aTTG```. It is impossible to determine which base was deleted and which one mutated to an ```A```, so both would be encoded as ```00010010``` (either substitution to ```A``` or deletion). Note that if the base with a substitution happens to mutate to the same base as the one that was deleted, then the mutation will be identified as a deletion only. For example, if ```AGCTTG``` actually mutates by a substitution and a deletion to ```Ac-TTG```, then the read will be ```ACTTG```. The read would appear to only contain a deletion (```A-CTTG```), when in reality that mutation was a substitution and an adjacent base identified as matching was actually deleted. Since only the read can be observed, not the mutations that created it, distinguishing the two cases is impossible. The deletion-only case is assumed by Bowtie2 because it is far more likely (and indeed, if it assumed otherwise, then every deletion would need to be considered to have an adjacent substitution).

A position adjacent to a deletion could be a substitution to base that is already within the deleted region. Although very unlikely, this could happen during alignment in a situation like ```TAGC``` -> ```TG```, if the penalty for opening two gaps (yielding ```T-G-```) exceeds the penalty for substituting one base plus opening one gap plus extending a gap once (yielding ```Tg--``` or ```T--g```); simplified, if ```abs(gap_open_penalty) >= abs(gap_extend_penalty + substitution_penalty)```. Resolving this situation by replacing the deletion with a match and the substitution with a deletion would require overwriting positions in the mutation vector and a more complicated algorithm that would only be needed in these rare cases. A much easier solution is to set the penalties for substitutions, gap openings, and gap extensions equal to each other during the alignment step, which would prevent this situation altogether.

A more common scenario that must be handled is that a position adjacent to a deletion is low quality (```N```), and thus whether the base in the read is a match or a substitution to any base other than the reference base cannot be determined. If the base is a match, then it would have all the properties of matches discussed above, namely it could swap with any position in the deletion with the same reference base. For example, if the reference were ```AGTTCG``` and the read ```AGN--G```, then ```AG-T-G``` would also be a legal permutation; note that the ```N``` has been replaced by a match (```T```) because this swap was performed under the assumption that the low-quality base in the read was really a ```T```. If it were instead a substitution to ```V``` (```01110000```), then it would follow the above rules for substitutions next to deletions, swapping with the ```T``` to yield ```AG-v-G```. It follows that every position in the deletion could also be a match or a substitution to anything but the reference base.

If a deletion is at the end of the region but not of the read (i.e. the read extends past the region of interest), then check the CIGAR operation of the base immediately outside of the deletion. If the operation is a match, then the deletion is kept. If it is a substitution, then the corresponding base in the read is checked, and the deletion is made ambiguous according to the previous description for either substitutions or low-quality bases. If a deletion is at the end of the read (next to a position that is not covered), then it is ambiguous whether the position was deleted from the cDNA or if the corresponding part of the RNA was not reverse transcribed or sequenced. Considering any deletions at the end of the read as deletions could cause a large pileup of deletions at the end. Thus deletions at the end of the read are set to not covered.

The above points about ambiguous deletions can be summarized as follows. A position within a deletion of any length has a legal move (i.e. a move that does not change the read sequence) if and only if swapping it with either position adjacent to the deletion would not create a new mutation. In the simple case of ```CC``` -> ```-C```, swapping the deletion with the base on its 3' side would yield ```C-```, which has the same sequence. However, if the same transformation were applied to ```AC``` -> ```-C```, then the result would be ```A-```, which has a different sequence. The same rule applies to longer deletions, such as ```CTCACAG``` -> ```----CAG```, which could be permuted to ```--C--AG``` and ```C----AG```. Clearly, if the deleted position swaps with a position that matches the reference, then the move is legal if and only if the reference bases are the same at the deleted position and the swapping position. However, if the position adjacent to the deletion is a substitution, then it can swap with any position in the deletion, since this swap would not create a new mutation (it would just exchange the positions of an existing deletion and substitution). For example, ```ACT``` -> ```G--``` could also be ```-G-``` or ```--G```; each contains two deletions and one substitution.

## Algorithm for creating a mutation vector

Here we devise a simple algorithm to compute all of the above normal and edge cases. This algorithm has three stages:
1. Construct a draft mutation vector and list all coordinates of insertions and deletions.
2. Determine all of the ambiguous positions in the mutation vector.
3. If the read is paired-end, then compute the consensus of the mutation vectors of the two mates.

### Stage 1
1. Initialize an empty mutation vector and empty lists of insertion and deletion positions.
2. If the read starts after the beginning of the region, pad the beginning of the mutation vector with missing bytes. If the read starts before, iterate through the CIGAR string until reaching the beginning of the region.
3. For each operation in the CIGAR string that lies within the region of interest:
    1. If the operation is a(n) ...
        - match (=), then append {length} match bytes to the mutation vector.
        - substitution (X), then append a substitution to A/C/G/T byte or low-quality byte to the mutation vector for each substitution.
        - deletion (D), then append {length} deletion bytes to the mutation vector. Also append the current positions in the reference and read sequences to the list of deletion positions.
        - insertion (I), then append the current positions in the reference and read sequences to the list of insertion positions.
        - soft clipping (S), do nothing.
        - Otherwise raise an error.
    2. Advance the positions in the reference and read sequences if the CIGAR operation consumed them.
4. If the read ends before the end of the region, pad the end of the mutation vector with missing bytes.
