---
header-includes: #allows you to add in your own Latex packages
- \usepackage{float} #use the 'float' package
- \floatplacement{figure}{H} #make every figure with caption = h

output:
  pdf_document:
    fig_caption: true
    keep_tex: true
    latex_engine: pdflatex
  html_document:
    df_print: paged
urlcolor: blue

graphics: yes
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
knitr::opts_chunk$set(cache = FALSE)
knitr::opts_chunk$set(warning = FALSE)
knitr::opts_chunk$set(out.extra = '')
#knitr::opts_chunk$set(fig.pos = 'H')
knitr::opts_knit$set(root.dir = "../")
```

## dada2 filtering reads

The first stage of the dada2 pipeline is filtering and trimming of reads. The number of reads
that remain for downstream analysis is dependent on the parameters that were set for filtering
and trimming. In most cases it would be expected that the vast majority of reads will remain
after this step. It is noteworthy that dada2 does not accept any "N" bases and so will remove
reads if there is an N in the sequence.

```{r include=FALSE, echo=FALSE}
library(ggplot2)
library(plotly)
library(gridExtra)
library(dplyr)
library(reshape)
library(gplots)
library(data.table)
```

### Number of reads input/output during filtering and trimming

Below is a summary of the number of input reads and the number of output reads for each sample.

```{r filtering, fig.cap='Number of input and output reads during filtering step', fig.height=4, fig.width=10,  echo=FALSE, message=FALSE}

#########################
# read relevant infiles
#########################

infiles = list.files(path="filtered.dir", pattern = "*summary.tsv")

###########################################
# create dataframe from all summary tables
###########################################

dfs <- list()
names.list <- c()
for (i in 1:length(infiles)){
    infile <- paste("filtered.dir", infiles[i], sep="/")
    df <- read.csv(infile, header=T, stringsAsFactors=F, sep="\t")
    dfs[[i]] <- df
    names.list <- append(basename(infile), names.list)
}
names(dfs) <- names.list
df <- bind_rows(dfs)

################################################
# define reads.in as the difference between the
# starting number and the finishing number. This
# enables visualisation in a stacked bar chart
################################################

df$reads.in <- df$reads.in - df$reads.out

# reshape and plot
df.m <- melt(df)

p1 <- ggplot(df.m, aes(x=reorder(sample, as.numeric(sample)), y=value, fill=variable))
p2 <- p1 + geom_bar(stat="identity")
p3 <- p2 + scale_fill_manual(values=c("red4", "blue4"))
p4 <- p3 + theme_bw() + theme(axis.text.x=element_text(angle=90)) + xlab("sample") + ylab("Number of reads")
ggplotly(p=p4)

```

## Learning the error model

Dada2 performs a step where it learns the sequencing error model. Taken from the tutorial:

The DADA2 algorithm makes use of a parametric error model (err) and every amplicon dataset has a different set of error rates.
The learnErrors method learns this error model from the data, by alternating estimation of the error rates and inference of sample
composition until they converge on a jointly consistent solution. As in many machine-learning problems, the algorithm must begin
with an initial guess, for which the maximum possible error rates in this data are used (the error rates if only the most abundant
sequence is correct and all the rest are errors).

In contrast to the dada2 tutorial [here](https://benjjneb.github.io/dada2/tutorial.html), and for the purposes of parallelisation,
we learn the error model using each sample separately. It should be noted that this may not be ideal in all situtations but it does
speed up data processing. This also means that we produce a plot for each sample separately. It is not feasible to display them
all here and so we just inspect one for the report purposes but all others are available where the piepline was run. Note that
at the moment this is restricted to the forward reads (so there is no error thrown when using single end data).


![Error model](errF.png)

## De-replication, sample inference, merging and chimera removal

The next stage of the dada2 pipeline involves dereplication, sample inference, merging (if paired-end) and chimera removal.
Again from the tutorial, dereplication combines all identical sequencing reads into into “unique sequences” with a corresponding
“abundance” equal to the number of reads with that unique sequence. These are then taken forward into the sample inference stage
and chimera removal. It is useful to see after this has been done how many sequences we are left with. The majority of reads
should contribute to the final overall counts.


```{r sample-inference, fig.cap='Number of input and output reads during filtering step', fig.height=8, fig.width=8,  echo=FALSE, message=FALSE}

#########################
# read relevant infiles
#########################

infiles = list.files(path="abundance.dir", pattern = "*summary.tsv")

###########################################
# create dataframe from all summary tables
###########################################

dfs <- list()
names.list <- c()
for (i in 1:length(infiles)){
    infile <- paste("abundance.dir", infiles[i], sep="/")
    df <- read.csv(infile, header=T, stringsAsFactors=F, sep="\t")
    dfs[[i]] <- df
    names.list <- append(basename(infile), names.list)
}
names(dfs) <- names.list
df <- bind_rows(dfs)

# reshape and plot
df.m <- melt(df)

p1 <- ggplot(df.m, aes(x=reorder(sample, as.numeric(sample)), y=value))
p2 <- p1 + geom_bar(stat="identity")
p3 <- p2 + theme_bw() + theme(axis.text.x=element_text(angle=90)) + xlab("sample") + ylab("Number of reads")
p4 <- p3 + facet_wrap(~variable)
ggplotly(p=p4)

```

## Number of ASVs called per sample and their prevalence

A useful metric is the number of ASVs that were called per sample even though we may not no beforehand the expected diversity in the samples we are analysing. In addition to simply counting the number of ASVs per sample we also plot the prevalence of these ASVs i.e. the proportion of samples that each ASV is observed in. By plotting the prevalence against the average relative abundance we get an idea of the presence of suprious ASVs i.e. low prevalence and low abundance.

```{r nasvs, fig.height=4, fig.width=10,  echo=FALSE, message=FALSE}

# read ASV counts and reformat
dat <- data.frame(fread("abundance.dir/taxa_abundances.tsv", header=TRUE, stringsAsFactors=F, sep="\t"))
rownames(dat) <- dat$featureID
dat <- dat[,2:ncol(dat)]

nasvs <- data.frame(colSums(dat > 0))
nasvs$sample <- factor(rownames(nasvs), levels=unique(rownames(nasvs)))
nasvs.m <- melt(nasvs)

# number of ASVs
p1 <- ggplot(nasvs.m, aes(x=reorder(sample, as.numeric(sample)), y=value))
p2 <- p1 + geom_bar(stat="identity")
p3 <- p2 + theme_bw()
p4 <- p3 + theme(axis.text.x=element_text(angle=90))
p5 <- p4 + ylab("number of ASVs")
ggplotly(p5)
```

```{r prevalence, fig.height=4, fig.width=10, echo=FALSE, message=FALSE}

# Functions lifted from AmpSeqKit - this will be a package in the future

###############
###############
###############

# prevalence
pprev <- function(counts){

    nsamples <- ncol(counts)
    prev <- (rowSums(counts > 0)/nsamples)*100
    prev.df <- data.frame(Taxon=rownames(counts), Prevalence=prev)
    return(prev.df)
    }

###############
###############
###############

# plotting
plotPrev <- function(prev.df, as.is=TRUE){

    if (as.is==TRUE){
        prev.df <- prev.df[order(prev.df$Prevalence, decreasing=TRUE),]
        p1 <- ggplot(prev.df, aes(x=Taxon, y=Prevalence))
        p2 <- p1 + geom_bar(stat="identity")
        p3 <- p2 + theme_bw()
        p4 <- p3 + theme(axis.text.x=element_text(angle=90))
    }else{
        p1 <- ggplot(prev.df, aes(x=Prevalence))
        p2 <- p1 + geom_histogram()
        p3 <- p2 + theme_bw()
        p4 <- p3
    }
    return(p4)
}

# calculate and plot histogram of prevalence
prev <- pprev(dat)
p6 <- plotPrev(prev, as.is=FALSE)

# relative abundance
relab <- function(counts){

    relab <- (sweep(counts, 2, colSums(counts), "/"))*100
    return(relab)
    }

# plot abundance vs. prevalence
ave.abundance <- rowMeans(relab(dat))
prev$abundance <- ave.abundance

p7 <- ggplot(prev, aes(x=abundance, y=Prevalence))
p8 <- p7 + geom_point()
p9 <- p8 + theme_bw()
p10 <- p9 + xlab("Mean relative abundance") + ylab("Prevalence (%)")

subplot(ggplotly(p=p6), ggplotly(p=p10), nrows=1)

```

## Taxonomic assignment

The next stage is to assign each of the amplicon sequence variants (ASV) to a taxonomic group. Below are plots of the taxonomic assignments for each sample (relative abundance at the phylum level) as well as the proportion of all ASVs that could be assigned at each taxonomic rank (phylum-species). We would expect (in most cases) that the majority of ASVs woild be assigned at high taxonomic ranks (e.g. phylum) and fewer at lower taxonomic ranks (e.g. species).


```{r taxonomy, fig.height=4, fig.width=10,  echo=FALSE, message=FALSE}

# taxonomic distributions
# phylum level only for visualisation purposes

p <- gsub(";c__.*", "", rownames(dat))
p <- gsub("ASV[0-9]*:k__[a-zA-Z]+;", "", p)
dp <- melt(aggregate(dat, list(p), sum))

# again lifted from AmpSeqKit.
plotBar <- function(x){
	p1 <- ggplot(x, aes(x=variable, y=value, fill=Group.1))
	p2 <- p1 + geom_bar(position="fill", stat="identity")
	p3 <- p2 + scale_y_continuous(labels = scales::percent)
	p4 <- p3 + theme_bw()
	p5 <- p4 + theme(axis.text.x=element_text(angle=90)) + ylab("% of reads")
	return(p5)
	}

# plot phylum level distributions
p11 <- plotBar(dp)
ggplotly(p11)
```

```{r nreads.in.taxa, fig.height=4, fig.width=5, echo=FALSE, message=FALSE}

# Plot number of asvs assigned at each level

taxonomy.abundances <- list.files(path="taxonomy_abundances.dir", pattern="*.tsv")
nasvs.assigned <- list()

for (i in 1:length(taxonomy.abundances)){
    dir <- "taxonomy_abundances.dir"
    inf <- taxonomy.abundances[i]
    tfile <- paste0(dir, "/", inf)
    abundances <- read.csv(tfile, header=T, stringsAsFactors=F, sep="\t", quote="", row.names=1)
    # get the first letter of file which is level
    level <- substr(inf, 1, 1)
    level.na <- paste0(level, "__", "NA")
    nhit <- nrow(abundances[grep(level.na, rownames(abundances), invert=TRUE),])
    n <- nrow(abundances)
    level <- unlist(strsplit(inf, "_"))[1]
    nassigned <- data.frame(percent.assigned=(nhit/n)*100, level=level)
    nasvs.assigned[[i]] <- nassigned
}
nasvs.assigned <- bind_rows(nasvs.assigned)
nasvs.assigned$level <- factor(nasvs.assigned$level, levels=c("phylum", "class", "order", "family", "genus", "species"))
p12 <- ggplot(nasvs.assigned, aes(x=level, y=percent.assigned, fill=level))
p13 <- p12 + geom_bar(stat="identity")
p14 <- p13 + theme_bw() 
p15 <- p14 + ylab("% assigned") + xlab("Taxonomic levels")
p16 <- p15 + scale_fill_manual(values=c("red3", "blue3", "green3", "orange3", "purple3", "grey", "black"))
ggplotly(p16)
```


