"""
This module is designed to provide functions to analyse qPCR data. 
It is designed for maximal user-friendliness and streamlined data-visualisation.

## Terminology
---

Let's talk some terminology first. There are a number of important terms that you will meet when using the `qpcr` module that you may feel have nothing to do with qPCR. 
Read on to learn what these terms mean and why they are important. 

### `qpcr` vs qPCR
Throughout this documentation, we will refer to the python module as `qpcr` in all lowercase, while we will talk about the qPCR experiment itself 
by using `qPCR` with uppercase. 

### File (or "datafile")
Let's start simple. A `file` is simply one of your input datafiles (shocker). For the `qpcr` module this means either a `csv` or an `excel` file. 
Files are the at the very basis of our data pipeline. 
However, `qpcr` does not strictly require that these files correspond to anything in particular from "qPCR vocabulary" _per se_ 
(although the basic assumption underlying the default settings
is that a single "regular" datafile contains exactly one single qPCR assay). 

The main `qpcr` classes are designed to work with datafiles that contain Ct values and replicate identifiers (see below for "replicates").
In the very most basic case, a simple datafile contains extactly an `"id"` or `"Name"` or whatever column and a `"Ct"` column (can also be named differently),
that store values from exactly _one single qPCR assay_. 

However, if your experimental setup looks differently, there are ways to adapt `qpcr` to handle different setups.
But if your data follows the above mentioned standard arrangements, you can think of a file as identical to a "qPCR assay".

### "Regular" vs "irregular" datafiles
`Regular` datafiles only contain the `id` and `Ct` columns and nothing else, storing values of exactly one qPCR assay. 
`Irregular` datafiles also have to have these two columns, but they allow for more irregular stuff
around these columns. 
Irregular files may contain multiple datasets / assays (see below). Check out the documentation of the `qpcr.Parsers` and `qpcr.Readers` for 
details on how to work with irregular and multi-assay datafiles.

### An `qpcr.Assay` and a "dataset" / "assay"
Let's start by the term "dataset". Well, a "dataset" is simply a collection or replicate identifiers and corresponding Ct values belonging together. 
By default this usually corresponds to a single qPCR assay, hence you will often encounter the term "assay" used instead of "dataset" to be more 
intuitive. However, the two terms are really interchangeable. The `qpcr.Assay` class is used to store the Ct values and replicate identifiers 
from a single dataset extracted from a datafile. 

### Replicates
As far as the `qpcr` module is concerned, each row within your datasets corresponds to one replicate. 
Hence, a _replicate_ is just a single pair of some identifier and a corresponding Ct value. 
That means that `qpcr` does not terminologically distinguish between multiplets (i.e. "true replicates" as an experimentor would understand them) and unicates. 
If you feel more comfortable with a term like "measurement" then you can also think of the replicates like that. 

### Groups (of Replicates)
This is one of the most important terms. In the previous paragraphs we already started to talk about replicates "belonging together". 
Groups of replicates are, as their name already implies, well, a number of replicates that somehow do belong together.
For most cases, if your datafiles contain one assay each, then the groups of replicates are most likely your different qPCR samples / experimental conditions. 
We talk about _groups_ of replicates instead of samples or conditions, primarily, because there might be different data setups so that these terms might not be always appropriate.
If your data follows default arrangements, however, then a _group of replicates_ is just what you would think of as a "qPCR sample". 
Groups are assigned a numeric index starting from 0, which is how they are identified by the classes. 
However, they also come with a text label called the `group_name` (you can manually set and re-set the group names as you like). 
Many classes such as the `qpcr.DataReader` will actually just use the term `names` instead of the full `group_names`. 
Whenever you see anything "names"-related it is (super-duper most likely) a reference to the `group_names`.

### Specifying (Groups of) Replicates
Here's the best part: usually, we don't necessarily need to do anything because `qpcr.Assay` objects are able to infer the groups of replicates in your data 
automatically from the replicate identifiers (yeah!). However, you will be asked to manually provide replicate settings in case this fails. 
In case you want to / have to manually specify replicate settings, an `qpcr.Assay` accepts an input `replicates` which is where you can specify this information. 

`replicates` can be either an `integer`, a `tuple`, or a `string`. Why's that? Well, normally we perform experiments as "triplicates", or "duplicates", or whatever multiplets.
Hence, if we always have the same number of replicates in each group (say all triplicates) we can simply specify this number as `replicates = 3`. 
However, some samples might only be done in unicates (such as the diluent sample), while others are triplicates.

In these cases your dataset does not have uniformly sized groups of replicates and a single number will not do to describe the groups of replicates. 
For these cases you can specify the number of replicates in each group separately as a `tuple` such as `replicates = (3,3,3,3,1)` or as a `string` "formula"
which allows you to avoid repeating the same number of replicates many times like `replicates = "3:4,1"`, which will translate into the same tuple as we specified manually. 
Check out the documentation of `qpcr.Assay.replicates` for more information on this. 

### `Delta-Ct` vs `Delta-Delta-Ct` vs `normalisation`
The default analysis workflow in $\Delta \Delta Ct$ analysis is to first calculate a $\Delta Ct$ using an intra-assay reference and then calculate the $\Delta \Delta Ct$ using a normaliser assay. 
The first $\Delta Ct$ step is performed by a class called `qpcr.Analyser` using its native method `qpcr.Analyser.DeltaCt`. 
Why just `DeltaCt` and not `DeltaDeltaCt`? Well, we call this second `Delta`-step in `DeltaDeltaCt` differently. 
We name it `normalisation`, and it is handled by a class called `qpcr.Normaliser` using its native method `qpcr.Normaliser.normalise`. 
So, as far as the `qpcr` module is concerned there is only `qpcr.Analyser.DeltaCt` which performs the first $\Delta Ct$, and `qpcr.Normaliser.normalise` which later handles the second "delta"-step to get to $\Delta \Delta Ct$. 
Of course, this means that the `qpcr.Normaliser` will need to have knowledge about which `qpcr.Assay` objects contain actual assays-of-interest and which ones contain normaliser-assays (specifying that is easy, though, so don't worry about that).


### The `anchor` and the "reference group"
Next to the "groups of replicates", this is probably one of the most important terms. The `anchor` is simply the intra-dataset reference used by the `qpcr.Analyser` to perform its first $\Delta Ct$. 
If your datafiles contain one assay each, and your groups of replicates are your qPCR samples, then you will likely have some "wildtype", "untreated", or "control" sample, right? 
Well, in `qpcr` terms that would be your _reference group_.
Usually your `anchor` is part of or generated from the Ct values of your _reference group_ (like their `mean` for instance).
By default it is assumed that your reference group is the _very first_ group of replicates. However, it's not a big problem if this is not the case, as you can specify different anchors easily.
So, again, the `anchor` is the dataset-internal reference value used for the first $\Delta Ct$.

### "assays" vs "normalisers"
You will likely encounter methods and/or arguments that speak of "assays" and "normalisers", especially with the `qpcr.Normaliser`. 
For all intents and purposes, an "assay" is simply one of your datasets (we know this already).
However, in practice "assays" are the short notation for specifically "assays-of-interest" 
(or more formally "datasets-of-interest"), while "normalisers" refer to your normaliser-assays (from housekeeping genes like ActinB for instance). 
But again, if your datafiles do not conform to standard data arrangements, do not be distracted from the terminology here.

You will also find that the term "assays" is used within the final results dataframe (when using the summary-statistics mode). 
In this setting "assays" refers to the assay-of-interst whose data was analysed according to the provided normaliser-assays. 
In fact, this is a new "hybrid" assay identifier taht includes the names of all the normaliser-assays used during computation (check out what the final results look like and it'll be immediately clear).

### "Samples"
You may find that there is also a term "sample" within `qpcr`'s vocabulary. 
As far as the `qpcr` module is concerned, the term "sample" is not very important in itself and usually appears in the context of "sample assays".
In this setting it is used interchangeably with "assays-of-interest". 
Actually, we try to phase out the term "sample" and it currently mainly appears in hidden auxiliary functions which have retained the term from earlier development versions.


## Some more basics 
----

#### The values stored by `qpcr`
Please, note at this point that, as described in more detail in the documentation of the `qpcr.Analyser`, Delta-Ct values are directly computed as 
exponential values $ \Delta Ct' = 2^{ - \Delta Ct}$, while normalisation later performs $ \mathrm{norm. } \Delta\Delta Ct = \\frac{  \Delta Ct'_s  }{  \Delta Ct'_n  }$, where $s$ is an assay of interest's Delta-Ct ($\Delta Ct'$) value of some replicate, 
and $n$ is the corresponding value of the normaliser assay. This is based on the mathemathical equivalence of $n^{  a - b  } \equiv \\frac{  n^{ a } } {  n^{ b } }$. 
Hence, while the documentation will continuously use the terms $\Delta Ct$ and $\Delta\Delta Ct$, they are in fact the exponential deriviative of the conventional values.

#### Modes of `normalisation`
The `qpcr.Normaliser` supports custom functions for normalisation. However, it also comes with three built-in methods to normalise sample-assays against normalisers.
These are accessible via the `mode` argument of the `qpcr.Normaliser.normalise` method, which can be set to `"pair-wise"` (default), `"combinatoric"`, or `"permutative"`. 
The default option `"pair-wise"` is computationally the fastest and will rigidly normalise replicates against their corresponding partner from the normaliser. I.e. first against first, 
second against second, etc. This mode is appropriate for multiplex qPCR experiments. For qPCR reactions that were pipetted individually, there is not reason to strictly only pair first
with first, second with second etc. For these cases there are other two options `"combinatoric"` and `"permutative"`. `"combinatoric"` normalisation will calculate all possible group-wise 
combinations of a sample-assay replicate with all available normaliser replicates of the same group. I.e. first against first, and against second, etc. This will generate $n^2$ values where 
$n$ is the number of replicates within a group. This mode is appropriate for small-scale datasets but will substantially increase computation times for larger datasets.   
`"permutative"` on the other hand will reflect the equivalence of replicates within a group through random permutations wihtin the normaliser replicates. Hence, 
first may be normalised against first, or second, etc. This normalisation method may by used iteratively to increase the accuracy. Replacement during permutations are allowed (although disabled by default).
If replacement is desired by the user, the probability of each replicate to be chosen will be weighted based on a fitted normal distribution. This method is appropriate for larger datasets for which combinatoric normalisation
is not desired. 

### `pipeline`s 
A `pipeline` is essentially any workflow that starts from one or multiple input datafiles and ultimately pops out some results table you are happy with.
Pipelines can be manually created by assembling the main `qpcr` classes, usually starting with a Reader, passing to an Analyser, to an Normaliser, and you're good to go.
When manually assembling your workflow you can extract your data at any point and perform your own computations on it as you like. However, if you wish to "just do some good ol' Delta-Delta-Ct"
there are pre-defined pipelines that will handle writing the workflow and only require a very basic setup. You can find these in the `qpcr.Pipes` submodule.

### "Results" = my final results?
Yes and no. Anything that is computed through any of the `qpcr` classes is called a "result" of some kind.
In practice as soon as you pass your data through a `qpcr.Normaliser` you generate "results" which are actually stored in a separate class called `qpcr.Results` (that's not so important, though).
So, when using the term "result" we usually mean just anything that was explicitly computed by `qpcr`. 

### `get`ting your data
Too many classes and objects? Well, no worries, the underlying data is stored as `pandas DataFrames`. To get your data from the clutches of the `qpcr` classes you can always use the `get()` method. 
`get` is pretty universal in the `qpcr` module, so whenever you want to extract your data, there's a `get()` method to help you.

### `link` vs `add` vs `pipe`
Different classes have slightly different methods of adding data to them. Classes that only accept one single data input (such as a single `qpcr.Assay` object or a single filepath as `string`)
usually have a `link()` method that, well, links the data to them. After that the classes are ready to perform whatever actions they can perform (an `qpcr.Analyser` would perform `DeltaCt()` for instance). 
Some classes such as the `qpcr.Analyser` have a wrapper that will call both their `link()` as well as their actual core-functional method together in one go. This wrapper is called `pipe()`. 
So for the `qpcr.Analyser` you could either manually use `link()` and then `DeltaCt()`, or simply call `pipe()` which does both for you. It is noteworthy that `pipe` methods actually _return_ whatever
their output is, which is *not* normally the case otherwise (normally you'd use the `get()` method to extract your data, see above). Most `qpcr.Readers` and `qpcr.Parsers` are also equipped with `pipe` methods.
Alright, we know about `link` and `pipe` now, what about `add`?  Classes that accept multiple inputs have `add` methods, which tells the class where exactly to store the input data. 
`add`-methods are especially implemented within the pre-defined analysis pipelines of the `qpcr.Pipes` submodule. You will probably often use the methods `add_assays()` and `add_normalisers()` if you plan on using these predefined pipelines.
However, these classes usually still have a `link()` method somewhere that you can use as well. For instance, the `qpcr.Normaliser` uses a `link` method to add both assays-of-interest and normaliser-assays simultaneously.

## Getting started
----
You can find useful tutorials and applied examples as `jupyter notebooks` [on GitHub](https://github.com/NoahHenrikKleinschmidt/qpcr/tree/main/Examples).

"""

import pandas as pd
import qpcr._auxiliary as aux
from qpcr._auxiliary import warnings as aw
import qpcr._auxiliary.defaults as defaults
import qpcr.Parsers as Parsers
import qpcr.Readers as Readers
import qpcr.Plotters as Plotters
import os
import numpy as np 
import scipy.stats as stats
from copy import deepcopy 
from io import StringIO
import re
import json

__pdoc__ = {
    "_CORE_Reader" : True
}

# default column names for raw Ct data files (don't change this!)
raw_col_names = defaults.raw_col_names

supported_filetypes = defaults.supported_filetypes

default_group_name = defaults.default_group_name

# the default columns that are structurally part of an Assay object
ref_cols = [ raw_col_names[0], "group", "group_name", defaults.default_dataset_header ]


# At some future point we will remove the _CORE_Reader from the 
# __init__ as it's now part of the Readers submodule...
class _CORE_Reader(aux._ID):
    """
    The class handling the core functions of the Reader class. 
    The standard qpcr.Reader inherits from this. 

    Note
    ----
    This implementation is deprecated. The _CORE_Reader has been moved
    (and further updated since) to the `qpcr.Readers` submodule. 

    """
    def __init__(self):
        super().__init__()
        self._src = None
        self._delimiter = None
        self._df = None

    def get(self):
        """
        Returns
        -------
        data : pd.DataFrame
            The dataframe from the datafile.
        """
        return self._df

    def n(self):
        """
        Returns
        -------
        n : int
            The number of replicates (entries) in the dataframe.
        """
        return len(self._df[raw_col_names[0]])


    def read(self, **kwargs):
        """
        Reads the given data file.

        If the data file is an Excel file replicates and their Ct values will be 
        extracted from the first excel sheet of the file. Note, this assumes by default
        that the replicates are headed by the label `"Name"` and the corresponding Ct values
        are headed by the label `"Ct"`. Both labels have to be on the same row. 

        If these labels do not match your excel file, you may
        specify `name_label` and `Ct_label` as additional arguments.
        """
        suffix = self._filesuffix()
        if suffix == "csv":
            try: 
                self._csv_read()
            except:
                # setup parser
                parser = Parsers.CsvParser()
                # check it file should be read transposed
                transpose = aux.from_kwargs("transpose", False, kwargs, rm = True)
                if transpose:
                    parser.transpose()
                
                # setup patterns and store assay-of-interest
                assay_pattern = aux.from_kwargs("assay_pattern", "Rotor-Gene", kwargs)
                assay_of_interest = aux.from_kwargs("assay", None, kwargs, rm=True)
                parser.assay_pattern(assay_pattern)
                
                # get data column labels
                id_label = aux.from_kwargs("id_label", "Name", kwargs, rm = True)
                ct_label = aux.from_kwargs("ct_label", "Ct", kwargs, rm = True)
                parser.labels(id_label,ct_label)
                
                # pipe the datafile through the parser
                parser.pipe(self._src, **kwargs)

                if len(parser.assays()) > 1:
                    if assay_of_interest is None: 
                        aw.HardWarning("Reader:cannot_read_multifile", file = self._src, assays = parser.assays())
                    self._df = parser.get(assay_of_interest)
                    self.id(assay_of_interest)
                else:
                    assay_of_interest = parser.assays()[0]
                    self._df = parser.get(assay_of_interest)
                    self.id(assay_of_interest)

        elif suffix == "xlsx":
            # setup parser
            parser = Parsers.ExcelParser()
            # check it file should be read transposed
            transpose = aux.from_kwargs("transpose", False, kwargs, rm = True)
            if transpose:
                parser.transpose()
            
            # check for sheet_name
            sheet_name = aux.from_kwargs("sheet_name", 0, kwargs, rm = True)

            # setup patterns and store assay-of-interest
            assay_pattern = aux.from_kwargs("assay_pattern", "Rotor-Gene", kwargs)
            assay_of_interest = aux.from_kwargs("assay", None, kwargs, rm=True)
            parser.assay_pattern(assay_pattern)
            
            # get data column labels
            id_label = aux.from_kwargs("id_label", "Name", kwargs, rm = True)
            ct_label = aux.from_kwargs("ct_label", "Ct", kwargs, rm = True)
            parser.labels(id_label,ct_label)

            # pipe the datafile through the parser
            parser.read(self._src, sheet_name = sheet_name)
            parser.parse(**kwargs)

            if len(parser.assays()) > 1:
                if assay_of_interest is None: 
                    aw.HardWarning("Reader:cannot_read_multifile", file = self._src, assays = parser.assays(), traceback = False)
                self._df = parser.get(assay_of_interest)
                self.id(assay_of_interest)
            else:
                assay_of_interest = parser.assays()[0]
                self._df = parser.get(assay_of_interest)
                self.id(assay_of_interest)
        
    def _csv_read(self, **kwargs):
        """
        Reads the given data file if it's a csv file
        """
        df = None
        try: 
            df = pd.read_csv(
                                self._src, 
                                sep = self._delimiter, 
                                header = aux.from_kwargs("header", 0, kwargs), 
                                names = raw_col_names
                            )
        except: 
            aw.HardWarning("Reader:cannot_read_csv", file = self._src)

        # check if a valid Ct column was found
        Ct = raw_col_names[1]
        full_valid_Ct_col = len(  df[ df[Ct] == df[Ct] ]  ) == len(df)
        if not full_valid_Ct_col:
            aw.HardWarning("Reader:cannot_read_csv", file = self._src)

        self._df = df

    def _filesuffix(self):
        """
        Returns the file-suffix of the provided file
        """
        try: 
            suffix = self._src.split(".")[-1]
        except: 
            pass
        return suffix

class Reader(_CORE_Reader):
    """
    Reads qpcr raw data files in csv or excel format to get a single dataset. 

    Note
    -----
    This implementation of the default Reader is now deprecated and will be removed
    at some point in the future. The new impelentation is the `qpcr.Readers.SingleReader`.
    Please, use that one instead.

    Input Data Files
    ----------------
    Valid input files are either regular `csv` files, or  irregular `csv` or `excel` files, 
    that specify assays by one replicate identifier column and one Ct value column.

    Irregular input files may specify multiple assays as separate tables, 
    one assay has to be selected using the `assay` argument. 
    Separate assay tables may be either below one another (separated by blank lines!)
    or besides one another (requires `transpose = True`).

    Parameters
    ----------
    filename : str
        A filepath to a raw data file.
        If the file is a `csv` file, it has to have two named columns; one for replicate names, one for Ct values. 
        Both csv (`,` spearated) and csv2 (`;` separated) are accepted.
        If the file is an `excel` file it the relevant sections of the spreadsheet are identified automatically. 
        But they require identifying headers. By default `Name` and `Ct` are assumed but these can be changed using 
        the `name_label` and `Ct_label` arguments that can be passed as kwargs (they will be forwarded to the `.read()` method). 

    **kwargs
        Any additional keyword arguments that shall be passed to the `read()` method which is immediately called during init.
    """
    def __init__(self, filename:str, **kwargs) -> pd.DataFrame: 
        super().__init__()
        self._src = filename
        if self._filesuffix() == "csv":
            self._delimiter = ";" if self._is_csv2() else ","
        self.read(**kwargs)

    def _is_csv2(self):
        """
        Tests if csv file is ; delimited (True) or common , (False)
        """
        with open(self._src, "r") as openfile: 
            content = openfile.read()
        if ";" in content: 
            return True
        return False

    def _has_header(self):
        """
        Checks if column headers are provided in the data file
        It does so by checking if the second element in the first row is numeric
        if it is numeric (returns None << False) no headers are presumed. Otherwise
        it returns 0 (as in first row has headers)...
        """
        with open(self._src, "r") as openfile: 
            content = openfile.read().split("\n")[0]
            content = content.split(self._delimiter)
        try: 
            second_col = content[1]
            second_col = float(second_col)
        except ValueError:
            return 0 # Headers in row 0
        return None  # no headers


# class _Qupid_Reader(_CORE_Reader):
#     """
#     This Reader class works with streamlit's UploadedFile class.
    
#     Note
#     -------
#     We have to use a little hack to make the UploadedFile readable.
#     We convert to a string and then back to a StringIO which we can then pass to pandas...
    
#     Parameters
#     ----------
#     file
#         A streamlit UploadedFile object
#     """
#     def __init__(self, file) -> pd.DataFrame: 
#         super().__init__()
#         self._filename = file.name
#         if self._filesuffix() == "csv":
#             self._content = file.read().decode()
#             self._src = StringIO(self._content)
#             self._delimiter = ";" if self._is_csv2() else ","
#         else: 
#             self._src = file
#         self.read()

#     def _filesuffix(self):
#         """
#         Returns the file-suffix of the provided file
#         """
#         suffix = self._filename.split(".")[-1]
#         return suffix

#     def _is_csv2(self):
#         """
#         Tests if csv file is ; delimited (True) or common , (False)
#         """
#         if ";" in self._content: 
#             return True
#         return False

#     def _has_header(self):
#         """
#         Checks if column headers are provided in the data file
#         It does so by checking if the second element in the first row is numeric
#         if it is numeric (returns None << False) no headers are presumed. Otherwise
#         it returns 0 (as in first row has headers)...
#         """
#         content = self._content.split("\n")[0]
#         content = content.split(self._delimiter)
#         try: 
#             second_col = content[1]
#             second_col = float(second_col)
#         except ValueError:
#             return 0 # Headers in row 0
#         return None  # no headers

class Assay(aux._ID):
    """
    The central storing unit of single datasets that were read from datafiles.
    An `qpcr.Assay` stores the replicate identifiers and Ct values, and also 
    groups these according to the `replicates` information (which is automatically
    inferred by default). Groups of replicates can be arbitrarily renamed by the user.

    Note
    -------
    The new implementation of the `qpcr.Assay` works directly with a DataFrame
    that was generated by any one of the `qpcr.Readers` or `qpcr.Parsers` 
    instead of a (now depcrecated) `qpcr.Reader` object. 
    However, a `qpcr.Reader` can still be passed as `df` argument and will be read-in using `link`. 
    Support for this will be removed at some point in the future. 

    Parameters
    ----------
    df : pandas.DataFrame
        A DataFrame produces by one of the `qpcr.Readers` 
        containing an `id` column for the replicate identifiers 
        and a `Ct` value column. 
    id : str
        The identifer of the assays (the Assay name, essentially). 

    replicates : int or tuple or str
            Can be an `integer` (equal group sizes, e.g. `3` for triplicates), 
            or a `tuple` (uneven group sizes, e.g. `(3,2,3)` if the second group is only a duplicate). 
            Another method to achieve the same thing is to specify a "formula" as a string of how to create a replicate tuple.
            The allowed structure of such a formula is `n:m,` where `n` is the number of replicates in a group and `m` is the number of times
            this pattern is repeated (if no `:m` is specified `:1` is assumed). See `qpcr.Assay.replicates` for an example. 

    group_names : list
        A list of names to use for the replicates groups. If replicates of the same group share the same identifier, then the 
        group will be inferred automatically. Otherwise, default group names will be set if no `group_names` are provided. 
    """
    def __init__(self, df : pd.DataFrame = None, id : str = None, replicates : (int or tuple or str) = None, group_names : list = None) -> dict:
        super().__init__()
        
        # FUTURE DROP HERE
        # check if a qpcr.Reader was supplied instead of a dataframe
        # drop this at some point
        if isinstance(df, Reader):
            self.link(df)
        else: 
            self._df = df
            if id is not None: self._id = id

        # setup length of the found data        
        self._length = None if self._df is None else len(self._df)

        # get replicates
        self._replicates = replicates

        # store names 
        self._names = group_names

        # setup the amplification efficiency
        self._efficiency = 1.0 
        self._eff = 2 * self._efficiency

        # if we got data, try to read it 
        if self._df is not None: 
            try: 
                self.replicates(self._replicates)
                self.group()
            except Exception as e:
                aw.SoftWarning("Assay:setup_not_grouped")
            
            # and try to change names, provided that we could group yet...
            if self._names is not None and self.groups() is not None: 
                self.rename(self._names)

    def efficiency( self, eff : float = None ):
        """
        Gets or sets the amplification efficiency of the Assay.

        Parameters
        -------
        eff : float
            A new efficiency to assign to the assay.

        Returns
        -------
        float 
            The currently assigned efficiency.
        """
        if isinstance( eff, (float, int) ):
            self._efficiency = float( eff ) 
            self._eff = 2 * self._efficiency
        return self._efficiency 

    def save(self, filename : str):
        """
        Saves the data from the `Assay` to a `csv` file.
        Parameters
        ----------
        filename : str
            The filename into which the assay should be stored.
            If this is a `directory`, then the assay `id` will automatically
            be used as filename. 
        """
        if os.path.isdir(filename):
            filename = os.path.join(filename, f"{self.id()}.csv")
        self.to_csv(filename, index = False)

    def get(self, copy : bool = False ):
        """
        Parameters
        -------
        copy : bool
            If `True` returns a deepcopy of the stored dataframe.

        Returns
        -------
        data : pandas.DataFrame
            The stored dataframe
        """
        if copy: 
            data = deepcopy( self._df )
        else: 
            data = self._df 
        return data

    def boxplot( self, mode : str = "interactive", **kwargs ):
        """
        A shortcut to call a `qpcr.Plotters.ReplicateBoxPlot` plotter
        to visualise the loaded replicates.

        Parameters
        -------
        mode : str
            The plotting mode. May be either "static" (matplotlib) or "interactive" (plotly).
        **kwargs
            Any additional keyword arguments to be passed to the plotter.

        Returns
        -------
        fig : plt.figure or plotly.figure
            The figure generated by `ReplicateBoxPlot`.
        """
        plotter = Plotters.ReplicateBoxPlot( mode = mode )
        plotter.link( self )
        fig = plotter.plot( **kwargs )
        return fig 



    def tile(self, n : int = 1):
        """
        Expands the dataframe to the square number of entries for each group.
        This is useful for combinatoric normalisation wherein each replicate is normalised
        against each replicate group-wise from the normaliser, instead of only its supposed partner value.
        
        Parameters
        -------
        n : int
            The number of tiles to produce. By default `1 tile` will effectively *square* the number of entries within the dataframe.
        """
        df = self.get()
        groups = self.groups()

        new = None

        for group in groups: 
            subset = df.query(f"group == {group}")
            length = len(subset) * n
            subset = pd.concat( [subset for i in range(length) ], ignore_index = True )
            if new is None:
                new = subset
            else:
                new = pd.concat( [new, subset], ignore_index = True )

        self.adopt( new, force = True)

    def stack(self, n : int = 2):
        """
        Expands the dataframe entry-wise `n` times. 

        Parameters
        -------
        n : int
            The number of stacks to produce. `1 stack` will introduce one more copy of each replicate.
            Note, `n == 1` will keep the current entries!
        """
        df = self.get()
        groups = self.groups()

        n = int(n)

        new = None
        if n > 1:
            for group in groups: 
                subset = df.query(f"group == {group}")
                length = n
                subset = pd.concat( [subset for i in range(length) ], ignore_index = True )
                if new is None:
                    new = subset
                else:
                    new = pd.concat( [new, subset], ignore_index = True )

            self.adopt( new, force = True)

    def Ct(self):
        """
        Returns
        ------
        Ct : pandas.Series
            A pandas Series with the assay's Ct values. The column is renamed 
            from "Ct" to the assay's `id`.
        """
        Ct = self._df[ raw_col_names[1] ]
        Ct.name = self.id()
        return Ct


    def dCt(self):
        """
        Returns
        -------
        dCt : pandas.Series
            A pandas Series with the computed Delta-Ct values. The column is renamed 
            from "dCt" to the assay's `id`.
        """
        dCt = self._df["dCt"]
        dCt.name = self.id()
        return dCt

    def ddCt(self):
        """
        Returns
        -------
        ddCt : pandas.DataFrame
            A pandas DataFrame with all Delta-Delta-Ct values that the Assay has stored. 
            All `"rel_{}"` columns are renamed to include the assay `id` to `"{id}_rel_{}"`.
        """
        # get all ddCt columns
        ddCt = [ i for i in self._df.columns if "rel_" in i ]
        id = self._id
        # make new names and generate renaming dictionary 
        new_names = [ f"{id}_{i}" for i in ddCt ]
        new_names = {  old : new for new, old in zip(new_names, ddCt)  }

        # get the data and rename
        ddCt = self._df[ ddCt ]
        if not isinstance(ddCt, pd.DataFrame):
            ddCt = pd.DataFrame(ddCt)
        ddCt = ddCt.rename(columns = new_names)
        
        return ddCt

    # FUTURE FEATURE HERE
    # def fc(self):
        # some method to also return the fold change columns... 


    def rename_cols(self, cols:dict):
        """
        Renames columns according to a dictionary as key -> value.

        Parameters
        ----------
        cols : dict
            A dictionary specifying old column names (keys) and new colums names (values).
        """
        self._df = self._df.rename(columns = cols)


    def link(self, Reader:Reader):
        """
        Links a `qpcr.Reader` object to the Assay.

        Note
        ------
        This is deprecated since the `qpcr.Reader` has been 
        replaced by the `qpcr.Readers.SingleReader`. 
        This method will be removed in the future.  

        Parameters
        ----------
        Reader : qpcr.Reader
            A qpcr.Reader object.
        """
        self._Reader = Reader
        self.adopt_id(Reader)
        df = self._Reader.get()
        self._df = df
        self._length = self._Reader.n()
        
    def n(self):
        """
        Returns 
        ------

        int 
            The number of entries (individual replicates) within the Assay.
        """
        return len( self._df )  # self._length

    def add_dCt(self, dCt : pd.Series): 
        """
        Adds results from Delta-Ct (first Delta-Ct performed by a `qpcr.Analyser`).

        Parameters
        -----------
        dCt : pandas.Series
            A pandas Series of Delta-Ct values that will be stored in a column `"dCt"`.
            Note, that each `Assay` can, of course, only store one single Delta-Ct column. 
        """
        self._df["dCt"] = dCt
    
    def add_ddCt(self, normaliser_id : str, ddCt : pd.Series):
        """
        Adds results from Delta-Delta-Ct ("normalisation" performed by a `qpcr.Normaliser`).
        These will be stored in a column named `"rel_{normaliser_id}"`. Hence, an Assay can store
        an arbitrary number of Delta-Delta-Ct columns against an arbitrary number of different normalisers. 
        
        Parameters
        ----------
        normaliser_id : str
            The id of the normaliser Assay used to compute the Delta-Delta-Ct values.
        ddCt : pandas.Series
            A pandas Series of Delta-Delta-Ct values.
        """
        name = f"rel_{normaliser_id}"
        self._df[name] = ddCt

    # FUTURE FEATURE HERE
    # some method to add fc columns here...

    def adopt(self, df : pd.DataFrame, force = False):
        """
        Adopts an externally computed dataframe as its own.
        This is supposed to be used when setting up new `qpcr.Assay` objects that do not 
        inherit data from one of the `qpcr.Readers`. If you wish to alter an existing `qpcr.Assay` use `force = True`.
        When doing this, please, make sure to retain the proper data structure!

        Parameters
        ----------
        df : pd.DataFrame
            A pandas DataFrame.
        force : bool
            If a dataframe is already stored the new dataframe will only be stored if `force = True`.
        """
        if self._df is None: 
            self._df = df
        elif force:
            self._df = df
        else:
            aw.HardWarning("Assay:no_data_adopted")
        self._length = len(self._df)

    def names(self, as_set = True):
        """
        Returns a set of the replicate group names (maintaing group order).

        Parameters
        ----------
        as_set : bool
            If `as_set = True` (default) it returns a set (as list without duplicates) 
            of assigned group names for replicate groups.
            If `as_set = False` it returns the full group_name column (including all repeated entries).
        
        Returns
        -------
        names : list or pd.Series
            The given group names of all replicate groups.
        """
        if "group_name" in self._df.columns: 
            if as_set:
                return aux.sorted_set(list(self._df["group_name"]))
            else: 
                return self._df["group_name"]
        else: 
            aw.SoftWarning("Assay:no_groupnames")
            return None
    
    def groups(self, as_set = True):
        """
        Returns a set of sample groups (numeric).

        Parameters
        ----------
        as_set : bool
            If `as_set = True` (default) it returns a set (as list without duplicates) 
            of assigned group names for replicate groups.
            If `as_set = False` it returns the full group_name column (including all repeated entries).
        
        Returns
        -------
        groups : list
            The given numeric group identifiers of all replicate groups.
        """
        if "group" in self._df.columns:
            groups = sorted(list(set(self._df["group"]))) if as_set else self._df["group"]
            return groups
        else:
            aw.SoftWarning("Assay:setup_not_grouped")
            return None

    def replicates(self, replicates : (int or tuple or str) = None):
        """
        Either sets or gets the replicates settings to be used for grouping
        Before they are assigned, replicates are vetted to ensure they cover all data entries.

        Parameters
        ----------
        replicates : int or tuple or str
            Can be an `integer` (equal group sizes, e.g. `3` for triplicates), 
            or a `tuple` (uneven group sizes, e.g. `(3,2,3)` if the second group is only a duplicate). 
            Another method to achieve the same thing is to specify a "formula" as a string of how to create a replicate tuple.
            The allowed structure of such a formula is `n:m,` where `n` is the number of replicates in a group and `m` is the number of times
            this pattern is repeated (if no `:m` is specified `:1` is assumed). 
            
            So, as an example, if there are 12 groups which are triplicates, but
            at the end there is one which only has a single replicate (like the commonly measured diluent qPCR sample), we could either specify the tuple
            individually as `replicates = (3,3,3,3,3,3,3,3,3,3,3,3,1)` or we use the formula to specify `replicates = "3:12,1"`. Of course, this works for
            any arbitrary setting such as `"3:5,2:5,10,3:12"` (which specifies five triplicates, followed by two duplicates, a single decaplicate, and twelve triplicates again – truly a dataset from another dimension)...
        """
        if replicates is not None and self._df is not None: 
            # convert a string formula to tuple if one was provided
            if isinstance(replicates, str): 
                replicates = self._reps_from_formula(replicates)
            # vet replicate coverage
            if self._vet_replicates(replicates):
                self._replicates = replicates
            else: 
                aw.HardWarning("Assay:reps_dont_cover", n_samples = self._length, reps = replicates)
        return self._replicates

    def group(self, infer_names = True):
        """
        Groups the data according to replicates-settings specified.

        Parameters
        ----------
        infer_names : bool
            Try to infer names of replicate groups based on the individual replicate sample identifiers.
            Note that this only works if all replicates have an identical sample name!
        """
        
        # generate group and group_names columns
        if isinstance(self._replicates, int):
            groups, group_names = self._make_equal_groups()            
        elif isinstance(self._replicates, tuple):
            groups, group_names = self._make_unequal_groups()
        else:
            if self._identically_named():
                groups = self._infer_replicates()
                group_names = [default_group_name.format(i) for i in groups]
            else: 
                aw.HardWarning("Assay:no_reps_inferred", assay = self.id())
        
        # add numeric group identifiers
        self._df["group"] = groups
        self._df["group_name"] = group_names
        
        if infer_names: #and self._names is None:
            # infer group names
            self._infer_names()
            

    def rename(self, names:(list or dict)):
        """
        Replaces the current names of the replicate groups 
        (stored in the "group_name" column).

        Parameters
        ----------
        names : list or dict
            Either a `list` (new names without repetitions) or `dict` (key = old name, value = new name) specifying new group names. 
            Group names only need to be specified once, and are applied to all replicate entries.
        """
        # get new group names based on list (index) or dict (key)
        if isinstance(names, (list, tuple, set)):
            new_names = self._rename_per_index(names)       
        elif isinstance(names, dict):
            new_names = self._rename_per_key(names)
        else:
            aw.SoftWarning("Assay:no_groupname_assignment", names = names)

        # update "group_name"
        self._df["group_name"] = new_names
        self._renamed = True

    def ignore(self, entries:tuple, drop = False):
        """
        Remove lines based on index from the dataframe.
        This is useful when removing corrupted data entries.

        Parameters
        ----------
        entries : tuple
            Tuple of row indices from the dataframe to drop.
        drop : bool 
            If True the provided entries will be entirely removed from the 
            dataset. If False, ignore entries will be set to NaN. 
        """
        if drop:
            self._df = self._df.drop(index = list(entries))
        else: 
            Cts = self.Ct()
            Cts = np.array( Cts )
            Cts[ entries ] = np.nan
            self._df["Ct"] = Cts
    
    def _reps_from_formula(self, replicates):
        """
        Generates a replicate tuple from a string formula. 
        See the docstring of `replicates()` for more info on the formula.

        Example:
        "3:4,1:4,2:3,9" -> (3, 3, 3, 3, 1, 1, 1, 1, 2, 2, 2, 9)
        """

        # split the formula and adjust standard formatting
        replicates = replicates.split(",")
        replicates = [i + ":1" if ":" not in i else i for i in replicates]

        # convert to numeric values and extend
        replicates = [np.array(i.split(":"), dtype = int) for i in replicates]
        replicates = [np.tile(i[0], i[1]) for i in replicates]
        
        # generate replicate tuple
        replicates = np.concatenate(replicates)
        replicates = tuple(replicates)
        
        return replicates

    def _infer_replicates(self):
        """
        Infers the replicate groups based on the replicate ids in case all replicates of the same group have the same name.
        """
        names = self._df[raw_col_names[0]]
        names_set = aux.sorted_set(names)
        groups = [i for i in range(len(names_set))]
        for name, group in zip(names_set, groups):
            names = names.replace(name, group)
        
        indices = np.array(names, dtype = int)
        return indices

    def _infer_names(self):
        """
        Infers replicate group names from the given replicate identifier column
        """
        if self._identically_named():
            self._df["group_name"] = self._df[raw_col_names[0]]
        elif self._names is None: 
            aw.SoftWarning("Assay:groupnames_not_inferred")

    def _identically_named(self):
        """
        Checks if all replicates in the same group have the same name / id
        It checks simply the first group, if that is identical then it's fine.
        """
        if "group" not in self._df.columns:
            names = self._df[raw_col_names[0]]
            names_set = aux.sorted_set(names)
            first_name = names_set[0]
            group0 = self._df.query(f"{raw_col_names[0]} == '{first_name}'")[raw_col_names[0]]
            entries = len(group0)
            all_identical = entries > 1                
        else: 
            group0 = self._df.query("group == 0")[raw_col_names[0]]
            all_identical = all(group0 == group0[0])
        return all_identical

    def _rename_per_key(self, names):
        """
        Generates new name list based on current names in "group_name" and uses string.replace()
        to update groupnames, based on key (old name) : value (new name) indexing. 
        Before applying it checks if all groups are covered by new names
        """
        current_names = aux.sorted_set(self._df["group_name"])
        all_groups_covered = len(names) == len(current_names)
        if all_groups_covered:
            current_names = list(self._df["group_name"])
            new_names = "$".join(current_names)
            for old_name, new_name in names.items():
                new_names = new_names.replace(old_name, new_name)
            new_names = new_names.split("$")       
            return new_names
        else:
            aw.HardWarning("Assay:groupnames_dont_colver", current_groups = current_names, new_received = names)

    def _rename_per_index(self, names):
        """
        Generates new name list based on current names in "group_names" and uses string.replace()
        to update groupnames to new names based on index (using a the order 
        of groups as is currently present in "group_name"). 
        """
        current_names_set = aux.sorted_set(self._df["group_name"])
        all_groups_covered = len(names) == len(current_names_set)
        if all_groups_covered:
            current_names = list(self._df["group_name"])
            new_names = "$".join(current_names)
            names = list(names)
            for old_name, new_name in zip(current_names_set, names):
                new_names = new_names.replace(old_name, new_name)
            new_names = new_names.split("$")
            return new_names
        else:
            aw.HardWarning("Assay:groupnames_dont_colver", current_groups = current_names_set, new_received = names)

    def _make_unequal_groups(self):
        """
        Returns two lists of [0,0,0,1,1,1] and 
        [Group0, Group0, Group0, Group1,...] 
        to cover all data entries.
        (this function works with a tuple for replicate group sizes)
        """
        groups = []
        group_names = []
        for rep, idx in zip(self._replicates, range(len(self._replicates))): 
            groups.extend([idx] * rep)
            group_names.extend([ default_group_name.format(idx) ] * rep)
        return groups, group_names

    def _make_equal_groups(self):
        """
        Returns two lists of [0,0,0,1,1,1] and 
        [Group0, Group0, Group0, Group1,...] 
        to cover all data entries.
        (this function works with an integer group size, 
        assuming all groups have the same size)
        """
        assays = self._length
        groups = []
        group_names = []
        slices = range(int(assays / self._replicates))
        for i in slices:
            groups.extend([i] * self._replicates)
            group_names.extend([ default_group_name.format(i) ] * self._replicates)
        return groups, group_names

    def _vet_replicates(self, replicates : (int or tuple)):
        """
        Checks if provided replicates will place all data entries into a group
        returns True if all replicates are covered, False if not...
        """
        current_entries = self._length
        verdict = None

        # for INT -> modulo will be 0 if all replicates are covered
        # for TUPLE -> sum(replicates) should cover all replicates...

        if isinstance(replicates, int):
            verdict = True if current_entries % replicates == 0 else False
        elif isinstance(replicates, tuple): 
            verdict = True if sum(replicates) == current_entries else False
        
        if verdict is None: 
            aw.HardWarning("Assay:reps_could_not_vet", reps = replicates)

        return verdict

"""
## Setting up a `qpcr.Assay`
---

Here is a manual example of creating a `qpcr.Assay` object. You can use either the `qpcr.DataReader` or any one of `qpcr.Readers` directly to 
read in your data and generate a pandas DataFrame. Note, the `qpcr.Readers` are already equipped with `make_Assay(s)` methods that will handle
setting up `qpcr.Assay` objects for you. 

However, setting up `qpcr.Assay`s manually can be as simple as:

```python
# get the dataframe from one of the qpcr.Readers
mydata = some_reader.get()

assay = Assay( df = mydata, id = "my_assay" )

```

If your replicate identifiers are the same for all replicates within each group then the groups are automatically inferred. And your assay is 
ready at this point already to be passed to an `qpcr.Analyser`. If not, you can specify the replicates manually like: 

```python
# manually specify triplicates during setup
assay = Assay( df = mydata, id = "my_assay", replicates = 3 )

# or you can change the replicates after initial setup like 
assay = Assay( df = mydata, id = "my_assay" )
assay.replicates(3)
assay.group()

```

"""

class SampleReader(Assay):
    """
    Sets up a Reader+Assay pipeline that reads in a single datafile and handles the 
    extracted dataset in a pandas dataframe. 
    Its `read()` method directly returns a `qpcr.Assay` object that can be piped to Analyser. 
    
    Note
    ----
    This is now deprecated and will be removed in a future version! Please, use the `qpcr.DataReader` instead.
    """
    def __init__(self):
        super().__init__()
        
        aw.SoftWarning("Versions:Deprecation", old = "SampleReader", new = "DataReader")

        self._replicates = None
        self._names = None
        self._Reader = None
        self._Assay = None

    # def __str__(self):
    #     header = f"qpcr.SampleReader ({self._id})"
    #     reps = f"Replicate settings:\t{self._replicates}"
    #     names = f"Name settings:\t\t{self._names}"

    #     header_line = max([len(i) for i in [header, reps, names]])
    #     header_line = "-" * header_line

    #     string = f"{header_line}\n{header}\n{header_line}\n\n{reps}\n{names}\n\n{header_line}"
    #     return string

    def replicates(self, replicates:(int or tuple)):
        """
        Set the replicates specifics to use for grouping.

        Parameters
        ----------
        replicates : int or tuple
            Can be an `integer` (equal group sizes, e.g. `3` for triplicates), 
            or a `tuple` (uneven group sizes, e.g. `(3,2,3)` if the second group is only a duplicate). 
        """
        self._replicates = replicates

    def names(self, names:(list or dict)):
        """
        Set names for replicates groups.

        Parameters
        ----------
        names : list or dict
            Either a `list` (new names without repetitions) or `dict` (key = old name, value = new name) specifying new group names. 
            Group names only need to be specified once, and are applied to all replicate entries.
        """
        self._names = names
        
    def read(self, filename, **kwargs):
        """
        Reads one raw datafile (csv or excel format).

        Parameters
        ----------
        filename : str
            A filepath to a raw data file.
            If the file is a `csv` file, it has to have two named columns; one for replicate names, one for Ct values. 
            Both csv (`,` spearated) and csv2 (`;` separated) are accepted.
            If the file is an `excel` file it the relevant sections of the spreadsheet are identified automatically. 
            But they require identifying headers. By default `Name` and `Ct` are assumed but these can be changed using 
            the `name_label` and `Ct_label` arguments that can be passed to `read()` as kwargs.
        **kwargs
            Any additional keyword arguments that should be passed to the `qpcr.Reader`.

        Returns
        -------
        Assay : qpcr.Assay
            A `qpcr.Assay` object containing the grouped and renamed data.
        """
        self._Reader = Reader(filename, **kwargs)
        self._Reader.id(aux.fileID(filename))

        self._Assay = Assay(self._Reader)
        self._Assay.adopt_id(self._Reader)

        if self._replicates is not None:
            self._Assay.replicates(self._replicates)
        else: 
            pass 
            # VITAL CHANGE HERE
            # since Assays can now infer replicates we 
            # don't raise an Error if no replicates are specified manually...
            # aw.HardWarning("SampleReader:no_reps_yet")

        if self._names is not None:
            self._Assay.group(infer_names = False)
            self._Assay.rename(self._names)
        else:
            self._Assay.group()
            
        return self._Assay

class DataReader(aux._ID):
    """
    Handles reading a single file containing input data
    for `qpcr`. 
    
    Note
    -----
    This is a top-level class that is designed
    as the central port through which data is read into `qpcr.Assay` objects
    from both regular and irregular, single- and multi-assay files.
    This is the suggested way to read your data for most users, which should
    work in most cases.
    
    However, due to the automated setup of the inferred Readers there may be cases where you 
    will either have a hard time or be unable to read your datafiles using the `DataReader`. 
    In such cases, don't try too long to make it work with the DataReader, 
    just use one of the `qpcr.Readers` or even `qpcr.Parsers`directly.  

    """
    def __init__(self):
        super().__init__()
        self._replicates = None
        self._names = None
        self._Reader = None             # the functional core will be either a Reader
        self._Data = {}                 # the _Data attribute will store any output from the Reader as a dictionary with filename : data structure.
        self._tmp_data = None           # this will not attempt to distinguish between assays / normalisers, or anything. It's just an archive of whatever data we got.
                                        # by default data is not stored by the DataReader, it's read only pipes through, but data can be stored using the .store() method.

    def Reader(self, Reader = None):
        """
        Gets or sets the functional core Reader    
        """
        if Reader is not None:
            self._Reader = Reader
        return self._Reader

    def clear(self):
        """
        Clears all data that was extracted
        """
        self._Data = {}
    
    def reset(self):
        """
        Resets the core Reader
        """
        self._Reader = None
    
    def prune(self):
        """
        Resets the DataReader completely
        """
        self.__init__()

    def store(self):
        """
        Will store the read data. 


        Note 
        -------
        `DataReader` does NOT have a specific data storage facility to distinguish between 
        assays / normalisers, data types, etc. It simply keeps a dictionary of `{filename : data}`
        that can be accessed. This is designed in case multiple files should be read using the same 
        `DataReader` to allow an easier access of the data in case the data outputs are of the same type.

        However, the main intended application of `DataReader` is to use the`read` method's returned data directly.
        """
        self._Data.update(self._tmp_data)
    
    def get(self):
        """
        Returns the stored data
        """
        return self._Data

    def read(self, filename : str, multi_assay : bool = False, big_table : bool = False, decorator : (bool or str) = None, reset = False, **kwargs):
        """
        Reads an input file and extracts available datasets using the
        specified `Reader` or by setting up an approproate `Reader`. 

        Parameters
        ----------
        filename : str
            A filepath to an input datafile.

        multi_assay : bool
            Set to `True` if the file contains multiple assays you wish to read.

        big_table : bool
            Set to `True` if the file is a "Big Table" file. 
            Check out the documentation of the `qpcr.Readers` for more 
            information on "Big Table" files.

        decorator : str or bool
            Set if the file is decorated. This can be set either to `True` for `multi_assay` and multi-sheet (excel) or `big_table` files,
            or it can be set to a valid `qpcr decorator` for single assay files or single-sheet files.
            Check out the documentation of the `qpcr.Parsers` for more information on decorators.

        reset : bool
            If multiple input files should be read but they do not all 
            adhere to the same filetype / datastructure, use `reset = True` 
            to set up a new Reader each time `read` is called.

        **kwargs
            Any additional keyword arguments to be passed to the core Reader.
            Note, while this tries to be utmost versatile there is a limitation
            to costumizibility through the kwargs. If you require streamlined datareading
            use dedicated `qpcr.Readers` and/or `qpcr.Parsers` directly.
        """
        self._src = filename
        # vet filesuffix
        suffix = self._filesuffix()
        if suffix not in supported_filetypes:
            aw.HardWarning("MultiReader:unknown_datafile", file = self._src)
        
        if reset or self._Reader is None: 
            self.reset()
            self._setup_Reader(
                                multi_assay = multi_assay, 
                                big_table = big_table,
                                decorator = decorator,
                                **kwargs
                            )
        
        # read file and return data
        data = self._Reader._DataReader( 
                                            filename = self._src, 
                                            decorator = decorator, 
                                            **kwargs 
                                    )

        self._tmp_data = {self._src : data}
        return data

    def _filesuffix(self):
        """
        Returns the filesuffix
        """
        return self._src.split(".")[-1]

    def _setup_Reader(self, **kwargs):
        """
        Sets up the core Reader 
        """
        suffix = self._filesuffix()

        use_multi = aux.from_kwargs("multi_assay", False, kwargs, rm = True)
        is_bigtable = aux.from_kwargs("big_table", False, kwargs, rm = True)

        if suffix == "csv":
            if not use_multi and not is_bigtable:
                reader = Readers.SingleReader()
            elif is_bigtable:
                reader = Readers.BigTableReader()
            else:
                reader = Readers.MultiReader()

        elif suffix == "xlsx":
            
            # check if not only a single sheet should be read
            multi_sheet = "sheet_name" not in kwargs

            if use_multi and multi_sheet:
                reader = Readers.MultiSheetReader()
            elif use_multi and not multi_sheet:
                reader = Readers.MultiReader()
            elif is_bigtable:
                reader = Readers.BigTableReader()
            else:
                reader = Readers.SingleReader()
        self._Reader = reader


class Results(aux._ID):
    """
    Handles a pandas dataframe for data and computed results from a `qpcr` class. 
    
    Note
    -----
    This is a central data collection that can inherit directly from `qpcr.Assay` objects and from 
    extrenally computed sources. Please, note that it will not perform extensive vetting on its data input, 
    so make sure to only provide proper data input when manually assembling your `qpcr.Results`!
    """
    def __init__(self):
        super().__init__()
        self._df = None
        self._Assay = None
        self._stats_results = {
                                "group" : [], 
                                defaults.default_dataset_header : [], 
                                "mean" : [], "stdev" : [], "median" : []
                            }
        self._stats_df = None

    def adopt_names(self, Assay:Assay):
        """
        Links an instance of Assay to be used as reference for group_names
        It copies the `group` and `group_name` columns to the results storing dataframe.
        This step can only be performed once!

        Parameters
        ----------
        Assay : qpcr.Assay
            A `qpcr.Assay` object whose group_name column will be copied.
        """
        if self.is_empty():
            self._df = Assay.get()
            self._drop_setup_cols()
        else:
            named_identically = all( self._df["group_name"] == Assay.get()["group_name"] )
            if not named_identically:
                aw.SoftWarning("Results:cannot_link")


    def add_Ct(self, assay : Assay):
        """
        Adds a `"Ct"` column with Delta-Ct values from an `qpcr.Assay`.
        It will store these as a new column using the Assay's `id` as header.

        Parameters
        -------
        assay : qpcr.Assay
            An `qpcr.Assay` object from which to import.
        """
        id = assay.id()
        Ct = assay.get()[ raw_col_names[1] ]
        Ct.name = id
        self.add(Ct)

    def add_dCt(self, assay : Assay):
        """
        Adds a `"dCt"` column with Delta-Ct values from an `qpcr.Assay`.
        It will store these as a new column using the Assay's `id` as header.

        Parameters
        -------
        assay : qpcr.Assay
            An `qpcr.Assay` object from which to import.
        """
        dCt = assay.dCt()
        self.add(dCt)

    def add_ddCt(self, assay : Assay):
        """
        Adds all `"rel_{}"` columns with Delta-Delta-Ct values from an `qpcr.Assay`.
        It will store these as new columns using the Assay's `id` + the `_rel_{}` composite id.

        Parameters
        -------
        assay : qpcr.Assay
            An `qpcr.Assay` object from which to import.
        """
        df = assay.ddCt()
        # add data
        self.add(df)

    # FUTURE FEATURE HERE
    # this feature will be for fold-change columns of second normalisation
    # currently this is not yet supported but is supposed to be an implemented feature of the 
    # Normaliser. The Normaliser is currently perfectly able to perform second normalisation but
    # does not yet have an integrated method of properly storing the results and there is also not
    # # yet a method for pairing assays together for second normalisation.
    # def add_fc(self, assay : Assay):
    #     """
    #     Adds all `"fc_{}"` columns with Delta-Delta-Ct values from an `qpcr.Assay`.
    #     It will store these as new columns using the Assay's `id` + the `_fc_{}` composite id.

    #     Parameters
    #     -------
    #     assay : qpcr.Assay
    #         An `qpcr.Assay` object from which to import.
    #     """
    #     id = assay.id()
    #     df = assay.get()
    #     # get the ddCt containing columns
    #     rel_cols = [ i for i in df.columns if "fc_" in i ]
    #     # generate new composite ids 
    #     new_names = [ f"{id}_{i}" for i in rel_cols ]
    #     # get ddCt columns 
    #     df = df[rel_cols]
    #     # rename the columns to include the assay id
    #     df = df.rename( columns = { new : old for new, old in zip(new_names, rel_cols) } )

    #     # add data
    #     self.add(df)

    def names(self, as_set = False):
        """
        Returns 
        -------
        names : list or None
            The adopted `group_names` 
            (only works if a `qpcr.Assay` has already been linked 
            using `adopt_names()`!)
        """
        if self._df is not None:
            names = self._df["group_name"]
            if as_set: names = aux.sorted_set(names)
            return names
        return None

    def get(self):
        """
        Returns 
        -------
        data : pd.DataFrame
            The results dataframe
        """
        return self._df

    def is_empty(self):
        """
        Checks if any results have been stored so far.

        Returns
        -------
        bool
            `True` if NO data is yet stored, else `False`.
        """
        return self._df is None

    def drop_groups(self, groups : (list or str)):
        """
        Removes specific groups of replicates from the DataFrame.

        Parameters
        ----------
        groups : list
            Either the numeric group identifiers or the group names
            of the groups to be removed, or a `regex` pattern defining which groups
            should be dropped (this is useful for systematically removing RT- groups etc.)
        """
        # check for regex pattern
        # and get corresponding group names 
        if isinstance(groups, str):
            groups = [i for i in self._df["group_name"] if re.match(groups, i) is not None]
        
        # get the right reference column and query to use to be 
        # used (either group or group_name)
        ref_query = "group != {group}" if isinstance( groups[0], int ) else "group_name != '{group}'"
        
        # remove groups from dataset
        for group in groups: 
            self._df = self._df.query(ref_query.format(group = group))

            # also drop from stats df
            if self._stats_df is not None:
                self._stats_df = self._stats_df.query(ref_query.format(group = group))
    

    def add(self, column:pd.Series, replace : bool = False):
        """
        Adds some new column of data.

        Note
        ----
        The `column` argument has to be named for this to work. However, there are 
        already implemented methods dedicated to adding specifically Delta-Ct, Delta-Delta-Ct or just
        Ct values to the Results.

        Parameters
        ----------
        column : pd.Series
            A named pandas Series or DataFrame that can be joined into the already
            stored dataframe.
        replace : bool
            In case results from a computation with the same identifiers are already stored
            no new data can be stored under that id. Either the new data must be renamed or
            `replace = True` must be set to overwrite the presently stored data. 
        """
        if isinstance(column, pd.Series):
            if column.name in self._df.columns:
                if not replace:  
                    aw.SoftWarning("Results:name_overlap", name = column.name)
                else: 
                    self._df[column.name] = column
            else: 
                self._df = self._df.join(column)
        else: 
            for i in column.columns:
                if i in self._df.columns: 
                    if not replace: 
                        aw.SoftWarning("Results:name_overlap", name = i )
                col = column[i]
                self._df[i] = col

    def merge(self, *Results):
        """
        Merges any number of other qpcr.Results objects into this one.
        The source id of the results is added as column-name suffix. 

        Parameters
        ----------
        *Results
            An arbitrary number of qpcr.Results objects.

        """
        new_df = self._df
        for R in Results: 
            R_df = R.get()

            # get only the delta-delta-Ct columns
            cols = [i for i in R_df.columns if i not in ref_cols]
            R_df = R_df[cols]


            # we merge the dataframes first without adding 
            # some new id suffix, only do so if this fails
            try: 
                new_df = pd.merge(new_df, R_df, 
                                    right_index = True, left_index = True, 
                                )
            except: 
                new_df = pd.merge(new_df, R_df, 
                                right_index = True, left_index = True, 
                                suffixes = [f"_{self.id()}", f"_{R.id()}"]
                            )
        self._df = new_df

    def drop_cols(self, *cols):
        """
        Drops all specified columns from the dataframes
        this is used for normaliser pre-processing.

        Parameters
        ----------
        *cols
            Any column names (as `str`) to be dropped.
            If no names are specified any/all `deltaCt` data-containing columns are dropped!
            If this is the case then the only columns retained are: `"group", "group_name", "id", "assay"`.
        """
        if cols == ():
            _to_drop = [c for c in self._df.columns if c not in [ "group", "group_name", raw_col_names[0], defaults.default_dataset_header ]]
        else:
            _to_drop = [c for c in cols if c in list(self._df.columns)]
        self._df = self._df.drop(columns = _to_drop)
        
    def rename_cols(self, cols:dict):
        """
        Renames columns according to a dictionary as key -> value.

        Parameters
        ----------
        cols : dict
            A dictionary specifying old column names (keys) and new colums names (values).
        """
        self._df = self._df.rename(columns = cols)


    def stats(self, recompute = False) -> pd.DataFrame:
        """
        Computes summary statistis about the replicate groups: 
        `Mean`, `Median`, and `StDev` of all replicate groups, for all datasets (assays).
        
        Parameters
        ----------
        recompute : bool
            Statistics will only be once unless recompute is set to `True`.

        Returns
        -------
        stats_df : pd.DataFrame
            A new dataframe containing the computed statistics for each replicate group.

        """
        default_dataset_header = defaults.default_dataset_header
        # if stats_df is already present, return but sorted according to assays, not groups (nicer for user to inspect)
        if self._stats_df is not None and not recompute:
            return self._stats_df.sort_values(default_dataset_header)
        elif recompute: 
            self._stats_results = {"group" : [], default_dataset_header : [], "mean" : [], "stdev" : [], "median" : []}
            self._stats_df = None

        # get groups and corresponding assay columns 
        groups = aux.sorted_set(list(self._df["group"]))
        assays = [c for c in self._df.columns if c not in ref_cols]
     
        # compute stats for all replicates per group
        for group in groups:
            group_subset = self._df.query(f"group == {group}")
            
            median = self._stat_var(group_subset, np.nanmedian)
            mean = self._stat_var(group_subset, np.nanmean)
            stdv = self._stat_var(group_subset, np.nanstd)
            self._add_stats(assays, group, median, mean, stdv)
            
        # add group names
        self._add_stats_names(assays)

        self._stats_df = pd.DataFrame(self._stats_results)
        return self._stats_df.sort_values(default_dataset_header)

    def save(self, path, df = True, stats = True):
        """
        Saves a csv file for each specified type of results.

        Parameters
        ----------
        path : str
            Path has to be a filepath if only one type of results shall be saved (i.e. either `df` or `stats`), 
            otherwise a path to the directory where both `df` and `stats` shall be saved.
        
        df : bool
            Save the results dataframe containing all replicate values (the full results).
            Default is `df = True`.
        
        stats : bool
            Save the results dataframe containing summary statistics for all replicate groups.
            Default is `stats = True`.
        
        """
        if df and stats and not os.path.isdir(path):
            aw.HardWarning("Results:save_need_dir")

        if df:
            # in case of raw results export we don't need the "assay" column as all 
            # assays are stored as separate columns anyaway, so it doesn't store any useful data
            _df = self._df
            if "assay" in _df.columns: _df = self._df.drop( columns = ["assay"] )
            self._save_single(path, _df, "_df")
        if stats:
            # compute stats if none have been computed yet...
            if self._stats_df is None:
                self.stats()
            self._save_single(path, self._stats_df, "_stats")

    def drop_rel(self):
        """
        Crops the `X_rel_Y` column-names of Delta-Delta-Ct results to just `X`.
        I.e. reduces back to the assay-of-interest name only.
        """
        colnames = self._df.columns
        to_change = {i : i.split("_rel_")[0] for i in colnames if "_rel_" in i }
        self.rename_cols(to_change)

        # also recompute the stats df with new names...
        if self._stats_df is not None: 
            self.stats(recompute = True)

    def split(self, reset_names = False, drop_rel = True):
        """
        Splits the stored results dataframe into separate qpcr.Results objects containing only a signle deltaCt column each.

        Parameters
        ----------
        reset_names : bool
            Resets the deltaCt column-name from `"X_rel_Y"` to just `"dCt"`.

        drop_rel : bool
            Crops `"X_rel_Y"` deltaCt column-names to just `"X"`. 

        Returns 
        -------
        objects : list
            A list of qpcr.Results objects containing only a single dCt column each (retaining group columns etc.)
        """
        shared_columns = [i for i in self._df.columns if i in ref_cols]
        dct_columns = [i for i in self._df.columns if i not in ref_cols]
        
        dfs = [self._df[shared_columns + [i]] for i in dct_columns]
        objects = [Results() for i in dfs]

        for o, df, dct_col in zip(objects, dfs, dct_columns): 
            o._df = df
            if reset_names:
                o.rename_cols({dct_col : "dCt"})
            if drop_rel: 
                o.drop_rel()

            o.id(dct_col)
        
        return objects


    def preview( self, kind : str = "AssayBars", mode : str = "static", **kwargs ):
        """
        A shortcut to call on a `qpcr.Plotters.PreviewResults` wrapper to visualise 
        the results.

        Parameters
        ----------
        kind : str
            The kind of Plotter to call. This can be any of the four wrapped 
            Plotters, e.g. `kind = "GroupBars"`.
        mode : str
            The plotting mode. May be either "static" (matplotlib) or "interactive" (plotly).

        Returns
        -------
        fig : plt.figure or plotly.figure
            The figure generated by `PreviewResults`.
        """
        preview_results = Plotters.PreviewResults( mode = mode, kind = kind )
        preview_results.params( **kwargs )
        preview_results.link( self )
        fig = preview_results.plot()
        return fig 

    def _save_single(self, path, src, suffix=""):
        """
        Saves either self._df or self._stats_df to a csv file based on a path
        (path can be either filename or directory)
        """
        filename = path if not os.path.isdir(path) else os.path.join(path, f"rel_{self.id()}{suffix}.csv")
        src.to_csv(filename, index = False)
        
    def _drop_setup_cols(self):
        """
        Removes unnnecessary columns from the df during self._df setup with link()
        """
        # drop the Ct columns
        relcols = [i for i in self._df.columns if "rel_" in i]
        self.drop_cols(
                        raw_col_names[1], "dCt", *relcols
                    )



    def _add_stats_names(self, samples):
        """
        Adds a group_name column to self._stats_result with appropriate
        repetition of group_names for each group of replicates...
        """
        self._stats_results["group_name"] = []
        group_names = aux.sorted_set(list(self._df["group_name"]))
        for group_name in group_names:
            self._stats_results["group_name"].extend([group_name] * len(samples))

    def _add_stats(self, samples, group, median, mean, stdv):
        """
        Adds new summary entries to self._stats_results
        """
        self._stats_results["group"].extend([group] * len(samples))
        self._stats_results["assay"].extend(samples)
        self._stats_results["median"].extend(median)
        self._stats_results["mean"].extend(mean)
        self._stats_results["stdev"].extend(stdv)


    def _stat_var(self, group_subset, func, **kwargs):
        """
        Performs a function (like mean or stdv) over all rows
        and returns the result as list with a float for each column in the df
        any function can be passed as long as it works with an iterable
        """
        # ignore group and group_name columns
        ignore = [raw_col_names[0], "group", "group_name", "assay"]
        all_cols = [g for g in group_subset.columns if g not in ignore]
        tmp = group_subset[all_cols]
        # compute stats based on func
        stats = []
        for col in tmp.columns:
            try: 
                stat = func(tmp[col], **kwargs)
            except: 
                stat = np.nan
            stats.append(stat)
        return stats
        

class Analyser(aux._ID):
    """
    Performs Single Delta-Ct (first normalisation 
    within dataset against the `anchor`) 
    """
    def __init__(self):
        super().__init__()
        self._Assay = None

        # default settings
        self._anchor = "first"
        self._ref_group = 0
        self._ref_group_col = "group" # used in case of "mean" anchor where the ref_group must be located either from a numeric (group) or string (group_name) id
        
        self._eff_src = self._Assay         # By default use the efficencies stored in the Assay directly 
                                            # this is set to self if self.efficiency() is called...
        self._efficiency = 1                # the formal effiency in percent
        self._eff = 2 * self._efficiency    # the actual doubplciation factor used for calculation
        self._deltaCt_function = self._get_deltaCt_function(exp = True)

    def get(self):
        """
        Returns 
        -------
        Assay : qpcr.Assay
            The analysed `qpcr.Assay` object that contains now deltaCT values.
        """
        return self._Assay

    def link(self, Assay:Assay):
        """
        Links a `qpcr.Assay` object to the Analyser.
       
        Parameters
        ----------
        Assay : qpcr.Assay
            A `qpcr.Assay` object containing data.
        
        """
        self._Assay = Assay

        # check if no efficiency was specifically set for the Analyser
        # and if not so, use the Assay's efficiency...
        if self._eff_src is not self: 
            self._eff_src = self._Assay

    def pipe(self, Assay:Assay, **kwargs) -> Results:
        """
        A quick one-step implementation of link + DeltaCt.

        Note
        ----
        This is the suggested application of the `qpcr.Analyser`.


        Parameters
        ----------
        Assay : qpcr.Assay
            A `qpcr.Assay` object to be linked to the Analyser for DeltaCt computation.
        
        **kwargs
            Any additional keyword arguments to be passed to the `DeltaCt()` method.

        Returns 
        -------
        assay : qpcr.Assay
            The same `qpcr.Assay` with computed Delta-Ct values. 

        """
        self.link(Assay)
        self.DeltaCt(**kwargs)
        assay = self.get()
        return assay

    def efficiency(self, e:float = None):
        """
        Sets an efficiency factor for externally calculated qPCR amplification efficiency.
        By default `efficiency = 1` (100%) is assumed.

        Note
        ----
        By default the efficiency is now (`qpcr 3.2.0`) handled by the `qpcr.Assay` objects directly.
        The Analyser will directly read the efficiency from the `qpcr.Assay`s. However,
        it is still possible to set the efficiency via the `qpcr.Analyser` in this fashion.

        Parameters
        ----------
        e : float
            An amplification efficiency factor. Default is `e = 1`, 
            which is then treated as `eff = 2 * e`, so `e = 1` corresponds to true duplication
            each cycle.

        Returns
        -------
        efficiency : float
            The current efficiency used.

        """
        # deprecation warning
        aw.SoftWarning( "blank", msg = "The use of efficiency() from the Analyser is deprecated and will be dropped in a future version! Please, set efficiencies directly in the Assay." )
        if isinstance(e, (int, float)):
            self._efficiency = float( e )
            self._eff = 2 * self._efficiency
            # and update the efficiency source 
            # to self instead of the assay...
            self._eff_src = self 
        return self._efficiency

    def anchor(self, anchor : (str or float or function) = None, group : (int or str) = 0):
        """
        Sets the anchor for DeltaCt for internal normalisation.

        Parameters
        ----------
        anchor : str or float or function
            The internal anchor for normalisation.
            This can be either `"first"` (default, the very first dataset entry),
            `"mean"` (mean of the reference group), 
            `"grouped"` (first entry for each replicate group), 
            any specified numeric value (as `float`), 
            or a `function` that will calculate the anchor and returns a single numeric value. 
            If you wish to use a function to compute the anchor, you can access the dataframe stored by the `qpcr.Assay` that is being analysed through the 
            `data` argument. `data` will be automatically forwarded to your custom anchor-function, unless you specify it directly. Please, make sure your function can handle
            `**kwargs` because any kwargs supplied during `DeltaCt()`-calling will be passed per default to both the anchor-function and DeltaCt-function. 
        group : int or str
            The reference group identifier. This can be either the numeric identifier or the `group_name`. This is only used for `anchor = "mean"`. 
            By default the first group is assumed. 
        
        Returns
        -------
        anchor 
            The currently selected `anchor`.
        ref_group
            The current reference group.
        """
        if anchor is not None:
            self._anchor = anchor
        if group != self._ref_group:
            self._ref_group = group
        
        # update column where to search for ref_group identifier
        if isinstance(group, str):
            self._ref_group_col = "group_name"
        else: 
            self._ref_group_col = "group"
        
        return self._anchor, self._ref_group


    def func(self, f:(str or function)):
        """
        Sets the function to be used for DeltaCt (optional)

        Parameters
        ----------
        f : str or function
            The function to be used for DeltaCt computation. Pre-defined functions are 
            either `"exponential"` (which uses  `dCt = eff ** ( -(s-r) )`, default), or `"linear"` 
            (uses `dCt = s-r`), where `s` is any replicate entry in the dataframe and `r` is the anchor. `eff = 2 * efficiency` is the 
            numeric duplication factor (default assumed `efficiency = 1`).
            It is also possible to assign any defined function that accepts one `float` Ct value `s` (1st!) and anchor `r` value (2nd!), 
            alongside any `kwargs` (which will be forwarded from DeltaCt()...). It must return also a single `float`. 
        """
        if f in ["exponential", "linear"]:
            f = True if f == "exponential" else False
            self._deltaCt_function = self._get_deltaCt_function(f)
        elif type(f) == type(aux.fileID):
            self._deltaCt_function = f
        else:
            aw.HardWarning("Analyser:cannot_set_func", func = f)

    def DeltaCt(self, **kwargs):
        """
        Calculates Delta-Ct for all groups within the dataframe.
        Any specifics such as `anchor` or `func` must have already been 
        set using the respective methods prior to calling `DeltaCt()`!

        Parameters
        ----------
        **kwargs
            Any additional keyword arguments that a custom DeltaCt function may require.
        """

        predefined = {
                        "first"     : self._DeltaCt_first_anchored,
                        "grouped"   : self._DeltaCt_grouped_anchored,
                        "mean"      : self._DeltaCt_mean_anchored,
                    }

        if self._anchor in predefined.keys():
        
            deltaCt_func = predefined[ self._anchor ]
            deltaCt_func(
                                            self._deltaCt_function, 
                                            **kwargs
                                    )
        
        elif isinstance(self._anchor, (float, int)): 
            self._DeltaCt_externally_anchored(
                                                self._anchor, 
                                                self._deltaCt_function, 
                                                **kwargs
                                            )
        elif type(self._anchor) == type(aux.fileID):
            self._DeltaCt_function_anchor(
                                            self._anchor, 
                                            self._deltaCt_function, 
                                            **kwargs
                                        )

    def _DeltaCt_function_anchor(self, anchor_function, deltaCt_function, **kwargs):
        """
        Performs DeltaCt using a function as anchor
        """
        df = self._Assay.get()
        # update a "data" argument into the 
        # kwargs for the anchor_function
        if "data" not in kwargs.keys(): 
            kwargs.update(dict(data = df))
        anchor = anchor_function(**kwargs)

        # apply deltaCt_function
        Ct = raw_col_names[1]
        dCt = df[Ct].apply(deltaCt_function, r = anchor, **kwargs)
        dCt.name = "dCt"
        # store results
        self._Assay.add_dCt(dCt)

    def _DeltaCt_mean_anchored(self, deltaCt_function, **kwargs):
        """
        Performs DeltaCt using the mean of the reference group as anchor
        """
        df = self._Assay.get()

        # get  reference group
        ref_query = "{} == {}" if isinstance(self._ref_group, int) else "{} == '{}'" 
        ref = df.query(
                        ref_query.format(  self._ref_group_col, self._ref_group  )
                    )
        # get Ct values from ref group and make anchor
        ref = ref[raw_col_names[1]]
        anchor = ref.mean()
        
        # apply DeltaCt function
        Ct = raw_col_names[1]
        dCt = df[Ct].apply(deltaCt_function, r = anchor, **kwargs)
        dCt.name = "dCt"
        # store results
        self._Assay.add_dCt(dCt)


    def _DeltaCt_externally_anchored(self, anchor:float, deltaCt_function, **kwargs):
        """
        Performs DeltaCt using a specified anchor
        """
        # get Ct column label 
        Ct = raw_col_names[1]
        df = self._Assay.get()

        # apply DeltaCt function
        dCt = df[Ct].apply(deltaCt_function, r = anchor, **kwargs)
        dCt.name = "dCt"
        # store results
        self._Assay.add_dCt(dCt)


    def _DeltaCt_grouped_anchored(self, deltaCt_function, **kwargs):
        """
        Performs DeltaCt using the first entry of each group as anchor
        """
        # get set of sample groups and dataset
        groups = self._Assay.groups()
        df = self._Assay.get()

        # get Ct column label
        Ct = raw_col_names[1]

        dCt = pd.Series()
        for group in groups: 
            group_subset = df.query(f"group == {group}").reset_index(drop = True)
            anchor = group_subset[Ct][0]
            delta_cts = group_subset[Ct].apply(deltaCt_function, r = anchor, **kwargs)
            dCt = dCt.append(delta_cts).reset_index(drop = True)
        
        dCt.name = "dCt"

        # store results
        self._Assay.add_dCt(dCt)

    def _DeltaCt_first_anchored(self, deltaCt_function, **kwargs):
        """
        Performs DeltaCt using the very first entry of the dataset as anchor
        """
        # get Ct column label
        Ct = raw_col_names[1]
        
        # get first available entry
        # we do this instead of just 0 
        # because the truly first entry 
        # might have been filtered out 
        df = self._Assay.get()
        first = list(df.index)[0]

        # get anchor
        anchor = df[Ct][ first ]

        # apply DeltaCt function
        dCt = df[Ct].apply(deltaCt_function, r = anchor, **kwargs)
        dCt.name = "dCt"
        # store results
        self._Assay.add_dCt(dCt)

    def _exp_DCt(self, s, r, **kwargs):
        """
        Calculates deltaCt exponentially
        """
        factor = s - r 
        return self._eff_src._eff **(-factor)

    def _simple_DCt(self, s, r, **kwargs):
        """
        Calculates deltaCt linearly
        """
        return s - r

    def _get_deltaCt_function(self, exp):
        """
        Returns the function to be used for DeltaCt based on 
        whether or not exponential shall be used.
        """
        if exp == True:
            dCt = self._exp_DCt
        else:
            dCt = self._simple_DCt
        return dCt


class Normaliser(aux._ID):
    """
    Handles the second step in Delta-Delta-Ct (normalisation against normaliser assays).
    
    Note
    -----
    This requires that all have been analysed in the same way before!
    """
    def __init__(self):
        super().__init__()

        self._Normalisers = []
        self._Assays = []
        
        self._Results = Results()
        
        # the actually used normaliser 
        # which will be a pre-processed version
        # from all supplied normalsier assays
        # by default this will be the averaged version
        # of all normaliser assays (but this can be changed)  
        self._normaliser = None

        # setup defaults
        self._prep_func = self._preprocess_normalisers
        self._norm_func = self._divide_by_normaliser

        # store the state if a custom norm func was provided
        self._norm_func_is_set = False


    def clear(self):
        """
        Will clear the presently stored results
        """
        self._Results = Results()

    
    def prune(self, assays = True, normalisers = True, results = True):
        """
        Will clear assays, normalisers, and/or results
        assays : bool
            Will clear any sample assays if True (default).
        
        results : bool
            Will clear any computed results if True (default).
        
        normalisers : bool
            Will clear any normalisers if True (default).
        """
        if assays:
            self._Assays = []
        if normalisers: 
            self._Normalisers = []
        if results: 
            self.clear()


    def get(self, copy=False):
        """
        Parameters
        ----------
        copy : bool
            Will return a deepcopy of the Results object if `copy = True` (default is `copy = False`).
        
        Returns
        -------
        Results : qpcr.Results
            A `qpcr.Results` object containing the normalised dataframe
        """
        if copy: 
            return deepcopy(self._Results)
        return self._Results
    
    def link(self, assays:(list or tuple or Analyser) = None, normalisers:(list or tuple or Analyser) = None):
        """
        Links either normalisers or assays-of-interest `qpcr.Assay` objects coming from the same `qpcr.Analyser`.

        Parameters
        ----------
        assays : list or tuple or qpcr.Analyser
            A list of `qpcr.Assay` objects coming from a `qpcr.Analyser` or the `qpcr.Analyser` itself. These assays will be normalised against a normaliser.
        
        normalisers : list or tuple or qpcr.Analyser
            A list of `qpcr.Assay` objects coming from a `qpcr.Analyser` or the `qpcr.Analyser` itself. These assays will be used as normalisers. These will be
            combined into one single pseudo-normaliser which will then be used to normalise the assays. The method of 
            combining the normalisers can be specified using the `qpcr.Normaliser.prep_func` method.
        """
        # convert to lists (since the _link methods really want lists)
        if assays is not None and not isinstance(assays, (list, tuple)): 
            assays = [assays]
        self._link_assays(assays)

        if normalisers is not None and not isinstance(normalisers, (list, tuple)): 
            normalisers = [normalisers]
        self._link_normaliser(normalisers)
    
    def prep_func(self, f = None):
        """
        Sets any defined function for combined normaliser pre-processing.
        If no `f` is provided, it returns the current `prep_func`.

        Parameters
        ----------
        f : function
            The function may accept one list of `qpcr.Assay` objects, and must return 
            either an `qpcr.Assay` object directly or a `pandas.Dataframe` (that will be migrated to an `qpcr.Assay`).
            The returned dataframe must contain a `"dCt"` column which stores the delta-Ct values ultimately used as 
            "normaliser assay".  
        """
        
        if aux.same_type(f, aux.fileID):
            self._prep_func = f
        elif f is None:
            return f
        else: 
            aw.HardWarning("Normaliser:cannot_set_prep_func", func = f)

    def norm_func(self, f = None):
        """
        Sets any defined function to perform normalisation of assays against normalisers.
        If no `f` is provided, it returns the current `norm_func`.

        Parameters
        ----------
        f : function
            The function may accept two `qpcr.Assay` objects (named `assay` and `normaliser` which will be forwarded from the `qpcr.Normaliser`). 
            The function may also accept one `pandas.DataFrame` (named `df`) containing two numeric columns of delta-Ct values from a sample-assay (named "s") and a normaliser-assay (named "n"),
            as well as a group identifier column (named "group"). 
            Whatever inputs it works with, it must return a named numeric `pandas.Series` of the same length as entries in the Assays' dataframes. 
            
            ##### Note
            Support for the dataframe direct usage will be dropped at some point in the future. 

            By default `s/n` is used, where `s` is a column of sample-assay deltaCt values, 
            and `n` is the corresponding `"dCt"` column from the normaliser.

        """
        if aux.same_type(f, aux.fileID):
            self._norm_func = f
            self._norm_func_is_set = True
        elif f is None:
            return f
        else: 
            aw.HardWarning("Normaliser:cannot_set_norm_func", func = f)

    def normalise(self, mode = "pair-wise", **kwargs):
        """
        Normalises all linked assays against the combined pseudo-normaliser 
        (by default, unless a custom `prep_func` has been specified), 
        and stores the results in a new Results object.

        Parameters
        ----------
        mode : str
            The normalisation mode to use. This can be either `pair-wise` (default), 
            or `combinatoric`, or `permutative`.
            `pair-wise` will normalise replicates only by their partner (i.e. first against first, 
            second by second, etc.). `combinatoric` will normalise all possible combinations of a replicate 
            with all partner replicates of the same group from a normaliser (i.e. first against first, then second, then third, etc.).
            This will generate `n^2` normalised Delta-Delta-Ct values, where `n` is the number of replicates in a group.
            `permutative` will scramble the normaliser replicates randomly and then normalise pair-wise. This mode supports 
            a parameter `k` which specifies the times this process should be repeated, thus generating `n * k` normalised Delta-Delta-Ct values.
            Also, through setting `replace = True` replacement may be allowed during normaliser scrambling.
            Note, this setting will be ignored if a custom `norm_func` is provided.

        **kwargs
            Any additional keyword arguments that may be passed to a custom 
            `norm_func` and `prep_func` (both will receive the kwargs!).
        """
        if self._normaliser is None: 
            self._normaliser = self._prep_func(self._Normalisers, **kwargs)
            self._vet_normaliser()

        if self._Assays == [] or self._normaliser is None:
            aw.SoftWarning("Normaliser:no_data_yet")
            
        # check which kind of norm_func we should use
        if not self._norm_func_is_set:
            # pair-wise is already default so we don't check...
            if mode == "combinatoric":
                self._norm_func = self._tile_normalise
                tiled = deepcopy(self._Assays[0])
                tiled.tile()
                self._Results.adopt_names( tiled )
                del tiled 
            elif mode == "permutative":
                self._norm_func = self._permutate_normalise
                n = aux.from_kwargs("k", 1, kwargs)
                tiled = deepcopy(self._Assays[0])
                tiled.stack(n)
                self._Results.adopt_names( tiled )
                del tiled 
            elif mode == "pair-wise":
                self._Results.adopt_names( self._Assays[0] )
        else:
            self._Results.adopt_names( self._Assays[0] )

        self._Results.drop_cols()

        # perform normalisation for each assay 
        for assay in self._Assays:

            # get data
            # assay_df = assay.get()

            # apply normalisation (delta-delta-Ct)
            normalised = self._norm_func_wrapper(
                                                    assay = assay, 
                                                    normaliser = self._normaliser, 
                                                    **kwargs
                                            )

            # # store results in _Results
            # self._store_to_Results(assay, normalised)

            # and store results also in the Assay itself
            assay.add_ddCt( self._normaliser.id(), normalised )

            # and store to results
            self._Results.add_ddCt(assay)

    def _vet_normaliser(self):
        """
        Checks if the normaliser is already a qpcr.Assay object, and if not
        convert it to one. 
        """
        if not isinstance(self._normaliser, Assay):
            tmp = Assay()
            tmp.adopt(self._normaliser)
            tmp.id("combined_normaliser")
            self._normaliser = tmp
            

    def _store_to_Results(self, assay, normalised):
        """
        Stores computed Delta-delta-ct (normalisation)
        into the _Results object.
        """
        column_name = f"{assay.id()}_rel_{self._normaliser.id()}"
        normalised = normalised.rename(column_name)
        self._Results.add(normalised)


    def _norm_func_wrapper(self, assay, normaliser, **kwargs):
        """
        The wrapper that will apply the _norm_func to the sample and normaliser dataframes and return a pandas series of normalised values
        """
        # for double normalised we want the same columns as dct and norm...

        # FUTURE DROP HERE
        # In the future we will not be creating the tmp_df 
        # directly anymore, but intead will only pass assay and normaliser. 

        sample_dCt = assay.dCt()
        groups = assay.groups( as_set = False )
        norm_dCt = normaliser.dCt()

        tmp_df = pd.DataFrame( dict( group = groups, s = sample_dCt, n = norm_dCt )  )

        results = self._norm_func(df = tmp_df, assay = assay, normaliser = normaliser, **kwargs)

        # this is the old call from before factoring out to Assays 
        # dCt_col, norm_col = self._prep_columns(sample_assay, dCt_col, norm_col)

        # tmp_df = normaliser.join(sample_assay, lsuffix="_s")
        # # tmp_df = sample_assay.join(normaliser, rsuffix = "_n")
        # results = self._norm_func(tmp_df[[dCt_col, norm_col]], **kwargs)
        return results

    def _tile_normalise(self, assay, normaliser, **kwargs):
        """
        Normalises assays and normalisers group wise, iteratively normalising
        each individual replicate against all replicates from the normaliser.
        This generates `n**2` normalised Delta-Delta-Ct values where `n` is the
        group size. 
        """

        # get the untiled data
        adf = assay.get()
        groups = assay.groups()
        ndf = normaliser.get()
        
        # get the column to draw data from
        col = aux.from_kwargs("col", "dCt", kwargs)

        # tile the assay
        assay.tile() 

        # generate results array
        ddCts = np.zeros( len( assay.get() ) )
       
        # now compute ddCt
        idx = 0
        for group in groups: 

            a_dCt = adf.query( f"group == {group}" )[ col ].to_numpy()
            n_dCt = ndf.query( f"group == {group}" )[ col ].to_numpy()
            
            for a in a_dCt:
                for n in n_dCt:

                    r = a / n
                    
                    try: 
                        ddCts[idx] = r 
                    except: 
                        break 

                    idx += 1

        ddCts = pd.Series( ddCts )       
        ddCts.name = "ddCt"
        return ddCts
        
    
    def _permutate_normalise(self, assay, normaliser, **kwargs):
        """
        Scrambles randomly the normaliser's replicate values group-wise 
        """
        # get the untiled data
        adf = assay.get()
        groups = assay.groups()
        ndf = normaliser.get()
        
        # get the column to draw data from
        col = aux.from_kwargs("col", "dCt", kwargs)

        # stack the assay
        # get the number of permutations to perform
        n = aux.from_kwargs("k", 1, kwargs)  
        assay.stack(n) 

        # get replace argument for random choice
        replace = aux.from_kwargs("replace", False, kwargs)

        # generate results array
        ddCts = np.zeros( len( assay.get() ) )
       
        # now compute ddCt
        idx = 0
        for group in groups: 
            for i in range(n):
                a_dCt = adf.query( f"group == {group}" )[ col ].to_numpy()
                n_dCt = ndf.query( f"group == {group}" )[ col ].to_numpy()
                
                # randomly scramble the normaliser replicates
                np.random.seed( defaults.default_seed )
                if replace:
                    # in case of replacement, we generate a normal 
                    # distribution from the replicate values and weigh
                    # the random.choice with the given probabilities of the values 
                    # being chosen. Since the probabilities do not themselves correspond to 1
                    # we normalise against the limited available subset to generate a probabilities
                    # array that sums up to 1.
                    mu, sd = stats.norm.fit(n_dCt)
                    probs = stats.norm.pdf(n_dCt, loc = mu, scale = sd)
                    probs = probs / np.sum(probs)
                    n_dCt = np.random.choice( n_dCt, size = n_dCt.size, replace = replace, p = probs )
                else:
                    n_dCt = np.random.choice( n_dCt, size = n_dCt.size, replace = replace )
                length = a_dCt.size
                r = a_dCt / n_dCt
                        
                try: 
                    ddCts[idx : idx + length] = r 
                except: 
                    break 

                idx += length

        ddCts = pd.Series( ddCts )       
        ddCts.name = "ddCt"
        return ddCts

    def _divide_by_normaliser(self, df, **kwargs):
        """
        Performs normalisation of sample s against normaliser n
        s and n are specified as two pandas dataframe columns
        Note, that the dataframe must ONLY contain these two columns, first the dCt sample, then the normaliser!
        (default _norm_func)
        """
        s, n = df["s"], df["n"]
        # this is the old call from before factoring out to Assays
        # dCt_col, norm_col = df.columns
        # s, n = df[dCt_col], df[norm_col]
        return s / n

    def _link_assays(self, assays):
        """
        Links any provided assays and checks their datatype in the process...
        """
        if assays is not None:
            for sample in assays: 
                if aux.same_type(sample, Assay()):
                    self._Assays.append(sample)

                elif aux.same_type(sample, Analyser()):
                    self._Assays.append(sample.get())
                
                else: 
                    aw.SoftWarning("Normaliser:unknown_data", s = sample)
                
    def _link_normaliser(self, normalisers):
        """
        Checks if normaliser is provided and has proper datatype to be added...
        """
        if normalisers is not None:
            for normaliser in normalisers:

                if aux.same_type(normaliser, Assay()):
                    self._Normalisers.append(normaliser)

                elif aux.same_type(normaliser, Analyser()):
                    self._Normalisers.append(normaliser.get())

                else: 
                    aw.SoftWarning("Normaliser:norm_unknown_data", s = normaliser)

    def _preprocess_normalisers(self, *args, **kwargs):
        """
        Averages the provided normalisers row-wise for all normalisers into a 
        single combined normaliser, that will be stored as a new Assay object.
        """

        # initialise new Results to store the dCt values form all normalisers
        combined = Results()
        
        # setup names using the first normaliser
        combined.adopt_names(self._Normalisers[0])
        combined.drop_cols("dCt")
        combined.adopt_id(self._Normalisers[0])

        # now add all dCt columns from all normalisers
        for norm in self._Normalisers:
            combined.add_dCt(norm)
        
        # remove the non-dCt columns as they would interfere with
        # pre-processing. But we keep the "group" column because some
        # custom prep_func may want to use the group references.
        # i.e. we just replace any "Ct" columns that may have smuggled in if 
        # a non-default process is used for assembly
        combined.drop_cols( raw_col_names[1] )

        # now generate the combined normaliser
        combined_normaliser = self._average(combined)
        combined_normaliser = combined_normaliser.rename("dCt")
        combined.add(combined_normaliser)
        
        # now assemble the normaliser into a qpcr.Assay
        combined = combined.get()
        normaliser = Assay()
        normaliser.adopt(combined)
        normaliser.adopt_id(self._Normalisers[0])


        self._normaliser = normaliser  
        if len(self._Normalisers) > 1:
            self._update_combined_id()
  
        # forward combined_id to self and _Results 
        self.adopt_id(self._normaliser)
        self._Results.adopt_id(self._normaliser)

        return self._normaliser

    def _update_combined_id(self):
        """
        Generates a new id based on all normaliser ids,
        joining them as a+b+c,...
        """
        ids = [N.id() for N in self._Normalisers]
        ids = "+".join(ids)
        self._normaliser.id_reset()
        self._normaliser.id(ids)
        

    def _average(self, combined):
        """
        Averages row-wise all Normaliser entries and 
        generates a series of their per-row means
        (default preprocess_normalisers function)
        """
        tmp_df = combined.get()

        # drop group as it is a numeric column 
        # and would otherwise skew the average
        if "group" in tmp_df.columns:
            tmp_df = tmp_df.drop(columns = ["group"]) 
        tmp_df = tmp_df.mean(axis = 1, numeric_only = True)

        return tmp_df


# FUTURE MOVE HERE
# NOTE: This should be moved to the Curves submodule once it will be made in a future
#       version to work directly with absorption and fluorescense curves.
class _EfficiencyCurve(aux._ID):
    """
    A helper class that will handle dilutions, ct values and the linreg model
    when newly computing efficiencies from assays.
    """
    def __init__(self, dilutions, ct_values, model, efficiency):
        super().__init__()
        self._dilutions = dilutions
        self._ct_values = ct_values
        self._model = model
        self._efficiency = efficiency
    
    def values(self):
        """
        Returns
        ------
        dilutions : np.ndarray
            The dilutions (x-values) used for efficiency calculation.
        ct_value : np.ndarray
            The underlying Ct values (y-values) used for efficiency calculation.
        """
        return self._dilutions, self._ct_values
    
    def model(self):
        """
        Returns
        -------
        model : stats.LinregressResult
            The linear regression model used for efficiency calculation.
        """
        return self._model
    
    def efficiency(self):
        """
        Returns
        -------
        eff : float
            The efficiency calculated from the stored data.
        """
        return self._efficiency
    

class Calibrator(aux._ID):
    """
    Calculates qPCR primer efficiency based on a dilution series.
    The dilution series may either be represented as an entire assay
    or as a subset of groups within an assay denoted as `calibrator : {some_name}`.
    In this mode, calibrator replicates will be removed after calibration is done.

    It is possible to specify the dilution steps directly in the groupnames as:
    `calibrator: {some_name}: dil` where `dil` is the inverse dilution step, e.g.
    `calibrator: my_sample: 2` for a `1 : 2` dilution or `calibrator: my_sample: 100`
    for a `1 : 100`. Note, this will have to be present in **each** groupname! 

    Alternatively, if no dilution is specified in the groupnames or they cannot be inferred
    for some other reason, it is possible to supply a dilution step via 
    the `qpcr.Calibrator.dilution` method. 
    """
    def __init__(self):
        super().__init__()
        self._eff_dict = {}             # stores the efficiencies as assay_id : efficiency
        self._computed_values = {}      # stores newly computed efficiency data as:
                                        # assay_id :  _EfficiencyCurve(...)
                                        #             The _EfficiencyCurve object stores:
                                        #             - dilutions
                                        #             - Ct values
                                        #             - the Linreg model
        self._dilution_step = None      # the dilution step(s) used 
        self._manual_dilution_set = False 
        self._orig_dilution = None 
        
        self._loaded_file = None


    def save( self, filename : str = None , mode : str = "write" ):
        """
        Saves the calculated efficiencies to a `csv` file.

        Parameters
        -------
        filename : str
            The filepath in which to store the efficiencies. If a file
            was already loaded then by default the same file will be 
            used to save values again.
        
        mode : str
            Can be either `"write"` to fully overwrite an existing file,
            with the newly computed data, or `"append"` to only add newly 
            computed efficiencies.
        """

        if filename is None and self._loaded_file is not None: 
            filename = self._loaded_file

        if mode == "write":
            self._save( filename, self._eff_dict )

        elif mode == "append":
            current = self._load(filename)
            new = { **current, **self._eff_dict }
            self._save( filename, new )

        else: 
            aw.SoftWarning("Calibrator:unknown_savemode", mode = mode )

    def load(self, filename, merge : bool = True, supersede : bool = False ):
        """
        Loads a `csv` file of previously computed efficiencies.

        Parameters
        -------
        filename : str
            The filepath to load efficiencies from.
        merge : bool
            In case efficiencies are already loaded, merge the
            new and existing ones. If `False` the current ones will
            be replaced completely.
        supersede : bool 
            In case efficiencies of the same assay are already loaded
            they will be overwritten by the newly incoming ones if `supersede = True`.
        """
        try: 
            current = self._load(filename)
            if merge:
                current = { **self._eff_dict, **current } if supersede else { **current, **self._eff_dict }
            self.adopt( current )
            self._loaded_file = filename
            return current
        except: 
            aw.HardWarning("Calibrator:unknown_filetype", filename = filename )

    def get( self, which = "efficiencies" ):
        """
        Returns
        -------
        dict
            Either the stored efficiencies (if 
            `which = "efficiencies"`) or 
            the computed values of newly computed
            efficiencies (if `which = "values"`).
        """
        if which == "efficiencies":
            return self.efficiencies()
        elif which == "values":
            return self.computed_values()
       
    def efficiencies( self ):
        """
        Returns
        ------
        dict
            The currently stored efficienies.
        """
        return self._eff_dict
    
    def computed_values( self ):
        """
        Returns
        ------
        dict
            The currently stored values from newly
            computed efficiencies.
        """ 
        return self._computed_values
    
    def merge( self, *filenames, outfile = None, adopt = True ):
        """
        Merges multiple efficiency files together into a single one.

        Parameters
        -------
        filenames : iterable
            Filepaths to load data from which should be merged together.
        
        outfile : str 
            The filepath in which to store the merged efficiencies.
            Not saved if set to `None`.
        
        adopt : bool
            Will adopt the merged dictionary as its own if `True` (default).

        Returns
        -------
        all_effiencies : dict
            The merged dictionary of all efficiencies from all files.
        """
        all_efficiencies = {}
        for filename in filenames: 
            new = self._load( filename )
            all_efficiencies = { **all_efficiencies, **new }

        if outfile is not None: 
            self._save( outfile, all_efficiencies )

        if adopt:
            self.adopt( all_efficiencies ) 

        return all_efficiencies
    
    def reset( self ):
        """
        Resets the Calibrator to initial settings. This will 
        clear all stored efficiency values and computed data!
        """
        self.__init__()

    def clear( self ):
        """
        Will clear all stored efficiency values and computed data.
        """
        self._eff_dict = {}
        self._computed_values = {}

    def adopt( self, effs : dict ):
        """
        Adopts an externally generated dictionary of `assay : efficiency`
        structure as its own.

        Parameters
        -------
        effs : dict
            A dictionary where keys are Assay Ids (`str`) 
            and values are `float` efficiencies.
        """
        if aux.same_type( effs, {} ):
            self._eff_dict = effs
        else: 
            aw.SoftWarning("Calibrator:cannot_adopt", effs = effs, eff_type = type(effs).__name__ )
    
    def dilution( self, step : float or np.ndarray or tuple = None ):
        """
        Gets or sets the dilution steps used. This must be a `float` fraction
        e.g. `0.5` for a `1 : 2` dilution series or `0.1` for a `1 : 10` series etc.
        If there are multiple steps because there is a gap in the dilution series. It is 
        necessary to supply a step for each group individually e.g. `[1,0.5,0.25,0.0625,0.03125]`.
        if there are 5 dilution steps (originally six but 0.125 was discarded).

        Note, both of the above also work with the inverse dilutions e.g. `2` or `[1,2,4,16,32]`.
        
        By default the `qpcr.Calibrator` tries to infer the dilutions automatically.
        This only works, however, if the calibrator groupnames specify `calibrator: {some name} : dil` where
        `dil` is the inverse dilution step (e.g. `calibrator: my_sample: 2` for a `1 : 2` dilution). Note,
        it is important that the dilution step is given as the inverse (i.e. *not* as `1:2 or 1/2` or something else! )

        Parameters
        ----------
        step : float or np.ndarray
            The dilution step used.  

        Returns
        -------
        dilution : float or np.ndarray
            The currently used dilution step.
        """
        
        dilution = self._dilution(step)
        self._orig_dilution = self._dilution_step

        # if the dilution is now set to a valid
        # value we set the _manual_dilution_set check to True
        if dilution is not None:
            self._manual_dilution_set = True 
        return dilution

        
    def pipe( self, assay : Assay, remove_calibrators : bool = True ):
        """
        A wrapper for calibrate / assign.

        This method will first try to assign pre-computed efficiencies
        and if no matching ones are found it will try to calculate a new efficiency
        from the assay. 

        Parameters
        ----------
        assay : qpcr.Assay
            A `qpcr.Assay` object.
        remove_calibrators : bool
            If calibrators are present in the assay alongside other groups, 
            remove the calibrator replicates after assignment or efficiency calculation. 
        """
        if self._eff_dict != {}:
            # first try to assign (will leave the assay unchanged if nothing is found)
            eff = self._get_efficiency( assay )
            if eff is not None: 
                assay = self.assign( assay, remove_calibrators = remove_calibrators )
            else: 
                try:
                    assay = self.calibrate( assay, remove_calibrators = remove_calibrators )
                except: 
                    aw.HardWarning("Calibrator:cannot_process_assay", id = assay.id() )
        else:
            try: 
                assay = self.calibrate( assay, remove_calibrators = remove_calibrators )
            except:
                aw.HardWarning("Calibrator:cannot_process_assay", id = assay.id() )

        return assay

    def calibrate( self, assay : Assay, remove_calibrators : bool = True ):
        """
        Computes an efficiency from an `qpcr.Assay` object.
    
        This method will try to compute a new efficiency. To do this, it will check autonomously if
        `calibrator : {}` replicates are present and use these for computation. If none are 
        found it will assume the entire assay is to be used as calibrator.

        Note
        ----
        Calibrators are searched for through the group `names` not the replicate ids!

        Parameters
        ----------
        assay : qpcr.Assay
            A `qpcr.Assay` object.

        remove_calibrators : bool
            If calibrators are present in the assay alongside other groups, 
            remove the calibrator replicates after efficiency calculation. 
        """

        # get the assay's groupnames and check for the calibrator prefix.
        names = assay.names( as_set = False ).unique()


        # check if any groups are declared as calibrators
        calibrators = np.array( [ self._has_calibrator_prefix(i) for i in names ] )
        has_calibrators = any( calibrators )

        # now get the relevant dataframe for the computation
        # this will either be the entire df (if no calibrator groups are present)
        # or just the subset of calibrators
        df = assay.get()

        if has_calibrators:
            df = self._subset_calibrators(names, calibrators, df)

        # get Ct column name
        ct_name = defaults.raw_col_names[1]
        # drop NaN cols as they are incompatible with linregress anyway...
        df = df[ df[ ct_name] == df[ ct_name ] ]

        # now sort the dataframe by Ct values as they need to strictly
        # increase for dilution series.
        df = df.sort_values( ct_name ).reset_index()
        df = df.rename( columns = { "index" : "orig_index" } )

        # now generate dilution steps ( i.e. "concentrations" )
        # to do that we first need to check if dilutions have not
        # been supplied, and then try to infer them based on the groupnames
        # if we got an input for dilution() we use  that to generate a 
        # dilution steps array...

        # NOTE: The non-log-scaled dilutions are now stored in self._dilution_steps
        #       while the log-scaled versions are returned. Hence, the dilutions 
        #       variable below is the log-scaled version!
        if not self._manual_dilution_set: 
            dilutions = self._infer_dilution_steps(df)
        else:
            dilutions = self._generate_dilution_steps(df)
        
        
        # now interpolate a line through the log dilutions and the ct values
        cts = df[ ct_name ].to_numpy()

        regression_line = stats.linregress( x = dilutions, y = cts )
        # and now compute the efficiency from the regression line
        efficiency = self._compute_efficiency(regression_line)

        # and now assign the efficiency to the assay
        assay.efficiency( efficiency )

        # save the efficiency in self._eff_dict
        # self._eff_dict.update( { assay.id() : assay.efficiency() } )
        self._eff_dict[ assay.id() ] = assay.efficiency()

        # and, finally, save the computed values and the efficiency
        self._save_computation( assay, dilutions, cts, regression_line )

        # now remove the calibrators from the assay
        # but only do so in case there were calibrators found!
        # If the entire assay was used, we do not delete the entries...
        if has_calibrators and remove_calibrators: 
            self._remove_calibrators(assay, df)

        return assay 
    
    def assign( self, assay : Assay, remove_calibrators : bool = True ):
        """
        Assigns an efficiency to an `qpcr.Assay` based on its Id.
        This requires that an efficiency corresponding to the Assay's Id
        is present in the currently loaded / computed effiencies.

        Parameters
        ----------
        assay : qpcr.Assay
            A `qpcr.Assay` object.

        remove_calibrators : bool
            If calibrators are present in the assay alongside other groups, 
            remove the calibrator replicates. 
        """
        eff = self._get_efficiency( assay )
        if eff is not None:
            # set assay's efficiency
            assay.efficiency( eff )

            # check if any groups are declared as calibrators
            # that should be removed from the df
            names = assay.names( as_set = False ).unique()
            calibrators = np.array( [ self._has_calibrator_prefix(i) for i in names ] )
            has_calibrators = any( calibrators )
            if has_calibrators and remove_calibrators: 
                df = assay.get()
                df = self._subset_calibrators(names, calibrators, df)
                df = df.sort_values( raw_col_names[1] ).reset_index()
                df = df.rename( columns = { "index" : "orig_index" } )
                self._remove_calibrators(assay, df)
        else:
            aw.SoftWarning("Calibrator:could_not_assign", id = assay.id() )
        return assay

    def plot( self, mode : str = "interactive", **kwargs ):
        """
        A shortcut to call a `qpcr.Plotters.EfficiencyLines` plotter
        to visualise the regression lines from de novo efficiency computations.

        Parameters
        -------
        mode : str
            The plotting mode. May be either "static" (matplotlib) or "interactive" (plotly).
        **kwargs
            Any additional keyword arguments to be passed to the plotter.

        Returns
        -------
        fig : plt.figure or plotly.figure
            The figure generated by `EfficiencyLines`.
        """
        plotter = Plotters.EfficiencyLines( mode = mode )
        plotter.link( self ) 
        fig = plotter.plot( **kwargs )
        return fig 

    def _dilution(self, step = None ):
        """
        The functional core of self.dilution() the only difference is
        that self.dilution also sets a boolean attribute self._manual_dilution_set to True...
        Which signals that the manually supplied dilutions should be used rather than that 
        they should be inferred from the dataset...
        """
        if step is not None: 
            unknown_datatype = not isinstance( step, (float, int, np.ndarray, pd.Series, tuple, list ) )
            if unknown_datatype:
                aw.HardWarning("Calibration:cannot_interpret_dilution", step = step, step_type = type(step).__name__ )
            
            # check if we need to convert to numpy array
            # because we have an iterable without math operations support.
            if isinstance( step, (tuple, list) ):
                step = np.array( step )

            # check for an ndarray and make sure to invert if 
            # the dilution steps are given as 2 4 instead of
            # 0.5 0.25 etc., also do the same for a single number...

            is_inverse_array = isinstance( step, (np.ndarray, pd.Series) ) and any( step > 1 )
            is_inverse_number = isinstance( step, (float, int) ) and step > 1
            need_inverse = is_inverse_array or is_inverse_number
            
            if need_inverse:
                step = 1 / step
            
            # and store new steps
            self._dilution_step = step

        return self._dilution_step

    def _remove_calibrators(self, assay, df):
        """
        Drops all calibrator replicates from the dataframe of the Assay.

        Note
        -------
        This leaves the index unchanged! Possible we might wish to also reset the 
        index during this step...
        """
        to_remove = df["orig_index"].to_numpy()
        index = np.zeros( assay.n() )
        for i in to_remove: index[i] = 1 
        index = np.argwhere( index == 1 )
        index = np.squeeze( index )
        assay.ignore( index, drop = True )

    def _save_computation(self, assay, dilutions, ct_values, linreg):
        """
        Creates a new entry in self._computed_values for the newly computed
        efficiency.
        """
        self._computed_values[ assay.id() ] = _EfficiencyCurve( 
                                                                dilutions = dilutions, 
                                                                ct_values = ct_values, 
                                                                model = linreg,
                                                                efficiency = assay.efficiency() 
                                                            )
        # also add the Id of the Assay to the _EfficiencyCurve
        self._computed_values[ assay.id() ].id( assay.id() )


    def _compute_efficiency(self, regression_line):
        """
        Calculates the efficiency from the regression line slope.
        """
        slope = regression_line.slope
        efficiency = -1 / slope
        efficiency = np.exp( efficiency  )
        efficiency -= 1 
        efficiency = round( efficiency, 4 )
        return efficiency

    def _subset_calibrators(self, names, calibrators, df):
        """
        Generates a dataframe subset containing only calibrator replicates.
        """
        # generate a total query formula for all found calibrators
        q = names[ calibrators ]
        q = "' or group_name == '".join(q)
        q = "group_name == '" + q + "'"
        df = df.query( q )
        return df

    def _infer_dilution_steps(self, df):
        """
        Infers the dilution steps from the group names if they are specified 
        as `calibrator: some_name: dilution` e.g. `calibrator: mysample: 5`. 
        """
        try: 
            # get dilution steps from groupnames in format calibrator: name : dil
            steps = df["group_name"].apply(   lambda x: float( x.split(":")[2] )   ) 
            steps = steps.to_numpy()

            # preprocess to get proper format
            self._dilution( steps )    
            
            # get and transform to log-scale
            dilutions = self._dilution()
            dilutions = np.log(dilutions)
            self._reset_dilution()
            return dilutions 
        except: 
            aw.HardWarning("Calibrator:could_not_infer_dilution")

    def _generate_dilution_steps(self, df):
        """
        Generates a numpy ndarray of log-scaled dilution steps
        for the calibrators.
        """
        # generate steps range
        # we shall use the concept of (dilution)^m to generate 
        # the dilution steps. To get m we use a re-anchored df["group"]
        # column

        self._reset_groups(df)
        steps = df[ "group" ]
        counts = df["group"].value_counts( sort = False )
        repeats = steps.size if isinstance( self._dilution(), float ) else counts

        # repeat the dilution steps to match the group replicate numbers
        dilutions = np.repeat( self._dilution(), repeats ) 

        # if we only had a single value for the dilution we 
        # also need to now scale the dilutions to generate the
        # actual dilution scale. If we already got a dilution series
        # as input, we must not do this as we otherwise doubly scale...
        if isinstance( self._dilution(), float ):
            dilutions = dilutions ** steps
   
        # save dilutions
        self._dilution( dilutions )

        # and transform to log scale 
        dilutions = np.log( dilutions )

        # and reset the diltion back to the what was 
        # originally set (or None from init)
        self._reset_dilution()
        return dilutions

    def _reset_dilution( self ):
        """
        Resets the to the default dilution step to ensure
        the same starting conditions are met for each assay 
        as it is passed through the Calibrator.
        """
        self._dilution_step = self._orig_dilution


    def _reset_groups(self, df):
        """
        Resets the numeric group identifiers to start continuously from 0.
        This method sets the initial group to 0 and then successively resets any
        gaps to match e.g. a 0,1,3 -> 0,1,2...
        """

        # get counts of each group_name in the df
        counts = df["group_name"].value_counts( sort = False )

        # generate new numeric identifiers for each group
        new_groups = np.arange( len(counts) )

        # transform to match the right repeats
        new_groups = np.repeat( new_groups, counts )
        
        # and set new groups
        df["group"] = new_groups

    def _has_calibrator_prefix( self, string ):
        """
        Checks if the calibrator prefix is the start of a string
        The string is a replicate group name in this case...
        """
        calibrator_prefix = defaults.calibrator_prefix
        return string.startswith( calibrator_prefix )

    def _get_efficiency( self, assay : Assay ):
        """
        Returns the efficiency from the currently loaded effiencies
        that corresponds to the assay's Id. Returns None if no match is present.
        """
        id = assay.id()
        effs = self.get()

        if id not in effs.keys():
            return None
        else:
            return effs[ id ]

    def _load(self, filename):
        """
        Loads a csv file but does not adobt the data as its own yet.
        Returns a dictionary.
        """
        # current = json.load( open(filename, "r") )
        current = pd.read_csv( filename )
        current = current.to_dict( "split" )[ "data" ]
        current = { id : eff for id, eff in current }
        return current

    def _save(self, filename, dict_to_save ):
        """
        Saves a dictionary to a csv file.
        """
        # json.dump( dict_to_save , open(filename, "w") )
        df = pd.DataFrame( dict_to_save, index = ["eff"] )
        df = df.transpose().reset_index()
        df.to_csv( filename, index = False )

if __name__ == "__main__":
    
    files = ["./Examples/Example Data/28S.csv", "./Examples/Example Data/actin.csv", "./Examples/Example Data/HNRNPL_nmd.csv", "./Examples/Example Data/HNRNPL_prot.csv"]
    # files = ["Example Data 2/28S.csv", "Example Data 2/actin.csv", "Example Data 2/HNRNPL_nmd.csv", "Example Data 2/HNRNPL_prot.csv"]

    groupnames = ["wt-", "wt+", "ko-", "ko+"]


    reader = DataReader()

    analyser = Analyser()
    analyser.anchor("first")

    normaliser = Normaliser()

    assays = []
    for file in files: 

        assay = reader.read(file, replicates = "6,6,6,6", names = groupnames)
        assay = analyser.pipe(assay)
        assays.append(assay)

    # first to files are normalisers
    normaliser.link(normalisers = assays[:2])

    # last to files are assays
    normaliser.link(assays = assays[2:])

    # def some_prepfunc(normalisers):
    #     first = normalisers[0].get()
    #     first = first["dCt"]
    #     return pd.DataFrame(dict(dCt_combined = first))
    
    # normaliser.prep_func(some_prepfunc)

    normaliser.normalise()

    print(assays[2].id())
    print(assays[2].get())
    
    result = normaliser.get()
    print(result.stats())

    result.preview()
    
    # alternatively we could link the analyser directly 
    # to get the Assay from there like

    # for file in files[2:]:

    #     assay = reader.read(file, replicates = 6)
    #     analyser.pipe(assay)
    #     normaliser1.link(assays = analyser)

    # for file in files[:2]:

    #     assay = reader.read(file, replicates = 6)
    #     analyser.pipe(assay)
    #     normaliser1.link(normalisers = analyser)

    # normaliser1.normalise()
    
    # result1 = normaliser1.get()

    # result1.drop_groups([3, 1])
    # print(result1.stats())

    # print(result.get() == result1.get())

# down here is some perliminary trial at making second
# normalisation... But this should be properly addressed at some point...

#     splitted = result.split(reset_names = False)

#     i, j = splitted
#     i.rename_cols({"HNRNPL_nmd": "HNRNPL"})
#     j.rename_cols({"HNRNPL_prot": "HNRNPL"})

# # print(i, j)
#     # print("-------")
#     # print(i.get()["HNRNPL"] / j.get()["HNRNPL"])
#     # print("-----")
    
    
    
#     sn = Normaliser()

#     sn.link(
#         assays = [splitted[0]], 
#         normalisers = [splitted[1]],
#     )
    
#     sn.normalise(dCt_col = "named", norm_col = "same")

#     print(sn.get().get())
#     print(sn.get().stats())

#     # # result.save("..")
    
#     # #result.add_names(samples)

#     # print(result.stats())


# if __name__ == "__main__":
    
#     srcfile = "__eff_data/Excel eff calculation.xlsx"


#     reader = DataReader()

#     assays, _ = reader.read( srcfile, multi_assay = True, assay_pattern = "Rotor-Gene" )

#     from qpcr.Plotters import EfficiencyCurves

#     plotter = EfficiencyCurves( "static" )

#     calibrator = Calibrator()

#     calibrator.dilution( (1,2,4,8,16,32,64) )

#     assays = [ calibrator.calibrate(i) for i in assays ]

#     print( [ assay.efficiency() for assay in assays ])

#     plotter.link( calibrator )
#     plotter.plot() 
#     print( calibrator.get() )