{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic example cytometry workflow\n",
    "\n",
    "Welcome to `cytoflow`!  I'm glad you're here.  The following is a heavily commented workflow for importing a few tubes of cytometry data and doing some (very) basic analysis.  The goal is to give you not only a taste of using the library for interactive work, but also some insight into the rationale for the way it's designed the way it is and the way it differs from existing projects.\n",
    "\n",
    "`cytoflow`'s goal is to enable *quantitative, reproducible* cytometry analysis.  Reproducibility between cytometry experiments is poor, due in part to differences in analysis between operators; but if all your analysis is in a `Jupyter` notebook (like this one!), then sharing and reuse of workflows is much easier.\n",
    "\n",
    "Let's look at a very basic experiment, containing only two tubes.  These two tubes contain cells that are expressing fluorescent proteins as the read-out of some cell state.  We'll assume that these two tubes are identical, except that one has been induced with 1 µM of the small molecule inducer Doxycycline (aka 'Dox') and the other tube has been induced with 10 µM Dox.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start by setting up Jupyter's plotting interface, then import the `cytoflow` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# in some buggy versions of the Jupyter notebook, this needs to be in its OWN CELL.\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cytoflow as flow\n",
    "\n",
    "# if your figures are too big or too small, you can scale them by changing matplotlib's DPI\n",
    "import matplotlib\n",
    "matplotlib.rc('figure', dpi = 160)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The central data structure in `cytoflow` is the `Experiment`, which is basically a `pandas.DataFrame` containing the events; its associated metadata (experimental conditions and the like); and some methods to manipulate them.\n",
    "\n",
    "You usually create an `Experiment` using an instance of the `ImportOp` class.  Start by defining two tubes, including their experimental conditions (ie, how much Dox is in each); then give those tubes, and a `dict` specifying the experimental conditions' names and types, to `ImportOp`.  Call the `apply()` function to get back the `Experiment` with all the data in it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "tube1 = flow.Tube(file = 'data/RFP_Well_A3.fcs',\n",
    "                  conditions = {'Dox' : 10.0})\n",
    "tube2 = flow.Tube(file='data/CFP_Well_A4.fcs',\n",
    "                  conditions = {'Dox' : 1.0})\n",
    "\n",
    "import_op = flow.ImportOp(conditions = {'Dox' : 'float'},\n",
    "                          tubes = [tube1, tube2])\n",
    "\n",
    "ex = import_op.apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have an `Experiment` instance, this is the last time you should ever think about tubes or wells.  Rather, think of your experiment as a very large set of single-cell measurements, each of which has some metadata associated with it; in this case, how much Dox the cell was exposed to.  `cytoflow` helps you focus your analysis on how those measurements change as the experimental conditions change, without worrying about what cells were in what tube.\n",
    "\n",
    "---\n",
    "Let's have a quick look at one of the fluorescence channels, `Y2-A`.  Instantiate a `HistogramView` and tell it which channel we're looking at, then call `plot` and pass it the experiment containing the data.  Remember, this is not a single tube, but rather all the data in the `Experiment`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 27196 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "hist = flow.HistogramView()\n",
    "hist.channel = 'Y2-A'\n",
    "hist.plot(ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmmm.  This plot is hard to interpret because most of the data is clustered around 0, in the linear range of the detector's response.  Let's re-plot using a different scale.  My favorite is `logicle`, which has a linear response around 0 and a log range elsewhere.  We specify the plot scale by setting the `scale` attribute of `HistogramView` to `logicle`; other options are `log` and `linear`.\n",
    "\n",
    "The cell below also demonstrates a different way to parameterize the `HistogramView` instance, by passing parameters to the constructor instead of setting the values of the instance's attributes after we make it.  Either way is correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 18816 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "hist = flow.HistogramView(channel = 'Y2-A',\n",
    "                          scale = 'logicle')\n",
    "hist.plot(ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ah, much better.  There is clearly a population of cells around 0 and a population of cells around about 5000.  But!  This is the entire `Experiment` -- is is one of the populations from the low-Dox tube and the other population from the high-Dox tube?\n",
    "\n",
    "Let's see if the histogram is different for the two different concentrations of inducer. `CytoFlow`'s plotting takes inspiration from Trellis plots (eg the lattice package in R): to split the data into subsets, you tell the plotting module which metadata \"facet\" you want to plot in the X, Y and color (hue) axes.\n",
    "\n",
    "This time, we tell `HistogramView` to make a separate plot for each different value of Dox and stack them on top of eachother by saying `yfacet = 'Dox'`; if we wanted the plots side-by-side, we would have said `xfacet = 'Dox'`.  \n",
    "\n",
    "Also note that this time, we don't keep around the `HistogramView` object; if we don't need to re-use it, we can just call `plot` right after the constructor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 43564 characters]\n",
      "text/plain": [
       "<Figure size 720x960 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flow.HistogramView(channel = 'Y2-A',\n",
    "                   scale = 'logicle',\n",
    "                   yfacet = 'Dox').plot(ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed, the two tubes have dramatically different histograms.  We could also plot them on top of eachother with different colors, by using `huefacet` instead of `yfacet`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 25812 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flow.HistogramView(channel = \"Y2-A\",\n",
    "                   scale = \"logicle\",\n",
    "                   huefacet = 'Dox').plot(ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, there's a clear difference between these tubes: one has a substantial population above about ~200 in the Y2-A, and the other doesn't.  What's the proportion of \"high\" cells?  Let's gate out the \"high\" population using a `ThresholdOp`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "thresh = flow.ThresholdOp(name = \"T\",\n",
    "                          channel = \"Y2-A\",\n",
    "                          threshold = 200)\n",
    "\n",
    "ex2 = thresh.apply(ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two important things to note here: first, an `Operation` (such as `ThresholdOp`) *does not operate in place*; instead, you create a `ThresholdOp` object, then call `apply()` to run it on an `Experiment`.  The `apply()` function returns a new `Experiment`.\n",
    "\n",
    "Second, a gate *does not remove events*; it simply *adds additional metadata* to the events already there.  You can see this if we look at the underlying `pandas.DataFrame` for `ex` and `ex2`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         B1-A        B1-H          B1-W   Dox       FSC-A       FSC-H  \\\n",
      "0 -127.094002  257.718353 -63040.300781  10.0  459.962982  437.354553   \n",
      "1  -70.234840  255.798340 -34034.042969  10.0 -267.174652  365.354553   \n",
      "2  -96.471756  313.398346 -41931.687500  10.0 -201.582336  501.354553   \n",
      "3   18.831570  277.669250   8514.489258  10.0  291.259888  447.029755   \n",
      "4  100.882095  255.756256  51291.074219  10.0 -397.168579  354.565308   \n",
      "\n",
      "           FSC-W      HDR-T        SSC-A        SSC-H          SSC-W  \\\n",
      "0  137847.578125   2.018511   840.091370   747.917847  147225.328125   \n",
      "1  -95849.679688  27.451754  3476.902344  3163.917969  144038.046875   \n",
      "2  -52700.828125  32.043865   480.270691   507.917877  123937.437500   \n",
      "3   85399.273438  79.327492  8026.275879  6741.838867  156043.484375   \n",
      "4 -146821.125000  79.731194  7453.750488   609.884277  262143.968750   \n",
      "\n",
      "         V2-A        V2-H           V2-W         Y2-A         Y2-H  \\\n",
      "0   41.593452  240.153854   22701.017578   109.946274   153.630051   \n",
      "1  103.437637  336.153870   40332.058594  5554.108398  4273.629883   \n",
      "2 -271.375580  256.153870 -138860.828125    81.835281   121.630051   \n",
      "3  -26.212378  207.677841  -16543.453125   -54.120304    98.122017   \n",
      "4   44.559933  216.036865   27035.013672   -10.542595   127.326027   \n",
      "\n",
      "            Y2-W  \n",
      "0   93802.468750  \n",
      "1  170344.203125  \n",
      "2   88188.023438  \n",
      "3  -72294.242188  \n",
      "4  -10852.761719  \n",
      "----\n",
      "         B1-A        B1-H          B1-W   Dox       FSC-A       FSC-H  \\\n",
      "0 -127.094002  257.718353 -63040.300781  10.0  459.962982  437.354553   \n",
      "1  -70.234840  255.798340 -34034.042969  10.0 -267.174652  365.354553   \n",
      "2  -96.471756  313.398346 -41931.687500  10.0 -201.582336  501.354553   \n",
      "3   18.831570  277.669250   8514.489258  10.0  291.259888  447.029755   \n",
      "4  100.882095  255.756256  51291.074219  10.0 -397.168579  354.565308   \n",
      "\n",
      "           FSC-W      HDR-T        SSC-A        SSC-H          SSC-W  \\\n",
      "0  137847.578125   2.018511   840.091370   747.917847  147225.328125   \n",
      "1  -95849.679688  27.451754  3476.902344  3163.917969  144038.046875   \n",
      "2  -52700.828125  32.043865   480.270691   507.917877  123937.437500   \n",
      "3   85399.273438  79.327492  8026.275879  6741.838867  156043.484375   \n",
      "4 -146821.125000  79.731194  7453.750488   609.884277  262143.968750   \n",
      "\n",
      "         V2-A        V2-H           V2-W         Y2-A         Y2-H  \\\n",
      "0   41.593452  240.153854   22701.017578   109.946274   153.630051   \n",
      "1  103.437637  336.153870   40332.058594  5554.108398  4273.629883   \n",
      "2 -271.375580  256.153870 -138860.828125    81.835281   121.630051   \n",
      "3  -26.212378  207.677841  -16543.453125   -54.120304    98.122017   \n",
      "4   44.559933  216.036865   27035.013672   -10.542595   127.326027   \n",
      "\n",
      "            Y2-W      T  \n",
      "0   93802.468750  False  \n",
      "1  170344.203125   True  \n",
      "2   88188.023438  False  \n",
      "3  -72294.242188  False  \n",
      "4  -10852.761719  False  \n"
     ]
    }
   ],
   "source": [
    "print(ex.data.head())\n",
    "print(\"----\")\n",
    "print(ex2.data.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In ex2, the `ThresholdOp` added a new column, named `T`, which has the same name as the `ThresholdOp`'s `name` attribute.  It is `True` if the `Y2-A` value is greater than 200, and `False` otherwise.  The `T` column is now a piece metadata *just like the Dox concentration* that we can use to facet the plots.  For example, the following plots different `Dox` concentrations in separate plots stacked on top of eachother (`yfacet = \"Dox\"`), and on each of these plots uses separate colors for the \"low\" and \"high\" populations (`huefacet = 'T'`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 46776 characters]\n",
      "text/plain": [
       "<Figure size 720x960 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flow.HistogramView(channel = \"Y2-A\",\n",
    "                   scale = \"logicle\",\n",
    "                   yfacet = \"Dox\",\n",
    "                   huefacet = 'T').plot(ex2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's also important to note that we can still access the underlying `pandas.Dataframe`: either by looking at the `Experiment.data` attribute, or by referencing a column directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    False\n",
       "1     True\n",
       "2    False\n",
       "3    False\n",
       "4    False\n",
       "5    False\n",
       "6     True\n",
       "7    False\n",
       "8     True\n",
       "9    False\n",
       "Name: T, dtype: bool"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex2[\"T\"].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the data is all stored in a single `pandas.Dataframe`, we can use the `pandas` API, and indeed the rest of the SciPy stack, to ask sophisticated questions of the underlying data.  For example, \"how many events of each Dox concentration were above the threshold?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dox   T    \n",
       "1.0   False    9946\n",
       "      True       54\n",
       "10.0  False    5561\n",
       "      True     4439\n",
       "dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex2.data.groupby(['Dox', 'T']).size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`cytoflow` can answer the same question for us using one of its statistics views.  In `cytoflow`, a *statistic* is a number that summarizes a population; one of the key features\n",
    "of `cytoflow` is that it makes it easy to see how these summary statistics change as your experimental conditions change.\n",
    "\n",
    "Several operations add statistics to an experiment; one of the most straightforward ones is `ChannelStatisticOp`.  It groups the experiment's data by the conditions specified in `by`, then applies `function` to `channel` for each group.\n",
    "\n",
    "In this case, we'll divide up the data into the subgroups `T == True & Dox == 1`; `T == True & Dox == 10`; `T == False & Dox == 1`; and `T == False & Dox == 10`.  Then, we'll apply the function `len` to the `Y2-A` channel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "ex3 = flow.ChannelStatisticOp(name = \"ByDox\",\n",
    "                             channel = \"Y2-A\",\n",
    "                             by = ['T', 'Dox'],\n",
    "                             function = len).apply(ex2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can look at the new statistic: `Experiment.statistics` is a dictionary whose keys are tuples and whose values are the computed statistics (stored as `MultiIndex`ed `pandas.Series`).  The first element in the tuple is the name of the operation that added it, and the second is defined by that operation.  In this case, it's `len`, the name of the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "T      Dox \n",
       "False  1.0     9946\n",
       "       10.0    5561\n",
       "True   1.0       54\n",
       "       10.0    4439\n",
       "Name: ByDox : len, dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex3.statistics[(\"ByDox\", \"len\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also plot statistics using one of the various statistics views, such as `BarChartView`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 26544 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flow.BarChartView(statistic = (\"ByDox\", \"len\"),\n",
    "                  variable = \"Dox\",\n",
    "                  huefacet = 'T').plot(ex3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Statistics are important enough that they get an entire notebook of examples; please see `Statistics.ipynb` for a more in-depth exploration.\n",
    "\n",
    "I hope this makes the semantics of the `cytoflow` package clear.  This was a pretty simplistic toy analysis; for more sophisticated examples, see the other accompanying Jupyter notebooks."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
