{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Statistics in `cytoflow`\n",
    "\n",
    "One of the most powerful concepts in `cytoflow` is that it makes it easy to summarize your data, then track how those subsets change as your experimental variables change.  This notebook demonstrates several different modules that create and plot statistics.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set up the Jupyter `matplotlib` integration, and import the `cytoflow` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# in some buggy versions of the Jupyter notebook, this needs to be in its OWN CELL.\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cytoflow as flow\n",
    "\n",
    "# if your figures are too big or too small, you can scale them by changing matplotlib's DPI\n",
    "import matplotlib\n",
    "matplotlib.rc('figure', dpi = 160)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the same data set as the **Yeast Dose Response** example notebook, with one variant: we load each tube three times, grabbing only 100 events from each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = {\n",
    "    \"Yeast_B1_B01.fcs\" : 5.0,\n",
    "    \"Yeast_B2_B02.fcs\" : 3.75,\n",
    "    \"Yeast_B3_B03.fcs\" : 2.8125,\n",
    "    \"Yeast_B4_B04.fcs\" : 2.109,\n",
    "    \"Yeast_B5_B05.fcs\" : 1.5820,\n",
    "    \"Yeast_B6_B06.fcs\" : 1.1865,\n",
    "    \"Yeast_B7_B07.fcs\" : 0.8899,\n",
    "    \"Yeast_B8_B08.fcs\" : 0.6674,\n",
    "    \"Yeast_B9_B09.fcs\" : 0.5,\n",
    "    \"Yeast_B10_B10.fcs\" : 0.3754,\n",
    "    \"Yeast_B11_B11.fcs\" : 0.2816,\n",
    "    \"Yeast_B12_B12.fcs\" : 0.2112,\n",
    "    \"Yeast_C1_C01.fcs\" : 0.1584,\n",
    "    \"Yeast_C2_C02.fcs\" : 0.1188,\n",
    "    \"Yeast_C3_C03.fcs\" : 0.0892,\n",
    "    \"Yeast_C4_C04.fcs\" : 0.0668,\n",
    "    \"Yeast_C5_C05.fcs\" : 0.05,\n",
    "    \"Yeast_C6_C06.fcs\" : 0.0376,\n",
    "    \"Yeast_C7_C07.fcs\" : 0.0282,\n",
    "    \"Yeast_C8_C08.fcs\" : 0.0211,\n",
    "    \"Yeast_C9_C09.fcs\" : 0.0159\n",
    "}\n",
    "\n",
    "tubes = []\n",
    "for filename, ip in inputs.items():\n",
    "    tubes.append(flow.Tube(file = \"data/\" + filename, conditions = {'IP' : ip, 'Replicate' : 1}))\n",
    "    tubes.append(flow.Tube(file = \"data/\" + filename, conditions = {'IP' : ip, 'Replicate' : 2}))\n",
    "    tubes.append(flow.Tube(file = \"data/\" + filename, conditions = {'IP' : ip, 'Replicate' : 3}))\n",
    "\n",
    "    \n",
    "ex = flow.ImportOp(conditions = {'IP' : \"float\", \"Replicate\" : \"int\"},\n",
    "                   tubes = tubes,\n",
    "                   events = 100).apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In `cytoflow`, a *statistic* is a value that summarizes something about some data.  `cytoflow` makes it easy to compute statistics for various subsets.  For example, if we expect the geometric mean of `FITC-A` channel to change as the `IP` variable changes, we can compute the geometric mean of the `FITC-A` channel of each different `IP` condition with the `ChannelStatisticOp` operation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "op = flow.ChannelStatisticOp(name = \"ByIP\",\n",
    "                             by = [\"IP\"],\n",
    "                             channel = \"FITC-A\",\n",
    "                             function = flow.geom_mean)\n",
    "ex2 = op.apply(ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This operation splits the data set by different values of `IP`, then applies the function `flow.geom_mean` to the `FITC-A` channel in each subset. The result is stored in the `statistics` attribute of an `Experiment`. The `statistics` attribute is a dictionary.  The keys are tuples, where the first element in the tuple is the name of the operation that created the statistic, and the second element is specific to the operation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys([('ByIP', 'geom_mean')])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex2.statistics.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Statistics1DOp` operation sets the second element of the tuple to the function name.  (You can override this by setting `Statistics1DOp.statistic_name`; this is useful if `function` is a lambda function.)\n",
    "\n",
    "The value of each entry in `Experiment.statistics` is a `pandas.Series`.  The series index is all the subsets for which the statistic was computed, and the contents are the values of the statstic itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IP    \n",
       "0.0159     106.760464\n",
       "0.0211     145.242912\n",
       "0.0282     152.152489\n",
       "0.0376     194.243504\n",
       "0.0500     240.788179\n",
       "0.0668     413.218589\n",
       "0.0892     670.798705\n",
       "0.1188     971.121453\n",
       "0.1584    1345.581190\n",
       "0.2112    1203.016726\n",
       "0.2816    1463.993731\n",
       "0.3754    1733.702533\n",
       "0.5000    1871.968527\n",
       "0.6674    2218.190937\n",
       "0.8899    2324.630935\n",
       "1.1865    2486.103908\n",
       "1.5820    2456.024025\n",
       "2.1090    2326.584475\n",
       "2.8125    2350.038991\n",
       "3.7500    2676.275419\n",
       "5.0000    2269.741514\n",
       "Name: ByIP : geom_mean, dtype: float64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex2.statistics[('ByIP', 'geom_mean')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also specify multiple variables to break data set into.  In the example above, `Statistics1DOp` lumps all events with the same value of `IP` together, but each amount of `IP` actually has three values of `Replicate` as well.  Let's apply `geom_mean` to each unique combination of `IP` and `Replicate`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IP      Replicate\n",
       "0.0159  1            109.294581\n",
       "        2            106.956580\n",
       "        3            104.099598\n",
       "0.0211  1            144.564288\n",
       "        2            147.712897\n",
       "        3            143.484638\n",
       "0.0282  1            155.558697\n",
       "        2            156.508248\n",
       "        3            144.679053\n",
       "0.0376  1            182.294878\n",
       "        2            206.147145\n",
       "        3            195.023849\n",
       "Name: ByIP : geom_mean, dtype: float64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "op = flow.ChannelStatisticOp(name = \"ByIP\",\n",
    "                             by = [\"IP\", \"Replicate\"],\n",
    "                             channel = \"FITC-A\",\n",
    "                             function = flow.geom_mean)\n",
    "ex2 = op.apply(ex)\n",
    "ex2.statistics[(\"ByIP\", \"geom_mean\")][0:12]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the `pandas.Series` now has a `MultiIndex`: there are values for each unique combination of `IP` and `Replicate`.\n",
    "\n",
    "---\n",
    "\n",
    "Now that we have computed a statistic, we can plot it with one of the statistics views.  We can use a bar chart:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "ename": "CytoflowViewError",
     "evalue": "('plot_name', \"You must use facets ['Replicate'] in either the plot variables or the plot name. Possible plot names: [1, 2, 3]\")",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mCytoflowViewError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-8-4102c59fbf66>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m flow.BarChartView(statistic = (\"ByIP\", \"geom_mean\"),\n\u001b[0;32m----> 2\u001b[0;31m                   variable = \"IP\").plot(ex2)\n\u001b[0m",
      "\u001b[0;32m~/src/cytoflow/cytoflow/views/bar_chart.py\u001b[0m in \u001b[0;36mplot\u001b[0;34m(self, experiment, plot_name, **kwargs)\u001b[0m\n\u001b[1;32m    129\u001b[0m         \"\"\"\n\u001b[1;32m    130\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 131\u001b[0;31m         \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mplot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mexperiment\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mplot_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    132\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    133\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_grid_plot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexperiment\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrid\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/src/cytoflow/cytoflow/views/base_views.py\u001b[0m in \u001b[0;36mplot\u001b[0;34m(self, experiment, plot_name, **kwargs)\u001b[0m\n\u001b[1;32m    859\u001b[0m                      \u001b[0mplot_name\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mplot_name\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    860\u001b[0m                      \u001b[0mscale\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mscale\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 861\u001b[0;31m                      **kwargs)\n\u001b[0m\u001b[1;32m    862\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    863\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_make_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexperiment\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/src/cytoflow/cytoflow/views/base_views.py\u001b[0m in \u001b[0;36mplot\u001b[0;34m(self, experiment, data, plot_name, **kwargs)\u001b[0m\n\u001b[1;32m    712\u001b[0m                                              \u001b[0;34m\"plot variables or the plot name. \"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    713\u001b[0m                                              \u001b[0;34m\"Possible plot names: {}\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 714\u001b[0;31m                                              .format(unused_names, list(groupby.groups.keys())))\n\u001b[0m\u001b[1;32m    715\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    716\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mplot_name\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgroupby\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgroups\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkeys\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mCytoflowViewError\u001b[0m: ('plot_name', \"You must use facets ['Replicate'] in either the plot variables or the plot name. Possible plot names: [1, 2, 3]\")"
     ]
    }
   ],
   "source": [
    "flow.BarChartView(statistic = (\"ByIP\", \"geom_mean\"),\n",
    "                  variable = \"IP\").plot(ex2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oops!  We forgot to specify how to plot the different `Replicate`s.  Each index of a statistic must be specified as either a variable or a facet of the plot, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 46256 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flow.BarChartView(statistic = (\"ByIP\", \"geom_mean\"),\n",
    "                  variable = \"IP\",\n",
    "                  huefacet = \"Replicate\").plot(ex2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A quick aside - sometimes you get ugly axes because of overlapping labels.  In this case, we want a wider plot.  While we can directly specify the height of a plot, we can't directly specify its width, only its aspect ratio (the ratio of the width to the height.)  `cytoflow` defaults to 1.5; in this case, let's widen it out to 4.  If this results in a plot that's wider than your browser, the Jupyter notebook will scale it down for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 52056 characters]\n",
      "text/plain": [
       "<Figure size 1920x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flow.BarChartView(statistic = (\"ByIP\", \"geom_mean\"),\n",
    "                  variable = \"IP\",\n",
    "                  huefacet = \"Replicate\").plot(ex2, aspect = 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bar charts are really best for categorical variables (with a modest number of categories.)  Let's do a line chart instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 64436 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flow.Stats1DView(statistic = (\"ByIP\", \"geom_mean\"),\n",
    "                 variable = \"IP\",\n",
    "                 variable_scale = 'log',\n",
    "                 huefacet = \"Replicate\").plot(ex2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Statistics views can also plot error bars; the error bars must *also* be a statistic, and they must have the same indices as the \"main\" statistic.  For example, let's plot the geometric mean and geometric standard deviation of each IP subset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 28164 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ex2 = flow.ChannelStatisticOp(name = \"ByIP\",\n",
    "                              by = [\"IP\"],\n",
    "                              channel = \"FITC-A\",\n",
    "                              function = flow.geom_mean).apply(ex)\n",
    "\n",
    "# While an arithmetic SD is usually plotted plus-or-minus the arithmetic mean, \n",
    "# a *geometric* SD is usually plotted (on a log scale!) multiplied-or-divided by the \n",
    "# geometric mean.  the function geom_sd_range is a convenience function that does this.\n",
    "\n",
    "ex3 = flow.ChannelStatisticOp(name = \"ByIP\",\n",
    "                              by = ['IP'],\n",
    "                              channel = \"FITC-A\",\n",
    "                              function = flow.geom_sd_range).apply(ex2)\n",
    "\n",
    "flow.Stats1DView(statistic = (\"ByIP\", \"geom_mean\"),\n",
    "                 variable = \"IP\",\n",
    "                 variable_scale = \"log\",\n",
    "                 scale = \"log\",\n",
    "                 error_statistic = (\"ByIP\", \"geom_sd_range\")).plot(ex3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The plot above shows how one statistic varies (on the Y axis) as a variable changes on the X axis.  We can also plot two statistics against eachother.  For example, we can ask if the geometric standard deviation varies as the geometric mean changes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 41496 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ex2 = flow.ChannelStatisticOp(name = \"ByIP\",\n",
    "                              by = [\"IP\"],\n",
    "                              channel = \"FITC-A\",\n",
    "                              function = flow.geom_mean).apply(ex)\n",
    "\n",
    "ex3 = flow.ChannelStatisticOp(name = \"ByIP\",\n",
    "                              by = ['IP'],\n",
    "                              channel = \"FITC-A\",\n",
    "                              function = flow.geom_sd).apply(ex2)\n",
    "\n",
    "flow.Stats2DView(variable = \"IP\",\n",
    "                 xstatistic = (\"ByIP\", \"geom_mean\"),\n",
    "                 ystatistic = (\"ByIP\", \"geom_sd\"),\n",
    "                 xscale = \"log\").plot(ex3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nope, guess not.  See the **TASBE Calibrated Flow Cytometry** notebook for more examples of 1D and 2D statistics views.\n",
    "\n",
    "---\n",
    "\n",
    "## Transforming statistics\n",
    "\n",
    "In addition to making statistics by applying summary functions to data, you can also apply functions to other statistics.  For example, a common question is \"What percentage of my events are in a particular gate?\"  We could, for instance, ask what percentage of events are above 1000 in the `FITC-A` channel, and how that varies by amount of `IP`. We start by defining a *threshold* gate with `ThresholdOp`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "thresh = flow.ThresholdOp(name = \"Above1000\",\n",
    "                          channel = \"FITC-A\",\n",
    "                          threshold = 1000)\n",
    "ex2 = thresh.apply(ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, the `Experiment` has a condition named `Above1000` that is `True` or `False` depending on whether that event's `FITC-A` channel is greater than 1000.  Next, we compute the total number of events in each subset with a unique combination of `Above1000` and `IP`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Above1000  IP    \n",
       "False      0.0159    299\n",
       "           0.0211    299\n",
       "           0.0282    299\n",
       "           0.0376    296\n",
       "           0.0500    299\n",
       "           0.0668    262\n",
       "           0.0892    200\n",
       "           0.1188    143\n",
       "           0.1584     76\n",
       "           0.2112    101\n",
       "           0.2816     76\n",
       "           0.3754     57\n",
       "           0.5000     53\n",
       "           0.6674     32\n",
       "           0.8899     19\n",
       "           1.1865     23\n",
       "           1.5820     21\n",
       "           2.1090     34\n",
       "           2.8125     25\n",
       "           3.7500     10\n",
       "           5.0000     29\n",
       "True       0.0159      1\n",
       "           0.0211      1\n",
       "           0.0282      1\n",
       "           0.0376      4\n",
       "           0.0500      1\n",
       "           0.0668     38\n",
       "           0.0892    100\n",
       "           0.1188    157\n",
       "           0.1584    224\n",
       "           0.2112    199\n",
       "           0.2816    224\n",
       "           0.3754    243\n",
       "           0.5000    247\n",
       "           0.6674    268\n",
       "           0.8899    281\n",
       "           1.1865    277\n",
       "           1.5820    279\n",
       "           2.1090    266\n",
       "           2.8125    275\n",
       "           3.7500    290\n",
       "           5.0000    271\n",
       "Name: Above1000 : len, dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex3 = flow.ChannelStatisticOp(name = \"Above1000\",\n",
    "                              by = [\"Above1000\", \"IP\"],\n",
    "                              channel = \"FITC-A\",\n",
    "                              function = len).apply(ex2)\n",
    "ex3.statistics[(\"Above1000\", \"len\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we compute the proportion of `Above1000 == True` for each value of `IP`.  `TransformStatisticOp` applies a function to subsets of a statistic; in this case, we're applying a `lambda` function to convert each `IP` subset from length into proportion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Above1000  IP    \n",
       "False      0.0159    0.996667\n",
       "           0.0211    0.996667\n",
       "           0.0282    0.996667\n",
       "           0.0376    0.986667\n",
       "           0.0500    0.996667\n",
       "           0.0668    0.873333\n",
       "           0.0892    0.666667\n",
       "           0.1188    0.476667\n",
       "Name: Above1000 : len, dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "ex4 = flow.TransformStatisticOp(name = \"Above1000\",\n",
    "                                statistic = (\"Above1000\", \"len\"),\n",
    "                                by = [\"IP\"],\n",
    "                                function = lambda a: pd.Series(a / a.sum()),\n",
    "                                statistic_name = \"proportion\").apply(ex3)\n",
    "\n",
    "ex4.statistics[(\"Above1000\", \"proportion\")][0:8]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Now we can plot the new statistic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 51696 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flow.Stats1DView(statistic = (\"Above1000\", \"proportion\"),\n",
    "                 variable = \"IP\",\n",
    "                 variable_scale = \"log\",\n",
    "                 huefacet = \"Above1000\").plot(ex4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that be default we get all the values in the statistic; in this case, it's both the proportion that's above 1000 (`True`) and the proportion that is below it (`False`).  We can pass a `subset` attribute to the `Stats1DView` to look at just one or the other. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/brian/src/cytoflow/cytoflow/views/base_views.py:751: CytoflowViewWarning: Only one value for level Above1000; dropping it.\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 31864 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flow.Stats1DView(statistic = (\"Above1000\", \"proportion\"),\n",
    "                 variable = \"IP\",\n",
    "                 variable_scale = \"log\",\n",
    "                 subset = \"Above1000 == True\").plot(ex4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the warning.  Because the index level `Above1000` is always `True` in this subset, it gets dropped from the index.  Note also that we don't have to specify it as a variable or facet anywhere.\n",
    "\n",
    "---\n",
    "\n",
    "## Statistics from data-driven modules\n",
    "\n",
    "One of the most exciting aspects of statistics in `cytoflow` is that other data-driven modules can add them to an `Experiment`, too.  For example, the `GaussianMixtureOp` adds several statistics for each component of the mixture model it fits, containing the mean, standard deviation and proportion of observations in each component:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = {\n",
    "    \"Yeast_B1_B01.fcs\" : 5.0,\n",
    "    \"Yeast_B2_B02.fcs\" : 3.75,\n",
    "    \"Yeast_B3_B03.fcs\" : 2.8125,\n",
    "    \"Yeast_B4_B04.fcs\" : 2.109,\n",
    "    \"Yeast_B5_B05.fcs\" : 1.5820,\n",
    "    \"Yeast_B6_B06.fcs\" : 1.1865,\n",
    "    \"Yeast_B7_B07.fcs\" : 0.8899,\n",
    "    \"Yeast_B8_B08.fcs\" : 0.6674,\n",
    "    \"Yeast_B9_B09.fcs\" : 0.5,\n",
    "    \"Yeast_B10_B10.fcs\" : 0.3754,\n",
    "    \"Yeast_B11_B11.fcs\" : 0.2816,\n",
    "    \"Yeast_B12_B12.fcs\" : 0.2112,\n",
    "    \"Yeast_C1_C01.fcs\" : 0.1584,\n",
    "    \"Yeast_C2_C02.fcs\" : 0.1188,\n",
    "    \"Yeast_C3_C03.fcs\" : 0.0892,\n",
    "    \"Yeast_C4_C04.fcs\" : 0.0668,\n",
    "    \"Yeast_C5_C05.fcs\" : 0.05,\n",
    "    \"Yeast_C6_C06.fcs\" : 0.0376,\n",
    "    \"Yeast_C7_C07.fcs\" : 0.0282,\n",
    "    \"Yeast_C8_C08.fcs\" : 0.0211,\n",
    "    \"Yeast_C9_C09.fcs\" : 0.0159\n",
    "}\n",
    "\n",
    "tubes = []\n",
    "for filename, ip in inputs.items():\n",
    "    tubes.append(flow.Tube(file = \"data/\" + filename, conditions = {'IP' : ip}))\n",
    "\n",
    "    \n",
    "ex = flow.ImportOp(conditions = {'IP' : \"float\"},\n",
    "                   tubes = tubes).apply()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 434060 characters]\n",
      "text/plain": [
       "<Figure size 2160x3360 with 21 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "op = flow.GaussianMixtureOp(name = \"Gauss\",\n",
    "                            channels = [\"FITC-A\"],\n",
    "                            scale = {\"FITC-A\" : 'logicle'},\n",
    "                            by = [\"IP\"],\n",
    "                            num_components = 1)\n",
    "op.estimate(ex)\n",
    "ex2 = op.apply(ex)\n",
    "op.default_view(xfacet = \"IP\").plot(ex2, col_wrap = 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys([('Gauss', 'mean'), ('Gauss', 'sigma'), ('Gauss', 'interval')])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex2.statistics.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IP      Component  Channel\n",
       "0.0159  1          FITC-A      114.867593\n",
       "0.0211  1          FITC-A      146.760865\n",
       "0.0282  1          FITC-A      161.831942\n",
       "0.0376  1          FITC-A      188.241811\n",
       "0.0500  1          FITC-A      252.533895\n",
       "0.0668  1          FITC-A      415.359435\n",
       "0.0892  1          FITC-A      675.335896\n",
       "0.1188  1          FITC-A      976.574632\n",
       "0.1584  1          FITC-A     1212.127491\n",
       "0.2112  1          FITC-A     1169.357113\n",
       "0.2816  1          FITC-A     1491.938821\n",
       "0.3754  1          FITC-A     1683.611714\n",
       "0.5000  1          FITC-A     1812.128528\n",
       "0.6674  1          FITC-A     2233.860489\n",
       "0.8899  1          FITC-A     2341.076532\n",
       "1.1865  1          FITC-A     2474.300525\n",
       "1.5820  1          FITC-A     2401.710245\n",
       "2.1090  1          FITC-A     2383.285932\n",
       "2.8125  1          FITC-A     2411.122019\n",
       "3.7500  1          FITC-A     2584.007182\n",
       "5.0000  1          FITC-A     2265.732416\n",
       "Name: Gauss : mean, dtype: float64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex2.statistics[(\"Gauss\", 'mean')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IP      Component  Channel\n",
       "0.0159  1          FITC-A       (56.94459866110697, 224.1783518669721)\n",
       "0.0211  1          FITC-A       (80.36534657785384, 265.7842407147643)\n",
       "0.0282  1          FITC-A       (88.27508462690886, 295.4218184072569)\n",
       "0.0376  1          FITC-A      (101.18497543667104, 350.4182139054138)\n",
       "0.0500  1          FITC-A     (127.95204222704295, 502.06564927968213)\n",
       "0.0668  1          FITC-A       (195.2351392293184, 894.5712089072493)\n",
       "0.0892  1          FITC-A     (310.61832832262314, 1484.6617729462246)\n",
       "0.1188  1          FITC-A       (437.6150673574889, 2200.017570462604)\n",
       "0.1584  1          FITC-A      (542.1024597233911, 2732.4410991530126)\n",
       "0.2112  1          FITC-A      (459.2685949581729, 3011.3302773605374)\n",
       "0.2816  1          FITC-A      (601.1140138003304, 3736.1069195737864)\n",
       "0.3754  1          FITC-A      (784.0312198090705, 3635.8937650247603)\n",
       "0.5000  1          FITC-A       (802.3248292365361, 4117.843046843494)\n",
       "0.6674  1          FITC-A      (1134.2954460750368, 4414.715040960627)\n",
       "0.8899  1          FITC-A      (1246.7333023687233, 4408.716470275269)\n",
       "1.1865  1          FITC-A        (1256.858649306397, 4886.57964348989)\n",
       "1.5820  1          FITC-A      (1151.7297619649949, 5027.751372108821)\n",
       "2.1090  1          FITC-A       (1161.983752683322, 4906.469724482335)\n",
       "2.8125  1          FITC-A     (1276.8361886540924, 4566.1348774856815)\n",
       "3.7500  1          FITC-A      (1277.3600413588333, 5244.721547284582)\n",
       "5.0000  1          FITC-A      (1046.8993940886585, 4925.741101423763)\n",
       "Name: Gauss : interval, dtype: object"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex2.statistics[(\"Gauss\", 'interval')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/brian/src/cytoflow/cytoflow/views/base_views.py:751: CytoflowViewWarning: Only one value for level Component; dropping it.\n",
      "/home/brian/src/cytoflow/cytoflow/views/base_views.py:751: CytoflowViewWarning: Only one value for level Channel; dropping it.\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 39248 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flow.Stats1DView(statistic = (\"Gauss\", \"mean\"),\n",
    "                 variable = \"IP\",\n",
    "                 variable_scale = \"log\",\n",
    "                 scale = \"log\",\n",
    "                 error_statistic = (\"Gauss\", \"interval\")).plot(ex2, shade_error = True)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
