{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Yeast Dosage-Response Curve\n",
    "\n",
    "This is data from a real experiment, measuring the dose response of an engineered yeast line to isopentyladenine, or IP.  The yeast is engineered with a basic GFP reporter, and GFP fluorescence is measured after 12 hours, at which time we expect the cells to be at steady-state.\n",
    "\n",
    "(Reference: [Chen et al, Nature Biotech 2005](http://www.nature.com/nbt/journal/v23/n12/abs/nbt1162.html))\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set up the Jupyter `matplotlib` integration, and import the `cytoflow` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# in some buggy versions of the Jupyter notebook, this needs to be in its OWN CELL.\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cytoflow as flow\n",
    "\n",
    "# if your figures are too big or too small, you can scale them by changing matplotlib's DPI\n",
    "import matplotlib\n",
    "matplotlib.rc('figure', dpi = 160)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we set up the `Experiment`.  This is primarily the mapping between the flow data (wells on a microtitre plate, here) and the metadata (experimental conditions, in this case the IP concentration.)\n",
    "\n",
    "For this experiment, I did a serial dilution with three dilutions per log.  Note that I specify that this metadata is of type `log`, which is represented internally as a `numpy` type `float`, but is plotted on a log scale when we go to visualize it.\n",
    "\n",
    "Also note that, in order to keep the size of the source distribution manageable, I've only included 1000 events from each tube in the example files.  The actual data set has 30,000 events from each, which gives a much prettier dose-response curve!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = {\n",
    "    \"Yeast_B1_B01.fcs\" : 5.0,\n",
    "    \"Yeast_B2_B02.fcs\" : 3.75,\n",
    "    \"Yeast_B3_B03.fcs\" : 2.8125,\n",
    "    \"Yeast_B4_B04.fcs\" : 2.109,\n",
    "    \"Yeast_B5_B05.fcs\" : 1.5820,\n",
    "    \"Yeast_B6_B06.fcs\" : 1.1865,\n",
    "    \"Yeast_B7_B07.fcs\" : 0.8899,\n",
    "    \"Yeast_B8_B08.fcs\" : 0.6674,\n",
    "    \"Yeast_B9_B09.fcs\" : 0.5,\n",
    "    \"Yeast_B10_B10.fcs\" : 0.3754,\n",
    "    \"Yeast_B11_B11.fcs\" : 0.2816,\n",
    "    \"Yeast_B12_B12.fcs\" : 0.2112,\n",
    "    \"Yeast_C1_C01.fcs\" : 0.1584,\n",
    "    \"Yeast_C2_C02.fcs\" : 0.1188,\n",
    "    \"Yeast_C3_C03.fcs\" : 0.0892,\n",
    "    \"Yeast_C4_C04.fcs\" : 0.0668,\n",
    "    \"Yeast_C5_C05.fcs\" : 0.05,\n",
    "    \"Yeast_C6_C06.fcs\" : 0.0376,\n",
    "    \"Yeast_C7_C07.fcs\" : 0.0282,\n",
    "    \"Yeast_C8_C08.fcs\" : 0.0211,\n",
    "    \"Yeast_C9_C09.fcs\" : 0.0159\n",
    "}\n",
    "\n",
    "tubes = []\n",
    "for filename, ip in inputs.items():\n",
    "    tubes.append(flow.Tube(file = \"data/\" + filename, conditions = {'IP' : ip}))\n",
    "    \n",
    "ex = flow.ImportOp(conditions = {'IP' : \"float\"},\n",
    "                   tubes = tubes).apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set `logicle` as the default plot scale:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "flow.set_default_scale(\"logicle\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step in most cytometry workflows is to filter out debris and aggregates based on morphological parameters (the forward- and side-scatter measurements.)  So let's have a look at the FSC-A and SSC-A channels.\n",
    "\n",
    "Remember, because we're not specifying otherwise, this is a plot of *all of the data* in the experiment, not a single tube!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/brian/src/cytoflow/cytoflow/utility/logicle_scale.py:307: CytoflowWarning: Channel FSC-A doesn't have any negative data. Try a log transform instead.\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 81096 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flow.ScatterplotView(xchannel = \"FSC-A\",\n",
    "                     ychannel = \"SSC-A\").plot(ex, alpha = 0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oops!  The biexponential scale is really intended for channels that have events around 0, with some negative events; let's use a log scale on both FSC-A and SSC-A in the future.  (The fluorescence channels should be fine on a `logicle` scale, though.)\n",
    "\n",
    "Because this is yeast growing in log phase on a drum roller, the distributions in FSC-A and SSC-A are pretty tight.  Let's fit a 2D gaussian to it, and gate out any data that is more than three standard deviations from the mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/brian/src/cytoflow/cytoflow/operations/gaussian.py:528: RuntimeWarning: invalid value encountered in less_equal\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 83380 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "g = flow.GaussianMixtureOp(name = \"Debris_Filter\",\n",
    "                           channels = [\"FSC-A\", \"SSC-A\"],\n",
    "                           scale = {'FSC-A' : 'log',\n",
    "                                    'SSC-A' : 'log'},\n",
    "                           num_components = 1,\n",
    "                           sigma = 3)\n",
    "g.estimate(ex)\n",
    "g.default_view().plot(ex, alpha = 0.05)\n",
    "\n",
    "ex2 = g.apply(ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One thing to note: if you've read the `Machine Learning` notebook, you know that a `GaussianMixtureOp` adds categorical metadata as well.  In this case, we have a new boolean column named `Debris_Filter_1`, which is `True` if the event is less than `sigma` from the centroid and `False` otherwise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FITC-A</th>\n",
       "      <th>FSC-A</th>\n",
       "      <th>FSC-H</th>\n",
       "      <th>FSC-W</th>\n",
       "      <th>IP</th>\n",
       "      <th>PerCP-Cy5-5-A</th>\n",
       "      <th>SSC-A</th>\n",
       "      <th>SSC-H</th>\n",
       "      <th>SSC-W</th>\n",
       "      <th>Time</th>\n",
       "      <th>Debris_Filter_1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2734.459961</td>\n",
       "      <td>3720.110107</td>\n",
       "      <td>3771.0</td>\n",
       "      <td>64651.589844</td>\n",
       "      <td>5.0</td>\n",
       "      <td>-306.440002</td>\n",
       "      <td>4036.360107</td>\n",
       "      <td>3915.0</td>\n",
       "      <td>67567.531250</td>\n",
       "      <td>2.3</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3316.320068</td>\n",
       "      <td>6256.010254</td>\n",
       "      <td>6119.0</td>\n",
       "      <td>67003.414062</td>\n",
       "      <td>5.0</td>\n",
       "      <td>-185.179993</td>\n",
       "      <td>2631.060059</td>\n",
       "      <td>2519.0</td>\n",
       "      <td>68451.437500</td>\n",
       "      <td>3.2</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3551.320068</td>\n",
       "      <td>5292.209961</td>\n",
       "      <td>4708.0</td>\n",
       "      <td>73668.281250</td>\n",
       "      <td>5.0</td>\n",
       "      <td>-234.059998</td>\n",
       "      <td>3729.919922</td>\n",
       "      <td>3376.0</td>\n",
       "      <td>72406.406250</td>\n",
       "      <td>9.2</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3110.459961</td>\n",
       "      <td>5934.479980</td>\n",
       "      <td>5855.0</td>\n",
       "      <td>66425.640625</td>\n",
       "      <td>5.0</td>\n",
       "      <td>395.739990</td>\n",
       "      <td>3499.619873</td>\n",
       "      <td>3419.0</td>\n",
       "      <td>67081.335938</td>\n",
       "      <td>12.7</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2560.560059</td>\n",
       "      <td>4605.700195</td>\n",
       "      <td>4631.0</td>\n",
       "      <td>65177.964844</td>\n",
       "      <td>5.0</td>\n",
       "      <td>-165.440002</td>\n",
       "      <td>3196.939941</td>\n",
       "      <td>3139.0</td>\n",
       "      <td>66745.671875</td>\n",
       "      <td>13.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        FITC-A        FSC-A   FSC-H         FSC-W   IP  PerCP-Cy5-5-A  \\\n",
       "0  2734.459961  3720.110107  3771.0  64651.589844  5.0    -306.440002   \n",
       "1  3316.320068  6256.010254  6119.0  67003.414062  5.0    -185.179993   \n",
       "2  3551.320068  5292.209961  4708.0  73668.281250  5.0    -234.059998   \n",
       "3  3110.459961  5934.479980  5855.0  66425.640625  5.0     395.739990   \n",
       "4  2560.560059  4605.700195  4631.0  65177.964844  5.0    -165.440002   \n",
       "\n",
       "         SSC-A   SSC-H         SSC-W  Time  Debris_Filter_1  \n",
       "0  4036.360107  3915.0  67567.531250   2.3             True  \n",
       "1  2631.060059  2519.0  68451.437500   3.2             True  \n",
       "2  3729.919922  3376.0  72406.406250   9.2             True  \n",
       "3  3499.619873  3419.0  67081.335938  12.7             True  \n",
       "4  3196.939941  3139.0  66745.671875  13.0             True  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex2.data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's look at the FITC-A channel, which is the one we're interested in.  First of all, just check histograms of the FITC-A distribution in each tube.  Make sure to set the `subset` trait to only include the events that aren't debris!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 188772 characters]\n",
      "text/plain": [
       "<Figure size 2160x3360 with 21 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flow.HistogramView(channel = \"FITC-A\",\n",
    "                   xfacet = \"IP\",\n",
    "                   subset = \"Debris_Filter_1 == True\").plot(ex2, col_wrap = 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sure enough, looking at the plots, it's clear that the histogram peak moves to the right as the IP concentration increases.  We can get a more compact representation of the same data with one of `cytoflow`'s fancier plots, a violin plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 84924 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flow.ViolinPlotView(variable = \"IP\",\n",
    "                    channel = \"FITC-A\",\n",
    "                    scale = \"logicle\",\n",
    "                    subset = \"Debris_Filter_1 == True\").plot(ex2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A quick aside - sometimes you get ugly axes because of overlapping labels.  In this case, we want a wider plot.  While we can directly specify the height of a plot, we can't directly specify its width, only its aspect ratio (the ratio of the width to the height.)  `cytoflow` defaults to 1.5; in this case, let's widen it out to 4.  If this results in a plot that's wider than your browser, the Jupyter notebook will scale it down for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 147296 characters]\n",
      "text/plain": [
       "<Figure size 1920x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flow.ViolinPlotView(variable = \"IP\",\n",
    "                    channel = \"FITC-A\",\n",
    "                    scale = \"logicle\",\n",
    "                    subset = \"Debris_Filter_1 == True\").plot(ex2, aspect = 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At least on my screen that's better, but a violin plot is really best for categorical data (with a modest number of categories.)  Instead, let's make a true does-response curve.  The `ChannelStatisticsOp` divides the data into subsets by different values of the `IP` variable and applies `flow.geom_mean` to each subset.  Then, the `Stats1DView` plots that statistic.\n",
    "\n",
    "(Confused?  Have a look at the `Statistics` example notebook for a more in-depth discussion.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 26600 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ex3 = flow.ChannelStatisticOp(name = \"ByIP\",\n",
    "                              channel = \"FITC-A\",\n",
    "                              by = [\"IP\"],\n",
    "                              function = flow.geom_mean,\n",
    "                              subset = \"Debris_Filter_1 == True\").apply(ex2)\n",
    "\n",
    "flow.Stats1DView(statistic = (\"ByIP\", \"geom_mean\"),\n",
    "                 scale = \"log\",\n",
    "                 variable = \"IP\",\n",
    "                 variable_scale = \"log\").plot(ex3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And of course, because the underlying data is just a `pandas.DataFrame`, we can manipulate it with the rest of the Scientific Python ecosystem.  Here, we get the compute the geometric means using `pandas.DataFrame.groupby`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FITC-A</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IP</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0.0159</th>\n",
       "      <td>110.209301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.0211</th>\n",
       "      <td>143.507986</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.0282</th>\n",
       "      <td>157.332377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.0376</th>\n",
       "      <td>184.959273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.0500</th>\n",
       "      <td>250.297256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.0668</th>\n",
       "      <td>413.781993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.0892</th>\n",
       "      <td>678.979138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.1188</th>\n",
       "      <td>955.664252</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.1584</th>\n",
       "      <td>1201.557123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.2112</th>\n",
       "      <td>1164.530618</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.2816</th>\n",
       "      <td>1503.467572</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.3754</th>\n",
       "      <td>1691.126815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.5000</th>\n",
       "      <td>1812.333210</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.6674</th>\n",
       "      <td>2219.928952</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0.8899</th>\n",
       "      <td>2330.923961</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.1865</th>\n",
       "      <td>2423.731503</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.5820</th>\n",
       "      <td>2362.649599</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2.1090</th>\n",
       "      <td>2341.478585</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2.8125</th>\n",
       "      <td>2422.780181</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3.7500</th>\n",
       "      <td>2549.603737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5.0000</th>\n",
       "      <td>2258.993753</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             FITC-A\n",
       "IP                 \n",
       "0.0159   110.209301\n",
       "0.0211   143.507986\n",
       "0.0282   157.332377\n",
       "0.0376   184.959273\n",
       "0.0500   250.297256\n",
       "0.0668   413.781993\n",
       "0.0892   678.979138\n",
       "0.1188   955.664252\n",
       "0.1584  1201.557123\n",
       "0.2112  1164.530618\n",
       "0.2816  1503.467572\n",
       "0.3754  1691.126815\n",
       "0.5000  1812.333210\n",
       "0.6674  2219.928952\n",
       "0.8899  2330.923961\n",
       "1.1865  2423.731503\n",
       "1.5820  2362.649599\n",
       "2.1090  2341.478585\n",
       "2.8125  2422.780181\n",
       "3.7500  2549.603737\n",
       "5.0000  2258.993753"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(ex2.data.query(\"Debris_Filter_1 == True\")[['IP', 'FITC-A']]\n",
    "         .groupby('IP')\n",
    "         .agg(flow.geom_mean))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
