{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning for Flow Cytometry\n",
    "\n",
    "One of the directions `cytoflow` is going in that I'm most excited about is the application of advanced machine learning methods to flow cytometry analysis.  After all, cytometry data is just a high-dimensional data set with many data points: making sense of it can take advantage of some of the sophisticated methods that have seen great success with other high-throughput biological data (such as microarrays.)\n",
    "\n",
    "The following notebook demonstrates a heavily-commented machine learning method, Gaussian Mixture Models, applied to the demo data set we worked with in \"Basic Cytometry.\"  Then, there are briefer examples of some of the other machine learning methods that are implemented in `cytoflow`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set up the Jupyter notebook and import `cytoflow`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# in some buggy versions of the Jupyter notebook, this needs to be in its OWN CELL.\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cytoflow as flow\n",
    "\n",
    "# if your figures are too big or too small, you can scale them by changing matplotlib's DPI\n",
    "import matplotlib\n",
    "matplotlib.rc('figure', dpi = 160)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have two `Tube`s of data that we specify were treated with two different concentrations of the inducer Doxycycline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "tube1 = flow.Tube(file = 'data/RFP_Well_A3.fcs',\n",
    "                  conditions = {\"Dox\" : 10.0})\n",
    "tube2 = flow.Tube(file='data/CFP_Well_A4.fcs',\n",
    "                  conditions = {\"Dox\" : 1.0})\n",
    "\n",
    "import_op = flow.ImportOp(conditions = {\"Dox\" : \"float\"},\n",
    "                          tubes = [tube1, tube2],\n",
    "                          channels = {'V2-A' : 'V2-A',\n",
    "                                      'Y2-A' : 'Y2-A'})\n",
    "\n",
    "ex = import_op.apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the histogram of the `Y2-A` channel (on a `logicle` plot scale)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 18816 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flow.HistogramView(scale = 'logicle',\n",
    "                   channel = 'Y2-A').plot(ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This data looks bimodal to me!  Not perfect Gaussians, but close enough that using a Gaussian Mixture Model will probably let us separate them.\n",
    "\n",
    "Let's use the `GaussianMixtureOp` to separate these two populations.  In operations that are paramterized by data (either an Experiment or some auxilliary FCS file), `cytoflow` separates the estimation of module parameters from their application.  Thus, after instantiating the operation, you call `estimate()` to estimate the model parameters.  Those parameters stay associated with the operation instance in the same way instances of `ThresholdOp` have the gate threshold as an instance attribute.\n",
    "\n",
    "Additionally, many modules, including `GaussianMixtureModelOp`, have a `default_view()` factory method that returns a diagnostic plot so you can check to see that the parameter estimation worked.  This is particularly important for unsupervised learning methods!  In this case, the `GaussianMixtureModelOp`'s `default_view()` returns a `View` that plots a histogram, colored by the component each event was assigned to, and an overlay of the Gaussian distributions on top of the histogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 37184 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "g = flow.GaussianMixtureOp(name = \"Gauss\",\n",
    "                           channels = [\"Y2-A\"],\n",
    "                           scale = {\"Y2-A\" : \"logicle\"},\n",
    "                           num_components = 2)\n",
    "\n",
    "g.estimate(ex)\n",
    "g.default_view().plot(ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Excellent.  It looks like the GMM found the two distributions we were looking for.  Let's call `apply()`, then use the operation's default view to plot the new `Experiment`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/brian/src/cytoflow/cytoflow/operations/base_op_views.py:341: CytoflowViewWarning: Setting 'huefacet' to 'Gauss'\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 44900 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ex2 = g.apply(ex)\n",
    "g.default_view().plot(ex2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you `apply()` the `GaussianMixtureModelOp`, it adds a new piece of metadata to each event in the data set:\n",
    "a condition whose name that's the same as the name of the operation (in this case, `Gauss`).  \n",
    "The values are `{Name}_{#}`, where `{Name}` is the name of the operation and `{#}` is which population had the  \n",
    "highest posterior probability.  So, in this example the events would be labeled `Gauss_1` or `Gauss_2`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Dox</th>\n",
       "      <th>V2-A</th>\n",
       "      <th>Y2-A</th>\n",
       "      <th>Gauss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10.0</td>\n",
       "      <td>41.593452</td>\n",
       "      <td>109.946274</td>\n",
       "      <td>Gauss_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10.0</td>\n",
       "      <td>103.437637</td>\n",
       "      <td>5554.108398</td>\n",
       "      <td>Gauss_2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10.0</td>\n",
       "      <td>-271.375580</td>\n",
       "      <td>81.835281</td>\n",
       "      <td>Gauss_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10.0</td>\n",
       "      <td>-26.212378</td>\n",
       "      <td>-54.120304</td>\n",
       "      <td>Gauss_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10.0</td>\n",
       "      <td>44.559933</td>\n",
       "      <td>-10.542595</td>\n",
       "      <td>Gauss_1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Dox        V2-A         Y2-A    Gauss\n",
       "0  10.0   41.593452   109.946274  Gauss_1\n",
       "1  10.0  103.437637  5554.108398  Gauss_2\n",
       "2  10.0 -271.375580    81.835281  Gauss_1\n",
       "3  10.0  -26.212378   -54.120304  Gauss_1\n",
       "4  10.0   44.559933   -10.542595  Gauss_1"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex2.data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use that new condition to plot or compute or otherwise operate on each of the populations separately:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 26448 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flow.HistogramView(channel = \"Y2-A\",\n",
    "                   scale = \"logicle\",\n",
    "                   huefacet = \"Gauss\").plot(ex2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's an important subtlety to notice here.  In the plot above, we set the data scale on the `HistogramView`, but prior to that we passed `scale = {\"Y2-A\" : \"logicle\"}` to the `GaussianMixtureOp` *operation*.  We did so in order to fit the gaussian model to the *scaled data*, as opposed to the raw data.  \n",
    "\n",
    "This is an example of a broader design goal: in order to enable more quantitative analysis, `cytoflow` does not re-scale the underlying data; rather, it transforms it before displaying it.  Frequently it is useful to perform the same transformation before doing data-driven things, so many modules that have an `estimate()` function also allow you to specify a `scale`.\n",
    "\n",
    "---\n",
    "\n",
    "A 1-dimensional gaussian mixture model works well if the populations are well-separated.  However, if they're closer together, you may only want to keep events that are \"clearly\" in one distribution or another.  One way to accomplish this by passing a `sigma` parameter to `GaussianMixtureOp`.  This doesn't change the behavior of `estimate()`, but when you `apply()` the operation it creates new conditions, one for each population.  The conditions are named `{Name}_{#}`, where `{Name}` is the name of the operation and `{#}` is the index of the\n",
    "population.  The value of the condition is `True` for an event if that event is within `sigma` standard deviations of the population mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 24200 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 24356 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "g = flow.GaussianMixtureOp(name = \"Gauss\",\n",
    "                           channels = [\"Y2-A\"],\n",
    "                           scale = {\"Y2-A\" : \"logicle\"},\n",
    "                           num_components = 2,\n",
    "                           sigma = 1)\n",
    "\n",
    "g.estimate(ex)\n",
    "ex2 = g.apply(ex)\n",
    "\n",
    "flow.HistogramView(channel = \"Y2-A\",\n",
    "                   huefacet = \"Gauss_1\",\n",
    "                   scale = \"logicle\").plot(ex2)\n",
    "\n",
    "flow.HistogramView(channel = \"Y2-A\",\n",
    "                   huefacet = \"Gauss_2\",\n",
    "                   scale = \"logicle\").plot(ex2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, mixtures are very close and separating them is difficult.  In such cases it may be better to filter the events based on the _posterior probability_ that they are actually members of the components to which they were assigned.  We can get this behavior by passing `posterior = True` as a parameter to `GaussianMixture1DOp`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/brian/src/cytoflow/cytoflow/operations/base_op_views.py:341: CytoflowViewWarning: Setting 'huefacet' to 'Gauss'\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 52292 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "g = flow.GaussianMixtureOp(name = \"Gauss\",\n",
    "                           channels = [\"V2-A\"],\n",
    "                           scale = {\"V2-A\" : \"logicle\"},\n",
    "                           num_components = 2,\n",
    "                           posteriors = True)\n",
    "\n",
    "g.estimate(ex)\n",
    "ex2 = g.apply(ex)\n",
    "g.default_view().plot(ex2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If `posteriors = True`, the `GaussianMixtureOp.apply()` adds another metadata column, `{Name}_{#}_Posterior`, that contains the posterior probability of each event in that component."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Dox</th>\n",
       "      <th>V2-A</th>\n",
       "      <th>Y2-A</th>\n",
       "      <th>Gauss</th>\n",
       "      <th>Gauss_1_posterior</th>\n",
       "      <th>Gauss_2_posterior</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10.0</td>\n",
       "      <td>41.593452</td>\n",
       "      <td>109.946274</td>\n",
       "      <td>Gauss_1</td>\n",
       "      <td>0.937407</td>\n",
       "      <td>0.062593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10.0</td>\n",
       "      <td>103.437637</td>\n",
       "      <td>5554.108398</td>\n",
       "      <td>Gauss_1</td>\n",
       "      <td>0.807580</td>\n",
       "      <td>0.192420</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10.0</td>\n",
       "      <td>-271.375580</td>\n",
       "      <td>81.835281</td>\n",
       "      <td>Gauss_1</td>\n",
       "      <td>0.999598</td>\n",
       "      <td>0.000402</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10.0</td>\n",
       "      <td>-26.212378</td>\n",
       "      <td>-54.120304</td>\n",
       "      <td>Gauss_1</td>\n",
       "      <td>0.983376</td>\n",
       "      <td>0.016624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10.0</td>\n",
       "      <td>44.559933</td>\n",
       "      <td>-10.542595</td>\n",
       "      <td>Gauss_1</td>\n",
       "      <td>0.933715</td>\n",
       "      <td>0.066285</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Dox        V2-A         Y2-A    Gauss  Gauss_1_posterior  \\\n",
       "0  10.0   41.593452   109.946274  Gauss_1           0.937407   \n",
       "1  10.0  103.437637  5554.108398  Gauss_1           0.807580   \n",
       "2  10.0 -271.375580    81.835281  Gauss_1           0.999598   \n",
       "3  10.0  -26.212378   -54.120304  Gauss_1           0.983376   \n",
       "4  10.0   44.559933   -10.542595  Gauss_1           0.933715   \n",
       "\n",
       "   Gauss_2_posterior  \n",
       "0           0.062593  \n",
       "1           0.192420  \n",
       "2           0.000402  \n",
       "3           0.016624  \n",
       "4           0.066285  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex2.data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use this second metadata column to filter out events with low posterior probabilities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Dox</th>\n",
       "      <th>V2-A</th>\n",
       "      <th>Y2-A</th>\n",
       "      <th>Gauss</th>\n",
       "      <th>Gauss_1_posterior</th>\n",
       "      <th>Gauss_2_posterior</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10.0</td>\n",
       "      <td>41.593452</td>\n",
       "      <td>109.946274</td>\n",
       "      <td>Gauss_1</td>\n",
       "      <td>0.937407</td>\n",
       "      <td>0.062593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10.0</td>\n",
       "      <td>-271.375580</td>\n",
       "      <td>81.835281</td>\n",
       "      <td>Gauss_1</td>\n",
       "      <td>0.999598</td>\n",
       "      <td>0.000402</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10.0</td>\n",
       "      <td>-26.212378</td>\n",
       "      <td>-54.120304</td>\n",
       "      <td>Gauss_1</td>\n",
       "      <td>0.983376</td>\n",
       "      <td>0.016624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10.0</td>\n",
       "      <td>44.559933</td>\n",
       "      <td>-10.542595</td>\n",
       "      <td>Gauss_1</td>\n",
       "      <td>0.933715</td>\n",
       "      <td>0.066285</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10.0</td>\n",
       "      <td>38.142925</td>\n",
       "      <td>34.791252</td>\n",
       "      <td>Gauss_1</td>\n",
       "      <td>0.941456</td>\n",
       "      <td>0.058544</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Dox        V2-A        Y2-A    Gauss  Gauss_1_posterior  Gauss_2_posterior\n",
       "0  10.0   41.593452  109.946274  Gauss_1           0.937407           0.062593\n",
       "1  10.0 -271.375580   81.835281  Gauss_1           0.999598           0.000402\n",
       "2  10.0  -26.212378  -54.120304  Gauss_1           0.983376           0.016624\n",
       "3  10.0   44.559933  -10.542595  Gauss_1           0.933715           0.066285\n",
       "4  10.0   38.142925   34.791252  Gauss_1           0.941456           0.058544"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex2.query(\"Gauss_1_posterior > 0.9 | Gauss_2_posterior > 0.9\").data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 23888 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flow.HistogramView(channel = \"V2-A\",\n",
    "                   huefacet = \"Gauss\",\n",
    "                   scale = \"logicle\",\n",
    "                   subset = \"Gauss_1_posterior > 0.9 | Gauss_2_posterior > 0.9\").plot(ex2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, sometimes you don't want to use the same model parameters for your entire data set.  Instead, you want to estimate different parameters for different subsets.  `cytoflow`'s data-driven modules allow you to do so with a `by` parameter, which _aggregates_ subsets before estimating model parameters.  You pass `by` an array of metadata columns, and the module estimates a new model for each unique subset of those metadata.\n",
    "\n",
    "For example, let's look at the `V2-A` channel faceted by `Dox`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 38784 characters]\n",
      "text/plain": [
       "<Figure size 720x960 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flow.HistogramView(channel = \"V2-A\", \n",
    "                   scale = \"logicle\",\n",
    "                   yfacet = \"Dox\").plot(ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's pretty clear that the `Dox == 1.0` condition and the `Dox == 10.0` condition are different; a single 2-component GMM won't fit both of them.  So let's fit a model to each unique value of `Dox`.  Note that `by` takes a *list* of metadata columns; you must pass it a list, even if there's only one element in the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/brian/src/cytoflow/cytoflow/operations/base_op_views.py:341: CytoflowViewWarning: Setting 'huefacet' to 'Gauss'\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 97248 characters]\n",
      "text/plain": [
       "<Figure size 720x960 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "g = flow.GaussianMixtureOp(name = \"Gauss\",\n",
    "                           channels = [\"V2-A\"],\n",
    "                           scale = {\"V2-A\" : \"logicle\"},\n",
    "                           num_components = 2,\n",
    "                           by = [\"Dox\"])\n",
    "\n",
    "g.estimate(ex)\n",
    "ex2 = g.apply(ex)\n",
    "g.default_view(yfacet = \"Dox\").plot(ex2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that the models for the two subsets are substantially different.\n",
    "\n",
    "It is important to note that while the estimated model parameters differ between subsets, it is not currently possible to specify different _module parameters_ across subsets.  For example, you can't specify that the `Dox = 1.0` subset have two GMM components, but `Dox == 10.0` have only one.  If we could _estimate_ the number of components, on the other hand, using (say) an AIC or BIC information criterion, then different subsets could have different numbers of components.  For this kind of unsupervized algorithm, see the `FlowPeaksOp` example below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2D Gaussian Mixture Models\n",
    "\n",
    "Did you notice how we were setting the `channels` attribute of `GaussianMixtureOp` to a one-element list?  That's because `GaussianMixtureOp` will work in any number of dimensions.  Here's a similar workflow in two channels instead of one:\n",
    "\n",
    "---\n",
    "\n",
    "Basic usage, assigning each event to one of the mixture components:  (the isolines in the `default_view()` are 1, 2 and 3 standard deviations away from the mean.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/brian/src/cytoflow/cytoflow/operations/base_op_views.py:341: CytoflowViewWarning: Setting 'huefacet' to 'Gauss'\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 240336 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "g = flow.GaussianMixtureOp(name = \"Gauss\",\n",
    "                           channels = [\"V2-A\", \"Y2-A\"],\n",
    "                           scale = {\"V2-A\" : \"logicle\",\n",
    "                                    \"Y2-A\" : \"logicle\"},\n",
    "                           num_components = 2)\n",
    "g.estimate(ex)\n",
    "ex2 = g.apply(ex)\n",
    "g.default_view().plot(ex2, alpha = 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Subsetting based on standard deviation.  Note: we use the Mahalnobis distance as a multivariate generalization of the number of standard deviations an event is from the mean of the multivariate gaussian."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/brian/src/cytoflow/cytoflow/operations/base_op_views.py:341: CytoflowViewWarning: Setting 'huefacet' to 'Gauss'\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 240336 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 167944 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "g = flow.GaussianMixtureOp(name = \"Gauss\",\n",
    "                           channels = [\"V2-A\", \"Y2-A\"],\n",
    "                           scale = {\"V2-A\" : \"logicle\",\n",
    "                                    \"Y2-A\" : \"logicle\"},\n",
    "                           num_components = 2,\n",
    "                           sigma = 2)\n",
    "\n",
    "g.estimate(ex)\n",
    "ex2 = g.apply(ex)\n",
    "\n",
    "g.default_view().plot(ex2, alpha = 0.1)\n",
    "\n",
    "flow.ScatterplotView(xchannel = \"V2-A\",\n",
    "                     ychannel = \"Y2-A\",\n",
    "                     xscale = \"logicle\",\n",
    "                     yscale = \"logicle\",\n",
    "                     huefacet = \"Gauss\",\n",
    "                     subset = \"Gauss_1 == True | Gauss_2 == True\").plot(ex2, alpha = 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gating based on posterior probabilities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/brian/src/cytoflow/cytoflow/operations/base_op_views.py:341: CytoflowViewWarning: Setting 'huefacet' to 'Gauss'\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 240336 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 197052 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "g = flow.GaussianMixtureOp(name = \"Gauss\",\n",
    "                           channels = [\"V2-A\", \"Y2-A\"],\n",
    "                           scale = {\"V2-A\" : \"logicle\",\n",
    "                                    \"Y2-A\" : \"logicle\"},\n",
    "                           num_components = 2,\n",
    "                           posteriors = True)\n",
    "\n",
    "g.estimate(ex)\n",
    "ex2 = g.apply(ex)\n",
    "\n",
    "g.default_view().plot(ex2, alpha = 0.1)\n",
    "\n",
    "flow.ScatterplotView(xchannel = \"V2-A\",\n",
    "                     ychannel = \"Y2-A\",\n",
    "                     xscale = \"logicle\",\n",
    "                     yscale = \"logicle\",\n",
    "                     huefacet = \"Gauss\",\n",
    "                     subset = \"Gauss_1_posterior > 0.99 | Gauss_2_posterior > 0.99\").plot(ex2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And multiple models for different subsets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/brian/src/cytoflow/cytoflow/operations/base_op_views.py:341: CytoflowViewWarning: Setting 'huefacet' to 'Gauss'\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 366452 characters]\n",
      "text/plain": [
       "<Figure size 720x960 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "g = flow.GaussianMixtureOp(name = \"Gauss\",\n",
    "                           channels = [\"V2-A\", \"Y2-A\"],\n",
    "                           scale = {\"V2-A\" : \"logicle\",\n",
    "                                    \"Y2-A\" : \"logicle\"},\n",
    "                           num_components = 2,\n",
    "                           by = ['Dox'])\n",
    "\n",
    "g.estimate(ex)\n",
    "ex2 = g.apply(ex)\n",
    "\n",
    "g.default_view(yfacet = \"Dox\").plot(ex2, alpha = 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## K-Means\n",
    "\n",
    "A gaussian mixture model can find clustered data, but it works best on clusters that are shaped like gaussian distributions (duh.)  K-means clustering is much more general.  As with the GMM operation, `cytoflow`'s K-means operation is N-dimensional and is parameterized very similarly.  Here's a demonstration in 2D."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/brian/src/cytoflow/cytoflow/operations/base_op_views.py:341: CytoflowViewWarning: Setting 'huefacet' to 'KMeans'\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 323368 characters]\n",
      "text/plain": [
       "<Figure size 720x960 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "k = flow.KMeansOp(name = \"KMeans\",\n",
    "                  channels = [\"V2-A\", \"Y2-A\"],\n",
    "                  scale = {\"V2-A\" : \"logicle\",\n",
    "                           \"Y2-A\" : \"logicle\"},\n",
    "                  num_clusters = 2,\n",
    "                  by = ['Dox'])\n",
    "\n",
    "k.estimate(ex)\n",
    "ex2 = k.apply(ex)\n",
    "k.default_view(yfacet = \"Dox\").plot(ex2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how the centroids are marked with blue stars.  The locations of the centroids are also saved as a new statistic:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{('KMeans', 'centers'): Dox   Cluster  Channel\n",
       " 1.0   1        V2-A        420.424332\n",
       "                Y2-A         29.304036\n",
       "       2        V2-A          1.276445\n",
       "                Y2-A         26.576759\n",
       " 10.0  1        V2-A         23.096459\n",
       "                Y2-A         14.118368\n",
       "       2        V2-A        127.770348\n",
       "                Y2-A       5113.999433\n",
       " dtype: float64}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex2.statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FlowPeaks\n",
    "\n",
    "Sometimes you want to cluster a data set and K-means just doesn't work.  (It generally works best when clusters are fairly compact.)  In such cases, the unsupervized clustering algorith `flowPeaks` may work better for you.  For example, the following FCS file (of an E. coli experiment) shows clear separation between the cells (upper population) and the particulate matter in the media (lower population.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "ex = flow.ImportOp(tubes = [flow.Tube(file = 'data/ecoli.fcs')]).apply()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 58652 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flow.ScatterplotView(xchannel = \"FSC-A\",\n",
    "                     xscale = 'log',\n",
    "                     ychannel = \"FSC-H\",\n",
    "                     yscale = 'log').plot(ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, K-means (even with more clusters than I want) isn't finding them properly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/brian/src/cytoflow/cytoflow/operations/base_op_views.py:341: CytoflowViewWarning: Setting 'huefacet' to 'KMeans'\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 77528 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "k = flow.KMeansOp(name = \"KMeans\",\n",
    "                  channels = [\"FSC-A\", \"FSC-H\"],\n",
    "                  scale = {\"FSC-A\" : \"log\",\n",
    "                           \"FSC-H\" : \"log\"},\n",
    "                  num_clusters = 3)\n",
    "\n",
    "k.estimate(ex)\n",
    "ex2 = k.apply(ex)\n",
    "k.default_view().plot(ex2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`flowPeaks` is an unsupervized learning strategy developed specifically for flow cytometry, and it works much better here.  It can automatically determine the number of \"natural\" clusters in a data set.  It is also much more computationally expensive -- if you have a large data set, be prepared to wait for a while!\n",
    "\n",
    "Another note -- there are a number of hyperparameters for this method that can dramatically change how it operates.  The defaults are good for many data sets; for the one below, I had to decrease `h0` from `1.0` to `0.5`.  See the documentation for details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/brian/src/cytoflow/cytoflow/operations/base_op_views.py:341: CytoflowViewWarning: Setting 'huefacet' to 'FP'\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 78728 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 64868 characters]\n",
      "text/plain": [
       "<Figure size 720x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fp = flow.FlowPeaksOp(name = \"FP\",\n",
    "                      channels = [\"FSC-A\", \"FSC-H\"],\n",
    "                      scale = {\"FSC-A\" : \"log\",\n",
    "                               \"FSC-H\" : \"log\"},\n",
    "                      h0 = 0.5)\n",
    "\n",
    "fp.estimate(ex)\n",
    "ex2 = fp.apply(ex)\n",
    "fp.default_view().plot(ex2)\n",
    "flow.ScatterplotView(xchannel = \"FSC-A\",\n",
    "                     xscale = \"log\",\n",
    "                     ychannel = \"FSC-H\",\n",
    "                     yscale = \"log\",\n",
    "                     huefacet = \"FP\").plot(ex2)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
