{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "32d6d560",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from dsc80_utils import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8c8a44b8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.simplefilter(action='ignore', category=FutureWarning)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce9e66d9",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# Lecture 3 – Aggregating\n",
    "\n",
    "## DSC 80, Winter 2025\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbab6fbf",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Announcements 📣\n",
    "\n",
    "- Lab 1 is due **tomorrow at 11:59pm.**\n",
    "    - The [**Welcome Survey**](https://forms.gle/9JdiAnu75D7T7MAu7) is part of your Lab 1; we won't count your Lab 1 as submitted unless your Welcome Survey is also submitted.\n",
    "- Project 1 is released.\n",
    "    - The checkpoint (Questions 1-7) is due on **Friday, Jan 17th**.\n",
    "    - The full project is due on **Friday, Jan 24th**.\n",
    "- Lab 2 will be released tomorrow."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e7a92cd",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Agenda\n",
    "\n",
    "- Data granularity and the `groupby` method.\n",
    "- `DataFrameGroupBy` objects and aggregation.\n",
    "- Other `DataFrameGroupBy` methods.\n",
    "- Pivot tables using the `pivot_table` method.\n",
    "\n",
    "You will need to code **a lot** today – make sure to pull the [course repository](https://github.com/dsc-courses/dsc80-2024-wi)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "642874dc",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "    <h3>Question 🤔 (Answer at <a href=\"https://dsc80.com/q\">dsc80.com/q)</h3>\n",
    "\n",
    "Code: `lec03-hello`\n",
    "\n",
    "**A question for you now: Have you set up your development environment? Have you started on Lab 1?**\n",
    "\n",
    "- A. Yes, I have set up my development environment. I've started Lab 1.\n",
    "- B. Yes, I have set up my development environment. I haven't started Lab 1.\n",
    "- C. No, I am running into difficulties setting up my environment.\n",
    "- D. No, I should really start getting set up!\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e65a8331",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Data granularity and the `groupby` method"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "860cdec9",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Example: Palmer Penguins\n",
    "\n",
    "<center><img src=\"imgs/lter_penguins.png\" width=60%>\n",
    "<i><a href=\"https://github.com/allisonhorst/palmerpenguins/blob/main/README.md\">Artwork by @allison_horst</a></i>\n",
    "\n",
    "</center>\n",
    "\n",
    "The dataset we'll work with for the rest of the lecture involves various measurements taken of three species of penguins in Antarctica."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9961f647",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>species</th>\n",
       "      <th>island</th>\n",
       "      <th>bill_length_mm</th>\n",
       "      <th>bill_depth_mm</th>\n",
       "      <th>flipper_length_mm</th>\n",
       "      <th>body_mass_g</th>\n",
       "      <th>sex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Adelie</td>\n",
       "      <td>Torgersen</td>\n",
       "      <td>39.1</td>\n",
       "      <td>18.7</td>\n",
       "      <td>181.0</td>\n",
       "      <td>3750.0</td>\n",
       "      <td>Male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Adelie</td>\n",
       "      <td>Torgersen</td>\n",
       "      <td>39.5</td>\n",
       "      <td>17.4</td>\n",
       "      <td>186.0</td>\n",
       "      <td>3800.0</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Adelie</td>\n",
       "      <td>Torgersen</td>\n",
       "      <td>40.3</td>\n",
       "      <td>18.0</td>\n",
       "      <td>195.0</td>\n",
       "      <td>3250.0</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>341</th>\n",
       "      <td>Gentoo</td>\n",
       "      <td>Biscoe</td>\n",
       "      <td>50.4</td>\n",
       "      <td>15.7</td>\n",
       "      <td>222.0</td>\n",
       "      <td>5750.0</td>\n",
       "      <td>Male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>342</th>\n",
       "      <td>Gentoo</td>\n",
       "      <td>Biscoe</td>\n",
       "      <td>45.2</td>\n",
       "      <td>14.8</td>\n",
       "      <td>212.0</td>\n",
       "      <td>5200.0</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>343</th>\n",
       "      <td>Gentoo</td>\n",
       "      <td>Biscoe</td>\n",
       "      <td>49.9</td>\n",
       "      <td>16.1</td>\n",
       "      <td>213.0</td>\n",
       "      <td>5400.0</td>\n",
       "      <td>Male</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>333 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    species     island  bill_length_mm  bill_depth_mm  flipper_length_mm  \\\n",
       "0    Adelie  Torgersen            39.1           18.7              181.0   \n",
       "1    Adelie  Torgersen            39.5           17.4              186.0   \n",
       "2    Adelie  Torgersen            40.3           18.0              195.0   \n",
       "..      ...        ...             ...            ...                ...   \n",
       "341  Gentoo     Biscoe            50.4           15.7              222.0   \n",
       "342  Gentoo     Biscoe            45.2           14.8              212.0   \n",
       "343  Gentoo     Biscoe            49.9           16.1              213.0   \n",
       "\n",
       "     body_mass_g     sex  \n",
       "0         3750.0    Male  \n",
       "1         3800.0  Female  \n",
       "2         3250.0  Female  \n",
       "..           ...     ...  \n",
       "341       5750.0    Male  \n",
       "342       5200.0  Female  \n",
       "343       5400.0    Male  \n",
       "\n",
       "[333 rows x 7 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "penguins = sns.load_dataset('penguins').dropna()\n",
    "penguins"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3791f7b7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "Here, each row corresponds to a single penguin, and each column corresponds to a different attribute (or feature) we have for each penguin. Data formatted in this way is called [tidy data](https://r4ds.had.co.nz/tidy-data.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c5a9cbf",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Granularity"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31238e88",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- Granularity refers to what each observation in a dataset represents.\n",
    "    - Fine: small details.\n",
    "    - Coarse: bigger picture."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6af071f0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- If you can control how your dataset is created, you should opt for **finer granularity**, i.e. for more detail.\n",
    "    - You can always remove details, but it's difficult to add detail that isn't already there.\n",
    "    - But obtaining fine-grained data can take more time/money."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "540a8603",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- Today, we'll focus on how to **remove** details from fine-grained data, in order to help us understand bigger-picture trends in our data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5e419eb",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Aggregating\n",
    "\n",
    "**Aggregating** is the act of combining many values into a single value."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e8fe6c7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- What is the mean `'body_mass_g'` for all penguins?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "accb653d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "np.float64(4207.057057057057)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "penguins['body_mass_g'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff0bce29",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- What is the mean `'body_mass_g'` **for each species**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e6bd5ce7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# ???"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60a2b631",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Naive approach: looping through unique values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "50496266",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Adelie       3706.16\n",
       "Chinstrap    3733.09\n",
       "Gentoo       5092.44\n",
       "dtype: float64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "species_map = pd.Series([], dtype=float)\n",
    "\n",
    "for species in penguins['species'].unique():\n",
    "    species_only = penguins.loc[penguins['species'] == species]\n",
    "    species_map.loc[species] = species_only['body_mass_g'].mean()\n",
    "\n",
    "species_map"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19109739",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- For each unique `'species'`, we make a pass through the entire dataset.\n",
    "    - The asymptotic runtime of this procedure is $\\Theta(ns)$, where $n$ is the number of rows and $s$ is the number of unique species."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "960159cd",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- While there are other loop-based solutions that only involve a single pass over the DataFrame, we'd like to avoid Python loops entirely, as they're slow."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5698fac",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Grouping\n",
    "\n",
    "A better solution, as we know from DSC 10, is to use the `groupby` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "c248efba",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "np.float64(4207.057057057057)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Before:\n",
    "penguins['body_mass_g'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "59a1739d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "species\n",
       "Adelie       3706.16\n",
       "Chinstrap    3733.09\n",
       "Gentoo       5092.44\n",
       "Name: body_mass_g, dtype: float64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# After:\n",
    "penguins.groupby('species')['body_mass_g'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad80eded",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "Somehow, the `groupby` method computes what we're looking for in just one line. How?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f8d0708",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### \"Split-apply-combine\" paradigm\n",
    "\n",
    "The `groupby` method involves three steps: **split**, **apply**, and **combine**. This is the same terminology that the [`pandas` documentation](https://pandas.pydata.org/docs/user_guide/groupby.html) uses.\n",
    "\n",
    "<center><img src=\"imgs/image_0.png\" width=40%></center>\n",
    "\n",
    "- **Split** breaks up and \"groups\" the rows of a DataFrame according to the specified **key**. There is one \"group\" for every unique value of the key.\n",
    "\n",
    "- **Apply** uses a function (e.g. aggregation, transformation, filtration) within the individual groups.\n",
    "\n",
    "- **Combine** stitches the results of these operations into an output DataFrame.\n",
    "\n",
    "- The split-apply-combine pattern can be **parallelized** to work on multiple computers or threads, by sending computations for each group to different processors."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cce87043",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### More examples\n",
    "\n",
    "Before we dive into the internals, let's look at a few more examples."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50e5db5f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "    <h3>Question 🤔 (Answer at <a href=\"http://dsc80.com/q\">dsc80.com/q)</h3>\n",
    "\n",
    "Code: `lec03-dream`\n",
    "\n",
    "What proportion of penguins of each `'species'` live on `'Dream'` island?\n",
    "</div>\n",
    "    \n",
    "Your output should look like:\n",
    "    \n",
    "    species\n",
    "    Adelie       0.38\n",
    "    Chinstrap    1.00\n",
    "    Gentoo       0.00"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1df4b98e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Fill this in, then respond on dsc80.com/q"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6bfa96e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## `DataFrameGroupBy` objects and aggregation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d6e2cd3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### `DataFrameGroupBy` objects\n",
    "\n",
    "We've just evaluated a few expressions of the following form."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "14ede341",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "species\n",
       "Adelie       38.82\n",
       "Chinstrap    48.83\n",
       "Gentoo       47.57\n",
       "Name: bill_length_mm, dtype: float64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "penguins.groupby('species')['bill_length_mm'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95ee51b5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "There are two method calls in the expression above: `.groupby('species')` and `.mean()`. What happens in the `.groupby()` call?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "16bdc987",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pandas.core.groupby.generic.DataFrameGroupBy object at 0x152c8f0a0>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "penguins.groupby('species')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "092a96d6",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Peeking under the hood\n",
    "\n",
    "If `df` is a DataFrame, then `df.groupby(key)` returns a `DataFrameGroupBy` object.\n",
    "\n",
    "This object represents the \"split\" in \"split-apply-combine\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "af9540c5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>species</th>\n",
       "      <th>body_mass_g</th>\n",
       "      <th>sex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Adelie</td>\n",
       "      <td>3750.0</td>\n",
       "      <td>Male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>156</th>\n",
       "      <td>Chinstrap</td>\n",
       "      <td>3725.0</td>\n",
       "      <td>Male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>308</th>\n",
       "      <td>Gentoo</td>\n",
       "      <td>4875.0</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Adelie</td>\n",
       "      <td>3800.0</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>258</th>\n",
       "      <td>Gentoo</td>\n",
       "      <td>4350.0</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>157</th>\n",
       "      <td>Chinstrap</td>\n",
       "      <td>3950.0</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>309</th>\n",
       "      <td>Gentoo</td>\n",
       "      <td>5550.0</td>\n",
       "      <td>Male</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       species  body_mass_g     sex\n",
       "0       Adelie       3750.0    Male\n",
       "156  Chinstrap       3725.0    Male\n",
       "308     Gentoo       4875.0  Female\n",
       "1       Adelie       3800.0  Female\n",
       "258     Gentoo       4350.0  Female\n",
       "157  Chinstrap       3950.0  Female\n",
       "309     Gentoo       5550.0    Male"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Simplified DataFrame for demonstration:\n",
    "penguins_small = penguins.iloc[[0, 150, 300, 1, 251, 151, 301], [0, 5, 6]]\n",
    "penguins_small"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b6c3ecfa",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pandas.core.groupby.generic.DataFrameGroupBy object at 0x152c8b310>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Creates one group for each unique value in the species column.\n",
    "penguin_groups = penguins_small.groupby('species')\n",
    "penguin_groups"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beae456a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "`DataFrameGroupBy` objects have a `groups` attribute, which is a dictionary in which the keys are group names and the values are lists of row labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "1526d3ff",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Adelie': [0, 1], 'Chinstrap': [156, 157], 'Gentoo': [308, 258, 309]}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "penguin_groups.groups"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2d52d95",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "`DataFrameGroupBy` objects also have a `get_group(key)` method, which returns a DataFrame with only the values for the given key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "5a8bcea7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>species</th>\n",
       "      <th>body_mass_g</th>\n",
       "      <th>sex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>156</th>\n",
       "      <td>Chinstrap</td>\n",
       "      <td>3725.0</td>\n",
       "      <td>Male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>157</th>\n",
       "      <td>Chinstrap</td>\n",
       "      <td>3950.0</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       species  body_mass_g     sex\n",
       "156  Chinstrap       3725.0    Male\n",
       "157  Chinstrap       3950.0  Female"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "penguin_groups.get_group('Chinstrap')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "6c3f0700",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>species</th>\n",
       "      <th>body_mass_g</th>\n",
       "      <th>sex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>156</th>\n",
       "      <td>Chinstrap</td>\n",
       "      <td>3725.0</td>\n",
       "      <td>Male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>157</th>\n",
       "      <td>Chinstrap</td>\n",
       "      <td>3950.0</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       species  body_mass_g     sex\n",
       "156  Chinstrap       3725.0    Male\n",
       "157  Chinstrap       3950.0  Female"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Same as the above!\n",
    "penguins_small.query('species == \"Chinstrap\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9472556",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "We usually don't use these attributes and methods, but they're useful in understanding how `groupby` works under the hood."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "178de68f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Aggregation\n",
    "\n",
    "- Once we create a `DataFrameGroupBy` object, we need to **apply** some function to each group, and **combine** the results.\n",
    "\n",
    "- The most common operation we apply to each group is an **aggregation**.\n",
    "    - Remember, aggregation is the act of combining many values into a single value.\n",
    "- To perform an aggregation, use an aggregation method on the `DataFrameGroupBy` object, e.g. `.mean()`, `.max()`, or `.median()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecf032ef",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "-"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins_small"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f9c1d82",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins_small.groupby('species')['body_mass_g'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9901725",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Whoa, what happened in the sex column?\n",
    "penguins_small.groupby('species').sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "239f8d70",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins_small.groupby('species').last()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfcfcbf3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins_small.groupby('species').max()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7abccbe",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Column independence\n",
    "\n",
    "Within each group, the aggregation method is applied to **each column independently**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0712c4fe",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins_small.groupby('species').max()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aea963e9",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "It **is not** telling us that there is a `'Male'` `'Adelie'` penguin with a `'body_mass_g'` of `3800.0`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b7bffd6",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# This penguin is Female!\n",
    "penguins_small.loc[(penguins['species'] == 'Adelie') & (penguins['body_mass_g'] == 3800.0)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68ae6955",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "    <h3>Question 🤔 (Answer at <a href=\"http://dsc80.com/q\">dsc80.com/q)</h3>\n",
    "\n",
    "Code: `lec03-heavy`\n",
    "\n",
    "Find the `species`, `island`, and `body_mass_g` of the heaviest `Male` and `Female` penguins in `penguins` (not `penguins_small`).\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58e1eaf1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code goes here."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd3d1b61",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Column selection and performance implications"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f07d4949",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- By default, the aggregator will be applied to **all** columns that it can be applied to.\n",
    "    - `max`, `min`, and `sum` are defined on strings, while `median` and `mean` are not."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94c613db",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- If we only care about one column, we can select that column before aggregating **to save time**.\n",
    "    - `DataFrameGroupBy` objects support `[]` notation, just like `DataFrame`s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1847d896",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Back to the big penguins dataset!\n",
    "penguins"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc7face1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Works, but involves wasted effort since the other columns had to be aggregated for no reason.\n",
    "penguins.groupby('species').sum()['bill_length_mm']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f1b1394",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# This is a SeriesGroupBy object!\n",
    "penguins.groupby('species')['bill_length_mm']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75185169",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Saves time!\n",
    "penguins.groupby('species')['bill_length_mm'].sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "632f5ff4",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "To demonstrate that the former is slower than the latter, we can use `%%timeit`. For reference, we'll also include our earlier `for`-loop-based solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc92adda",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%timeit\n",
    "penguins.groupby('species').sum()['bill_length_mm']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2bb5cc3e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%timeit\n",
    "penguins.groupby('species')['bill_length_mm'].sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc3f9090",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%timeit\n",
    "species_map = pd.Series([], dtype=float)\n",
    "\n",
    "for species in penguins['species'].unique():\n",
    "    species_only = penguins.loc[penguins['species'] == species]\n",
    "    species_map.loc[species] = species_only['body_mass_g'].mean()\n",
    "\n",
    "species_map"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2a190d3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Takeaways\n",
    "\n",
    "- It's important to understand _what_ each piece of your code evaluates to – in the first two timed examples, the code is almost identical, but the performance is quite different.\n",
    "\n",
    "    ```py\n",
    "    # Slower\n",
    "    penguins.groupby('species').sum()['bill_length_mm']\n",
    "\n",
    "    # Faster\n",
    "    penguins.groupby('species')['bill_length_mm'].sum()\n",
    "    ```\n",
    "\n",
    "- The `groupby` method is much quicker than `for`-looping over the DataFrame in Python. It can often produce results using just a **single, fast pass** over the data, updating the sum, mean, count, min, or other aggregate for each group along the way.\n",
    "- You should **always** select the columns you want after `groupby`, unless you really know what you're doing!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc94fa49",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Beyond default aggregation methods\n",
    "\n",
    "- There are many built-in aggregation methods.\n",
    "- What if you want to apply different aggregation methods to different columns?\n",
    "- What if the aggregation method you want to use doesn't already exist in `pandas`?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d50e51cf",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### The `aggregate` method\n",
    "\n",
    "- The `DataFrameGroupBy` object has a general `aggregate` method, which aggregates using one or more operations.\n",
    "    - Remember, aggregation is the act of combining many values into a single value.\n",
    "- There are many ways of using `aggregate`; refer to [the documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.core.groupby.DataFrameGroupBy.aggregate.html) for a comprehensive list.\n",
    "- Example arguments:\n",
    "    - A single function.\n",
    "    - A list of functions.\n",
    "    - A dictionary mapping column names to functions.\n",
    "- Per [the documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.agg.html), `agg` is an alias for `aggregate`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc402b8b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Example\n",
    "\n",
    "How many penguins are there of each `'species'`, and what is the mean `'body_mass_g'` of each `'species'`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f425d5e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "(penguins\n",
    " .groupby('species')\n",
    " ['body_mass_g']\n",
    " .aggregate(['count', 'mean'])\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec2fe848",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Example\n",
    "\n",
    "What is the maximum `'bill_length_mm'` of each `'species'`, and which `'island'`s is each `'species'` found on?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f274f133",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "(penguins\n",
    " .groupby('species')\n",
    " .aggregate({'bill_length_mm': 'max', 'island': 'unique'})\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab8a7443",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Example\n",
    "\n",
    "What is the **interquartile range** of the `'body_mass_g'` of each `'species'`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a969c21",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Here, the argument to agg is a function,\n",
    "# which takes in a pd.Series and returns a scalar.\n",
    "\n",
    "def iqr(s):\n",
    "    return np.percentile(s, 75) - np.percentile(s, 25)\n",
    "\n",
    "(penguins\n",
    " .groupby('species')\n",
    " ['body_mass_g']\n",
    " .agg(iqr)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6c726b3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Other `DataFrameGroupBy` methods"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "478cae01",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Split-apply-combine, revisited\n",
    "\n",
    "When we introduced the split-apply-combine pattern, the \"apply\" step involved **aggregation** – our final DataFrame had one row for each group.\n",
    "\n",
    "<center><img src=\"imgs/image_0.png\" width=40%></center>\n",
    "\n",
    "Instead of aggregating during the apply step, we could instead perform a:\n",
    "\n",
    "- **Transformation**, in which we perform operations to every value within each group.\n",
    "\n",
    "- **Filtration**, in which we keep only the groups that satisfy some condition."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f57a9d0b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Transformations\n",
    "\n",
    "Suppose we want to convert the `'body_mass_g'` column to to z-scores (i.e. standard units):\n",
    "\n",
    "$$z(x_i) = \\frac{x_i - \\text{mean of } x}{\\text{SD of } x}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc70cbb6",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def z_score(x):\n",
    "    return (x - x.mean()) / x.std(ddof=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d600ae5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "z_score(penguins['body_mass_g'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a85c12f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Transformations within groups\n",
    "\n",
    "- Now, what if we wanted the z-score within each group?\n",
    "\n",
    "- To do so, we can use the `transform` method on a `DataFrameGroupBy` object. The `transform` method takes in a function, which itself takes in a Series and returns a new Series.\n",
    "\n",
    "- A transformation produces a DataFrame or Series of the same size – it is **not** an aggregation!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82da9c72-63de-4988-a0bc-371699fb9bbc",
   "metadata": {
    "temporary": true
   },
   "source": [
    "### Heaviest Penguins\n",
    "\n",
    "Question:\n",
    "\n",
    "Find the `species`, `island`, and `body_mass_g` of the heaviest `Male` and `Female` penguins in `penguins` (not `penguins_small`).\n",
    "\n",
    "Answer:\n",
    "```python\n",
    "import pandas as pd\n",
    "\n",
    "penguins.loc[penguins.groupby('sex')['body_mass_g'].idxmax()][['species', 'island', 'body_mass_g']]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6301a61f-b496-46f8-9985-0001786dabe1",
   "metadata": {
    "temporary": true
   },
   "source": [
    "### Transformation Practice\n",
    "\n",
    "Question:\n",
    "\n",
    "Use the `.transform` method to create a new column called `bill_length_mm_zscore` in the `penguins` DataFrame containing the z-scores of `bill_length_mm` for each penguin species.  Display the first 8 rows of the resulting DataFrame.\n",
    "\n",
    "Answer:\n",
    "```python\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "def z_score(x):\n",
    "    return (x - x.mean()) / x.std(ddof=0)\n",
    "bill_length_zscore = (penguins\n",
    "                      .groupby('species')\n",
    "                      ['bill_length_mm']\n",
    "                      .transform(z_score))\n",
    "penguins = penguins.assign(bill_length_mm_zscore = bill_length_zscore)\n",
    "display_df(penguins, rows=8)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a99a25b-a1ea-404e-acd5-89538444815a",
   "metadata": {
    "temporary": true
   },
   "source": [
    "### Transformation Practice\n",
    "\n",
    "Question: \n",
    "Using the `penguins` DataFrame, compute the z-score of the `bill_length_mm` column within each species group.  Assign the result to a new column called `z_bill`. Display the first 8 rows of the resulting DataFrame.\n",
    "Answer: \n",
    "z_bill = (penguins\n",
    "          .groupby('species')\n",
    "          ['bill_length_mm']\n",
    "          .transform(z_score))\n",
    "penguins = penguins.assign(z_bill=z_bill)\n",
    "display_df(penguins, rows=8)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba7859cc-12c4-49b0-a97f-457122c74420",
   "metadata": {
    "temporary": true
   },
   "source": [
    "### Heaviest Penguins\n",
    "\n",
    "Question:\n",
    "Find the `species`, `island`, and `body_mass_g` of the heaviest `Male` and `Female` penguins in `penguins` (not `penguins_small`).\n",
    "Answer:\n",
    "```python\n",
    "penguins.loc[penguins.groupby('sex')['body_mass_g'].idxmax()][['sex', 'species', 'island', 'body_mass_g']]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3872682f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "z_mass = (penguins\n",
    "          .groupby('species')\n",
    "          ['body_mass_g']\n",
    "          .transform(z_score))\n",
    "z_mass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bd014d2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins.assign(z_mass=z_mass)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "499fd16f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "display_df(penguins.assign(z_mass=z_mass), rows=8)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17ace31a",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Note that above, penguin 340 has a larger `'body_mass_g'` than penguin 0, but a lower `'z_mass'`.\n",
    "- Penguin 0 has an above average `'body_mass_g'` among `'Adelie'` penguins.\n",
    "- Penguin 340 has a below average `'body_mass_g'` among `'Gentoo'` penguins. Remember from earlier that the average `'body_mass_g'` of `'Gentoo'` penguins is much higher than for other species."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61b31afc",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins.groupby('species')['body_mass_g'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ff3bc14",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Filtering groups\n",
    "\n",
    "- To keep only the groups that satisfy a particular condition, use the `filter` method on a `DataFrameGroupBy` object.\n",
    "\n",
    "- The `filter` method takes in a function, which itself takes in a DataFrame/Series and return a single Boolean. The result is a new DataFrame/Series with only the groups for which the filter function returned `True`.\n",
    "\n",
    "For example, suppose we want only the `'species'` whose average `'bill_length_mm'` is above 39."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "447eeb69",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "(penguins\n",
    " .groupby('species')\n",
    " .filter(lambda df: df['bill_length_mm'].mean() > 39)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a372cbf2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "No more `'Adelie'`s!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3d21c74",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "Or, as another example, suppose we only want `'species'` with at least 100 penguins:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a88db646",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "(penguins\n",
    " .groupby('species')\n",
    " .filter(lambda df: df.shape[0] > 100)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4891c387",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "No more `'Chinstrap'`s!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5519947a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "    <h3>Question 🤔 (Answer at <a href=\"http://dsc80.com/q\">dsc80.com/q)</h3>\n",
    "\n",
    "Code: `lec03-aggs`\n",
    "\n",
    "Answer the following questions about grouping:\n",
    "\n",
    "- In `.agg(fn)`, what is the input to `fn`? What is the output of `fn`?\n",
    "- In `.transform(fn)`, what is the input to `fn`? What is the output of `fn`?\n",
    "- In `.filter(fn)`, what is the input to `fn`? What is the output of `fn`?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e376960",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Grouping with multiple columns\n",
    "\n",
    "When we group with multiple columns, one group is created for **every unique combination** of elements in the specified columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72b3489d-b988-4987-bff1-c9a7df126e8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "penguins"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a87917b8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "species_and_island = (\n",
    "    penguins\n",
    "    .groupby(['species', 'island'])\n",
    "    [['bill_length_mm', 'body_mass_g']]\n",
    "    .mean()\n",
    ")\n",
    "species_and_island"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db94be88",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Grouping and indexes\n",
    "\n",
    "- The `groupby` method creates an index based on the specified columns.\n",
    "- When grouping by multiple columns, the resulting DataFrame has a `MultiIndex`.\n",
    "- Advice: When working with a `MultiIndex`, use `reset_index` or set `as_index=False` in `groupby`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ade0a51",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "species_and_island"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "771e57f5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "species_and_island['body_mass_g']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb58b9eb",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "species_and_island.loc['Adelie']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bc553eb",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "species_and_island.loc[('Adelie', 'Torgersen')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca4039ed",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "species_and_island.reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b35fbd1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "(penguins\n",
    " .groupby(['species', 'island'], as_index=False)\n",
    " [['bill_length_mm', 'body_mass_g']]\n",
    " .mean()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ffc9679",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "    <h3>Question 🤔 (Answer at <a href=\"http://dsc80.com/q\">dsc80.com/q)</h3>\n",
    "\n",
    "Code: `lec03-names`\n",
    "\n",
    "Find the most popular `Male` and `Female` baby `Name` for each `Year` in `baby`. **Exclude** `Year`s where there were fewer than 1 million births recorded.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73e04e4b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "baby_path = Path('data') / 'baby.csv'\n",
    "baby = pd.read_csv(baby_path)\n",
    "baby"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dcabd1d5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code goes here."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c92b024d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Pivot tables using the `pivot_table` method"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "191b9073",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Pivot tables: an extension of grouping\n",
    "\n",
    "Pivot tables are a compact way to display tables for humans to read:\n",
    "\n",
    "<table border=\"1\" class=\"dataframe\">\n",
    "  <thead>\n",
    "    <tr style=\"text-align: right;\">\n",
    "      <th>Sex</th>\n",
    "      <th>F</th>\n",
    "      <th>M</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>Year</th>\n",
    "      <th></th>\n",
    "      <th></th>\n",
    "    </tr>\n",
    "  </thead>\n",
    "  <tbody>\n",
    "    <tr>\n",
    "      <th>2018</th>\n",
    "      <td>1698373</td>\n",
    "      <td>1813377</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>2019</th>\n",
    "      <td>1675139</td>\n",
    "      <td>1790682</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>2020</th>\n",
    "      <td>1612393</td>\n",
    "      <td>1721588</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>2021</th>\n",
    "      <td>1635800</td>\n",
    "      <td>1743913</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>2022</th>\n",
    "      <td>1628730</td>\n",
    "      <td>1733166</td>\n",
    "    </tr>\n",
    "  </tbody>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "064305ac",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- Notice that each value in the table is a sum over the counts, split by year and sex.\n",
    "- **You can think of pivot tables as grouping using two columns, then \"pivoting\" one of the group labels into columns.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b6c0de2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### `pivot_table`\n",
    "\n",
    "The `pivot_table` DataFrame method aggregates a DataFrame using two columns. To use it:\n",
    "\n",
    "```py\n",
    "df.pivot_table(index=index_col,\n",
    "               columns=columns_col,\n",
    "               values=values_col,\n",
    "               aggfunc=func)\n",
    "```\n",
    "The resulting DataFrame will have:\n",
    "- One row for every unique value in `index_col`.\n",
    "- One column for every unique value in `columns_col`.\n",
    "- Values determined by applying `func` on values in `values_col`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99835266-8508-49f2-ad90-d9d831c695ce",
   "metadata": {
    "temporary": true
   },
   "source": [
    "### Granularity and the `groupby` Method\n",
    "\n",
    "Question:\n",
    "Explain the concept of data granularity.  Illustrate how the process of aggregation, using the `groupby` method, helps transition from fine-grained to coarse-grained data in the context of the Palmer Penguins dataset.  What are the advantages and disadvantages of working with fine-grained versus coarse-grained data?\n",
    "Answer:\n",
    "Data granularity refers to the level of detail in a dataset. Fine-grained data contains many observations with specific details, while coarse-grained data summarizes information at a higher level.  The Palmer Penguins dataset initially represents a fine-grained view, with each row detailing individual penguins' attributes.  Aggregation, via the `groupby` method, allows us to summarize information, moving towards coarse-grained data. For example, grouping by 'species' and aggregating 'body_mass_g' using `.mean()` provides the average body mass for each penguin species. This coarser representation gives a bigger-picture summary that hides individual penguin data. Advantages of fine-grained data include capturing more detail and allowing for richer analysis.  However, it requires more storage space and can be more complex to manage and analyze.  Coarse-grained data is simpler to visualize and comprehend but may mask important fine-grained variations and potentially miss important information if aggregation is done inappropriately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d891326",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "last_5_years = baby.query('Year >= 2018')\n",
    "last_5_years"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fcb474ed",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "last_5_years.pivot_table(\n",
    "    index='Year',\n",
    "    columns='Sex',\n",
    "    values='Count',\n",
    "    aggfunc='sum',\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e77809b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Look at the similarity to the snippet above!\n",
    "(last_5_years\n",
    " .groupby(['Year', 'Sex'])\n",
    " [['Count']]\n",
    " .sum()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5668bc16",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Example\n",
    "\n",
    "Find the number of penguins per `'island'` and `'species'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1aa031f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "099f7869",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins.pivot_table(\n",
    "    index='species',\n",
    "    columns='island',\n",
    "    values='bill_length_mm', # Choice of column here doesn't actually matter!\n",
    "    aggfunc='count',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3f69a47",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "Note that there is a `NaN` at the intersection of `'Biscoe'` and `'Chinstrap'`, because there were no Chinstrap penguins on Biscoe Island.\n",
    "\n",
    "We can either use the `fillna` method afterwards or the `fill_value` argument to fill in `NaN`s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0867a074",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins.pivot_table(\n",
    "    index='species',\n",
    "    columns='island',\n",
    "    values='bill_length_mm',\n",
    "    aggfunc='count',\n",
    "    fill_value=0,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e56de78",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Reshaping\n",
    "\n",
    "- `pivot_table` reshapes DataFrames from \"long\" to \"wide\".\n",
    "- Other DataFrame reshaping methods:\n",
    "    - `melt`: Un-pivots a DataFrame. Very useful in data cleaning.\n",
    "    - `pivot`: Like `pivot_table`, but doesn't do aggregation.\n",
    "    - `stack`: Pivots multi-level columns to multi-indices.\n",
    "    - `unstack`: Pivots multi-indices to columns.\n",
    "    - Google and the documentation are your friends!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b7a9157-5dfa-48ab-a0d4-756993b8269e",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    <h3>We will most likely end lecture here.</h3>\n",
    "</div>\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8508a69b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Distributions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "244a4c6f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Joint distribution\n",
    "\n",
    "When using `aggfunc='count'`, a pivot table describes the **joint distribution** of two categorical variables. This is also called a **contingency table**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "657cb701",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "counts = penguins.pivot_table(\n",
    "    index='species',\n",
    "    columns='sex',\n",
    "    values='body_mass_g',\n",
    "    aggfunc='count',\n",
    "    fill_value=0\n",
    ")\n",
    "counts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "125b3d37",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "We can normalize the DataFrame by dividing by the total number of penguins. The resulting numbers can be interpreted as **probabilities** that a randomly selected penguin from the dataset belongs to a given combination of species and sex."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef27fe23",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "joint = counts / counts.sum().sum()\n",
    "joint"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bf70932",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Marginal probabilities\n",
    "\n",
    "If we sum over one of the axes, we can compute **marginal probabilities**, i.e. unconditional probabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ae4e508",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "joint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8539e9a6",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Recall, joint.sum(axis=0) sums across the rows,\n",
    "# which computes the sum of the **columns**.\n",
    "joint.sum(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f5a59a1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "joint.sum(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3abcbba0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "For instance, the second Series tells us that a randomly selected penguin has a 0.36 chance of being of species `'Gentoo'`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fb270de",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Conditional probabilities\n",
    "\n",
    "Using `counts`, how might we compute conditional probabilities like $$P(\\text{species } = \\text{\"Adelie\"} \\mid \\text{sex } = \\text{\"Female\"})?$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ba45ff3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "counts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43ad126b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "$$\\begin{align*}\n",
    "P(\\text{species} = c \\mid \\text{sex} = x) &= \\frac{\\# \\: (\\text{species} = c \\text{ and } \\text{sex} = x)}{\\# \\: (\\text{sex} = x)}\n",
    "\\end{align*}$$\n",
    "\n",
    "<details>\n",
    "    <summary>➡️ Click <b>here</b> to see more of a derivation.</summary>\n",
    "$$\\begin{align*}\n",
    "P(\\text{species} = c \\mid \\text{sex} = x) &= \\frac{P(\\text{species} = c \\text{ and } \\text{sex} = x)}{P(\\text{sex = }x)} \\\\\n",
    "&= \\frac{\\frac{\\# \\: (\\text{species } = \\: c \\text{ and } \\text{sex } = \\: x)}{N}}{\\frac{\\# \\: (\\text{sex } = \\: x)}{N}} \\\\\n",
    "&= \\frac{\\# \\: (\\text{species} = c \\text{ and } \\text{sex} = x)}{\\# \\: (\\text{sex} = x)}\n",
    "\\end{align*}$$\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff143cc6",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "**Answer**: To find conditional probabilities of **`'species'` given `'sex'`**, divide by **column sums**. To find conditional probabilities of **`'sex'` given `'species'`**, divide by **row sums**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d373b71c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Conditional probabilities\n",
    "\n",
    "To find conditional probabilities of **`'species'` given `'sex'`**, divide by **column sums**. To find conditional probabilities of **`'sex'` given `'species'`**, divide by **row sums**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13c67c6c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e995c56",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "counts.sum(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08bae3f5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "The conditional distribution of **`'species'` given `'sex'`** is below. Note that in this new DataFrame, the `'Female'` and `'Male'` columns each sum to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac092c84",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "counts / counts.sum(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82ff00a4",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "For instance, the above DataFrame tells us that the probability that a randomly selected penguin is of `'species'` `'Adelie'` **given** that they are of `'sex'` `'Female'` is 0.442424."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1860aca1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "    <h3>Question 🤔 (Answer at <a href=\"http://dsc80.com/q\">dsc80.com/q)</h3>\n",
    "\n",
    "Code: `lec03-dist`\n",
    "\n",
    "Find the conditional distribution of `'sex'` given `'species'`.  \n",
    "\n",
    "**_Hint_**: Use `.T`.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f96daece",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code goes here."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30f5cf4d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Summary, next time"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b500565",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Summary\n",
    "\n",
    "- Grouping allows us to change the level of granularity in a DataFrame.\n",
    "- Grouping involves three steps – split, apply, and combine.\n",
    "    - Usually, what is applied is an aggregation, but it could be a transformation or filtration.\n",
    "- `pivot_table` aggregates data based on two categorical columns, and reshapes the result to be \"wide\" instead of \"long\"."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9c3bd04",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Next time\n",
    "\n",
    "- Simpson's paradox.\n",
    "- Merging.\n",
    "    - Review [this diagram](https://dsc10.com/diagrams/#merging) from DSC 10!\n",
    "- The pitfalls of the `apply` method."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  },
  "livereveal": {
   "scroll": true
  },
  "rise": {
   "transition": "none"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
