{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "32d6d560",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from dsc80_utils import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8c8a44b8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.simplefilter(action='ignore', category=FutureWarning)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce9e66d9",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# Lecture 3 – Aggregating\n",
    "\n",
    "## DSC 80, Winter 2025\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbab6fbf",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Announcements 📣\n",
    "\n",
    "- Lab 1 is due **tomorrow at 11:59pm.**\n",
    "    - The [**Welcome Survey**](https://forms.gle/9JdiAnu75D7T7MAu7) is part of your Lab 1; we won't count your Lab 1 as submitted unless your Welcome Survey is also submitted.\n",
    "- Project 1 is released.\n",
    "    - The checkpoint (Questions 1-7) is due on **Friday, Jan 17th**.\n",
    "    - The full project is due on **Friday, Jan 24th**.\n",
    "- Lab 2 will be released tomorrow."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e7a92cd",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Agenda\n",
    "\n",
    "- Data granularity and the `groupby` method.\n",
    "- `DataFrameGroupBy` objects and aggregation.\n",
    "- Other `DataFrameGroupBy` methods.\n",
    "- Pivot tables using the `pivot_table` method.\n",
    "\n",
    "You will need to code **a lot** today – make sure to pull the [course repository](https://github.com/dsc-courses/dsc80-2024-wi)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "642874dc",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "    <h3>Question 🤔 (Answer at <a href=\"https://dsc80.com/q\">dsc80.com/q)</h3>\n",
    "\n",
    "Code: `lec03-hello`\n",
    "\n",
    "**A question for you now: Have you set up your development environment? Have you started on Lab 1?**\n",
    "\n",
    "- A. Yes, I have set up my development environment. I've started Lab 1.\n",
    "- B. Yes, I have set up my development environment. I haven't started Lab 1.\n",
    "- C. No, I am running into difficulties setting up my environment.\n",
    "- D. No, I should really start getting set up!\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e65a8331",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Data granularity and the `groupby` method"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "860cdec9",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Example: Palmer Penguins\n",
    "\n",
    "<center><img src=\"imgs/lter_penguins.png\" width=60%>\n",
    "<i><a href=\"https://github.com/allisonhorst/palmerpenguins/blob/main/README.md\">Artwork by @allison_horst</a></i>\n",
    "\n",
    "</center>\n",
    "\n",
    "The dataset we'll work with for the rest of the lecture involves various measurements taken of three species of penguins in Antarctica."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9961f647",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "penguins = sns.load_dataset('penguins').dropna()\n",
    "penguins"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3791f7b7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "Here, each row corresponds to a single penguin, and each column corresponds to a different attribute (or feature) we have for each penguin. Data formatted in this way is called [tidy data](https://r4ds.had.co.nz/tidy-data.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c5a9cbf",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Granularity"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31238e88",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- Granularity refers to what each observation in a dataset represents.\n",
    "    - Fine: small details.\n",
    "    - Coarse: bigger picture."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6af071f0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- If you can control how your dataset is created, you should opt for **finer granularity**, i.e. for more detail.\n",
    "    - You can always remove details, but it's difficult to add detail that isn't already there.\n",
    "    - But obtaining fine-grained data can take more time/money."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "540a8603",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- Today, we'll focus on how to **remove** details from fine-grained data, in order to help us understand bigger-picture trends in our data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5e419eb",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Aggregating\n",
    "\n",
    "**Aggregating** is the act of combining many values into a single value."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e8fe6c7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- What is the mean `'body_mass_g'` for all penguins?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "accb653d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins['body_mass_g'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff0bce29",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- What is the mean `'body_mass_g'` **for each species**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6bd5ce7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# ???"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60a2b631",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Naive approach: looping through unique values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50496266",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "species_map = pd.Series([], dtype=float)\n",
    "\n",
    "for species in penguins['species'].unique():\n",
    "    species_only = penguins.loc[penguins['species'] == species]\n",
    "    species_map.loc[species] = species_only['body_mass_g'].mean()\n",
    "\n",
    "species_map"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19109739",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- For each unique `'species'`, we make a pass through the entire dataset.\n",
    "    - The asymptotic runtime of this procedure is $\\Theta(ns)$, where $n$ is the number of rows and $s$ is the number of unique species."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "960159cd",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- While there are other loop-based solutions that only involve a single pass over the DataFrame, we'd like to avoid Python loops entirely, as they're slow."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5698fac",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Grouping\n",
    "\n",
    "A better solution, as we know from DSC 10, is to use the `groupby` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c248efba",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Before:\n",
    "penguins['body_mass_g'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59a1739d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# After:\n",
    "penguins.groupby('species')['body_mass_g'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad80eded",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "Somehow, the `groupby` method computes what we're looking for in just one line. How?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f8d0708",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### \"Split-apply-combine\" paradigm\n",
    "\n",
    "The `groupby` method involves three steps: **split**, **apply**, and **combine**. This is the same terminology that the [`pandas` documentation](https://pandas.pydata.org/docs/user_guide/groupby.html) uses.\n",
    "\n",
    "<center><img src=\"imgs/image_0.png\" width=40%></center>\n",
    "\n",
    "- **Split** breaks up and \"groups\" the rows of a DataFrame according to the specified **key**. There is one \"group\" for every unique value of the key.\n",
    "\n",
    "- **Apply** uses a function (e.g. aggregation, transformation, filtration) within the individual groups.\n",
    "\n",
    "- **Combine** stitches the results of these operations into an output DataFrame.\n",
    "\n",
    "- The split-apply-combine pattern can be **parallelized** to work on multiple computers or threads, by sending computations for each group to different processors."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b1a7169-1af5-43c7-abfd-81f926367d90",
   "metadata": {
    "temporary": true
   },
   "source": [
    "### Heaviest Penguins\n",
    "\n",
    "Question:\n",
    "Find the `species`, `island`, and `body_mass_g` of the heaviest `Male` and `Female` penguins in `penguins` (not `penguins_small`).\n",
    "Answer:\n",
    "```python\n",
    "penguins.loc[penguins.groupby('sex')['body_mass_g'].idxmax()][['sex', 'species', 'island', 'body_mass_g']]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cce87043",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### More examples\n",
    "\n",
    "Before we dive into the internals, let's look at a few more examples."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50e5db5f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "    <h3>Question 🤔 (Answer at <a href=\"http://dsc80.com/q\">dsc80.com/q)</h3>\n",
    "\n",
    "Code: `lec03-dream`\n",
    "\n",
    "What proportion of penguins of each `'species'` live on `'Dream'` island?\n",
    "</div>\n",
    "    \n",
    "Your output should look like:\n",
    "    \n",
    "    species\n",
    "    Adelie       0.38\n",
    "    Chinstrap    1.00\n",
    "    Gentoo       0.00"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1df4b98e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Fill this in, then respond on dsc80.com/q"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6bfa96e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## `DataFrameGroupBy` objects and aggregation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d6e2cd3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### `DataFrameGroupBy` objects\n",
    "\n",
    "We've just evaluated a few expressions of the following form."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14ede341",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins.groupby('species')['bill_length_mm'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95ee51b5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "There are two method calls in the expression above: `.groupby('species')` and `.mean()`. What happens in the `.groupby()` call?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16bdc987",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins.groupby('species')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "092a96d6",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Peeking under the hood\n",
    "\n",
    "If `df` is a DataFrame, then `df.groupby(key)` returns a `DataFrameGroupBy` object.\n",
    "\n",
    "This object represents the \"split\" in \"split-apply-combine\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af9540c5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Simplified DataFrame for demonstration:\n",
    "penguins_small = penguins.iloc[[0, 150, 300, 1, 251, 151, 301], [0, 5, 6]]\n",
    "penguins_small"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6c3ecfa",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Creates one group for each unique value in the species column.\n",
    "penguin_groups = penguins_small.groupby('species')\n",
    "penguin_groups"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beae456a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "`DataFrameGroupBy` objects have a `groups` attribute, which is a dictionary in which the keys are group names and the values are lists of row labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1526d3ff",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguin_groups.groups"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2d52d95",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "`DataFrameGroupBy` objects also have a `get_group(key)` method, which returns a DataFrame with only the values for the given key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a8bcea7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguin_groups.get_group('Chinstrap')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c3f0700",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Same as the above!\n",
    "penguins_small.query('species == \"Chinstrap\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9472556",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "We usually don't use these attributes and methods, but they're useful in understanding how `groupby` works under the hood."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "178de68f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Aggregation\n",
    "\n",
    "- Once we create a `DataFrameGroupBy` object, we need to **apply** some function to each group, and **combine** the results.\n",
    "\n",
    "- The most common operation we apply to each group is an **aggregation**.\n",
    "    - Remember, aggregation is the act of combining many values into a single value.\n",
    "- To perform an aggregation, use an aggregation method on the `DataFrameGroupBy` object, e.g. `.mean()`, `.max()`, or `.median()`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45789d69",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "Let's look at some examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecf032ef",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "-"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins_small"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f9c1d82",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins_small.groupby('species')['body_mass_g'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9901725",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Whoa, what happened in the sex column?\n",
    "penguins_small.groupby('species').sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "239f8d70",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins_small.groupby('species').last()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfcfcbf3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins_small.groupby('species').max()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7abccbe",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Column independence\n",
    "\n",
    "Within each group, the aggregation method is applied to **each column independently**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0712c4fe",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins_small.groupby('species').max()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aea963e9",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "It **is not** telling us that there is a `'Male'` `'Adelie'` penguin with a `'body_mass_g'` of `3800.0`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b7bffd6",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# This penguin is Female!\n",
    "penguins_small.loc[(penguins['species'] == 'Adelie') & (penguins['body_mass_g'] == 3800.0)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68ae6955",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "    <h3>Question 🤔 (Answer at <a href=\"http://dsc80.com/q\">dsc80.com/q)</h3>\n",
    "\n",
    "Code: `lec03-heavy`\n",
    "\n",
    "Find the `species`, `island`, and `body_mass_g` of the heaviest `Male` and `Female` penguins in `penguins` (not `penguins_small`).\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58e1eaf1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code goes here."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd3d1b61",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Column selection and performance implications"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f07d4949",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- By default, the aggregator will be applied to **all** columns that it can be applied to.\n",
    "    - `max`, `min`, and `sum` are defined on strings, while `median` and `mean` are not."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94c613db",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- If we only care about one column, we can select that column before aggregating **to save time**.\n",
    "    - `DataFrameGroupBy` objects support `[]` notation, just like `DataFrame`s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1847d896",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Back to the big penguins dataset!\n",
    "penguins"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc7face1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Works, but involves wasted effort since the other columns had to be aggregated for no reason.\n",
    "penguins.groupby('species').sum()['bill_length_mm']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f1b1394",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# This is a SeriesGroupBy object!\n",
    "penguins.groupby('species')['bill_length_mm']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75185169",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Saves time!\n",
    "penguins.groupby('species')['bill_length_mm'].sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "632f5ff4",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "To demonstrate that the former is slower than the latter, we can use `%%timeit`. For reference, we'll also include our earlier `for`-loop-based solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc92adda",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%timeit\n",
    "penguins.groupby('species').sum()['bill_length_mm']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2bb5cc3e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%timeit\n",
    "penguins.groupby('species')['bill_length_mm'].sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc3f9090",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%timeit\n",
    "species_map = pd.Series([], dtype=float)\n",
    "\n",
    "for species in penguins['species'].unique():\n",
    "    species_only = penguins.loc[penguins['species'] == species]\n",
    "    species_map.loc[species] = species_only['body_mass_g'].mean()\n",
    "\n",
    "species_map"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2a190d3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Takeaways\n",
    "\n",
    "- It's important to understand _what_ each piece of your code evaluates to – in the first two timed examples, the code is almost identical, but the performance is quite different.\n",
    "\n",
    "    ```py\n",
    "    # Slower\n",
    "    penguins.groupby('species').sum()['bill_length_mm']\n",
    "\n",
    "    # Faster\n",
    "    penguins.groupby('species')['bill_length_mm'].sum()\n",
    "    ```\n",
    "\n",
    "- The `groupby` method is much quicker than `for`-looping over the DataFrame in Python. It can often produce results using just a **single, fast pass** over the data, updating the sum, mean, count, min, or other aggregate for each group along the way.\n",
    "- You should **always** select the columns you want after `groupby`, unless you really know what you're doing!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc94fa49",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Beyond default aggregation methods\n",
    "\n",
    "- There are many built-in aggregation methods.\n",
    "- What if you want to apply different aggregation methods to different columns?\n",
    "- What if the aggregation method you want to use doesn't already exist in `pandas`?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d50e51cf",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### The `aggregate` method\n",
    "\n",
    "- The `DataFrameGroupBy` object has a general `aggregate` method, which aggregates using one or more operations.\n",
    "    - Remember, aggregation is the act of combining many values into a single value.\n",
    "- There are many ways of using `aggregate`; refer to [the documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.core.groupby.DataFrameGroupBy.aggregate.html) for a comprehensive list.\n",
    "- Example arguments:\n",
    "    - A single function.\n",
    "    - A list of functions.\n",
    "    - A dictionary mapping column names to functions.\n",
    "- Per [the documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.agg.html), `agg` is an alias for `aggregate`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbba0933-2045-4755-9307-ed42f41da644",
   "metadata": {
    "temporary": true
   },
   "source": [
    "### Proportion of Penguins on Dream Island\n",
    "\n",
    "Question:\n",
    "What proportion of penguins of each `'species'` live on `'Dream'` island?\n",
    "Answer:\n",
    "```python\n",
    "penguins.groupby(['species', 'island']).size().unstack().fillna(0).apply(lambda x: x/x.sum(), axis=1).loc['Dream']\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc402b8b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Example\n",
    "\n",
    "How many penguins are there of each `'species'`, and what is the mean `'body_mass_g'` of each `'species'`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f425d5e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "(penguins\n",
    " .groupby('species')\n",
    " ['body_mass_g']\n",
    " .aggregate(['count', 'mean'])\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec2fe848",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Example\n",
    "\n",
    "What is the maximum `'bill_length_mm'` of each `'species'`, and which `'island'`s is each `'species'` found on?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f274f133",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "(penguins\n",
    " .groupby('species')\n",
    " .aggregate({'bill_length_mm': 'max', 'island': 'unique'})\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab8a7443",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Example\n",
    "\n",
    "What is the **interquartile range** of the `'body_mass_g'` of each `'species'`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1a969c21",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'penguins' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[1], line 7\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21miqr\u001b[39m(s):\n\u001b[1;32m      5\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m np\u001b[38;5;241m.\u001b[39mpercentile(s, \u001b[38;5;241m75\u001b[39m) \u001b[38;5;241m-\u001b[39m np\u001b[38;5;241m.\u001b[39mpercentile(s, \u001b[38;5;241m25\u001b[39m)\n\u001b[0;32m----> 7\u001b[0m (\u001b[43mpenguins\u001b[49m\n\u001b[1;32m      8\u001b[0m  \u001b[38;5;241m.\u001b[39mgroupby(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mspecies\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m      9\u001b[0m  [\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mbody_mass_g\u001b[39m\u001b[38;5;124m'\u001b[39m]\n\u001b[1;32m     10\u001b[0m  \u001b[38;5;241m.\u001b[39magg(iqr)\n\u001b[1;32m     11\u001b[0m )\n",
      "\u001b[0;31mNameError\u001b[0m: name 'penguins' is not defined"
     ]
    }
   ],
   "source": [
    "# Here, the argument to agg is a function,\n",
    "# which takes in a pd.Series and returns a scalar.\n",
    "\n",
    "def iqr(s):\n",
    "    return np.percentile(s, 75) - np.percentile(s, 25)\n",
    "\n",
    "(penguins\n",
    " .groupby('species')\n",
    " ['body_mass_g']\n",
    " .agg(iqr)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6d04b89-dea9-4bc5-8b4a-4c01f455f1f7",
   "metadata": {
    "temporary": true
   },
   "source": [
    "### Calculating the range of body mass for penguin species\n",
    "\n",
    "Question:\n",
    "\n",
    "Write a function that calculates the range (maximum value minus minimum value) of a Pandas Series. Then, use this function with the `.agg` method to calculate the range of body mass for each penguin species.\n",
    "\n",
    "Answer:\n",
    "```python\n",
    "def range_fn(s):\n",
    "    return s.max() - s.min()\n",
    "\n",
    "(penguins\n",
    " .groupby('species')\n",
    " ['body_mass_g']\n",
    " .agg(range_fn)\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6c726b3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Other `DataFrameGroupBy` methods"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "478cae01",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Split-apply-combine, revisited\n",
    "\n",
    "When we introduced the split-apply-combine pattern, the \"apply\" step involved **aggregation** – our final DataFrame had one row for each group.\n",
    "\n",
    "<center><img src=\"imgs/image_0.png\" width=40%></center>\n",
    "\n",
    "Instead of aggregating during the apply step, we could instead perform a:\n",
    "\n",
    "- **Transformation**, in which we perform operations to every value within each group.\n",
    "\n",
    "- **Filtration**, in which we keep only the groups that satisfy some condition."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f57a9d0b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Transformations\n",
    "\n",
    "Suppose we want to convert the `'body_mass_g'` column to to z-scores (i.e. standard units):\n",
    "\n",
    "$$z(x_i) = \\frac{x_i - \\text{mean of } x}{\\text{SD of } x}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc70cbb6",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def z_score(x):\n",
    "    return (x - x.mean()) / x.std(ddof=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d600ae5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "z_score(penguins['body_mass_g'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a85c12f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Transformations within groups\n",
    "\n",
    "- Now, what if we wanted the z-score within each group?\n",
    "\n",
    "- To do so, we can use the `transform` method on a `DataFrameGroupBy` object. The `transform` method takes in a function, which itself takes in a Series and returns a new Series.\n",
    "\n",
    "- A transformation produces a DataFrame or Series of the same size – it is **not** an aggregation!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3872682f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "z_mass = (penguins\n",
    "          .groupby('species')\n",
    "          ['body_mass_g']\n",
    "          .transform(z_score))\n",
    "z_mass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bd014d2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins.assign(z_mass=z_mass)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "499fd16f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "display_df(penguins.assign(z_mass=z_mass), rows=8)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17ace31a",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Note that above, penguin 340 has a larger `'body_mass_g'` than penguin 0, but a lower `'z_mass'`.\n",
    "- Penguin 0 has an above average `'body_mass_g'` among `'Adelie'` penguins.\n",
    "- Penguin 340 has a below average `'body_mass_g'` among `'Gentoo'` penguins. Remember from earlier that the average `'body_mass_g'` of `'Gentoo'` penguins is much higher than for other species."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61b31afc",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins.groupby('species')['body_mass_g'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ff3bc14",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Filtering groups\n",
    "\n",
    "- To keep only the groups that satisfy a particular condition, use the `filter` method on a `DataFrameGroupBy` object.\n",
    "\n",
    "- The `filter` method takes in a function, which itself takes in a DataFrame/Series and return a single Boolean. The result is a new DataFrame/Series with only the groups for which the filter function returned `True`.\n",
    "\n",
    "For example, suppose we want only the `'species'` whose average `'bill_length_mm'` is above 39."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "447eeb69",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "(penguins\n",
    " .groupby('species')\n",
    " .filter(lambda df: df['bill_length_mm'].mean() > 39)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a372cbf2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "No more `'Adelie'`s!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3d21c74",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "Or, as another example, suppose we only want `'species'` with at least 100 penguins:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a88db646",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "(penguins\n",
    " .groupby('species')\n",
    " .filter(lambda df: df.shape[0] > 100)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4891c387",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "No more `'Chinstrap'`s!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5519947a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "    <h3>Question 🤔 (Answer at <a href=\"http://dsc80.com/q\">dsc80.com/q)</h3>\n",
    "\n",
    "Code: `lec03-aggs`\n",
    "\n",
    "Answer the following questions about grouping:\n",
    "\n",
    "- In `.agg(fn)`, what is the input to `fn`? What is the output of `fn`?\n",
    "- In `.transform(fn)`, what is the input to `fn`? What is the output of `fn`?\n",
    "- In `.filter(fn)`, what is the input to `fn`? What is the output of `fn`?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e376960",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Grouping with multiple columns\n",
    "\n",
    "When we group with multiple columns, one group is created for **every unique combination** of elements in the specified columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72b3489d-b988-4987-bff1-c9a7df126e8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "penguins"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a87917b8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "species_and_island = (\n",
    "    penguins\n",
    "    .groupby(['species', 'island'])\n",
    "    [['bill_length_mm', 'body_mass_g']]\n",
    "    .mean()\n",
    ")\n",
    "species_and_island"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db94be88",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Grouping and indexes\n",
    "\n",
    "- The `groupby` method creates an index based on the specified columns.\n",
    "- When grouping by multiple columns, the resulting DataFrame has a `MultiIndex`.\n",
    "- Advice: When working with a `MultiIndex`, use `reset_index` or set `as_index=False` in `groupby`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ade0a51",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "species_and_island"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "771e57f5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "species_and_island['body_mass_g']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb58b9eb",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "species_and_island.loc['Adelie']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bc553eb",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "species_and_island.loc[('Adelie', 'Torgersen')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca4039ed",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "species_and_island.reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b35fbd1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "(penguins\n",
    " .groupby(['species', 'island'], as_index=False)\n",
    " [['bill_length_mm', 'body_mass_g']]\n",
    " .mean()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ffc9679",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "    <h3>Question 🤔 (Answer at <a href=\"http://dsc80.com/q\">dsc80.com/q)</h3>\n",
    "\n",
    "Code: `lec03-names`\n",
    "\n",
    "Find the most popular `Male` and `Female` baby `Name` for each `Year` in `baby`. **Exclude** `Year`s where there were fewer than 1 million births recorded.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73e04e4b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "baby_path = Path('data') / 'baby.csv'\n",
    "baby = pd.read_csv(baby_path)\n",
    "baby"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dcabd1d5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code goes here."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c92b024d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Pivot tables using the `pivot_table` method"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "191b9073",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Pivot tables: an extension of grouping\n",
    "\n",
    "Pivot tables are a compact way to display tables for humans to read:\n",
    "\n",
    "<table border=\"1\" class=\"dataframe\">\n",
    "  <thead>\n",
    "    <tr style=\"text-align: right;\">\n",
    "      <th>Sex</th>\n",
    "      <th>F</th>\n",
    "      <th>M</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>Year</th>\n",
    "      <th></th>\n",
    "      <th></th>\n",
    "    </tr>\n",
    "  </thead>\n",
    "  <tbody>\n",
    "    <tr>\n",
    "      <th>2018</th>\n",
    "      <td>1698373</td>\n",
    "      <td>1813377</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>2019</th>\n",
    "      <td>1675139</td>\n",
    "      <td>1790682</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>2020</th>\n",
    "      <td>1612393</td>\n",
    "      <td>1721588</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>2021</th>\n",
    "      <td>1635800</td>\n",
    "      <td>1743913</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>2022</th>\n",
    "      <td>1628730</td>\n",
    "      <td>1733166</td>\n",
    "    </tr>\n",
    "  </tbody>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "064305ac",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- Notice that each value in the table is a sum over the counts, split by year and sex.\n",
    "- **You can think of pivot tables as grouping using two columns, then \"pivoting\" one of the group labels into columns.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b6c0de2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### `pivot_table`\n",
    "\n",
    "The `pivot_table` DataFrame method aggregates a DataFrame using two columns. To use it:\n",
    "\n",
    "```py\n",
    "df.pivot_table(index=index_col,\n",
    "               columns=columns_col,\n",
    "               values=values_col,\n",
    "               aggfunc=func)\n",
    "```\n",
    "The resulting DataFrame will have:\n",
    "- One row for every unique value in `index_col`.\n",
    "- One column for every unique value in `columns_col`.\n",
    "- Values determined by applying `func` on values in `values_col`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d891326",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "last_5_years = baby.query('Year >= 2018')\n",
    "last_5_years"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fcb474ed",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "last_5_years.pivot_table(\n",
    "    index='Year',\n",
    "    columns='Sex',\n",
    "    values='Count',\n",
    "    aggfunc='sum',\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e77809b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Look at the similarity to the snippet above!\n",
    "(last_5_years\n",
    " .groupby(['Year', 'Sex'])\n",
    " [['Count']]\n",
    " .sum()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5668bc16",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Example\n",
    "\n",
    "Find the number of penguins per `'island'` and `'species'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1aa031f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "099f7869",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins.pivot_table(\n",
    "    index='species',\n",
    "    columns='island',\n",
    "    values='bill_length_mm', # Choice of column here doesn't actually matter!\n",
    "    aggfunc='count',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3f69a47",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "Note that there is a `NaN` at the intersection of `'Biscoe'` and `'Chinstrap'`, because there were no Chinstrap penguins on Biscoe Island.\n",
    "\n",
    "We can either use the `fillna` method afterwards or the `fill_value` argument to fill in `NaN`s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0867a074",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguins.pivot_table(\n",
    "    index='species',\n",
    "    columns='island',\n",
    "    values='bill_length_mm',\n",
    "    aggfunc='count',\n",
    "    fill_value=0,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e56de78",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Reshaping\n",
    "\n",
    "- `pivot_table` reshapes DataFrames from \"long\" to \"wide\".\n",
    "- Other DataFrame reshaping methods:\n",
    "    - `melt`: Un-pivots a DataFrame. Very useful in data cleaning.\n",
    "    - `pivot`: Like `pivot_table`, but doesn't do aggregation.\n",
    "    - `stack`: Pivots multi-level columns to multi-indices.\n",
    "    - `unstack`: Pivots multi-indices to columns.\n",
    "    - Google and the documentation are your friends!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b7a9157-5dfa-48ab-a0d4-756993b8269e",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    <h3>We will most likely end lecture here.</h3>\n",
    "</div>\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8508a69b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Distributions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "244a4c6f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Joint distribution\n",
    "\n",
    "When using `aggfunc='count'`, a pivot table describes the **joint distribution** of two categorical variables. This is also called a **contingency table**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "657cb701",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "counts = penguins.pivot_table(\n",
    "    index='species',\n",
    "    columns='sex',\n",
    "    values='body_mass_g',\n",
    "    aggfunc='count',\n",
    "    fill_value=0\n",
    ")\n",
    "counts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "125b3d37",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "We can normalize the DataFrame by dividing by the total number of penguins. The resulting numbers can be interpreted as **probabilities** that a randomly selected penguin from the dataset belongs to a given combination of species and sex."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef27fe23",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "joint = counts / counts.sum().sum()\n",
    "joint"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bf70932",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Marginal probabilities\n",
    "\n",
    "If we sum over one of the axes, we can compute **marginal probabilities**, i.e. unconditional probabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ae4e508",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "joint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8539e9a6",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Recall, joint.sum(axis=0) sums across the rows,\n",
    "# which computes the sum of the **columns**.\n",
    "joint.sum(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f5a59a1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "joint.sum(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3abcbba0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "For instance, the second Series tells us that a randomly selected penguin has a 0.36 chance of being of species `'Gentoo'`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fb270de",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Conditional probabilities\n",
    "\n",
    "Using `counts`, how might we compute conditional probabilities like $$P(\\text{species } = \\text{\"Adelie\"} \\mid \\text{sex } = \\text{\"Female\"})?$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ba45ff3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "counts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43ad126b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "$$\\begin{align*}\n",
    "P(\\text{species} = c \\mid \\text{sex} = x) &= \\frac{\\# \\: (\\text{species} = c \\text{ and } \\text{sex} = x)}{\\# \\: (\\text{sex} = x)}\n",
    "\\end{align*}$$\n",
    "\n",
    "<details>\n",
    "    <summary>➡️ Click <b>here</b> to see more of a derivation.</summary>\n",
    "$$\\begin{align*}\n",
    "P(\\text{species} = c \\mid \\text{sex} = x) &= \\frac{P(\\text{species} = c \\text{ and } \\text{sex} = x)}{P(\\text{sex = }x)} \\\\\n",
    "&= \\frac{\\frac{\\# \\: (\\text{species } = \\: c \\text{ and } \\text{sex } = \\: x)}{N}}{\\frac{\\# \\: (\\text{sex } = \\: x)}{N}} \\\\\n",
    "&= \\frac{\\# \\: (\\text{species} = c \\text{ and } \\text{sex} = x)}{\\# \\: (\\text{sex} = x)}\n",
    "\\end{align*}$$\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff143cc6",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "**Answer**: To find conditional probabilities of **`'species'` given `'sex'`**, divide by **column sums**. To find conditional probabilities of **`'sex'` given `'species'`**, divide by **row sums**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d373b71c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Conditional probabilities\n",
    "\n",
    "To find conditional probabilities of **`'species'` given `'sex'`**, divide by **column sums**. To find conditional probabilities of **`'sex'` given `'species'`**, divide by **row sums**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13c67c6c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e995c56",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "counts.sum(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08bae3f5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "The conditional distribution of **`'species'` given `'sex'`** is below. Note that in this new DataFrame, the `'Female'` and `'Male'` columns each sum to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac092c84",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "counts / counts.sum(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82ff00a4",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "For instance, the above DataFrame tells us that the probability that a randomly selected penguin is of `'species'` `'Adelie'` **given** that they are of `'sex'` `'Female'` is 0.442424."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1860aca1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "    <h3>Question 🤔 (Answer at <a href=\"http://dsc80.com/q\">dsc80.com/q)</h3>\n",
    "\n",
    "Code: `lec03-dist`\n",
    "\n",
    "Find the conditional distribution of `'sex'` given `'species'`.  \n",
    "\n",
    "**_Hint_**: Use `.T`.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f96daece",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code goes here."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30f5cf4d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Summary, next time"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b500565",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Summary\n",
    "\n",
    "- Grouping allows us to change the level of granularity in a DataFrame.\n",
    "- Grouping involves three steps – split, apply, and combine.\n",
    "    - Usually, what is applied is an aggregation, but it could be a transformation or filtration.\n",
    "- `pivot_table` aggregates data based on two categorical columns, and reshapes the result to be \"wide\" instead of \"long\"."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9c3bd04",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Next time\n",
    "\n",
    "- Simpson's paradox.\n",
    "- Merging.\n",
    "    - Review [this diagram](https://dsc10.com/diagrams/#merging) from DSC 10!\n",
    "- The pitfalls of the `apply` method."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  },
  "livereveal": {
   "scroll": true
  },
  "rise": {
   "transition": "none"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
