{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploration"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction <a class=\"anchor\" id=\"section_0\"></a>\n",
    "\n",
    "To study and quantify human behavior using longitudinal multimodal digital data, it is essential to get to know the data well first. These data from various sources or sensors, such as smartphones and watches and activity trackers, yields data with different types and properties. The data may be a mixture of categorical, ordinal and numerical data, typically consisting of time series measured for multiple subjetcs from different groups. While the data is typically dense, it is also heterogenous and contains lots of missing values. Therefore, the analysis has to be conducted on many different levels.  \n",
    "\n",
    "This notebook introduces the Niimpy toolbox exploration module, which seeks to address the aforementioned issues. The module has functionalities for exploratory data analysis (EDA) of digital behavioral data. The module aims to produce a summary of the data characteristics, inspecting the structures underlying the data, to detecting patterns and changes in the patterns, and to assess the data quality (e.g., missing data, outliers). This information is highly essential for assessing data validity, data filtering and selection, and for data preprocessing. The module includes functions for [plotting catogorical data](#section_1), [data counts](#section_2), [timeseries lineplots](#section_3), [punchcards](#section_4) and [visualizing missing data.](#section_5)\n",
    "\n",
    "Exploration module functions are supposed to run after [data preprocessing](https://digitraceslab.github.io/niimpy/user_guide/preprocessing/), but they can be run also on the raw observations.\n",
    "All the functions are implemented by using [Plotly Python Open sourde Library](https://plotly.com/python/). Plotly enables interactive visualizations which in turn makers it easier to explore different aspects of the data (e.g.,specific timerange and summary statistics).\n",
    "\n",
    "This notebook uses several sample dataframes for module demonstration. The sample data is already preprocessed, or will be preprocessed in notebook sections before visualizations. When the sample data is loaded, some of the key characteristics of the data are displayed. \n",
    "\n",
    "All eploration module functions require the data to follow [data schema.](https://digitraceslab.github.io/niimpy/schema/) defined in the Niimpy toolbox [documentation](https://digitraceslab.github.io/niimpy/). The user must ensure that the input data follows the specified schema.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "### Sub-module overview\n",
    "\n",
    "The following table shows accepted data types, visualization functions and the purpose of each exploration sub-module. All submodules are located inside `niimpy/exploration/eda` -folder.\n",
    "\n",
    "| Sub-module | Data type | Functions | For what |\n",
    "| :----------- | :----------- | :------------ | :------------ | \n",
    "| catogorical.py | Categorical |  Barplot | Observations counts and distributions |\n",
    "| countplot.py | Categorical* / Numerical | Barplot/Boxplot | Observation counts and distibutions |\n",
    "| lineplot.py | Numerical| Lineplot | Trend, cyclicity, patterns | \n",
    "| punchcard.py | Categorical* / Numerical | Heatmap | Temporal patterns of counts or values  |\n",
    "| missingness.py | Categorical / Numerical | Barplot / Heatmap | Missing data patterns |"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data types denoted with \\* are not compatible with every function within the module.\n",
    "***\n",
    "### *NOTES*\n",
    "\n",
    "This notebook uses following definitions referring to data:\n",
    "* *Feature* refers to dataframe column that stores observations (e.g., numerical sensor values, questionnaire answers)\n",
    "* *User* refers to unique identifier for each subject in the data. Dataframe should also have a column named as `user`.\n",
    "* *Group* refers to unique group idenfier. If subjects are grouped, dataframe shoudl have a column named as `group`.\n",
    "\n",
    "***"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imports\n",
    "Here we import modules needed for running this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import plotly\n",
    "import plotly.graph_objects as go\n",
    "import plotly.express as px\n",
    "import plotly.io as pio\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "import niimpy\n",
    "from niimpy import config\n",
    "from niimpy.preprocessing import survey \n",
    "from niimpy.exploration import setup_dataframe\n",
    "from niimpy.exploration.eda import categorical, countplot, lineplot, missingness, punchcard"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plotly settings\n",
    "Next code block defines default settings for [plotly](https://plotly.com/python/) visualizations. Feel free to adjust the settings according to your needs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "pio.renderers.default = \"png\"\n",
    "pio.templates.default = \"seaborn\"\n",
    "px.defaults.template = \"ggplot2\"\n",
    "px.defaults.color_continuous_scale = px.colors.sequential.RdBu\n",
    "px.defaults.width = 1200\n",
    "px.defaults.height = 482\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) Categorical plot <a class=\"anchor\" id=\"section_1\"></a>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section introduces categorical plot module visualizes **categorical data**, such as questionnaire data responses.<br>\n",
    "We will demonstrate functions by using a mock survey dataframe, containing answers for:\n",
    "* *Patient Health Questionnaire-2 (PHQ-2)*\n",
    "* *Perceived Stress Scale (PSS10)*\n",
    "* *Generalized Anxiety Disorder-2 (GAD-2)*\n",
    "\n",
    "The data will be preprocessed, and then it's basic characteristics will be summarized before visualizations."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1) Reading the data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll start by importing the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>age</th>\n",
       "      <th>gender</th>\n",
       "      <th>Little interest or pleasure in doing things.</th>\n",
       "      <th>Feeling down; depressed or hopeless.</th>\n",
       "      <th>Feeling nervous; anxious or on edge.</th>\n",
       "      <th>Not being able to stop or control worrying.</th>\n",
       "      <th>In the last month; how often have you felt that you were unable to control the important things in your life?</th>\n",
       "      <th>In the last month; how often have you felt confident about your ability to handle your personal problems?</th>\n",
       "      <th>In the last month; how often have you felt that things were going your way?</th>\n",
       "      <th>In the last month; how often have you been able to control irritations in your life?</th>\n",
       "      <th>In the last month; how often have you felt that you were on top of things?</th>\n",
       "      <th>In the last month; how often have you been angered because of things that were outside of your control?</th>\n",
       "      <th>In the last month; how often have you felt difficulties were piling up so high that you could not overcome them?</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>20</td>\n",
       "      <td>Male</td>\n",
       "      <td>several-days</td>\n",
       "      <td>more-than-half-the-days</td>\n",
       "      <td>not-at-all</td>\n",
       "      <td>nearly-every-day</td>\n",
       "      <td>almost-never</td>\n",
       "      <td>sometimes</td>\n",
       "      <td>fairly-often</td>\n",
       "      <td>never</td>\n",
       "      <td>sometimes</td>\n",
       "      <td>very-often</td>\n",
       "      <td>fairly-often</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>32</td>\n",
       "      <td>Male</td>\n",
       "      <td>more-than-half-the-days</td>\n",
       "      <td>more-than-half-the-days</td>\n",
       "      <td>not-at-all</td>\n",
       "      <td>several-days</td>\n",
       "      <td>never</td>\n",
       "      <td>never</td>\n",
       "      <td>very-often</td>\n",
       "      <td>sometimes</td>\n",
       "      <td>never</td>\n",
       "      <td>fairly-often</td>\n",
       "      <td>never</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>15</td>\n",
       "      <td>Male</td>\n",
       "      <td>more-than-half-the-days</td>\n",
       "      <td>not-at-all</td>\n",
       "      <td>several-days</td>\n",
       "      <td>not-at-all</td>\n",
       "      <td>never</td>\n",
       "      <td>very-often</td>\n",
       "      <td>very-often</td>\n",
       "      <td>fairly-often</td>\n",
       "      <td>never</td>\n",
       "      <td>never</td>\n",
       "      <td>almost-never</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>35</td>\n",
       "      <td>Female</td>\n",
       "      <td>not-at-all</td>\n",
       "      <td>nearly-every-day</td>\n",
       "      <td>not-at-all</td>\n",
       "      <td>several-days</td>\n",
       "      <td>very-often</td>\n",
       "      <td>fairly-often</td>\n",
       "      <td>very-often</td>\n",
       "      <td>never</td>\n",
       "      <td>sometimes</td>\n",
       "      <td>never</td>\n",
       "      <td>fairly-often</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>23</td>\n",
       "      <td>Male</td>\n",
       "      <td>more-than-half-the-days</td>\n",
       "      <td>not-at-all</td>\n",
       "      <td>more-than-half-the-days</td>\n",
       "      <td>several-days</td>\n",
       "      <td>almost-never</td>\n",
       "      <td>very-often</td>\n",
       "      <td>almost-never</td>\n",
       "      <td>sometimes</td>\n",
       "      <td>sometimes</td>\n",
       "      <td>very-often</td>\n",
       "      <td>never</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user  age  gender Little interest or pleasure in doing things.  \\\n",
       "0     1   20    Male                                 several-days   \n",
       "1     2   32    Male                      more-than-half-the-days   \n",
       "2     3   15    Male                      more-than-half-the-days   \n",
       "3     4   35  Female                                   not-at-all   \n",
       "4     5   23    Male                      more-than-half-the-days   \n",
       "\n",
       "  Feeling down; depressed or hopeless. Feeling nervous; anxious or on edge.  \\\n",
       "0              more-than-half-the-days                           not-at-all   \n",
       "1              more-than-half-the-days                           not-at-all   \n",
       "2                           not-at-all                         several-days   \n",
       "3                     nearly-every-day                           not-at-all   \n",
       "4                           not-at-all              more-than-half-the-days   \n",
       "\n",
       "  Not being able to stop or control worrying.  \\\n",
       "0                            nearly-every-day   \n",
       "1                                several-days   \n",
       "2                                  not-at-all   \n",
       "3                                several-days   \n",
       "4                                several-days   \n",
       "\n",
       "  In the last month; how often have you felt that you were unable to control the important things in your life?  \\\n",
       "0                                       almost-never                                                              \n",
       "1                                              never                                                              \n",
       "2                                              never                                                              \n",
       "3                                         very-often                                                              \n",
       "4                                       almost-never                                                              \n",
       "\n",
       "  In the last month; how often have you felt confident about your ability to handle your personal problems?  \\\n",
       "0                                          sometimes                                                          \n",
       "1                                              never                                                          \n",
       "2                                         very-often                                                          \n",
       "3                                       fairly-often                                                          \n",
       "4                                         very-often                                                          \n",
       "\n",
       "  In the last month; how often have you felt that things were going your way?  \\\n",
       "0                                       fairly-often                            \n",
       "1                                         very-often                            \n",
       "2                                         very-often                            \n",
       "3                                         very-often                            \n",
       "4                                       almost-never                            \n",
       "\n",
       "  In the last month; how often have you been able to control irritations in your life?  \\\n",
       "0                                              never                                     \n",
       "1                                          sometimes                                     \n",
       "2                                       fairly-often                                     \n",
       "3                                              never                                     \n",
       "4                                          sometimes                                     \n",
       "\n",
       "  In the last month; how often have you felt that you were on top of things?  \\\n",
       "0                                          sometimes                           \n",
       "1                                              never                           \n",
       "2                                              never                           \n",
       "3                                          sometimes                           \n",
       "4                                          sometimes                           \n",
       "\n",
       "  In the last month; how often have you been angered because of things that were outside of your control?  \\\n",
       "0                                         very-often                                                        \n",
       "1                                       fairly-often                                                        \n",
       "2                                              never                                                        \n",
       "3                                              never                                                        \n",
       "4                                         very-often                                                        \n",
       "\n",
       "  In the last month; how often have you felt difficulties were piling up so high that you could not overcome them?  \n",
       "0                                       fairly-often                                                                \n",
       "1                                              never                                                                \n",
       "2                                       almost-never                                                                \n",
       "3                                       fairly-often                                                                \n",
       "4                                              never                                                                "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = niimpy.read_csv(config.SURVEY_PATH, tz='Europe/Helsinki')\n",
    "df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then check some basic descriptive statistics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1000.000000</td>\n",
       "      <td>1000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>500.500000</td>\n",
       "      <td>26.911000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>288.819436</td>\n",
       "      <td>4.992595</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>12.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>250.750000</td>\n",
       "      <td>23.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>500.500000</td>\n",
       "      <td>27.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>750.250000</td>\n",
       "      <td>30.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1000.000000</td>\n",
       "      <td>43.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              user          age\n",
       "count  1000.000000  1000.000000\n",
       "mean    500.500000    26.911000\n",
       "std     288.819436     4.992595\n",
       "min       1.000000    12.000000\n",
       "25%     250.750000    23.000000\n",
       "50%     500.500000    27.000000\n",
       "75%     750.250000    30.000000\n",
       "max    1000.000000    43.000000"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataframe's columns are raw questions from a survey. Some questions belong to a specific category, so we will annotate them with ids. The id is constructed from a prefix (the questionnaire category: GAD, PHQ, PSQI etc.), followed by the question number (1,2,3). Similarly, we will also convert the answers to meaningful numerical values.\n",
    "\n",
    "**Note:** It's important that the dataframe follows the below schema before passing into niimpy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>age</th>\n",
       "      <th>gender</th>\n",
       "      <th>PHQ2_1</th>\n",
       "      <th>PHQ2_2</th>\n",
       "      <th>GAD2_1</th>\n",
       "      <th>GAD2_2</th>\n",
       "      <th>PSS10_2</th>\n",
       "      <th>PSS10_4</th>\n",
       "      <th>PSS10_5</th>\n",
       "      <th>PSS10_6</th>\n",
       "      <th>PSS10_7</th>\n",
       "      <th>PSS10_8</th>\n",
       "      <th>PSS10_9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>20</td>\n",
       "      <td>Male</td>\n",
       "      <td>several-days</td>\n",
       "      <td>more-than-half-the-days</td>\n",
       "      <td>not-at-all</td>\n",
       "      <td>nearly-every-day</td>\n",
       "      <td>almost-never</td>\n",
       "      <td>sometimes</td>\n",
       "      <td>fairly-often</td>\n",
       "      <td>never</td>\n",
       "      <td>sometimes</td>\n",
       "      <td>very-often</td>\n",
       "      <td>fairly-often</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>32</td>\n",
       "      <td>Male</td>\n",
       "      <td>more-than-half-the-days</td>\n",
       "      <td>more-than-half-the-days</td>\n",
       "      <td>not-at-all</td>\n",
       "      <td>several-days</td>\n",
       "      <td>never</td>\n",
       "      <td>never</td>\n",
       "      <td>very-often</td>\n",
       "      <td>sometimes</td>\n",
       "      <td>never</td>\n",
       "      <td>fairly-often</td>\n",
       "      <td>never</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>15</td>\n",
       "      <td>Male</td>\n",
       "      <td>more-than-half-the-days</td>\n",
       "      <td>not-at-all</td>\n",
       "      <td>several-days</td>\n",
       "      <td>not-at-all</td>\n",
       "      <td>never</td>\n",
       "      <td>very-often</td>\n",
       "      <td>very-often</td>\n",
       "      <td>fairly-often</td>\n",
       "      <td>never</td>\n",
       "      <td>never</td>\n",
       "      <td>almost-never</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>35</td>\n",
       "      <td>Female</td>\n",
       "      <td>not-at-all</td>\n",
       "      <td>nearly-every-day</td>\n",
       "      <td>not-at-all</td>\n",
       "      <td>several-days</td>\n",
       "      <td>very-often</td>\n",
       "      <td>fairly-often</td>\n",
       "      <td>very-often</td>\n",
       "      <td>never</td>\n",
       "      <td>sometimes</td>\n",
       "      <td>never</td>\n",
       "      <td>fairly-often</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>23</td>\n",
       "      <td>Male</td>\n",
       "      <td>more-than-half-the-days</td>\n",
       "      <td>not-at-all</td>\n",
       "      <td>more-than-half-the-days</td>\n",
       "      <td>several-days</td>\n",
       "      <td>almost-never</td>\n",
       "      <td>very-often</td>\n",
       "      <td>almost-never</td>\n",
       "      <td>sometimes</td>\n",
       "      <td>sometimes</td>\n",
       "      <td>very-often</td>\n",
       "      <td>never</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user  age  gender                   PHQ2_1                   PHQ2_2  \\\n",
       "0     1   20    Male             several-days  more-than-half-the-days   \n",
       "1     2   32    Male  more-than-half-the-days  more-than-half-the-days   \n",
       "2     3   15    Male  more-than-half-the-days               not-at-all   \n",
       "3     4   35  Female               not-at-all         nearly-every-day   \n",
       "4     5   23    Male  more-than-half-the-days               not-at-all   \n",
       "\n",
       "                    GAD2_1            GAD2_2       PSS10_2       PSS10_4  \\\n",
       "0               not-at-all  nearly-every-day  almost-never     sometimes   \n",
       "1               not-at-all      several-days         never         never   \n",
       "2             several-days        not-at-all         never    very-often   \n",
       "3               not-at-all      several-days    very-often  fairly-often   \n",
       "4  more-than-half-the-days      several-days  almost-never    very-often   \n",
       "\n",
       "        PSS10_5       PSS10_6    PSS10_7       PSS10_8       PSS10_9  \n",
       "0  fairly-often         never  sometimes    very-often  fairly-often  \n",
       "1    very-often     sometimes      never  fairly-often         never  \n",
       "2    very-often  fairly-often      never         never  almost-never  \n",
       "3    very-often         never  sometimes         never  fairly-often  \n",
       "4  almost-never     sometimes  sometimes    very-often         never  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert column name to id, based on provided mappers from niimpy\n",
    "column_map = {**survey.PHQ2_MAP, **survey.PSQI_MAP, **survey.PSS10_MAP, **survey.PANAS_MAP, **survey.GAD2_MAP}\n",
    "df = survey.clean_survey_column_names(df)\n",
    "df = df.rename(column_map, axis = 1)\n",
    "df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `convert_survey_to_numerical_answer` helper method to convert the answers into a numerical value. We use the `ID_MAP_PREFIX` mapping dictionary provided by `Niimpy`, which describes how each text answer should be mapped to a number. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>age</th>\n",
       "      <th>gender</th>\n",
       "      <th>PHQ2_1</th>\n",
       "      <th>PHQ2_2</th>\n",
       "      <th>GAD2_1</th>\n",
       "      <th>GAD2_2</th>\n",
       "      <th>PSS10_2</th>\n",
       "      <th>PSS10_4</th>\n",
       "      <th>PSS10_5</th>\n",
       "      <th>PSS10_6</th>\n",
       "      <th>PSS10_7</th>\n",
       "      <th>PSS10_8</th>\n",
       "      <th>PSS10_9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>20</td>\n",
       "      <td>Male</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>32</td>\n",
       "      <td>Male</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>15</td>\n",
       "      <td>Male</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>35</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>23</td>\n",
       "      <td>Male</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user  age  gender  PHQ2_1  PHQ2_2  GAD2_1  GAD2_2  PSS10_2  PSS10_4  \\\n",
       "0     1   20    Male       1       2       0       3        1        2   \n",
       "1     2   32    Male       2       2       0       1        0        0   \n",
       "2     3   15    Male       2       0       1       0        0        4   \n",
       "3     4   35  Female       0       3       0       1        4        3   \n",
       "4     5   23    Male       2       0       2       1        1        4   \n",
       "\n",
       "   PSS10_5  PSS10_6  PSS10_7  PSS10_8  PSS10_9  \n",
       "0        3        0        2        4        3  \n",
       "1        4        2        0        3        0  \n",
       "2        4        3        0        0        1  \n",
       "3        4        0        2        0        3  \n",
       "4        1        2        2        4        0  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Transform raw answers to numerical values\n",
    "num_df = survey.convert_survey_to_numerical_answer(\n",
    "    df, id_map=survey.ID_MAP_PREFIX, use_prefix=True\n",
    ")\n",
    "num_df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each of these surveys, the overall score is calculated as the sum of the numerical value. We can calculate this for each survey using the `sum_survey_scores` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>PHQ2</th>\n",
       "      <th>PSS10</th>\n",
       "      <th>GAD2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>15</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>16</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>14</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user  PHQ2  PSS10  GAD2\n",
       "0     1     3     15     3\n",
       "1     2     4      9     1\n",
       "2     3     2     12     1\n",
       "3     4     3     16     1\n",
       "4     5     2     14     3"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum_df = survey.sum_survey_scores(num_df, [\"PHQ2\", \"PSS10\", \"GAD2\"])\n",
    "sum_df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1. Questionnaire summary <a class=\"anchor\" id=\"section_1_1\"></a>\n",
    "\n",
    "We can now make some plots for the preprocessed data frame. First, we can display the summary for the specific question (*PHQ-2* first question)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 35544 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = categorical.questionnaire_summary(num_df, \n",
    "                                        question = 'PHQ2_1',\n",
    "                                        title='PHQ2 question: Little interest or pleasure in doing things / <br> answer value frequencies', \n",
    "                                        xlabel='value',\n",
    "                                        ylabel='count',\n",
    "                                        width=600,\n",
    "                                        height=400)\n",
    "fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The figure shows that the answer values (from 0 to 3) almost uniform in distribution."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2. Questionnaire grouped summary <a class=\"anchor\" id=\"section_1_2\"></a>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also display the summary for each subgroup (*gender*)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 40940 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = categorical.questionnaire_grouped_summary(num_df, \n",
    "                                                question='PSS10_9', \n",
    "                                                group='gender', \n",
    "                                                title='PSS10_9 Question / <br> Score frequency distributions by group',\n",
    "                                                xlabel='score',\n",
    "                                                ylabel='count',\n",
    "                                                width=800,\n",
    "                                                height=400)\n",
    "fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The figure shows that the differences between subgroups are not very large."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3. Questionnaire grouped summary score distribution <a class=\"anchor\" id=\"section_1_3\"></a>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With some quick preprocessing, we can display the score distribution of each questionaire.\n",
    "\n",
    "We'll extract *PSS-10* questionnaire answers from the dataframe, using the `sum_survey_scores` function from the `niimpy.preprocessing.survey` module, and set the gender variable from the original dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>PSS10</th>\n",
       "      <th>gender</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>Male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>Male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "      <td>Male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>16</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>14</td>\n",
       "      <td>Male</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user  PSS10  gender\n",
       "0     1     15    Male\n",
       "1     2      9    Male\n",
       "2     3     12    Male\n",
       "3     4     16  Female\n",
       "4     5     14    Male"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum_df = survey.sum_survey_scores(num_df, [\"PSS10\"])\n",
    "sum_df[\"gender\"] = num_df[\"gender\"]\n",
    "sum_df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then visualize aggregated summary score distributions, grouped by gender:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 38928 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = categorical.questionnaire_grouped_summary(sum_df, \n",
    "                                                question='PSS10', \n",
    "                                                group='gender', \n",
    "                                                title='PSS10',\n",
    "                                                xlabel='score', \n",
    "                                                ylabel='count',\n",
    "                                                width=800, \n",
    "                                                height=400)\n",
    "fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The figure shows that the grouped summary score distrubutions are close to each other."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2) Countplot <a class=\"anchor\" id=\"section_2\"></a>\n",
    "This section introduces Countplot module. The module contain functions for user and group level observation count (number of datapoints per user or group) visualization and observation value distributions.<br>\n",
    "Observation counts use barplots for user level and a boxplots for group level visualizations. Boxplots are used for group level value distributions.<br>\n",
    "The module assumes that the visualized **data is numerical**."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data\n",
    "We will use sample from <a href=\"https://studentlife.cs.dartmouth.edu/dataset.html\">StudentLife dataset</a> to demonstrate the module functions. The sample contains hourly aggregated activity data (values from 0 to 5, where 0 corresponds to no activity, and 5 to high activity) and group information based on pre- and post-study PHQ-9 test scores. Study subjects have been grouped by the depression symptom severity into groups: *none*, *mild*, *moderate*, *moderately severe*, and *severe*. Preprocessed data sample is included in the Niimpy toolbox *sampledata* folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load data\n",
    "sl = niimpy.read_csv(config.SL_ACTIVITY_PATH, tz='Europe/Helsinki')\n",
    "sl.set_index('timestamp',inplace=True)\n",
    "sl.index = pd.to_datetime(sl.index)\n",
    "sl_loc = sl.tz_localize(None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>activity</th>\n",
       "      <th>group</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>timestamp</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2013-03-27 06:00:00</th>\n",
       "      <td>u00</td>\n",
       "      <td>2</td>\n",
       "      <td>none</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-03-27 07:00:00</th>\n",
       "      <td>u00</td>\n",
       "      <td>1</td>\n",
       "      <td>none</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-03-27 08:00:00</th>\n",
       "      <td>u00</td>\n",
       "      <td>2</td>\n",
       "      <td>none</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-03-27 09:00:00</th>\n",
       "      <td>u00</td>\n",
       "      <td>3</td>\n",
       "      <td>none</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-03-27 10:00:00</th>\n",
       "      <td>u00</td>\n",
       "      <td>4</td>\n",
       "      <td>none</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    user  activity group\n",
       "timestamp                               \n",
       "2013-03-27 06:00:00  u00         2  none\n",
       "2013-03-27 07:00:00  u00         1  none\n",
       "2013-03-27 08:00:00  u00         2  none\n",
       "2013-03-27 09:00:00  u00         3  none\n",
       "2013-03-27 10:00:00  u00         4  none"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sl_loc.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before visualizations, we'll inspect the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>activity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>55907.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.750264</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.298238</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>5.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           activity\n",
       "count  55907.000000\n",
       "mean       0.750264\n",
       "std        1.298238\n",
       "min        0.000000\n",
       "25%        0.000000\n",
       "50%        0.000000\n",
       "75%        1.000000\n",
       "max        5.000000"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sl_loc.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['none', 'severe', 'mild', 'moderately severe', 'moderate'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sl_loc.group.unique()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1. User level observation count <a class=\"anchor\" id=\"section_2_1\"></a>\n",
    "At first we visualize the number of observations for each subject."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 96724 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = countplot.countplot(sl, \n",
    "                          fig_title='Activity event counts by user', \n",
    "                          plot_type='count', \n",
    "                          points='all',\n",
    "                          aggregation='user', \n",
    "                          user=None, \n",
    "                          column=None,\n",
    "                          binning=False)\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The barplot shows that there are differences in user total activity counts. The user *u24* has the lowest event count of 710 and users *u02* and *u59* have the highest count of 1584."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2. Group level observation count <a class=\"anchor\" id=\"section_2_2\"></a>\n",
    "Next we'll inspect group level daily activity event count distributions by using boxplots. For the improved clarity, we select a timerange of one week from the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 79612 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sl_one_week = sl_loc.loc['2013-03-28':'2013-4-3']\n",
    "\n",
    "fig = countplot.countplot(sl_one_week, \n",
    "                          fig_title='Group level daily activity event count distributions', \n",
    "                          plot_type='value', \n",
    "                          points='all',\n",
    "                          aggregation='group', \n",
    "                          user=None, \n",
    "                          column='activity',\n",
    "                          binning='D')\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The boxplot shows some variability in group level event count distributions across the days spanning from Mar 28 to Apr 3 2013."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3. Group level value distributions <a class=\"anchor\" id=\"section_2_3\"></a>\n",
    "Finally we visualize group level activity value distributions for whole time range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 56832 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = countplot.countplot(sl,\n",
    "                          fig_title='Group level activity score distributions', \n",
    "                          plot_type='value', \n",
    "                          points='outliers',\n",
    "                          aggregation='group', \n",
    "                          user=None,\n",
    "                          column='activity',\n",
    "                          binning=False)\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The boxplot shows that activity score distribution for groups *mild* and *moderately severe* differ from the rest."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Lineplot <a class=\"anchor\" id=\"section_3\"></a>\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section introduces Lineplot module functions. We use the same <a href=\"https://studentlife.cs.dartmouth.edu/dataset.html\">StudentLife dataset</a> derived activity data as in previous section."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1. Lineplot <a class=\"anchor\" id=\"section_3_1\"></a>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lineplot functions display **numerical feature values** on time axis. The user can optionally resample (downsample) and smoothen the data for better visual clarity."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1.1. Single user single feature <a class=\"anchor\" id=\"section_3_1_1\"></a>\n",
    "At first, we'll visualize single user single feature data, without resampling or smoothing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 115540 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = lineplot.timeplot(sl_loc,\n",
    "                        users=['u01'],\n",
    "                        columns=['activity'],\n",
    "                        title='User: {} activity lineplot'.format('u01'),\n",
    "                        xlabel='Date',\n",
    "                        ylabel='Value',\n",
    "                        resample=False,\n",
    "                        interpolate=False,\n",
    "                        window=1,\n",
    "                        reset_index=False)\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The figure showing all the activity datapoints is difficult to interpet. By zooming in the time range, the daily patters come apparent. There is no or low activity during the night."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1.2. Single user single feature index resetted <a class=\"anchor\" id=\"section_3_1_2\"></a>\n",
    "Next, we'll plot visualize the same data using resampling by hour, and 24 hour rolling window smoothing for improved visualization clarity. We also reset the index, showing now hours from the first activity feature observation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 130652 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = lineplot.timeplot(sl_loc,\n",
    "                        users=['u00'],\n",
    "                        columns=['activity'],\n",
    "                        title='User: {} activity lineplot / <br> resetted index'.format('u01'),\n",
    "                        xlabel='Date',\n",
    "                        ylabel='Value',\n",
    "                        resample='h',\n",
    "                        interpolate=True,\n",
    "                        window=24,\n",
    "                        reset_index=True)\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By zooming in the smoothed lineplot, daily activity patterns are easier to detect."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1.3. Single user single feature, aggregated by day <a class=\"anchor\" id=\"section_3_1_3\"></a>\n",
    "Next visualization shows resamplig by day and 7 day rolling window smoothing, making the activity time series trend visible.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 97928 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = lineplot.timeplot(sl_loc,\n",
    "                        users=['u00'],\n",
    "                        columns=['activity'],\n",
    "                        title='User: {} activity lineplot / <br> rolling window (7 days) smoothing'.format('u01'),\n",
    "                        xlabel='Date',\n",
    "                        ylabel='Value',\n",
    "                        resample='D',\n",
    "                        interpolate=True,\n",
    "                        window=7)\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Daily aggregated and smoothed data makes the user activity trend visible. There is a peak at May 9 and the crest at May 23."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2. Multiple subjects single feature <a class=\"anchor\" id=\"section_3_2\"></a>\n",
    "The following visualization superimposes three subject's activity on same figure. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 123692 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = lineplot.timeplot(sl_loc,\n",
    "                        users=['u00','u01'],\n",
    "                        columns=['activity'],\n",
    "                        title='User: {}, {} activity lineplot / <br> rolling window (7 days) smoothing'.format('u00','u01'),\n",
    "                        xlabel='Date',\n",
    "                        ylabel='Value',\n",
    "                        resample='D',\n",
    "                        interpolate=True,\n",
    "                        window=7)\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The figure shows that the user daily averaged activity is quite similar in the beginning of inspected time range. In first two weeks of May, the activity shows opposing trends (user *u00* activity increases and user *u01* decreases). "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3. Group level hourly averages <a class=\"anchor\" id=\"section_3_3\"></a>\n",
    "Next we'll compare group level hourly average activity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 145996 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = lineplot.timeplot(sl_loc,\n",
    "                        users='Group',\n",
    "                        columns=['activity'],\n",
    "                        title='User group activity / <br> hourly averages',\n",
    "                        xlabel='Date',\n",
    "                        ylabel='Value',\n",
    "                        resample='D',\n",
    "                        interpolate=True,\n",
    "                        window=7,\n",
    "                        reset_index=False,\n",
    "                        by='hour')\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The time plot reveals that the hourly averaged group level activity follows circadian rhytmn (less activity during the night). Moderately severe group seems to be least active group during the latter half of the day. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.4. Group level weekday averages <a class=\"anchor\" id=\"section_3_4\"></a>\n",
    "And finally, "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 103600 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = lineplot.timeplot(sl_loc,\n",
    "                        users='Group',\n",
    "                        columns=['activity'],\n",
    "                        title='User Activity',\n",
    "                        xlabel='Date',\n",
    "                        ylabel='Value',\n",
    "                        resample='D',\n",
    "                        interpolate=True,\n",
    "                        window=7,\n",
    "                        reset_index=False,\n",
    "                        by='weekday')\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The timeplot shows that there is some differences between the average group level activity, e.g., group *mild* being more active than *moderately severe*. Additionally, activity during Sundays is at lower level in comparison with weekdays."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Punchcard <a class=\"anchor\" id=\"section_4\"></a>\n",
    "This section introduces Punchcard module functions. The functions aggregate the data and show the averaged value for each timepoint. <br> We use the same StudentLife dataset derived activity data as in two previous sections."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1. Single user punchcard <a class=\"anchor\" id=\"section_4_1\"></a>\n",
    "At first we visualize one daily aggregated mean activity for single subject. We'll change the plot color to grayscale for improved clarity.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "px.defaults.color_continuous_scale = px.colors.sequential.gray"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 44744 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = punchcard.punchcard_plot(sl, \n",
    "                               user_list=['u00'],\n",
    "                               columns=['activity'],\n",
    "                               title=\"User {} activity punchcard\".format('u00'),\n",
    "                               resample='D',\n",
    "                               normalize=False,\n",
    "                               agg_func=np.mean,\n",
    "                               timerange=False)\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The punchcard reveals that May 5th has the highest average activity and May 18th, 20th, and 21th have the lowest activity.  "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2. Multiple user punchcard <a class=\"anchor\" id=\"section_4_2\"></a>\n",
    "Next, we'll visualize mean activity for multiple subjects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 60128 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = punchcard.punchcard_plot(sl, \n",
    "                               user_list=['u00','u01','u02'],\n",
    "                               columns=['activity'],\n",
    "                               title=\"Users {}, {}, and {} activity punchcard\".format('u00','u01','u02'),\n",
    "                               resample='D',\n",
    "                               normalize=False,\n",
    "                               agg_func=np.mean,\n",
    "                               timerange=False)\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The punchard allows comparison of daily average activity for multiple subjects. It seems that there is not evident common pattern in the activity."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.3. Single user punchcard showing two features <a class=\"anchor\" id=\"section_4_3\"></a>\n",
    "Lastly, we'll visualize daily aggregated single user activity side by side with activity of previous week. <br>\n",
    "We start by shifting the activity by one week and by adding it to the original dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "sl_loc['previous_week_activity'] = sl_loc['activity'].shift(periods=7, fill_value=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 82868 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = punchcard.punchcard_plot(sl_loc, \n",
    "                               user_list=['u00'],\n",
    "                               columns=['activity','previous_week_activity'],\n",
    "                               title=\"User {} activity and previous week activity punchcard\".format('u00'),\n",
    "                               resample='D',\n",
    "                               normalize=False,\n",
    "                               agg_func=np.mean,\n",
    "                               timerange=False)\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The punchcard show weekly repeating patterns in subjects activity."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5) Missingness <a class=\"anchor\" id=\"section_5\"></a>\n",
    "This sections introduces Missingness module for missing data inspection. The module features data missingness visualizations by frequency and by timepoint.<br>\n",
    "Additionally, it offers an option for missing data correlation visualization.   "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For data missingness visualizations, we'll create a mock dataframe with missing values using `niimpy.util.create_missing_dataframe` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_m = setup_dataframe.create_missing_dataframe(nrows=2*24*14, ncols=5, density=0.7, index_type='dt', freq='10min')\n",
    "df_m.columns = ['User_1','User_2','User_3','User_4','User_5',]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will quickly inspect the dataframe before the visualizations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>User_1</th>\n",
       "      <th>User_2</th>\n",
       "      <th>User_3</th>\n",
       "      <th>User_4</th>\n",
       "      <th>User_5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2022-01-01 00:00:00</th>\n",
       "      <td>95.449550</td>\n",
       "      <td>NaN</td>\n",
       "      <td>63.620984</td>\n",
       "      <td>84.779703</td>\n",
       "      <td>10.134786</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-01-01 00:10:00</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>16.542671</td>\n",
       "      <td>30.059590</td>\n",
       "      <td>48.531147</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-01-01 00:20:00</th>\n",
       "      <td>NaN</td>\n",
       "      <td>39.576178</td>\n",
       "      <td>62.931902</td>\n",
       "      <td>14.169682</td>\n",
       "      <td>30.127549</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-01-01 00:30:00</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.209067</td>\n",
       "      <td>76.309938</td>\n",
       "      <td>64.975323</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-01-01 00:40:00</th>\n",
       "      <td>8.108314</td>\n",
       "      <td>68.563101</td>\n",
       "      <td>11.451206</td>\n",
       "      <td>67.939258</td>\n",
       "      <td>51.267042</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-01-05 15:10:00</th>\n",
       "      <td>29.951958</td>\n",
       "      <td>87.768113</td>\n",
       "      <td>3.413069</td>\n",
       "      <td>NaN</td>\n",
       "      <td>71.765351</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-01-05 15:20:00</th>\n",
       "      <td>66.755720</td>\n",
       "      <td>54.265762</td>\n",
       "      <td>NaN</td>\n",
       "      <td>41.359552</td>\n",
       "      <td>85.334592</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-01-05 15:30:00</th>\n",
       "      <td>65.078709</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>25.390743</td>\n",
       "      <td>27.214379</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-01-05 15:40:00</th>\n",
       "      <td>43.437510</td>\n",
       "      <td>93.718360</td>\n",
       "      <td>3.737043</td>\n",
       "      <td>93.463233</td>\n",
       "      <td>61.265460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-01-05 15:50:00</th>\n",
       "      <td>52.235363</td>\n",
       "      <td>3.903921</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13.510993</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>672 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                        User_1     User_2     User_3     User_4     User_5\n",
       "2022-01-01 00:00:00  95.449550        NaN  63.620984  84.779703  10.134786\n",
       "2022-01-01 00:10:00        NaN        NaN  16.542671  30.059590  48.531147\n",
       "2022-01-01 00:20:00        NaN  39.576178  62.931902  14.169682  30.127549\n",
       "2022-01-01 00:30:00        NaN        NaN   3.209067  76.309938  64.975323\n",
       "2022-01-01 00:40:00   8.108314  68.563101  11.451206  67.939258  51.267042\n",
       "...                        ...        ...        ...        ...        ...\n",
       "2022-01-05 15:10:00  29.951958  87.768113   3.413069        NaN  71.765351\n",
       "2022-01-05 15:20:00  66.755720  54.265762        NaN  41.359552  85.334592\n",
       "2022-01-05 15:30:00  65.078709        NaN        NaN  25.390743  27.214379\n",
       "2022-01-05 15:40:00  43.437510  93.718360   3.737043  93.463233  61.265460\n",
       "2022-01-05 15:50:00  52.235363   3.903921        NaN        NaN  13.510993\n",
       "\n",
       "[672 rows x 5 columns]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>User_1</th>\n",
       "      <th>User_2</th>\n",
       "      <th>User_3</th>\n",
       "      <th>User_4</th>\n",
       "      <th>User_5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>467.000000</td>\n",
       "      <td>459.000000</td>\n",
       "      <td>505.000000</td>\n",
       "      <td>460.000000</td>\n",
       "      <td>461.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>52.139713</td>\n",
       "      <td>49.797069</td>\n",
       "      <td>49.664041</td>\n",
       "      <td>47.506335</td>\n",
       "      <td>51.783715</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>28.177069</td>\n",
       "      <td>29.022774</td>\n",
       "      <td>29.374702</td>\n",
       "      <td>28.842758</td>\n",
       "      <td>27.133421</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.115046</td>\n",
       "      <td>1.356182</td>\n",
       "      <td>1.018417</td>\n",
       "      <td>1.055085</td>\n",
       "      <td>1.089756</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>28.128912</td>\n",
       "      <td>24.049961</td>\n",
       "      <td>23.363264</td>\n",
       "      <td>22.883851</td>\n",
       "      <td>28.179801</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>54.304329</td>\n",
       "      <td>52.259609</td>\n",
       "      <td>48.483288</td>\n",
       "      <td>43.471076</td>\n",
       "      <td>52.341558</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>75.519421</td>\n",
       "      <td>74.290567</td>\n",
       "      <td>77.026377</td>\n",
       "      <td>74.314602</td>\n",
       "      <td>74.746701</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>99.517549</td>\n",
       "      <td>99.943515</td>\n",
       "      <td>99.674461</td>\n",
       "      <td>99.967264</td>\n",
       "      <td>99.863755</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           User_1      User_2      User_3      User_4      User_5\n",
       "count  467.000000  459.000000  505.000000  460.000000  461.000000\n",
       "mean    52.139713   49.797069   49.664041   47.506335   51.783715\n",
       "std     28.177069   29.022774   29.374702   28.842758   27.133421\n",
       "min      1.115046    1.356182    1.018417    1.055085    1.089756\n",
       "25%     28.128912   24.049961   23.363264   22.883851   28.179801\n",
       "50%     54.304329   52.259609   48.483288   43.471076   52.341558\n",
       "75%     75.519421   74.290567   77.026377   74.314602   74.746701\n",
       "max     99.517549   99.943515   99.674461   99.967264   99.863755"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_m.describe()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.1. Data frequency by feature <a class=\"anchor\" id=\"section_5_1\"></a>\n",
    "\n",
    "First, we create a histogram to visualize data frequency per column. Here, frequency of 1 indicates no missing data points and 0 that all data points are missing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 43212 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = missingness.bar(df_m,\n",
    "                      xaxis_title='User',\n",
    "                      yaxis_title='Frequency')\n",
    "fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data frequency is nearly similar for each user, *User_5* having the highest frequency."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2. Average frequency by user <a class=\"anchor\" id=\"section_5_2\"></a>\n",
    "Next, we will show average data frequency for all users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 49936 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = missingness.bar(df_m, \n",
    "                      sampling_freq='30min',\n",
    "                      xaxis_title='Time',\n",
    "                      yaxis_title='Frequency')\n",
    "fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The overall data frequency suggests no clear pattern for data missingness."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.3. Missingness matrix <a class=\"anchor\" id=\"section_5_3\"></a>\n",
    "We can also create a missingness matrix visualization for the dataframe. The nullity matrix show data missingess by a timepoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 45808 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = missingness.matrix(df_m,\n",
    "                         sampling_freq='30min',\n",
    "                         xaxis_title=\"User ID\",\n",
    "                         yaxis_title=\"Time\")\n",
    "fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.4. Missing data correlations <a class=\"anchor\" id=\"section_5_4\"></a>\n",
    "\n",
    "Finally, we plot a heatmap to display the correlations between missing data.\n",
    "\n",
    "Correlation ranges from -1 to 1: \n",
    "* -1 means that if one variable appears then the other will be missing. \n",
    "* 0 means that there is no correlation between the missingness of two variables. \n",
    "* 1 means that the two variables will always appear together."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data\n",
    "For the correlations, we use *NYC collision factors* [sample data](https://github.com/ResidentMario/missingno). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "collisions = pd.read_csv(\"https://raw.githubusercontent.com/ResidentMario/missingno-data/master/nyc_collision_factors.csv\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we'll inspect the data frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DATE</th>\n",
       "      <th>TIME</th>\n",
       "      <th>BOROUGH</th>\n",
       "      <th>ZIP CODE</th>\n",
       "      <th>LATITUDE</th>\n",
       "      <th>LONGITUDE</th>\n",
       "      <th>LOCATION</th>\n",
       "      <th>ON STREET NAME</th>\n",
       "      <th>CROSS STREET NAME</th>\n",
       "      <th>OFF STREET NAME</th>\n",
       "      <th>...</th>\n",
       "      <th>CONTRIBUTING FACTOR VEHICLE 1</th>\n",
       "      <th>CONTRIBUTING FACTOR VEHICLE 2</th>\n",
       "      <th>CONTRIBUTING FACTOR VEHICLE 3</th>\n",
       "      <th>CONTRIBUTING FACTOR VEHICLE 4</th>\n",
       "      <th>CONTRIBUTING FACTOR VEHICLE 5</th>\n",
       "      <th>VEHICLE TYPE CODE 1</th>\n",
       "      <th>VEHICLE TYPE CODE 2</th>\n",
       "      <th>VEHICLE TYPE CODE 3</th>\n",
       "      <th>VEHICLE TYPE CODE 4</th>\n",
       "      <th>VEHICLE TYPE CODE 5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11/10/2016</td>\n",
       "      <td>16:11:00</td>\n",
       "      <td>BROOKLYN</td>\n",
       "      <td>11208.0</td>\n",
       "      <td>40.662514</td>\n",
       "      <td>-73.872007</td>\n",
       "      <td>(40.6625139, -73.8720068)</td>\n",
       "      <td>WORTMAN AVENUE</td>\n",
       "      <td>MONTAUK AVENUE</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>Failure to Yield Right-of-Way</td>\n",
       "      <td>Unspecified</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TAXI</td>\n",
       "      <td>PASSENGER VEHICLE</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11/10/2016</td>\n",
       "      <td>05:11:00</td>\n",
       "      <td>MANHATTAN</td>\n",
       "      <td>10013.0</td>\n",
       "      <td>40.721323</td>\n",
       "      <td>-74.008344</td>\n",
       "      <td>(40.7213228, -74.0083444)</td>\n",
       "      <td>HUBERT STREET</td>\n",
       "      <td>HUDSON STREET</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>Failure to Yield Right-of-Way</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>PASSENGER VEHICLE</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>04/16/2016</td>\n",
       "      <td>09:15:00</td>\n",
       "      <td>BROOKLYN</td>\n",
       "      <td>11201.0</td>\n",
       "      <td>40.687999</td>\n",
       "      <td>-73.997563</td>\n",
       "      <td>(40.6879989, -73.9975625)</td>\n",
       "      <td>HENRY STREET</td>\n",
       "      <td>WARREN STREET</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>Lost Consciousness</td>\n",
       "      <td>Lost Consciousness</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>PASSENGER VEHICLE</td>\n",
       "      <td>VAN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>04/15/2016</td>\n",
       "      <td>10:20:00</td>\n",
       "      <td>QUEENS</td>\n",
       "      <td>11375.0</td>\n",
       "      <td>40.719228</td>\n",
       "      <td>-73.854542</td>\n",
       "      <td>(40.7192276, -73.8545422)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>67-64 FLEET STREET</td>\n",
       "      <td>...</td>\n",
       "      <td>Failure to Yield Right-of-Way</td>\n",
       "      <td>Failure to Yield Right-of-Way</td>\n",
       "      <td>Failure to Yield Right-of-Way</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>PASSENGER VEHICLE</td>\n",
       "      <td>PASSENGER VEHICLE</td>\n",
       "      <td>PASSENGER VEHICLE</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>04/15/2016</td>\n",
       "      <td>10:35:00</td>\n",
       "      <td>BROOKLYN</td>\n",
       "      <td>11210.0</td>\n",
       "      <td>40.632147</td>\n",
       "      <td>-73.952731</td>\n",
       "      <td>(40.6321467, -73.9527315)</td>\n",
       "      <td>BEDFORD AVENUE</td>\n",
       "      <td>CAMPUS ROAD</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>Failure to Yield Right-of-Way</td>\n",
       "      <td>Failure to Yield Right-of-Way</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>PASSENGER VEHICLE</td>\n",
       "      <td>PASSENGER VEHICLE</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 26 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         DATE      TIME    BOROUGH  ZIP CODE   LATITUDE  LONGITUDE  \\\n",
       "0  11/10/2016  16:11:00   BROOKLYN   11208.0  40.662514 -73.872007   \n",
       "1  11/10/2016  05:11:00  MANHATTAN   10013.0  40.721323 -74.008344   \n",
       "2  04/16/2016  09:15:00   BROOKLYN   11201.0  40.687999 -73.997563   \n",
       "3  04/15/2016  10:20:00     QUEENS   11375.0  40.719228 -73.854542   \n",
       "4  04/15/2016  10:35:00   BROOKLYN   11210.0  40.632147 -73.952731   \n",
       "\n",
       "                    LOCATION  ON STREET NAME CROSS STREET NAME  \\\n",
       "0  (40.6625139, -73.8720068)  WORTMAN AVENUE    MONTAUK AVENUE   \n",
       "1  (40.7213228, -74.0083444)   HUBERT STREET     HUDSON STREET   \n",
       "2  (40.6879989, -73.9975625)    HENRY STREET     WARREN STREET   \n",
       "3  (40.7192276, -73.8545422)             NaN               NaN   \n",
       "4  (40.6321467, -73.9527315)  BEDFORD AVENUE       CAMPUS ROAD   \n",
       "\n",
       "      OFF STREET NAME  ...  CONTRIBUTING FACTOR VEHICLE 1  \\\n",
       "0                 NaN  ...  Failure to Yield Right-of-Way   \n",
       "1                 NaN  ...  Failure to Yield Right-of-Way   \n",
       "2                 NaN  ...             Lost Consciousness   \n",
       "3  67-64 FLEET STREET  ...  Failure to Yield Right-of-Way   \n",
       "4                 NaN  ...  Failure to Yield Right-of-Way   \n",
       "\n",
       "   CONTRIBUTING FACTOR VEHICLE 2  CONTRIBUTING FACTOR VEHICLE 3  \\\n",
       "0                    Unspecified                            NaN   \n",
       "1                            NaN                            NaN   \n",
       "2             Lost Consciousness                            NaN   \n",
       "3  Failure to Yield Right-of-Way  Failure to Yield Right-of-Way   \n",
       "4  Failure to Yield Right-of-Way                            NaN   \n",
       "\n",
       "   CONTRIBUTING FACTOR VEHICLE 4  CONTRIBUTING FACTOR VEHICLE 5  \\\n",
       "0                            NaN                            NaN   \n",
       "1                            NaN                            NaN   \n",
       "2                            NaN                            NaN   \n",
       "3                            NaN                            NaN   \n",
       "4                            NaN                            NaN   \n",
       "\n",
       "   VEHICLE TYPE CODE 1 VEHICLE TYPE CODE 2 VEHICLE TYPE CODE 3  \\\n",
       "0                 TAXI   PASSENGER VEHICLE                 NaN   \n",
       "1    PASSENGER VEHICLE                 NaN                 NaN   \n",
       "2    PASSENGER VEHICLE                 VAN                 NaN   \n",
       "3    PASSENGER VEHICLE   PASSENGER VEHICLE   PASSENGER VEHICLE   \n",
       "4    PASSENGER VEHICLE   PASSENGER VEHICLE                 NaN   \n",
       "\n",
       "  VEHICLE TYPE CODE 4 VEHICLE TYPE CODE 5  \n",
       "0                 NaN                 NaN  \n",
       "1                 NaN                 NaN  \n",
       "2                 NaN                 NaN  \n",
       "3                 NaN                 NaN  \n",
       "4                 NaN                 NaN  \n",
       "\n",
       "[5 rows x 26 columns]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "collisions.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DATE                              object\n",
       "TIME                              object\n",
       "BOROUGH                           object\n",
       "ZIP CODE                         float64\n",
       "LATITUDE                         float64\n",
       "LONGITUDE                        float64\n",
       "LOCATION                          object\n",
       "ON STREET NAME                    object\n",
       "CROSS STREET NAME                 object\n",
       "OFF STREET NAME                   object\n",
       "NUMBER OF PERSONS INJURED          int64\n",
       "NUMBER OF PERSONS KILLED           int64\n",
       "NUMBER OF PEDESTRIANS INJURED      int64\n",
       "NUMBER OF PEDESTRIANS KILLED       int64\n",
       "NUMBER OF CYCLISTS INJURED       float64\n",
       "NUMBER OF CYCLISTS KILLED        float64\n",
       "CONTRIBUTING FACTOR VEHICLE 1     object\n",
       "CONTRIBUTING FACTOR VEHICLE 2     object\n",
       "CONTRIBUTING FACTOR VEHICLE 3     object\n",
       "CONTRIBUTING FACTOR VEHICLE 4     object\n",
       "CONTRIBUTING FACTOR VEHICLE 5     object\n",
       "VEHICLE TYPE CODE 1               object\n",
       "VEHICLE TYPE CODE 2               object\n",
       "VEHICLE TYPE CODE 3               object\n",
       "VEHICLE TYPE CODE 4               object\n",
       "VEHICLE TYPE CODE 5               object\n",
       "dtype: object"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "collisions.dtypes"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will then inspect the basic statistics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ZIP CODE</th>\n",
       "      <th>LATITUDE</th>\n",
       "      <th>LONGITUDE</th>\n",
       "      <th>NUMBER OF PERSONS INJURED</th>\n",
       "      <th>NUMBER OF PERSONS KILLED</th>\n",
       "      <th>NUMBER OF PEDESTRIANS INJURED</th>\n",
       "      <th>NUMBER OF PEDESTRIANS KILLED</th>\n",
       "      <th>NUMBER OF CYCLISTS INJURED</th>\n",
       "      <th>NUMBER OF CYCLISTS KILLED</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>6919.000000</td>\n",
       "      <td>7303.000000</td>\n",
       "      <td>7303.000000</td>\n",
       "      <td>7303.000000</td>\n",
       "      <td>7303.000000</td>\n",
       "      <td>7303.000000</td>\n",
       "      <td>7303.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>10900.746640</td>\n",
       "      <td>40.717653</td>\n",
       "      <td>-73.921406</td>\n",
       "      <td>0.350678</td>\n",
       "      <td>0.000959</td>\n",
       "      <td>0.133644</td>\n",
       "      <td>0.000822</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>551.568724</td>\n",
       "      <td>0.069437</td>\n",
       "      <td>0.083317</td>\n",
       "      <td>0.707873</td>\n",
       "      <td>0.030947</td>\n",
       "      <td>0.362129</td>\n",
       "      <td>0.028653</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>10001.000000</td>\n",
       "      <td>40.502341</td>\n",
       "      <td>-74.248277</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>10310.000000</td>\n",
       "      <td>40.670865</td>\n",
       "      <td>-73.980744</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>11211.000000</td>\n",
       "      <td>40.723260</td>\n",
       "      <td>-73.933888</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>11355.000000</td>\n",
       "      <td>40.759527</td>\n",
       "      <td>-73.864463</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>11694.000000</td>\n",
       "      <td>40.909628</td>\n",
       "      <td>-73.702590</td>\n",
       "      <td>16.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           ZIP CODE     LATITUDE    LONGITUDE  NUMBER OF PERSONS INJURED  \\\n",
       "count   6919.000000  7303.000000  7303.000000                7303.000000   \n",
       "mean   10900.746640    40.717653   -73.921406                   0.350678   \n",
       "std      551.568724     0.069437     0.083317                   0.707873   \n",
       "min    10001.000000    40.502341   -74.248277                   0.000000   \n",
       "25%    10310.000000    40.670865   -73.980744                   0.000000   \n",
       "50%    11211.000000    40.723260   -73.933888                   0.000000   \n",
       "75%    11355.000000    40.759527   -73.864463                   1.000000   \n",
       "max    11694.000000    40.909628   -73.702590                  16.000000   \n",
       "\n",
       "       NUMBER OF PERSONS KILLED  NUMBER OF PEDESTRIANS INJURED  \\\n",
       "count               7303.000000                    7303.000000   \n",
       "mean                   0.000959                       0.133644   \n",
       "std                    0.030947                       0.362129   \n",
       "min                    0.000000                       0.000000   \n",
       "25%                    0.000000                       0.000000   \n",
       "50%                    0.000000                       0.000000   \n",
       "75%                    0.000000                       0.000000   \n",
       "max                    1.000000                       3.000000   \n",
       "\n",
       "       NUMBER OF PEDESTRIANS KILLED  NUMBER OF CYCLISTS INJURED  \\\n",
       "count                   7303.000000                         0.0   \n",
       "mean                       0.000822                         NaN   \n",
       "std                        0.028653                         NaN   \n",
       "min                        0.000000                         NaN   \n",
       "25%                        0.000000                         NaN   \n",
       "50%                        0.000000                         NaN   \n",
       "75%                        0.000000                         NaN   \n",
       "max                        1.000000                         NaN   \n",
       "\n",
       "       NUMBER OF CYCLISTS KILLED  \n",
       "count                        0.0  \n",
       "mean                         NaN  \n",
       "std                          NaN  \n",
       "min                          NaN  \n",
       "25%                          NaN  \n",
       "50%                          NaN  \n",
       "75%                          NaN  \n",
       "max                          NaN  "
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "collisions.describe()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we will visualize the nullity (how strongly the presence or absence of one variable affects the presence of another) correlations by a heatmap and a dendrogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 120304 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = missingness.heatmap(collisions)\n",
    "fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The nullity heatmap and dendrogram reveals a data correlation structure, e.g., *vehicle type codes* and *contributing factor vehicle* are highly correlated. Features having complete data are not shown on the figure."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
