{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nkF_4ePZHNzg"
   },
   "source": [
    "# Exploring the EvalRS dataset\n",
    "\n",
    "This notebook provides some basic EDA on the dataset of the [EvalRS](https://reclist.io/cikm2022-cup/) challenge.\n",
    "Please refer to [our paper](https://arxiv.org/abs/2207.05772) for all the details.\n",
    "\n",
    "With the goal of building a *all-around* recommender system, we likely want to understand how users, tracks and listening events are distributed, with a special case to *long tails* and rare (or new events). Yeah, you can thank us for the tip later.\n",
    "\n",
    "Before we start, we need to check we are inside the right virtual environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-08-04T18:34:13.645302Z",
     "iopub.status.busy": "2022-08-04T18:34:13.644435Z",
     "iopub.status.idle": "2022-08-04T18:35:00.150666Z",
     "shell.execute_reply": "2022-08-04T18:35:00.148978Z",
     "shell.execute_reply.started": "2022-08-04T18:34:13.645133Z"
    },
    "id": "dljiNmSJjr-u"
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "# check we are using the right interpreter with the right RecList version\n",
    "!which python\n",
    "!pip install -r ../requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "Mv-L2X8Fj0fw",
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "sys.path.insert(0, '../')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "_-FxCXfaoFNU",
    "tags": []
   },
   "outputs": [],
   "source": [
    "from evaluation.EvalRSRunner import ChallengeDataset\n",
    "\n",
    "import gc\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "sns.set_theme(\"notebook\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we have to download the dataset (~600MB) -- actually, you'll download it just the first time!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hij5x-hOoHKx",
    "outputId": "857b78b1-e056-4836-8442-877deb0e721f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LFM dataset already downloaded. Skipping download.\n",
      "Loading dataset.\n",
      "Generating folds.\n"
     ]
    }
   ],
   "source": [
    "dataset = ChallengeDataset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And get a sample fold (see details on our Bootstrapping Cross-Validation evaluation strategy to know more about folds)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "K-UJp05VoJVD",
    "outputId": "6d6f97fc-dcf8-4904-86b5-5d6853945ac0"
   },
   "outputs": [],
   "source": [
    "train, test = dataset.get_sample_train_test()\n",
    "train.shape, test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "n5z3WQ1iQF_Z",
    "outputId": "71918ba2-f965-44fd-9068-7bd28cec7d53"
   },
   "outputs": [],
   "source": [
    "train.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So. We have ~7M training listening records. Each tells us who (*user_id*) listened to what (*track_it*), when (*timestamp*), and how many times (*user_track_count*).\n",
    "\n",
    "Let's now run some preprocessing to get started."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert the timestamp in a more convenient format\n",
    "train[\"timestamp\"] = pd.to_datetime(train[\"timestamp\"], unit=\"s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add a column for Hour Of the Day\n",
    "train[\"hod\"] = train[\"timestamp\"].dt.hour"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are ready to do some EDA."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yxlmpAMSQAYD"
   },
   "source": [
    "## EDA on Tracks\n",
    "\n",
    "Let's begin with a few statistics about tracks. We can inspect how many *unique* tracks are there! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "percentiles = np.arange(1, step=0.1)\n",
    "gb_track = train.groupby(\"track_id\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q0: what's the Track distribution?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "track_records = gb_track.size()\n",
    "track_records.describe(percentiles)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are ~230K distinct tracks. But wait. We know there is a `timestamp` feature, does that mean that a user can play the same track multiple times? Programmatically, we should check if the pair <`user`,`track`> does appear multiple times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(train.groupby([\"user_id\", \"track_id\"]).size()) == train.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The two sizes correspond: we can assume there are no <`user`,`track`> duplicates. Indeed, the `user_track_count` value lets us know the play count!\n",
    "\n",
    "We also obtained a new information: `track_records` contains the number of distinct users that played a given track. Let's now observe the distribution graphically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(12,8))\n",
    "ax.hist(track_records)\n",
    "ax.set_yscale(\"log\")\n",
    "ax.set_xlabel(\"Distinct Users\")\n",
    "ax.set_ylabel(\"Track Count (log)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the distribution in percentiles again. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "nQk3NbVYpsE2",
    "outputId": "1bd40fd4-a366-43da-e316-6449648beca0"
   },
   "outputs": [],
   "source": [
    "track_records.describe(percentiles)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tracks are listened by 30 distinct users on the average, and that 90% of them has less than 58 distinct users that played it. Further, there are some outliers: the maximum number of users is ~1600. Is this a very popular track or a dataset artifact? Let's find out.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "K = 10\n",
    "top_by_user = track_records.iloc[track_records.argsort().values[:-(K+1):-1]]\n",
    "top_by_user.name = \"distinct_user_count\"\n",
    "\n",
    "# Retrieve the task information from the `dataset` handler variable\n",
    "dataset.df_tracks.loc[top_by_user.index].join(top_by_user)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, these song seem pretty popular, right? But something more came up... look at position **3** and **6**: there are two different versions of *Get Lucky*. Is this something you want your model to factor in at training time? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q1: How many times does a user play the same song?**\n",
    "\n",
    "We can use the handy `user_track_count` to measure how frequently users get engaged by songs (remember to take a look at all the columns in `dataset.df_users`, there are very userful variable there)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "avg_play_count = gb_track[\"user_track_count\"].mean()\n",
    "avg_play_count.describe(percentiles)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the average, a user repeats a given track ~2.6 times. Again, let's inspect our tail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "top_by_user = avg_play_count.iloc[avg_play_count.argsort().values[:-(K+1):-1]]\n",
    "top_by_user.name = \"avg_play_count\"\n",
    "dataset.df_tracks.loc[top_by_user.index].join(top_by_user)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here it is a new piece of information: the top 10 tracks by number of play counts is completely different than the one by number of distinct users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cleanup..\n",
    "del gb_track\n",
    "del avg_play_count\n",
    "del track_records\n",
    "del top_by_user\n",
    "_ = gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EDA on Users \n",
    "\n",
    "Let's move to users now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gb_user = train.groupby(\"user_id\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_records = gb_user.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_records.describe(percentiles=percentiles)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are almost 30K distinct users in our dataset. The average number of song they played is 232 with some binge listener that made it to 494.\n",
    "\n",
    "What about their demographics?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_users = train.join(dataset.df_users, on=\"user_id\", rsuffix=\"_data\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gb_train_users = train_users.groupby(\"user_id\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gb_train_users.first().gender.value_counts(normalize=True, dropna=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Only half of our users are annotated with (binary) gender, and there is a significant skew towards males."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "users_by_country = gb_train_users.first().country.value_counts(normalize=True, sort=True, ascending=False, dropna=False)\n",
    "users_by_country"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "top_countries = gb_train_users.first().country.value_counts(dropna=True, sort=True, ascending=False)[:10]\n",
    "top_countries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(12,8))\n",
    "ax.bar(x=top_countries.index, height=top_countries.values)\n",
    "ax.set_xlabel(\"Country\")\n",
    "ax.set_ylabel(\"Count\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 158 distinct countries but only half of the users are annotated with that information. Most of the users was logged from the United States."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "del gb_user\n",
    "del gb_train_users\n",
    "del top_countries\n",
    "_ = gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analysis on listening events\n",
    "\n",
    "We have explored relationships between users and tracks. But what about the playing time and hour? \n",
    "\n",
    "For example, one might search for any correlation between listening events and hour of the day. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q0: Is there any listening pattern across hours of the day?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(12,8))\n",
    "ax = sns.countplot(data=train_users, x=\"hod\")\n",
    "ax.set_xlabel(\"Hour of the Day\")\n",
    "ax.set_ylabel(\"Count\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that the `timestamp` attribute is in Unix time, i.e. it is absolute in terms of timezone. The chart is showing us that events are mainly located in a single timezone, as the events decrease in the morning and increase in late afternoon.   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q1: Is there any listening pattern across hours of the day, separately by gender?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(12,8))\n",
    "ax = sns.countplot(data=train_users, x=\"hod\", hue=\"gender\")\n",
    "ax.set_xlabel(\"Hour of the Day\")\n",
    "ax.set_ylabel(\"Count\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Judging by the chart, there seem to be no evident difference. Well, beware!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
