{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random model tutorial\n",
    "\n",
    "\n",
    "In this notebook, we present a more verbose version of the standard submission.py script, with the aim of explaining in detail how the main abstractions work and showing how easy it is to partecipate in the challenge. \n",
    "\n",
    "_NOTE_: this notebook is meant as a coding guide to the evaluation script, and a walk-through baseline submission to explain how to partecipate in the challenge. While you're free to experiment with this or other notebooks and even submit to the leaderboard from here, the _final_ submission should comply with the template scripts, as explained in the README.\n",
    "\n",
    "Please contact the organizers on Slack should you have any doubt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check we are using the right interpreter with the right RecList version\n",
    "!which python\n",
    "!pip install -r ../requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "sys.path.insert(0, '../')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Basic imports, read the credentials from the env file_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "load_dotenv('../upload.env')\n",
    "\n",
    "EMAIL = os.getenv('EMAIL')  # the e-mail you used to sign up\n",
    "assert EMAIL != '' and EMAIL is not None\n",
    "BUCKET_NAME = os.getenv('BUCKET_NAME') # you received it in your e-mail\n",
    "PARTICIPANT_ID = os.getenv('PARTICIPANT_ID') # you received it in your e-mail\n",
    "AWS_ACCESS_KEY = os.getenv('AWS_ACCESS_KEY') # you received it in your e-mail\n",
    "AWS_SECRET_KEY = os.getenv('AWS_SECRET_KEY') # you received it in your e-mail"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Specify some other global variables to improve local iteration and debugging, for example setting a LIMIT to work with a smaller, faster test set_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "LIMIT = 1000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_NOTE: as long as there is a limit specified, the runner won't upload results: make sure to have LIMIT=0 when you want to submit to the leaderboard!_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from evaluation.EvalRSRunner import EvalRSRunner\n",
    "from evaluation.EvalRSRunner import ChallengeDataset\n",
    "from reclist.abstractions import RecModel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Declare our model, in this case, a random generator: any model needs to include an implementation of \"train\" \"predict\", taking user IDs as input and returning a DataFrame with predictions as output._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyModel(RecModel):\n",
    "    \n",
    "    def __init__(self, items: pd.DataFrame, top_k: int=100, **kwargs):\n",
    "        super(MyModel, self).__init__()\n",
    "        self.items = items\n",
    "        self.top_k = top_k\n",
    "        # kwargs may contain additional arguments in case, for example, you\n",
    "        # have data augmentation strategies\n",
    "        print(\"Received additional arguments: {}\".format(kwargs))\n",
    "        return\n",
    "\n",
    "    def train(self, train_df: pd.DataFrame):\n",
    "        \"\"\"\n",
    "        Implement here your training logic. Since our example method is a simple random model,\n",
    "        we actually don't use any training data to build the model, but you should ;-)\n",
    "\n",
    "        At the end of training, make sure the class contains a trained model you can use in the predict method.\n",
    "        \"\"\"\n",
    "        print(train_df.head(1))\n",
    "        print(\"Training completed!\")\n",
    "        return \n",
    "\n",
    "    def predict(self, user_ids: pd.DataFrame) -> pd.DataFrame:\n",
    "        \"\"\"\n",
    "        \n",
    "        This function takes as input all the users that we want to predict the top-k items for, and \n",
    "        returns all the predicted songs.\n",
    "\n",
    "        While in this example is just a random generator, the same logic in your implementation \n",
    "        would allow for batch predictions of all the target data points.\n",
    "        \n",
    "        \"\"\"\n",
    "        k = self.top_k\n",
    "        num_users = len(user_ids)\n",
    "        pred = self.items.sample(n=k*num_users, replace=True).index.values\n",
    "        pred = pred.reshape(num_users, k)\n",
    "        pred = np.concatenate((user_ids[['user_id']].values, pred), axis=1)\n",
    "        return pd.DataFrame(pred, columns=['user_id', *[str(i) for i in range(k)]]).set_index('user_id')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Get the dataset and inspect the basic entities: tracks, users, and the interaction dataset_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "dataset = ChallengeDataset(force_download=False)  # note, if YES, the dataset will be donwloaded again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.df_tracks.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.df_users.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, test = dataset.get_sample_train_test()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(train.shape, test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_When we are happy with our model class, we can instantiate it and then initialize the runner with our credentials_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_model = MyModel(\n",
    "    items=dataset.df_tracks,\n",
    "    # kwargs may contain additional arguments that you wish to use\n",
    "    my_custom_argument='my_custom_argument' \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "runner = EvalRSRunner(\n",
    "    dataset=dataset,\n",
    "    aws_access_key_id=AWS_ACCESS_KEY,\n",
    "    aws_secret_access_key=AWS_SECRET_KEY,\n",
    "    participant_id=PARTICIPANT_ID,\n",
    "    bucket_name=BUCKET_NAME,\n",
    "    email=EMAIL\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Finally, we run the evaluation code: remember, if LIMIT is not 0, your submission won't be uploaded but the loop may still be useful for you to debug / iterate locally_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "runner.evaluate(model=my_model, limit=LIMIT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Customizing RecList for your submission\n",
    "\n",
    "A huge motivation behind the Challenge is building as a community shareable insights in the form of working tests for our use case.\n",
    "\n",
    "While your leaderboard score is ONLY influenced by the official tests as stated in the evaluation README, we ask that your final submission must also include custom tests that you found helpful / insightful when improving your model.\n",
    "\n",
    "The snippet below shows a working example of how to _extend_ the default RecList with additional tests, and run the same evaluation code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from reclist.abstractions import rec_test\n",
    "from evaluation.EvalRSRecList import EvalRSRecList\n",
    "\n",
    "class myRecList(EvalRSRecList):\n",
    "    \n",
    "    @rec_test(test_type='custom_test')\n",
    "    def lucky_user_test(self):\n",
    "        \"\"\"\n",
    "        Custom test, returning my lucky user from the catalog\n",
    "        \"\"\"\n",
    "        from random import choice\n",
    "\n",
    "        return {\n",
    "          \"luck_user\": str(choice(self._x_test['user_id'].unique())) \n",
    "        }\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Re-run the evaluation with the additional test, which gets executed together with the default ones that produce the leaderboard score._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "runner.evaluate(\n",
    "    model=my_model, \n",
    "    limit=1,\n",
    "    custom_RecList=myRecList\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "source": [
    "### Final submission to the committee\n",
    "\n",
    "Since this is a code competition, you'll be required to submit your repository for statistical verification of your scores. \n",
    "\n",
    "Please consult the README carefully to make sure your project complies with the rules and follows the provided template script."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
