{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49a1a69c-8605-47b1-842b-005b8f0e89ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "#default_exp data.datasets.m5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c3dbb4c-07aa-4421-a8f1-3cc36f4c9777",
   "metadata": {},
   "source": [
    "# M5 dataset\n",
    "\n",
    "> Download and evaluate the M5 dataset.\n",
    "\n",
    "The 2020 M5 competition was organized by the Makridakis Open Forecasting Center was organized by University of Nicosia and hosted in Kaggle. The main task consisted in providing hierarchical point and probabilistic predictions for 28 days ahead for the sales of Walmart stores in three US States (California, Texas, and Wisconsin). The dataset is organized at the item, department, product and store level details. Additionally it contains temporal covariates like price, promotions, special event and calendar variables.\n",
    "\n",
    "\n",
    "[University of Nicosia. The M5 competition: Estimate the unit sales of Walmart retail goods. Kaggle Competition, 2020.](https://www.kaggle.com/c/m5-forecasting-accuracy/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1c7e36a-58cf-44c5-9638-c1aa3c108cc0",
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "from nbdev import *\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4c83ec4-30a2-4229-9d54-b836f762f5ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "import os\n",
    "from dataclasses import dataclass\n",
    "from typing import Dict, List, Optional, Tuple, Union\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from nixtlats.data.datasets.utils import download_file, Info"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e30abbd7-91b7-43af-904c-975445af26d4",
   "metadata": {},
   "source": [
    "## Download data class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ede0cfa5-fe76-43fb-b835-cefc69ececd0",
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "@dataclass\n",
    "class M5:\n",
    "    \n",
    "    # original data available from Kaggle directly\n",
    "    # pip install kaggle --upgrade\n",
    "    # kaggle competitions download -c m5-forecasting-accuracy\n",
    "    source_url: str = 'https://github.com/Nixtla/m5-forecasts/raw/main/datasets/m5.zip'\n",
    "    \n",
    "    @staticmethod\n",
    "    def download(directory: str) -> None:\n",
    "        \"\"\"Downloads M5 Competition Dataset.\"\"\"\n",
    "        path = f'{directory}/m5/datasets'\n",
    "        if not os.path.exists(path):\n",
    "            download_file(directory=path,\n",
    "                          source_url=M5.source_url,\n",
    "                          decompress=True)\n",
    "            \n",
    "    @staticmethod\n",
    "    def load(directory: str, cache: bool = True) -> Tuple[pd.DataFrame, \n",
    "                                                          pd.DataFrame, \n",
    "                                                          pd.DataFrame]:\n",
    "        \"\"\"Downloads and loads M5 data.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        directory: str\n",
    "            Directory where data will be downloaded.\n",
    "        cache: bool\n",
    "            If `True` saves and loads.\n",
    "            \n",
    "        Notes\n",
    "        -----\n",
    "        [1] Returns train+test sets.\n",
    "        [2] Based on https://www.kaggle.com/lemuz90/m5-preprocess.\n",
    "        \"\"\"\n",
    "        path = f'{directory}/m5/datasets'\n",
    "        file_cache = f'{path}/m5.p'\n",
    "        \n",
    "        if os.path.exists(file_cache) and cache:\n",
    "            Y_df, X_df, S_df = pd.read_pickle(file_cache)\n",
    "            \n",
    "            return Y_df, X_df, S_df\n",
    "        \n",
    "        M5.download(directory)\n",
    "        # Calendar data\n",
    "        cal_dtypes = {\n",
    "            'wm_yr_wk': np.uint16,\n",
    "            'event_name_1': 'category',\n",
    "            'event_type_1': 'category',\n",
    "            'event_name_2': 'category',\n",
    "            'event_type_2': 'category',\n",
    "            'snap_CA': np.uint8,\n",
    "            'snap_TX': np.uint8,\n",
    "            'snap_WI': np.uint8,\n",
    "        }\n",
    "        cal = pd.read_csv(f'{path}/calendar.csv', \n",
    "                          dtype=cal_dtypes, \n",
    "                          usecols=list(cal_dtypes.keys()) + ['date'], \n",
    "                          parse_dates=['date'])\n",
    "        cal['d'] = np.arange(cal.shape[0]) + 1\n",
    "        cal['d'] = 'd_' + cal['d'].astype('str')\n",
    "        cal['d'] = cal['d'].astype('category')\n",
    "        \n",
    "        event_cols = [k for k in cal_dtypes if k.startswith('event')]\n",
    "        for col in event_cols:\n",
    "            cal[col] = cal[col].cat.add_categories('nan').fillna('nan')\n",
    "        \n",
    "        # Prices\n",
    "        prices_dtypes = {\n",
    "            'store_id': 'category',\n",
    "            'item_id': 'category',\n",
    "            'wm_yr_wk': np.uint16,\n",
    "            'sell_price': np.float32\n",
    "        }\n",
    "\n",
    "        prices = pd.read_csv(f'{path}/sell_prices.csv', \n",
    "                             dtype=prices_dtypes)\n",
    "        \n",
    "        # Sales\n",
    "        sales_dtypes = {\n",
    "            'item_id': prices.item_id.dtype,\n",
    "            'dept_id': 'category',\n",
    "            'cat_id': 'category',\n",
    "            'store_id': 'category',\n",
    "            'state_id': 'category',\n",
    "            **{f'd_{i+1}': np.float32 for i in range(1969)}\n",
    "        }\n",
    "        # Reading train and test sets\n",
    "        sales_train = pd.read_csv(f'{path}/sales_train_evaluation.csv', \n",
    "                                  dtype=sales_dtypes)\n",
    "        sales_test = pd.read_csv(f'{path}/sales_test_evaluation.csv', \n",
    "                                 dtype=sales_dtypes)\n",
    "        sales = sales_train.merge(sales_test, how='left', \n",
    "                                  on=['item_id', 'dept_id', 'cat_id', 'store_id', 'state_id'])\n",
    "        sales['id'] = sales[['item_id', 'store_id']].astype(str).agg('_'.join, axis=1).astype('category')\n",
    "        # Long format\n",
    "        long = sales.melt(id_vars=['id', 'item_id', 'dept_id', 'cat_id', 'store_id', 'state_id'], \n",
    "                          var_name='d', value_name='y')\n",
    "        long['d'] = long['d'].astype(cal.d.dtype)\n",
    "        long = long.merge(cal, on=['d'])\n",
    "        long = long.merge(prices, on=['store_id', 'item_id', 'wm_yr_wk'])\n",
    "        long = long.drop(columns=['d', 'wm_yr_wk'])\n",
    "        \n",
    "        def first_nz_mask(values, index):\n",
    "            \"\"\"Return a boolean mask where the True starts at the first non-zero value.\"\"\"\n",
    "            mask = np.full(values.size, True)\n",
    "            for idx, value in enumerate(values):\n",
    "                if value == 0:\n",
    "                    mask[idx] = False\n",
    "                else:\n",
    "                    break\n",
    "            return mask\n",
    "        \n",
    "        long = long.sort_values(['id', 'date'], ignore_index=True)\n",
    "        keep_mask = long.groupby('id')['y'].transform(first_nz_mask, engine='numba')\n",
    "        long = long[keep_mask.astype(bool)]\n",
    "        long.rename(columns={'id': 'unique_id', 'date': 'ds'}, inplace=True)\n",
    "        Y_df = long.filter(items=['unique_id', 'ds', 'y'])\n",
    "        cats = ['item_id', 'dept_id', 'cat_id', 'store_id', 'state_id']\n",
    "        S_df = long.filter(items=['unique_id'] + cats)\n",
    "        S_df = S_df.drop_duplicates(ignore_index=True)\n",
    "        X_df = long.drop(columns=['y'] + cats)\n",
    "        \n",
    "        if cache:\n",
    "            pd.to_pickle((Y_df, X_df, S_df), file_cache)\n",
    "        \n",
    "        return Y_df, X_df, S_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f11ef7c6-4878-48e8-bc88-a17b23a91d77",
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_df, X_df, S_df = M5.load('./data')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f409f99-0306-4a71-bbb5-c4ce19a273e2",
   "metadata": {},
   "source": [
    "## Test number of series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3920d220-646d-410a-be2c-010b0e7ec089",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_series = 30_490\n",
    "assert Y_df['unique_id'].unique().size == n_series\n",
    "assert X_df['unique_id'].unique().size == n_series\n",
    "assert S_df.shape[0] == 30_490"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a576b88-13e5-485d-8696-a331e0b6909b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>unique_id</th>\n",
       "      <th>ds</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>FOODS_1_001_CA_1</td>\n",
       "      <td>2011-01-29</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>FOODS_1_001_CA_1</td>\n",
       "      <td>2011-01-30</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>FOODS_1_001_CA_1</td>\n",
       "      <td>2011-01-31</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>FOODS_1_001_CA_1</td>\n",
       "      <td>2011-02-01</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>FOODS_1_001_CA_1</td>\n",
       "      <td>2011-02-02</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          unique_id         ds    y\n",
       "0  FOODS_1_001_CA_1 2011-01-29  3.0\n",
       "1  FOODS_1_001_CA_1 2011-01-30  0.0\n",
       "2  FOODS_1_001_CA_1 2011-01-31  0.0\n",
       "3  FOODS_1_001_CA_1 2011-02-01  1.0\n",
       "4  FOODS_1_001_CA_1 2011-02-02  4.0"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98e3e6a0-3aef-46ff-aafe-ae016dd030ef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>unique_id</th>\n",
       "      <th>ds</th>\n",
       "      <th>event_name_1</th>\n",
       "      <th>event_type_1</th>\n",
       "      <th>event_name_2</th>\n",
       "      <th>event_type_2</th>\n",
       "      <th>snap_CA</th>\n",
       "      <th>snap_TX</th>\n",
       "      <th>snap_WI</th>\n",
       "      <th>sell_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>FOODS_1_001_CA_1</td>\n",
       "      <td>2011-01-29</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>FOODS_1_001_CA_1</td>\n",
       "      <td>2011-01-30</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>FOODS_1_001_CA_1</td>\n",
       "      <td>2011-01-31</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>FOODS_1_001_CA_1</td>\n",
       "      <td>2011-02-01</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>FOODS_1_001_CA_1</td>\n",
       "      <td>2011-02-02</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          unique_id         ds event_name_1 event_type_1 event_name_2  \\\n",
       "0  FOODS_1_001_CA_1 2011-01-29          nan          nan          nan   \n",
       "1  FOODS_1_001_CA_1 2011-01-30          nan          nan          nan   \n",
       "2  FOODS_1_001_CA_1 2011-01-31          nan          nan          nan   \n",
       "3  FOODS_1_001_CA_1 2011-02-01          nan          nan          nan   \n",
       "4  FOODS_1_001_CA_1 2011-02-02          nan          nan          nan   \n",
       "\n",
       "  event_type_2  snap_CA  snap_TX  snap_WI  sell_price  \n",
       "0          nan        0        0        0         2.0  \n",
       "1          nan        0        0        0         2.0  \n",
       "2          nan        0        0        0         2.0  \n",
       "3          nan        1        1        0         2.0  \n",
       "4          nan        1        0        1         2.0  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7326a877-309b-4cf7-8b8c-1d07fbd2ccf9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>unique_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>dept_id</th>\n",
       "      <th>cat_id</th>\n",
       "      <th>store_id</th>\n",
       "      <th>state_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>FOODS_1_001_CA_1</td>\n",
       "      <td>FOODS_1_001</td>\n",
       "      <td>FOODS_1</td>\n",
       "      <td>FOODS</td>\n",
       "      <td>CA_1</td>\n",
       "      <td>CA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>FOODS_1_001_CA_2</td>\n",
       "      <td>FOODS_1_001</td>\n",
       "      <td>FOODS_1</td>\n",
       "      <td>FOODS</td>\n",
       "      <td>CA_2</td>\n",
       "      <td>CA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>FOODS_1_001_CA_3</td>\n",
       "      <td>FOODS_1_001</td>\n",
       "      <td>FOODS_1</td>\n",
       "      <td>FOODS</td>\n",
       "      <td>CA_3</td>\n",
       "      <td>CA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>FOODS_1_001_CA_4</td>\n",
       "      <td>FOODS_1_001</td>\n",
       "      <td>FOODS_1</td>\n",
       "      <td>FOODS</td>\n",
       "      <td>CA_4</td>\n",
       "      <td>CA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>FOODS_1_001_TX_1</td>\n",
       "      <td>FOODS_1_001</td>\n",
       "      <td>FOODS_1</td>\n",
       "      <td>FOODS</td>\n",
       "      <td>TX_1</td>\n",
       "      <td>TX</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          unique_id      item_id  dept_id cat_id store_id state_id\n",
       "0  FOODS_1_001_CA_1  FOODS_1_001  FOODS_1  FOODS     CA_1       CA\n",
       "1  FOODS_1_001_CA_2  FOODS_1_001  FOODS_1  FOODS     CA_2       CA\n",
       "2  FOODS_1_001_CA_3  FOODS_1_001  FOODS_1  FOODS     CA_3       CA\n",
       "3  FOODS_1_001_CA_4  FOODS_1_001  FOODS_1  FOODS     CA_4       CA\n",
       "4  FOODS_1_001_TX_1  FOODS_1_001  FOODS_1  FOODS     TX_1       TX"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "S_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71a715f2-2879-4663-b164-8eb28015929b",
   "metadata": {},
   "source": [
    "## Evaluation class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cebfe34e-4174-48f1-8dee-459281c8846d",
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class M5Evaluation:\n",
    "    \n",
    "    levels: dict =  dict(\n",
    "        Level1=['total'],\n",
    "        Level2=['state_id'],\n",
    "        Level3=['store_id'],\n",
    "        Level4=['cat_id'],\n",
    "        Level5=['dept_id'],\n",
    "        Level6=['state_id', 'cat_id'],\n",
    "        Level7=['state_id', 'dept_id'],\n",
    "        Level8=['store_id', 'cat_id'],\n",
    "        Level9=['store_id', 'dept_id'],\n",
    "        Level10=['item_id'],\n",
    "        Level11=['state_id', 'item_id'],\n",
    "        Level12=['item_id', 'store_id']\n",
    "    )\n",
    "    \n",
    "    @staticmethod\n",
    "    def load_benchmark(directory: str,\n",
    "                       source_url: Optional[str] = None,\n",
    "                       validation: bool = False) -> np.ndarray:\n",
    "        \"\"\"Downloads and loads a bechmark forecasts.\n",
    "        \n",
    "        Parameters\n",
    "        ----------\n",
    "        directory: str\n",
    "            Directory where data will be downloaded.\n",
    "        source_url: str, optional\n",
    "            Optional benchmark url obtained from \n",
    "            https://github.com/Nixtla/m5-forecasts/tree/master/forecasts.\n",
    "            If `None` returns the M5 winner. \n",
    "                validation: bool\n",
    "        Wheter return validation forecasts.\n",
    "            Default False, return test forecasts.\n",
    "        \n",
    "        Returns\n",
    "        -------\n",
    "        benchmark: numpy array\n",
    "            Numpy array of shape (n_series, horizon).\n",
    "        \"\"\"\n",
    "        path = f'{directory}/m5/datasets'\n",
    "        if source_url is not None:\n",
    "            filename = source_url.split('/')[-1].replace('.rar', '.csv')\n",
    "            filepath = f'{path}/{filename}'\n",
    "            if not os.path.exists(filepath):\n",
    "                download_file(path, source_url, decompress=True)\n",
    "            \n",
    "        else:\n",
    "            source_url = 'https://github.com/Nixtla/m5-forecasts/raw/main/forecasts/0001 YJ_STU.zip'\n",
    "            return M5Evaluation.load_benchmark(directory, source_url, validation)\n",
    "        \n",
    "        benchmark = pd.read_csv(filepath)\n",
    "        mask = benchmark['id'].str.endswith('validation') \n",
    "        if validation:\n",
    "            benchmark = benchmark[mask]\n",
    "            benchmark['id'] = benchmark['id'].str.replace('_validation', '')\n",
    "        else:\n",
    "            benchmark = benchmark[~mask]\n",
    "            benchmark['id'] = benchmark['id'].str.replace('_evaluation', '')\n",
    "            \n",
    "        benchmark = benchmark.sort_values('id', ignore_index=True)\n",
    "        benchmark.rename(columns={'id': 'unique_id'}, inplace=True)\n",
    "        *_, s_df = M5.load(directory)\n",
    "        benchmark = benchmark.merge(s_df, how='left',\n",
    "                                    on=['unique_id'])\n",
    "        \n",
    "        return benchmark\n",
    "    \n",
    "    @staticmethod\n",
    "    def aggregate_levels(y_hat: pd.DataFrame, \n",
    "                         categories: pd.DataFrame = None) -> pd.DataFrame:\n",
    "        \"\"\"Aggregates the 30_480 series to get 42_840.\"\"\"\n",
    "        y_hat_cat = y_hat.assign(total='Total')\n",
    "\n",
    "        df_agg = []\n",
    "        for level, agg in M5Evaluation.levels.items():\n",
    "            df = y_hat_cat.groupby(agg).sum().reset_index()\n",
    "            renamer = dict(zip(agg, ['Agg_Level_1', 'Agg_Level_2']))\n",
    "            df.rename(columns=renamer, inplace=True)\n",
    "            df.insert(0, 'Level_id', level)\n",
    "            df_agg.append(df)\n",
    "        df_agg = pd.concat(df_agg)\n",
    "        df_agg = df_agg.fillna('X')\n",
    "        df_agg = df_agg.set_index(['Level_id', 'Agg_Level_1', 'Agg_Level_2'])\n",
    "        df_agg.columns = [f'd_{i+1}' for i in range(df_agg.shape[1])]\n",
    "\n",
    "        return df_agg\n",
    "    \n",
    "    @staticmethod\n",
    "    def evaluate(directory: str, \n",
    "                 y_hat: Union[pd.DataFrame, str],\n",
    "                 validation: bool = False) -> pd.DataFrame:\n",
    "        \"\"\"Evaluates y_hat according to M4 methodology.\n",
    "        \n",
    "        Parameters\n",
    "        ----------\n",
    "        directory: str\n",
    "            Directory where data will be downloaded.\n",
    "        validation: bool\n",
    "            Wheter perform validation evaluation.\n",
    "            Default False, return test evaluation.\n",
    "        y_hat: pandas datafrae, str\n",
    "            Forecasts as wide pandas dataframe with columns\n",
    "            ['unique_id'] and forecasts or\n",
    "            benchmark url from\n",
    "            https://github.com/Nixtla/m5-forecasts/tree/main/forecasts.\n",
    "            \n",
    "        Returns\n",
    "        -------\n",
    "        evaluation: pandas dataframe\n",
    "            DataFrame with columns OWA, SMAPE, MASE\n",
    "            and group as index.\n",
    "        \"\"\"\n",
    "        if isinstance(y_hat, str):\n",
    "            y_hat = M5Evaluation.load_benchmark(directory, y_hat, validation)\n",
    "    \n",
    "        M5.download(directory)\n",
    "        path = f'{directory}/m5/datasets'\n",
    "        if validation:\n",
    "            weights = pd.read_csv(f'{path}/weights_validation.csv')\n",
    "            sales = pd.read_csv(f'{path}/sales_train_vaidation.csv')\n",
    "            y_test = pd.read_csv(f'{path}/sales_test_vaidation.csv')\n",
    "        else:\n",
    "            weights = pd.read_csv(f'{path}/weights_evaluation.csv')\n",
    "            sales = pd.read_csv(f'{path}/sales_train_evaluation.csv')\n",
    "            y_test = pd.read_csv(f'{path}/sales_test_evaluation.csv')\n",
    "            \n",
    "        # sales\n",
    "        sales = M5Evaluation.aggregate_levels(sales)\n",
    "        def scale(x):\n",
    "            x = x.values\n",
    "            x = x[np.argmax(x!=0):]\n",
    "            scale = ((x[1:] - x[:-1]) ** 2).mean()\n",
    "            return scale\n",
    "        scales = sales.agg(scale, 1).rename('scale').reset_index()\n",
    "        \n",
    "        # y_test\n",
    "        y_test = M5Evaluation.aggregate_levels(y_test)\n",
    "        \n",
    "        #y_hat\n",
    "        y_hat = M5Evaluation.aggregate_levels(y_hat)\n",
    "        \n",
    "        score = (y_test - y_hat) ** 2\n",
    "        score = score.mean(1)\n",
    "        score = score.rename('rmse').reset_index()\n",
    "        score = score.merge(weights, how='left', \n",
    "                            on=['Level_id', 'Agg_Level_1', 'Agg_Level_2'])\n",
    "        score = score.merge(scales, how='left', \n",
    "                            on=['Level_id', 'Agg_Level_1', 'Agg_Level_2'])\n",
    "        score['wrmsse'] = (score['rmse'] / score['scale']).pow(1 / 2) * score['weight']\n",
    "        score = score.groupby('Level_id')[['wrmsse']].sum()\n",
    "        score = score.loc[M5Evaluation.levels.keys()]\n",
    "        total = score.mean().rename('Total').to_frame().T\n",
    "        score = pd.concat([total, score])\n",
    "        \n",
    "        return score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f243a243-3b80-4773-9801-00975b53cb34",
   "metadata": {},
   "source": [
    "### URL-based evaluation\n",
    "\n",
    "The method `evaluate` from the class `M5Evaluation` can receive a url of a [submission to the M5 competiton](https://github.com/Nixtla/m5-forecasts/tree/main/forecasts).  \n",
    "\n",
    "The results compared to the on-the-fly evaluation were obtained from the [official evaluation](https://github.com/Mcompetitions/M5-methods/blob/master/Scores%20and%20Ranks.xlsx)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3baf1f52-3b31-48c9-a755-ae01180ce0ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "m5_winner_url = 'https://github.com/Nixtla/m5-forecasts/raw/main/forecasts/0001 YJ_STU.zip'\n",
    "winner_evaluation = M5Evaluation.evaluate('data', m5_winner_url)\n",
    "# Test of the same evaluation as the original one\n",
    "test_close(winner_evaluation.loc['Total'].item(), 0.520, eps=1e-3)\n",
    "winner_evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16c7d00c-fda4-4609-a985-523a433e45bc",
   "metadata": {},
   "source": [
    "### Pandas-based evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1aae1209-943d-438c-a895-56ab2a390e80",
   "metadata": {},
   "source": [
    "Also the method `evaluate` can recevie a pandas DataFrame of forecasts.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02539952-0ab3-446b-8498-c30b6a786182",
   "metadata": {},
   "outputs": [],
   "source": [
    "m5_second_place_url = 'https://github.com/Nixtla/m5-forecasts/raw/main/forecasts/0002 Matthias.zip'\n",
    "m5_second_place_forecasts = M5Evaluation.load_benchmark('data', m5_second_place_url)\n",
    "second_place_evaluation = M5Evaluation.evaluate('data', m5_second_place_forecasts)\n",
    "# Test of the same evaluation as the original one\n",
    "test_close(second_place_evaluation.loc['Total'].item(), 0.528, eps=1e-3)\n",
    "second_place_evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "215ada69-0da2-41a7-92d6-e27bcaac6752",
   "metadata": {},
   "source": [
    "# Kaggle-Competition-M5 References\n",
    "\n",
    "The evaluation metric of the Favorita Kaggle competition was the normalized weighted root mean squared logarithmic error (NWRMSLE).\n",
    "Perishable items have a score weight of 1.25; otherwise, the weight is 1.0.\n",
    "\n",
    "$$ NWRMSLE = \\sqrt{\\frac{\\sum^{n}_{i=1} w_{i}\\left(log(\\hat{y}_{i}+1)  - log(y_{i}+1)\\right)^{2}}{\\sum^{n}_{i=1} w_{i}}}$$\n",
    "\n",
    "Kaggle Competition Forecasting Methods                                                              | 16D ahead NWRMSLE\n",
    ":-------------------------------------------------------------------------------------------------: | :-------: \n",
    "[LGBM](https://www.kaggle.com/shixw125/1st-place-lgb-model-public-0-506-private-0-511/comments) [1] | 0.5091   | \n",
    "[Seq2Seq WaveNet](https://arxiv.org/abs/1803.04037) [2]                                             | 0.5129   |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "813fbb27-c5f2-44e6-a3a0-8cef208d4c73",
   "metadata": {},
   "source": [
    "1.\t[Corporación Favorita. Corporación favorita grocery sales forecasting. Kaggle Competition Leaderboard, 2018.](https://www.kaggle.com/c/favorita-grocery-sales-forecasting/leaderboard)\n",
    "2.\t[Glib Kechyn, Lucius Yu, Yangguang Zang, and Svyatoslav Kechyn.  Sales forecasting using wavenet within the framework of the Favorita Kaggle competition. Computing Research Repository, abs/1803.04037, 2018](https://arxiv.org/abs/1803.04037)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "nixtla",
   "language": "python",
   "name": "nixtla"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
