{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Working with the Models Subpackage\n",
    "----------------------------------\n",
    "\n",
    "The ``models`` subpackage is crafted to offer a comprehensive suite of tools for creating and managing various machine learning models within the ``MED3pa`` package.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the ModelFactory Class\n",
    "The `ModelFactory` class within the `models` subpackage offers a streamlined approach to creating machine learning models, either from predefined configurations or from serialized states. Here’s how to leverage this functionality effectively:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Importing Necessary Modules\n",
    "Start by importing the required classes and utilities for model management:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import os\n",
    "\n",
    "sys.path.insert(0, os.path.abspath(os.path.join(os.getcwd(), '..')))\n",
    "\n",
    "from pprint import pprint\n",
    "from MED3pa.models import factories\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Creating an Instance of ModelFactory\n",
    "Instantiate the `ModelFactory`, which serves as your gateway to generating various model instances:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "factory = factories.ModelFactory()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Discovering Supported Models\n",
    "Before creating a model, check which models are currently supported by the factory:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Supported models: ['XGBoostModel']\n"
     ]
    }
   ],
   "source": [
    "print(\"Supported models:\", factory.get_supported_models())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Creating a Model using the factory\n",
    "There are mainly two ways to create a model using the factory, from hyperparameters or from a serialized (pickled) file.\n",
    "\n",
    "#### Creating a model from hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data_preparation_strategy': 'ToDmatrixStrategy',\n",
      " 'model': 'XGBoostModel',\n",
      " 'model_type': 'Booster',\n",
      " 'params': {'colsample_bytree': 0.8,\n",
      "            'device': 'cpu',\n",
      "            'eta': 0.1,\n",
      "            'eval_metric': 'auc',\n",
      "            'max_depth': 6,\n",
      "            'min_child_weight': 1,\n",
      "            'nthread': 4,\n",
      "            'objective': 'binary:logistic',\n",
      "            'subsample': 0.8,\n",
      "            'tree_method': 'hist'},\n",
      " 'pickled_model': False}\n"
     ]
    }
   ],
   "source": [
    "xgb_params = {\n",
    "    'objective': 'binary:logistic',\n",
    "    'eval_metric': 'auc',\n",
    "    'eta': 0.1,\n",
    "    'max_depth': 6,\n",
    "    'subsample': 0.8,\n",
    "    'colsample_bytree': 0.8,\n",
    "    'min_child_weight': 1,\n",
    "    'nthread': 4,\n",
    "    'tree_method': 'hist',\n",
    "    'device': 'cpu'\n",
    "}\n",
    "\n",
    "xgb_model = factory.create_model_with_hyperparams('XGBoostModel', xgb_params)\n",
    "pprint(xgb_model.get_info())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loading a Model from a Serialized State\n",
    "For pre-trained models, we can make use of the `create_model_from_pickled` method to load a model from its serialized (pickled) state. You only need to specify the path to this pickled file. This function will examine the pickled file and extract all necessary information:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data_preparation_strategy': 'ToDmatrixStrategy',\n",
      " 'model': 'XGBoostModel',\n",
      " 'model_type': 'Booster',\n",
      " 'params': {'alpha': 0,\n",
      "            'base_score': 0.3500931,\n",
      "            'boost_from_average': 1,\n",
      "            'booster': 'gbtree',\n",
      "            'cache_opt': 1,\n",
      "            'colsample_bylevel': 1,\n",
      "            'colsample_bynode': 1,\n",
      "            'colsample_bytree': 0.824717641,\n",
      "            'debug_synchronize': 0,\n",
      "            'device': 'cpu',\n",
      "            'disable_default_eval_metric': 0,\n",
      "            'eta': 0.0710294247,\n",
      "            'eval_metric': ['auc'],\n",
      "            'fail_on_invalid_gpu_id': 0,\n",
      "            'gamma': 0.302559406,\n",
      "            'grow_policy': 'depthwise',\n",
      "            'interaction_constraints': '',\n",
      "            'lambda': 1,\n",
      "            'learning_rate': 0.0710294247,\n",
      "            'max_bin': 256,\n",
      "            'max_cached_hist_node': 65536,\n",
      "            'max_cat_threshold': 64,\n",
      "            'max_cat_to_onehot': 4,\n",
      "            'max_delta_step': 0,\n",
      "            'max_depth': 9,\n",
      "            'max_leaves': 0,\n",
      "            'min_child_weight': 1,\n",
      "            'min_split_loss': 0.302559406,\n",
      "            'monotone_constraints': '()',\n",
      "            'multi_strategy': 'one_output_per_tree',\n",
      "            'n_jobs': 0,\n",
      "            'nthread': 0,\n",
      "            'num_boost_rounds': 30,\n",
      "            'num_class': 0,\n",
      "            'num_feature': 8,\n",
      "            'num_target': 1,\n",
      "            'objective': 'reg:squarederror',\n",
      "            'process_type': 'default',\n",
      "            'random_state': 0,\n",
      "            'refresh_leaf': 1,\n",
      "            'reg_alpha': 0,\n",
      "            'reg_lambda': 1,\n",
      "            'sampling_method': 'uniform',\n",
      "            'scale_pos_weight': 1,\n",
      "            'seed': 0,\n",
      "            'seed_per_iteration': 0,\n",
      "            'sketch_ratio': 2,\n",
      "            'sparse_threshold': 0.2,\n",
      "            'subsample': 0.817121327,\n",
      "            'tree_method': 'auto',\n",
      "            'updater': 'grow_quantile_histmaker',\n",
      "            'updater_seq': 'grow_quantile_histmaker',\n",
      "            'validate_parameters': 1},\n",
      " 'pickled_model': True}\n"
     ]
    }
   ],
   "source": [
    "xgb_model_pkl = factory.create_model_from_pickled('./models/diabetes_xgb_model.pkl')\n",
    "pprint(xgb_model_pkl.get_info())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the Model Class\n",
    "In this section, we will learn how to train, predict, and evaluate a machine learning model. For this, we will directly use the created model from the previous section.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Training the Model\n",
    "Generate Training and Validation Data:\n",
    "\n",
    "Prepare the data for training and validation. The following example generates synthetic data for demonstration purposes:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "np.random.seed(0)\n",
    "X_train = np.random.randn(1000, 10)\n",
    "y_train = np.random.randint(0, 2, 1000)\n",
    "X_val = np.random.randn(1000, 10)\n",
    "y_val = np.random.randint(0, 2, 1000)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When training a model, you can specify additional `training_parameters`. If they are not specified, the model will use the initialization parameters. You can also specify whether you'd like to balance the training classes.\n",
    "\n",
    "If a validation set is provided, the Model will use it for validation and then outputs the evaluation results on the set.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation Results:\n",
      "logloss: 16.82\n"
     ]
    }
   ],
   "source": [
    "training_params = {\n",
    "    'eval_metric': 'logloss',\n",
    "    'eta': 0.1,\n",
    "    'max_depth': 6\n",
    "}\n",
    "xgb_model.train(X_train, y_train, X_val, y_val, training_params, balance_train_classes=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Predicting Using the Trained Model\n",
    "Model Prediction:\n",
    "\n",
    "Once the model is trained, use it to predict labels or probabilities on a new dataset. This step demonstrates predicting binary labels for the test data. The `return_proba` parameter specifies whether to return the `predicted_probabilities` or the `predicted_labels`. The labels are calculated based on the `threshold`:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test = np.random.randn(1000, 10)\n",
    "y_test = np.random.randint(0, 2, 1000)\n",
    "y_pred = xgb_model.predict(X_test, return_proba=False, threshold=0.5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Evaluating the Model\n",
    "Evaluate the model's performance using various metrics to understand its effectiveness in making predictions. The supported metrics include Accuracy, AUC, Precision, Recall, and F1 Score, among others. The `evaluate` method will handle the model predictions and then evaluate the model based on these predictions. You only need to specify the test data.\n",
    "\n",
    "To retrieve the list of supported `classification_metrics`, you can use `ClassificationEvaluationMetrics.supported_metrics()`:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Supported evaluation metrics: ['Accuracy', 'BalancedAccuracy', 'Precision', 'Recall', 'F1Score', 'Specificity', 'Sensitivity', 'Auc', 'LogLoss', 'Auprc', 'NPV', 'PPV', 'MCC']\n",
      "Evaluation Results:\n",
      "Auc: 0.51\n",
      "Accuracy: 0.50\n"
     ]
    }
   ],
   "source": [
    "from MED3pa.models import ClassificationEvaluationMetrics\n",
    "\n",
    "# Display supported metrics\n",
    "print(\"Supported evaluation metrics:\", ClassificationEvaluationMetrics.supported_metrics())\n",
    "\n",
    "# Evaluate the model\n",
    "evaluation_results = xgb_model.evaluate(X_test, y_test, eval_metrics=['Auc', 'Accuracy'], print_results=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Retrieving Model Information\n",
    "The `get_info` method provides detailed information about the model, including its type, parameters, data preparation strategy, and whether it's a pickled model. This is useful for understanding the configuration and state of the model:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data_preparation_strategy': 'ToDmatrixStrategy',\n",
      " 'model': 'XGBoostModel',\n",
      " 'model_type': 'Booster',\n",
      " 'params': {'colsample_bytree': 0.8,\n",
      "            'device': 'cpu',\n",
      "            'eta': 0.1,\n",
      "            'eval_metric': 'logloss',\n",
      "            'max_depth': 6,\n",
      "            'min_child_weight': 1,\n",
      "            'nthread': 4,\n",
      "            'objective': 'binary:logistic',\n",
      "            'subsample': 0.8,\n",
      "            'tree_method': 'hist'},\n",
      " 'pickled_model': False}\n"
     ]
    }
   ],
   "source": [
    "model_info = xgb_model.get_info()\n",
    "pprint(model_info)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: Saving Model Information\n",
    "You can save the model by using the `save` method, which will save the underlying model instance as a pickled file, and the model's information as a .json file:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb_model.save(\"./models/saved_model\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "MED3pa",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
