{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Working with the Detectron Subpackage\n",
    "-----------------------------------------\n",
    "\n",
    "This tutorial guides you through the process of setting up and running the Detectron experiment using the `detectron` subpackage, which is designed to assess the robustness of predictive models against covariate shifts in datasets.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Setting up the Datasets\n",
    "First, configure the `DatasetsManager` to manage the necessary datasets for the experiment. \n",
    "To execute the detectron experiment, we need the `training_dataset` used to train the model, alongside the `validation_dataset`. We also need an unseen `reference_dataset` that is driven from the model's knowledge domain, and finally the dataset we want to inspect if it is possibly shifted, the `test_dataset`:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import os\n",
    "\n",
    "sys.path.insert(0, os.path.abspath(os.path.join(os.getcwd(), '..')))\n",
    "\n",
    "from MED3pa.datasets import DatasetsManager\n",
    "\n",
    "# Initialize the DatasetsManager\n",
    "datasets = DatasetsManager()\n",
    "\n",
    "# Load datasets for training, validation, reference, and testing\n",
    "datasets.set_from_file(dataset_type=\"training\", file='./data/train_data.csv', target_column_name='Outcome')\n",
    "datasets.set_from_file(dataset_type=\"validation\", file='./data/val_data.csv', target_column_name='Outcome')\n",
    "datasets.set_from_file(dataset_type=\"reference\", file='./data/test_data.csv', target_column_name='Outcome')\n",
    "datasets.set_from_file(dataset_type=\"testing\", file='./data/test_data_shifted_0.1.csv', target_column_name='Outcome')\n",
    "\n",
    "\n",
    "datasets2 = DatasetsManager()\n",
    "\n",
    "# Load datasets for training, validation, reference, and testing\n",
    "datasets2.set_from_file(dataset_type=\"training\", file='./data/train_data.csv', target_column_name='Outcome')\n",
    "datasets2.set_from_file(dataset_type=\"validation\", file='./data/val_data.csv', target_column_name='Outcome')\n",
    "datasets2.set_from_file(dataset_type=\"reference\", file='./data/test_data.csv', target_column_name='Outcome')\n",
    "datasets2.set_from_file(dataset_type=\"testing\", file='./data/test_data_shifted_1.6.csv', target_column_name='Outcome')\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Configuring the Model\n",
    "Next, utilize the `ModelFactory` to load a pre-trained model, setting it as the base model manager for the experiment. Alternatively, you can train your own model and use it:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from MED3pa.models import BaseModelManager, ModelFactory\n",
    "\n",
    "# Initialize the model factory and load the pre-trained model\n",
    "factory = ModelFactory()\n",
    "model = factory.create_model_from_pickled(\"./models/diabetes_xgb_model.pkl\")\n",
    "\n",
    "# Set the base model using BaseModelManager\n",
    "base_model_manager = BaseModelManager()\n",
    "base_model_manager.set_base_model(model=model)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Running the Detectron Experiment\n",
    "Execute the Detectron experiment with the specified datasets and base model. You can also specify other parameters:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:17<00:00,  5.81it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on reference set completed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:17<00:00,  5.74it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on testing set completed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:18<00:00,  5.52it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on reference set completed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:19<00:00,  5.04it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on testing set completed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from MED3pa.detectron import DetectronExperiment\n",
    "\n",
    "# Execute the Detectron experiment\n",
    "experiment_results = DetectronExperiment.run(datasets=datasets, base_model_manager=base_model_manager)\n",
    "experiment_results2 = DetectronExperiment.run(datasets=datasets2, base_model_manager=base_model_manager)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Analyzing the Results\n",
    "Finally, evaluate the outcomes of the experiment using different strategies to determine the probability of a shift in dataset distributions:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'shift_probability': 0.8139612188365651,\n",
       "  'test_statistic': 9.83157894736842,\n",
       "  'baseline_mean': 7.231578947368421,\n",
       "  'baseline_std': 1.4396829444740318,\n",
       "  'significance_description': {'unsignificant shift': 15.789473684210526,\n",
       "   'small': 8.421052631578947,\n",
       "   'moderate': 36.84210526315789,\n",
       "   'large': 38.94736842105263},\n",
       "  'Strategy': 'enhanced_disagreement_strategy'},\n",
       " {'p_value': 1.95388272998725e-18,\n",
       "  'u_statistic': 1259.5,\n",
       "  'significance_description': {'unsignificant shift': 15.789473684210526,\n",
       "   'small': 8.421052631578947,\n",
       "   'moderate': 36.84210526315789,\n",
       "   'large': 38.94736842105263},\n",
       "  'Strategy': 'mannwhitney_strategy'},\n",
       " {'p_value': 0.05,\n",
       "  'test_statistic': 8,\n",
       "  'baseline_mean': 12.47,\n",
       "  'baseline_std': 1.920702996301094,\n",
       "  'Strategy': 'original_disagreement_strategy'}]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from MED3pa.detectron.strategies import *\n",
    "\n",
    "# Analyze the results using the disagreement strategies\n",
    "test_strategies = [\"enhanced_disagreement_strategy\", \"mannwhitney_strategy\", \"original_disagreement_strategy\"]\n",
    "experiment_results.analyze_results(test_strategies)\n",
    "experiment_results2.analyze_results(test_strategies)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: Saving the Results\n",
    "You can save the experiment results using the ``save`` method, while specifying the path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment_results.save(\"./detectron_experiment_results\")\n",
    "experiment_results2.save(\"./detectron_experiment_results2\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6: Comparing the Results\n",
    "You can compare between two detectron experiments using the ``DetectronComparaison`` class, as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import os\n",
    "\n",
    "sys.path.insert(0, os.path.abspath(os.path.join(os.getcwd(), '..')))\n",
    "\n",
    "from MED3pa.detectron.comparaison import DetectronComparison\n",
    "\n",
    "comparaison = DetectronComparison(\"./detectron_experiment_results\", \"./detectron_experiment_results2\")\n",
    "comparaison.compare_experiments()\n",
    "comparaison.save(\"./detectron_experiment_comparaison\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "MED3pa",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
