{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Working with the med3pa Subpackage\n",
    "-----------------------------------------\n",
    "\n",
    "This tutorial guides you through the process of setting up and running comprehensive experiments using the `med3pa` subpackage. It includes steps to execute MED3pa experiment with `Med3paExperiment` and the combination of MED3pa and Detectron using `Med3paDetectronExperiment`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running the MED3pa Experiment\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Setting up the Datasets\n",
    "First, configure the `DatasetsManager`. In the case of MED3pa only experiment you only need to set the DatasetManager with either `testing` and `reference` dataset:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import os\n",
    "\n",
    "sys.path.insert(0, os.path.abspath(os.path.join(os.getcwd(), '..')))\n",
    "\n",
    "from MED3pa.datasets import DatasetsManager\n",
    "\n",
    "# Initialize the DatasetsManager\n",
    "datasets = DatasetsManager()\n",
    "\n",
    "# Load datasets for reference, and testing\n",
    "datasets.set_from_file(dataset_type=\"training\", file='./data/train_data.csv', target_column_name='Outcome')\n",
    "datasets.set_from_file(dataset_type=\"validation\", file='./data/val_data.csv', target_column_name='Outcome')\n",
    "datasets.set_from_file(dataset_type=\"reference\", file='./data/test_data.csv', target_column_name='Outcome')\n",
    "datasets.set_from_file(dataset_type=\"testing\", file='./data/test_data_shifted_0.1.csv', target_column_name='Outcome')\n",
    "\n",
    "\n",
    "# Initialize the DatasetsManager\n",
    "datasets2 = DatasetsManager()\n",
    "\n",
    "# Load datasets for reference, and testing\n",
    "datasets2.set_from_file(dataset_type=\"training\", file='./data/train_data.csv', target_column_name='Outcome')\n",
    "datasets2.set_from_file(dataset_type=\"validation\", file='./data/val_data.csv', target_column_name='Outcome')\n",
    "datasets2.set_from_file(dataset_type=\"reference\", file='./data/test_data.csv', target_column_name='Outcome')\n",
    "datasets2.set_from_file(dataset_type=\"testing\", file='./data/test_data_shifted_1.6.csv', target_column_name='Outcome')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Configuring the Model\n",
    "Next, utilize the `ModelFactory` to load a pre-trained model, and set it as the base model for the experiment. Alternatively, you can train your own model and use it:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from MED3pa.models import BaseModelManager, ModelFactory\n",
    "\n",
    "# Initialize the model factory and load the pre-trained model\n",
    "factory = ModelFactory()\n",
    "model = factory.create_model_from_pickled(\"./models/diabetes_xgb_model.pkl\")\n",
    "\n",
    "# Set the base model using BaseModelManager\n",
    "base_model_manager = BaseModelManager()\n",
    "base_model_manager.set_base_model(model=model)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Running the MED3pa Experiment\n",
    "Execute the MED3PA experiment with the specified datasets and base model. You can also specify other parameters as needed. See the documentation of the subpackage for more information about the parameters.\n",
    "\n",
    "The experiment outputs two structure one for the reference set and the other for the testing set, both containing files indicating the extracted profiles at different declaration rates, the performance of the model on these profiles..etc.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running MED3pa Experiment on the reference set:\n",
      "<MED3pa.datasets.masked.MaskedDataset object at 0x000001CF90443F90>\n",
      "IPC Model training complete.\n",
      "IPC Model optimization complete.\n",
      "Individualized confidence scores calculated.\n",
      "APC Model training complete.\n",
      "APC Model optimization complete.\n",
      "Aggregated confidence scores calculated.\n",
      "Mixed confidence scores calculated.\n",
      "Results extracted for minimum_samples_ratio =  0\n",
      "Results extracted for minimum_samples_ratio =  5\n",
      "Results extracted for minimum_samples_ratio =  10\n",
      "Running MED3pa Experiment on the test set:\n",
      "IPC Model training complete.\n",
      "IPC Model optimization complete.\n",
      "Individualized confidence scores calculated.\n",
      "APC Model training complete.\n",
      "APC Model optimization complete.\n",
      "Aggregated confidence scores calculated.\n",
      "Mixed confidence scores calculated.\n",
      "Results extracted for minimum_samples_ratio =  0\n",
      "Results extracted for minimum_samples_ratio =  5\n",
      "Results extracted for minimum_samples_ratio =  10\n",
      "Running MED3pa Experiment on the reference set:\n",
      "<MED3pa.datasets.masked.MaskedDataset object at 0x000001CF9043B410>\n",
      "IPC Model training complete.\n",
      "IPC Model optimization complete.\n",
      "Individualized confidence scores calculated.\n",
      "APC Model training complete.\n",
      "APC Model optimization complete.\n",
      "Aggregated confidence scores calculated.\n",
      "Mixed confidence scores calculated.\n",
      "Results extracted for minimum_samples_ratio =  0\n",
      "Results extracted for minimum_samples_ratio =  5\n",
      "Results extracted for minimum_samples_ratio =  10\n",
      "Running MED3pa Experiment on the test set:\n",
      "IPC Model training complete.\n",
      "IPC Model optimization complete.\n",
      "Individualized confidence scores calculated.\n",
      "APC Model training complete.\n",
      "APC Model optimization complete.\n",
      "Aggregated confidence scores calculated.\n",
      "Mixed confidence scores calculated.\n",
      "Results extracted for minimum_samples_ratio =  0\n",
      "Results extracted for minimum_samples_ratio =  5\n",
      "Results extracted for minimum_samples_ratio =  10\n"
     ]
    }
   ],
   "source": [
    "from MED3pa.med3pa import Med3paExperiment\n",
    "\n",
    "# Define parameters for the experiment\n",
    "ipc_params = {'n_estimators': 100}\n",
    "ipc_grid = {\n",
    "    'n_estimators': [50, 100, 200],\n",
    "    'max_depth': [2, 3, 4, 5],\n",
    "    'min_samples_leaf':[1, 2, 4]\n",
    "\n",
    "}\n",
    "apc_grid = {\n",
    "    'max_depth': [2, 3, 4, 5],\n",
    "    'min_samples_leaf':[1, 2, 4]\n",
    "}\n",
    "apc_params = {'max_depth': 3}\n",
    "med3pa_metrics = ['Auc', 'Accuracy', 'BalancedAccuracy']\n",
    "\n",
    "apc_grid2 = {\n",
    "    'max_depth': [2, 3, 4, 5],\n",
    "    'min_samples_leaf':[1, 2, 4]\n",
    "}\n",
    "apc_params2 = {'max_depth': 3}\n",
    "\n",
    "# Execute the MED3PA experiment\n",
    "results = Med3paExperiment.run(\n",
    "                                datasets_manager=datasets,\n",
    "                                base_model_manager=base_model_manager,\n",
    "                                uncertainty_metric=\"absolute_error\",\n",
    "                                ipc_type='RandomForestRegressor',\n",
    "                                ipc_params=ipc_params,\n",
    "                                apc_params=apc_params,\n",
    "                                ipc_grid_params=ipc_grid,\n",
    "                                apc_grid_params=apc_grid,\n",
    "                                samples_ratio_min=0,\n",
    "                                samples_ratio_max=10,\n",
    "                                samples_ratio_step=5,\n",
    "                                med3pa_metrics=med3pa_metrics,\n",
    "                                evaluate_models=True,\n",
    "                                mode='mpc',\n",
    "                                )\n",
    "\n",
    "BaseModelManager.reset()\n",
    "# Initialize the model factory and load the pre-trained model\n",
    "factory = ModelFactory()\n",
    "model = factory.create_model_from_pickled(\"./models/diabetes_xgb_model.pkl\")\n",
    "\n",
    "# Set the base model using BaseModelManager\n",
    "base_model_manager = BaseModelManager()\n",
    "base_model_manager.set_base_model(model=model)\n",
    "\n",
    "results2 = Med3paExperiment.run(\n",
    "                                datasets_manager=datasets2,\n",
    "                                base_model_manager=base_model_manager,\n",
    "                                uncertainty_metric=\"absolute_error\",\n",
    "                                ipc_type='RandomForestRegressor',\n",
    "                                ipc_params=ipc_params,\n",
    "                                apc_params=apc_params2,\n",
    "                                ipc_grid_params=ipc_grid,\n",
    "                                apc_grid_params=apc_grid2,\n",
    "                                samples_ratio_min=0,\n",
    "                                samples_ratio_max=10,\n",
    "                                samples_ratio_step=5,\n",
    "                                med3pa_metrics=med3pa_metrics,\n",
    "                                evaluate_models=True,\n",
    "                                mode='mpc',\n",
    "                                )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Analyzing and Saving the Results\n",
    "After running the experiment, you can analyze and save the results using the returned `Med3paResults` instance:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the results to a specified directory\n",
    "results.save(file_path='./med3pa_experiment_results')\n",
    "results2.save(file_path='./med3pa_experiment_results_2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additonnally, you can save the instances the IPC and APC models as pickled files:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "results.save_models(file_path='./med3pa_experiment_results_models')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Running experiments from pretrained models\n",
    "If you don't want to re-train new APC and IPC models in your experiment, you can directly use a previously saved instances. as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running MED3pa Experiment on the reference set:\n",
      "<MED3pa.datasets.masked.MaskedDataset object at 0x000001CF90443F90>\n",
      "Loaded a pretrained IPC model.\n",
      "Individualized confidence scores calculated.\n",
      "Loaded a pretrained APC model.\n",
      "Aggregated confidence scores calculated.\n",
      "Mixed confidence scores calculated.\n",
      "Results extracted for minimum_samples_ratio =  0\n",
      "Results extracted for minimum_samples_ratio =  5\n",
      "Results extracted for minimum_samples_ratio =  10\n",
      "Running MED3pa Experiment on the test set:\n",
      "Loaded a pretrained IPC model.\n",
      "Individualized confidence scores calculated.\n",
      "Loaded a pretrained APC model.\n",
      "Aggregated confidence scores calculated.\n",
      "Mixed confidence scores calculated.\n",
      "Results extracted for minimum_samples_ratio =  0\n",
      "Results extracted for minimum_samples_ratio =  5\n",
      "Results extracted for minimum_samples_ratio =  10\n",
      "Running MED3pa Experiment on the reference set:\n",
      "<MED3pa.datasets.masked.MaskedDataset object at 0x000001CF9043B410>\n",
      "Loaded a pretrained IPC model.\n",
      "Individualized confidence scores calculated.\n",
      "Loaded a pretrained APC model.\n",
      "Aggregated confidence scores calculated.\n",
      "Mixed confidence scores calculated.\n",
      "Results extracted for minimum_samples_ratio =  0\n",
      "Results extracted for minimum_samples_ratio =  5\n",
      "Results extracted for minimum_samples_ratio =  10\n",
      "Running MED3pa Experiment on the test set:\n",
      "Loaded a pretrained IPC model.\n",
      "Individualized confidence scores calculated.\n",
      "Loaded a pretrained APC model.\n",
      "Aggregated confidence scores calculated.\n",
      "Mixed confidence scores calculated.\n",
      "Results extracted for minimum_samples_ratio =  0\n",
      "Results extracted for minimum_samples_ratio =  5\n",
      "Results extracted for minimum_samples_ratio =  10\n"
     ]
    }
   ],
   "source": [
    "from MED3pa.med3pa import Med3paExperiment\n",
    "from MED3pa.med3pa.uncertainty import AbsoluteError\n",
    "\n",
    "# Define parameters for the experiment\n",
    "ipc_params = {'n_estimators': 100}\n",
    "apc_params = {'max_depth': 3}\n",
    "med3pa_metrics = ['Auc', 'Accuracy', 'BalancedAccuracy']\n",
    "\n",
    "# Execute the MED3PA experiment\n",
    "results = Med3paExperiment.run(\n",
    "                                datasets_manager=datasets,\n",
    "                                base_model_manager=base_model_manager,\n",
    "                                uncertainty_metric=\"absolute_error\",\n",
    "                                ipc_type='RandomForestRegressor',\n",
    "                                pretrained_ipc='./med3pa_experiment_results_models/ipc_model.pkl',\n",
    "                                pretrained_apc='./med3pa_experiment_results_models/apc_model.pkl',\n",
    "                                samples_ratio_min=0,\n",
    "                                samples_ratio_max=10,\n",
    "                                samples_ratio_step=5,\n",
    "                                med3pa_metrics=med3pa_metrics,\n",
    "                                evaluate_models=True,\n",
    "                                use_ref_models=True\n",
    "                                )\n",
    "\n",
    "results2 = Med3paExperiment.run(\n",
    "                                datasets_manager=datasets2,\n",
    "                                base_model_manager=base_model_manager,\n",
    "                                uncertainty_metric=\"absolute_error\",\n",
    "                                ipc_type='RandomForestRegressor',\n",
    "                                pretrained_ipc='./med3pa_experiment_results_models/ipc_model.pkl',\n",
    "                                pretrained_apc='./med3pa_experiment_results_models/apc_model.pkl',\n",
    "                                samples_ratio_min=0,\n",
    "                                samples_ratio_max=10,\n",
    "                                samples_ratio_step=5,\n",
    "                                med3pa_metrics=med3pa_metrics,\n",
    "                                evaluate_models=True,\n",
    "                                use_ref_models=True\n",
    "                                )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the results to a specified directory\n",
    "results.save(file_path='./med3pa_experiment_results_pretrained')\n",
    "results2.save(file_path='./med3pa_experiment_results_2_pretrained')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Comparing two experiments\n",
    "You can compare between two experiments bu using the ``Med3paComparaison`` class, this class works as follows:\n",
    "- the two experiments need to be of the same type, either ``Med3paExperiment`` or ``Med3paDetectronExperiment``.\n",
    "- if the two experiments were executed using the same tree structure, or the same apc/ipc models, the profiles will also be compared.\n",
    "- if the experiments are of type ``Med3paDetectronExperiment``, the detectron results will be also compared."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import os\n",
    "\n",
    "sys.path.insert(0, os.path.abspath(os.path.join(os.getcwd(), '..')))\n",
    "\n",
    "from MED3pa.med3pa.comparaison import Med3paComparison\n",
    "\n",
    "comparaison = Med3paComparison('./med3pa_experiment_results', './med3pa_experiment_results_2')\n",
    "comparaison.compare_experiments()\n",
    "comparaison.save('./med3pa_comparaison_results')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running the MED3pa and Detectron Experiment\n",
    "You can also run an experiment that combines the forces of Detectron in covariate shift detection with MED3pa problematic profiles extraction using the `Med3paDetectronExperiment` class. To be able to run this experiment, all datasets of the `DatasetsManager` should be set, alongside the ``BaseModelManager``. This experiment will run MED3pa experiment on the `testing` and `reference` sets and then run the `detectron` experiment on the `testing` set as a whole, and then on the **extracted profiles** from MED3pa:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import os\n",
    "\n",
    "sys.path.insert(0, os.path.abspath(os.path.join(os.getcwd(), '..')))\n",
    "\n",
    "from MED3pa.datasets import DatasetsManager\n",
    "\n",
    "# Initialize the DatasetsManager\n",
    "datasets = DatasetsManager()\n",
    "\n",
    "# Load datasets for reference, and testing\n",
    "datasets.set_from_file(dataset_type=\"training\", file='./data/train_data.csv', target_column_name='Outcome')\n",
    "datasets.set_from_file(dataset_type=\"validation\", file='./data/val_data.csv', target_column_name='Outcome')\n",
    "datasets.set_from_file(dataset_type=\"reference\", file='./data/test_data.csv', target_column_name='Outcome')\n",
    "datasets.set_from_file(dataset_type=\"testing\", file='./data/test_data_shifted_0.1.csv', target_column_name='Outcome')\n",
    "\n",
    "\n",
    "# Initialize the DatasetsManager\n",
    "datasets2 = DatasetsManager()\n",
    "\n",
    "# Load datasets for reference, and testing\n",
    "datasets2.set_from_file(dataset_type=\"training\", file='./data/train_data.csv', target_column_name='Outcome')\n",
    "datasets2.set_from_file(dataset_type=\"validation\", file='./data/val_data.csv', target_column_name='Outcome')\n",
    "datasets2.set_from_file(dataset_type=\"reference\", file='./data/test_data.csv', target_column_name='Outcome')\n",
    "datasets2.set_from_file(dataset_type=\"testing\", file='./data/test_data_shifted_1.6.csv', target_column_name='Outcome')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from MED3pa.models import BaseModelManager, ModelFactory\n",
    "\n",
    "BaseModelManager.reset()\n",
    "# Initialize the model factory and load the pre-trained model\n",
    "factory = ModelFactory()\n",
    "model = factory.create_model_from_pickled(\"./models/diabetes_xgb_model.pkl\")\n",
    "\n",
    "# Set the base model using BaseModelManager\n",
    "base_model_manager = BaseModelManager()\n",
    "base_model_manager.set_base_model(model=model)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<MED3pa.datasets.masked.MaskedDataset object at 0x000001CF91B1A210>\n",
      "Running MED3pa Experiment on the reference set:\n",
      "<MED3pa.datasets.masked.MaskedDataset object at 0x000001CF91B1A210>\n",
      "IPC Model training complete.\n",
      "Individualized confidence scores calculated.\n",
      "Loaded a pretrained APC model.\n",
      "Aggregated confidence scores calculated.\n",
      "Mixed confidence scores calculated.\n",
      "Results extracted for minimum_samples_ratio =  0\n",
      "Results extracted for minimum_samples_ratio =  5\n",
      "Results extracted for minimum_samples_ratio =  10\n",
      "Running MED3pa Experiment on the test set:\n",
      "IPC Model training complete.\n",
      "Individualized confidence scores calculated.\n",
      "Loaded a pretrained APC model.\n",
      "Aggregated confidence scores calculated.\n",
      "Mixed confidence scores calculated.\n",
      "Results extracted for minimum_samples_ratio =  0\n",
      "Results extracted for minimum_samples_ratio =  5\n",
      "Results extracted for minimum_samples_ratio =  10\n",
      "Running Global Detectron Experiment:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:18<00:00,  5.40it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on reference set completed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:19<00:00,  5.13it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on testing set completed.\n",
      "Running Profiled Detectron Experiment:\n",
      "Running Detectron on Profile: *, \n",
      "Profile length in reference:  116 Profile length in test 116\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:21<00:00,  4.65it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on reference set completed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:17<00:00,  5.74it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on testing set completed.\n",
      "Running Detectron on Profile: *, Age <= -0.8201243877410889\n",
      "Profile length in reference:  28 Profile length in test 28\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:14<00:00,  6.73it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on reference set completed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:14<00:00,  7.13it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on testing set completed.\n",
      "Running Detectron on Profile: *, Age <= -0.8201243877410889 & BloodPressure <= 0.4599476158618927\n",
      "Profile length in reference:  22 Profile length in test 22\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:12<00:00,  8.29it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on reference set completed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:11<00:00,  8.51it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on testing set completed.\n",
      "Running Detectron on Profile: *, Age > -0.8201243877410889\n",
      "Profile length in reference:  88 Profile length in test 88\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:20<00:00,  4.98it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on reference set completed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:19<00:00,  5.13it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on testing set completed.\n",
      "Running Detectron on Profile: *, Age > -0.8201243877410889 & DiabetesPedigreeFunction <= -0.5517899990081787\n",
      "Profile length in reference:  31 Profile length in test 30\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:20<00:00,  4.88it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on reference set completed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:20<00:00,  4.83it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on testing set completed.\n",
      "Running Detectron on Profile: *, Age > -0.8201243877410889 & DiabetesPedigreeFunction > -0.5517899990081787\n",
      "Profile length in reference:  57 Profile length in test 58\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:19<00:00,  5.08it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on reference set completed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:17<00:00,  5.56it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on testing set completed.\n",
      "<MED3pa.datasets.masked.MaskedDataset object at 0x000001CFAD6F56D0>\n",
      "Running MED3pa Experiment on the reference set:\n",
      "<MED3pa.datasets.masked.MaskedDataset object at 0x000001CFAD6F56D0>\n",
      "IPC Model training complete.\n",
      "Individualized confidence scores calculated.\n",
      "Loaded a pretrained APC model.\n",
      "Aggregated confidence scores calculated.\n",
      "Mixed confidence scores calculated.\n",
      "Results extracted for minimum_samples_ratio =  0\n",
      "Results extracted for minimum_samples_ratio =  5\n",
      "Results extracted for minimum_samples_ratio =  10\n",
      "Running MED3pa Experiment on the test set:\n",
      "IPC Model training complete.\n",
      "Individualized confidence scores calculated.\n",
      "Loaded a pretrained APC model.\n",
      "Aggregated confidence scores calculated.\n",
      "Mixed confidence scores calculated.\n",
      "Results extracted for minimum_samples_ratio =  0\n",
      "Results extracted for minimum_samples_ratio =  5\n",
      "Results extracted for minimum_samples_ratio =  10\n",
      "Running Global Detectron Experiment:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:17<00:00,  5.78it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on reference set completed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:17<00:00,  5.81it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on testing set completed.\n",
      "Running Profiled Detectron Experiment:\n",
      "Running Detectron on Profile: *, \n",
      "Profile length in reference:  116 Profile length in test 116\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:13<00:00,  7.26it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on reference set completed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:15<00:00,  6.56it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on testing set completed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "d:\\MED3pa-package\\MED3pa\\MED3pa\\detectron\\experiment.py:207: UserWarning: The reference set is smaller than twice the testing set, this may lead to poor calibration\n",
      "  warn(\"The reference set is smaller than twice the testing set, this may lead to poor calibration\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running Detectron on Profile: *, Age <= -0.8201243877410889\n",
      "Profile length in reference:  28 Profile length in test 28\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:09<00:00, 10.14it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on reference set completed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:11<00:00,  8.35it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on testing set completed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "d:\\MED3pa-package\\MED3pa\\MED3pa\\detectron\\experiment.py:207: UserWarning: The reference set is smaller than twice the testing set, this may lead to poor calibration\n",
      "  warn(\"The reference set is smaller than twice the testing set, this may lead to poor calibration\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running Detectron on Profile: *, Age <= -0.8201243877410889 & BloodPressure <= 0.4599476158618927\n",
      "Profile length in reference:  22 Profile length in test 22\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:08<00:00, 12.35it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on reference set completed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:11<00:00,  9.04it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on testing set completed.\n",
      "Running Detectron on Profile: *, Age > -0.8201243877410889\n",
      "Profile length in reference:  88 Profile length in test 88\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:14<00:00,  6.68it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on reference set completed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:17<00:00,  5.84it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on testing set completed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "d:\\MED3pa-package\\MED3pa\\MED3pa\\detectron\\experiment.py:207: UserWarning: The reference set is smaller than twice the testing set, this may lead to poor calibration\n",
      "  warn(\"The reference set is smaller than twice the testing set, this may lead to poor calibration\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running Detectron on Profile: *, Age > -0.8201243877410889 & DiabetesPedigreeFunction <= -0.5517899990081787\n",
      "Profile length in reference:  31 Profile length in test 46\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:19<00:00,  5.08it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on reference set completed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:18<00:00,  5.47it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on testing set completed.\n",
      "Running Detectron on Profile: *, Age > -0.8201243877410889 & DiabetesPedigreeFunction > -0.5517899990081787\n",
      "Profile length in reference:  57 Profile length in test 42\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:13<00:00,  7.34it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on reference set completed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "running seeds: 100%|██████████| 100/100 [00:15<00:00,  6.25it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detectron execution on testing set completed.\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "import os\n",
    "\n",
    "sys.path.insert(0, os.path.abspath(os.path.join(os.getcwd(), '..')))\n",
    "\n",
    "from MED3pa.med3pa import Med3paDetectronExperiment\n",
    "from MED3pa.detectron.strategies import EnhancedDisagreementStrategy\n",
    "\n",
    "# Execute the integrated MED3PA and Detectron experiment\n",
    "med3pa_detectron_results = Med3paDetectronExperiment.run(\n",
    "    datasets=datasets,\n",
    "    base_model_manager=base_model_manager,\n",
    "    uncertainty_metric=\"absolute_error\",\n",
    "    samples_size=20,\n",
    "    ensemble_size=10,\n",
    "    num_calibration_runs=100,\n",
    "    patience=3,\n",
    "    test_strategies=\"enhanced_disagreement_strategy\",\n",
    "    pretrained_apc='./med3pa_experiment_results_models/apc_model.pkl',\n",
    "    allow_margin=False,\n",
    "    margin=0.05,\n",
    "    samples_ratio_min=0,\n",
    "    samples_ratio_max=10,\n",
    "    samples_ratio_step=5,\n",
    "    evaluate_models=True,\n",
    ")\n",
    "\n",
    "med3pa_detectron_results2 = Med3paDetectronExperiment.run(\n",
    "    datasets=datasets2,\n",
    "    base_model_manager=base_model_manager,\n",
    "    uncertainty_metric=\"absolute_error\",\n",
    "    samples_size=20,\n",
    "    samples_size_profiles=20,\n",
    "    ensemble_size=10,\n",
    "    num_calibration_runs=100,\n",
    "    patience=3,\n",
    "    test_strategies=\"enhanced_disagreement_strategy\",\n",
    "    pretrained_apc='./med3pa_experiment_results_models/apc_model.pkl',\n",
    "    allow_margin=False,\n",
    "    margin=0.05,\n",
    "    samples_ratio_min=0,\n",
    "    samples_ratio_max=10,\n",
    "    samples_ratio_step=5,\n",
    "    evaluate_models=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the results to a specified directory\n",
    "med3pa_detectron_results.save(file_path='./med3pa_detectron_experiment_results/')\n",
    "med3pa_detectron_results2.save(file_path='./med3pa_detectron_experiment_results2/')\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "det3pa",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
