{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Working with datasets subpackage\n",
    "--------------------------------\n",
    "The ``datasets`` subpackage is designed to provide robust and flexible data loading and management functionalities tailored for machine learning models. \n",
    "This tutorial will guide you through using this subpackage to handle and prepare your data efficiently.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the DatasetsManager Class\n",
    "The `DatasetsManager` class in the `MED3pa.datasets` submodule is designed to facilitate the management of various datasets needed for model training and evaluation. This tutorial provides a step-by-step guide on setting up and using the `DatasetsManager` to handle data efficiently."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Importing the DatasetsManager\n",
    "First, import the `DatasetsManager` from the `MED3pa.datasets` submodule:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import os\n",
    "\n",
    "sys.path.insert(0, os.path.abspath(os.path.join(os.getcwd(), '..')))\n",
    "\n",
    "from MED3pa.datasets import DatasetsManager\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Creating an Instance of DatasetsManager\n",
    "Create an instance of `DatasetsManager`. This instance will manage all operations related to datasets:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "manager = DatasetsManager()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Loading Datasets\n",
    "With the `DatasetsManager`, you can load various segments of your base model datasets, such as training, validation, reference, and testing datasets. You don't need to load all datasets at once. Provide the path to your dataset and the name of the target column:\n",
    "\n",
    "#### Loading from File\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "manager.set_from_file(dataset_type=\"training\", file='./data/train_data.csv', target_column_name='Outcome')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loading from NumPy Arrays\n",
    "You can also load the datasets as NumPy arrays. For this, you need to specify the features, true labels, and column labels as a list (excluding the target column) if they are not already set.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('./data/val_data.csv')\n",
    "\n",
    "# Extract labels and features\n",
    "X_val = df.drop(columns='Outcome').values\n",
    "y_val = df['Outcome'].values\n",
    "\n",
    "# Example of setting data from numpy arrays\n",
    "manager.set_from_data(dataset_type=\"validation\", observations=X_val, true_labels=y_val)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Ensuring Feature Consistency\n",
    "Upon loading the first dataset, the `DatasetsManager` automatically extracts and stores the names of features. You can retrieve the list of these features using:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracted features : ['Pregnancies', 'Glucose', 'BloodPressure', 'SkinThickness', 'Insulin', 'BMI', 'DiabetesPedigreeFunction', 'Age']\n"
     ]
    }
   ],
   "source": [
    "features = manager.get_column_labels()\n",
    "print(\"Extracted features :\", features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: Retrieving Data\n",
    "Retrieve the loaded data in different formats as needed.\n",
    "\n",
    "#### As NumPy Arrays\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Observations shape: (537, 8)\n",
      "Labels shape: (537,)\n",
      "\n",
      "First 5 rows of features:\n",
      "[[-0.8362943  -0.80005088 -0.53576428 -0.15714558 -0.18973183 -1.06015343\n",
      "  -0.61421636 -0.94861028]\n",
      " [ 0.39072767 -0.49054341  0.12804365  0.55361931  2.13020339  0.64646721\n",
      "  -0.90973787 -0.43466673]\n",
      " [-1.14304979  0.43797901 -0.09322566  1.39361417  1.47853619  1.35537117\n",
      "  -0.30699103 -0.77729576]\n",
      " [ 0.08397217  0.31417602 -0.09322566  0.03669939  0.74866893  0.14760887\n",
      "  -0.90681191 -0.43466673]\n",
      " [-0.8362943  -0.5524449  -2.19528409  1.13515422  0.02749057  1.48664968\n",
      "  -0.83951493 -0.00638043]]\n",
      "\n",
      "First 5 labels:\n",
      "[0 0 1 0 0]\n"
     ]
    }
   ],
   "source": [
    "observations, labels = manager.get_dataset_by_type(dataset_type=\"training\")\n",
    "\n",
    "# Print the shape of features and labels to verify they have been loaded\n",
    "print(f\"Observations shape: {observations.shape}\")\n",
    "print(f\"Labels shape: {labels.shape}\")\n",
    "\n",
    "print(\"\\nFirst 5 rows of features:\")\n",
    "print(observations[:5])\n",
    "\n",
    "print(\"\\nFirst 5 labels:\")\n",
    "print(labels[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### As a MaskedDataset Instance\n",
    "To work with the data encapsulated in a `MaskedDataset` instance, which includes more functionalities, retrieve it by setting `return_instance` to `True`:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_dataset = manager.get_dataset_by_type(dataset_type=\"training\", return_instance=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6: Getting a Summary\n",
    "You can print a summary of the `DatasetsManager` to see the status of the datasets:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training_set: {'file_path': './data/train_data.csv', 'num_samples': 537, 'num_observations': 8, 'has_pseudo_labels': False, 'has_pseudo_probabilities': False, 'has_confidence_scores': False}\n",
      "validation_set: {'file_path': None, 'num_samples': 115, 'num_observations': 8, 'has_pseudo_labels': False, 'has_pseudo_probabilities': False, 'has_confidence_scores': False}\n",
      "reference_set: Not set\n",
      "testing_set: Not set\n",
      "column_labels: ['Pregnancies', 'Glucose', 'BloodPressure', 'SkinThickness', 'Insulin', 'BMI', 'DiabetesPedigreeFunction', 'Age']\n"
     ]
    }
   ],
   "source": [
    "manager.summarize()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 7: Saving and Resetting Datasets\n",
    "You can save a specific dataset to a CSV file or reset all datasets managed by the `DatasetsManager`.\n",
    "\n",
    "#### Save to CSV\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "manager.save_dataset_to_csv(dataset_type=\"training\", file_path='./data/saved_train_data.csv')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Reset Datasets\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training_set: Not set\n",
      "validation_set: Not set\n",
      "reference_set: Not set\n",
      "testing_set: Not set\n",
      "column_labels: Not set\n"
     ]
    }
   ],
   "source": [
    "manager.reset_datasets()\n",
    "manager.summarize()  # Verify that all datasets are reset\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the MaskedDataset Class\n",
    "The `MaskedDataset` class, a crucial component of the `MED3pa.datasets` submodule, facilitates nuanced data operations that are essential for custom data manipulation and model training processes. This tutorial details common usage scenarios of the `MaskedDataset`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Importing Necessary Modules\n",
    "Begin by importing the `MaskedDataset` and `DatasetsManager`, along with NumPy for additional data operations:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from MED3pa.datasets import MaskedDataset, DatasetsManager\n",
    "import numpy as np\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Loading Data with DatasetsManager\n",
    "Retrieve the dataset as a `MaskedDataset` instance:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "manager = DatasetsManager()\n",
    "manager.set_from_file(dataset_type=\"training\", file='./data/train_data.csv', target_column_name='Outcome')\n",
    "training_dataset = manager.get_dataset_by_type(dataset_type=\"training\", return_instance=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Performing Operations on MaskedDataset\n",
    "Once you have your dataset loaded as a `MaskedDataset` instance, you can perform various operations:\n",
    "\n",
    "#### Cloning the Dataset\n",
    "Create a copy of the dataset to ensure the original data remains unchanged during experimentation:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "cloned_instance = training_dataset.clone()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sampling the Dataset\n",
    "You can either sample the data uniformely, or randomely. The uniform sampling prioritize the least sampled element which is useful when we sample a dataset multiple times, while the random sampling samples the data randomely :\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "sampled_instance = training_dataset.sample_uniform(N=20, seed=42)\n",
    "sampled_instance_rand = training_dataset.sample_random(N=20, seed=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Refining the Dataset\n",
    "Refine the dataset based on a boolean mask, which is useful for filtering out unwanted data points:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Remaining samples 259\n"
     ]
    }
   ],
   "source": [
    "mask = np.random.rand(len(training_dataset)) > 0.5\n",
    "remaining_samples = training_dataset.refine(mask=mask)\n",
    "\n",
    "print(\"Remaining samples\", remaining_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting Pseudo Labels and Probabilities\n",
    "Set pseudo labels and probabilities for the dataset, for this you only need to pass the pseudo_probabilities along with the threshold to extract the pseudo_labels from:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "pseudo_probs = np.random.rand(len(training_dataset))\n",
    "training_dataset.set_pseudo_probs_labels(pseudo_probabilities=pseudo_probs, threshold=0.5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting Feature Vectors and Labels\n",
    "Retrieve the feature vectors, true labels, and pseudo labels:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Features shape: (259, 8)\n",
      "True Labels shape: (259,)\n",
      "Pseudo Labels shape: (259,)\n",
      "\n",
      "First 5 rows of features:\n",
      "[[ 0.39072767 -0.49054341  0.12804365  0.55361931  2.13020339  0.64646721\n",
      "  -0.90973787 -0.43466673]\n",
      " [-1.14304979  0.43797901 -0.09322566  1.39361417  1.47853619  1.35537117\n",
      "  -0.30699103 -0.77729576]\n",
      " [ 0.08397217  0.31417602 -0.09322566  0.03669939  0.74866893  0.14760887\n",
      "  -0.90681191 -0.43466673]\n",
      " [-0.22278332  0.22132378  0.45994761 -1.3202154  -0.6936878  -1.42773326\n",
      "  -0.59080872  1.87807928]\n",
      " [-1.14304979  0.12847154 -0.09322566 -1.3202154  -0.6936878  -0.95513062\n",
      "  -0.77221796 -1.03426754]]\n",
      "\n",
      "First 5 true labels:\n",
      "[0 1 0 0 0]\n",
      "\n",
      "First 5 pseudo labels:\n",
      "[False  True  True False  True]\n"
     ]
    }
   ],
   "source": [
    "observations = training_dataset.get_observations()\n",
    "true_labels = training_dataset.get_true_labels()\n",
    "pseudo_labels = training_dataset.get_pseudo_labels()\n",
    "\n",
    "# Print the shape of features, true labels, and pseudo labels to verify they have been loaded\n",
    "print(f\"Features shape: {observations.shape}\")\n",
    "print(f\"True Labels shape: {true_labels.shape}\")\n",
    "print(f\"Pseudo Labels shape: {pseudo_labels.shape}\")\n",
    "\n",
    "# Optionally, print the first few rows of features and labels\n",
    "print(\"\\nFirst 5 rows of features:\")\n",
    "print(observations[:5])\n",
    "\n",
    "print(\"\\nFirst 5 true labels:\")\n",
    "print(true_labels[:5])\n",
    "\n",
    "print(\"\\nFirst 5 pseudo labels:\")\n",
    "print(pseudo_labels[:5])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting Confidence Scores\n",
    "Get the confidence scores if available:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confidence Scores shape: (259,)\n",
      "\n",
      "First 5 confidence scores:\n",
      "[0.23590603 0.16552051 0.18632088 0.83749079 0.33214641]\n"
     ]
    }
   ],
   "source": [
    "confidence_scores = np.random.rand(len(training_dataset))\n",
    "training_dataset.set_confidence_scores(confidence_scores=confidence_scores)\n",
    "\n",
    "confidence_scores = training_dataset.get_confidence_scores()\n",
    "print(f\"Confidence Scores shape: {confidence_scores.shape}\")\n",
    "\n",
    "# Optionally, print the first few confidence scores\n",
    "print(\"\\nFirst 5 confidence scores:\")\n",
    "print(confidence_scores[:5])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving the dataset\n",
    "You can save the dataset as a .csv file, but using `save_to_csv` and providing the path this will save the observations, true_labels, pseudo_labels and pseudo_probabilities, alongside confidence_scores if they were set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_dataset.save_to_csv(\"./data/saved_from_masked.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting Dataset Information\n",
    "Get detailed information about the dataset, or you can directly use `summarize`:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of samples: 259\n",
      "Number of observations: 8\n",
      "Has pseudo labels: True\n",
      "Has pseudo probabilities: True\n",
      "Has confidence scores: True\n"
     ]
    }
   ],
   "source": [
    "training_dataset.summarize()\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "MED3pa",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
