{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Metrics specific to imbalanced learning\n\nSpecific metrics have been developed to evaluate classifier which\nhas been trained using imbalanced data. :mod:`imbalanced_ensemble` provides mainly\ntwo additional metrics which are not implemented in :mod:`sklearn`: (i)\ngeometric mean (:func:`imbalanced_ensemble.metrics.geometric_mean_score`) \nand (ii) index balanced accuracy (:func:`imbalanced_ensemble.metrics.make_index_balanced_accuracy`).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Adapted from imbalanced-learn\n# Authors: Guillaume Lemaitre <g.lemaitre58@gmail.com>\n# License: MIT"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "print(__doc__)\n\nRANDOM_STATE = 42"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "First, we will generate some imbalanced dataset.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.datasets import make_classification\n\nX, y = make_classification(\n    n_classes=3,\n    class_sep=2,\n    weights=[0.1, 0.9],\n    n_informative=10,\n    n_redundant=1,\n    flip_y=0,\n    n_features=20,\n    n_clusters_per_class=4,\n    n_samples=5000,\n    random_state=RANDOM_STATE,\n)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We will split the data into a training and testing set.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.model_selection import train_test_split\n\nX_train, X_test, y_train, y_test = train_test_split(\n    X, y, stratify=y, random_state=RANDOM_STATE\n)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We will create a pipeline made of a :class:`~imbalanced_ensemble.sampler.over_sampling.SMOTE`\nover-sampler followed by a :class:`~sklearn.svm.LinearSVC` classifier.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from imbalanced_ensemble.pipeline import make_pipeline\nfrom imbalanced_ensemble.sampler.over_sampling import SMOTE\nfrom sklearn.svm import LinearSVC\n\nmodel = make_pipeline(\n    SMOTE(random_state=RANDOM_STATE), LinearSVC(random_state=RANDOM_STATE)\n)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now, we will train the model on the training set and get the prediction\nassociated with the testing set. Be aware that the resampling will happen\nonly when calling `fit`: the number of samples in `y_pred` is the same than\nin `y_test`.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "model.fit(X_train, y_train)\ny_pred = model.predict(X_test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The geometric mean corresponds to the square root of the product of the\nsensitivity and specificity. Combining the two metrics should account for\nthe balancing of the dataset.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from imbalanced_ensemble.metrics import geometric_mean_score\n\nprint(f\"The geometric mean is {geometric_mean_score(y_test, y_pred):.3f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The index balanced accuracy can transform any metric to be used in\nimbalanced learning problems.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from imbalanced_ensemble.metrics import make_index_balanced_accuracy\n\nalpha = 0.1\ngeo_mean = make_index_balanced_accuracy(alpha=alpha, squared=True)(geometric_mean_score)\n\nprint(\n    f\"The IBA using alpha={alpha} and the geometric mean: \"\n    f\"{geo_mean(y_test, y_pred):.3f}\"\n)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "alpha = 0.5\ngeo_mean = make_index_balanced_accuracy(alpha=alpha, squared=True)(geometric_mean_score)\n\nprint(\n    f\"The IBA using alpha={alpha} and the geometric mean: \"\n    f\"{geo_mean(y_test, y_pred):.3f}\"\n)"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}