{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Compare ensemble classifiers using resampling\n\nEnsemble classifiers have shown to improve classification performance compare\nto single learner. However, they will be affected by class imbalance. This\nexample shows the benefit of balancing the training set before to learn\nlearners. We are making the comparison with non-balanced ensemble methods.\n\nWe make a comparison using the balanced accuracy and geometric mean which are\nmetrics widely used in the literature to evaluate models learned on imbalanced\nset.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Authors: Guillaume Lemaitre <g.lemaitre58@gmail.com>\n# License: MIT"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "print(__doc__)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Load an imbalanced dataset\n\nWe will load the UCI SatImage dataset which has an imbalanced ratio of 9.3:1\n(number of majority sample for a minority sample). The data are then split\ninto training and testing.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from imblearn.datasets import fetch_datasets\nfrom sklearn.model_selection import train_test_split\n\nsatimage = fetch_datasets()[\"satimage\"]\nX, y = satimage.data, satimage.target\nX_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Classification using a single decision tree\n\nWe train a decision tree classifier which will be used as a baseline for the\nrest of this example.\n\nThe results are reported in terms of balanced accuracy and geometric mean\nwhich are metrics widely used in the literature to validate model trained on\nimbalanced set.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.tree import DecisionTreeClassifier\n\ntree = DecisionTreeClassifier()\ntree.fit(X_train, y_train)\ny_pred_tree = tree.predict(X_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import balanced_accuracy_score\nfrom imblearn.metrics import geometric_mean_score\n\nprint(\"Decision tree classifier performance:\")\nprint(\n    f\"Balanced accuracy: {balanced_accuracy_score(y_test, y_pred_tree):.2f} - \"\n    f\"Geometric mean {geometric_mean_score(y_test, y_pred_tree):.2f}\"\n)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import seaborn as sns\nfrom sklearn.metrics import plot_confusion_matrix\n\nsns.set_context(\"poster\")\n\ndisp = plot_confusion_matrix(tree, X_test, y_test, colorbar=False)\n_ = disp.ax_.set_title(\"Decision tree\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Classification using bagging classifier with and without sampling\n\nInstead of using a single tree, we will check if an ensemble of decsion tree\ncan actually alleviate the issue induced by the class imbalancing. First, we\nwill use a bagging classifier and its counter part which internally uses a\nrandom under-sampling to balanced each boostrap sample.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.ensemble import BaggingClassifier\nfrom imblearn.ensemble import BalancedBaggingClassifier\n\nbagging = BaggingClassifier(n_estimators=50, random_state=0)\nbalanced_bagging = BalancedBaggingClassifier(n_estimators=50, random_state=0)\n\nbagging.fit(X_train, y_train)\nbalanced_bagging.fit(X_train, y_train)\n\ny_pred_bc = bagging.predict(X_test)\ny_pred_bbc = balanced_bagging.predict(X_test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Balancing each bootstrap sample allows to increase significantly the balanced\naccuracy and the geometric mean.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "print(\"Bagging classifier performance:\")\nprint(\n    f\"Balanced accuracy: {balanced_accuracy_score(y_test, y_pred_bc):.2f} - \"\n    f\"Geometric mean {geometric_mean_score(y_test, y_pred_bc):.2f}\"\n)\nprint(\"Balanced Bagging classifier performance:\")\nprint(\n    f\"Balanced accuracy: {balanced_accuracy_score(y_test, y_pred_bbc):.2f} - \"\n    f\"Geometric mean {geometric_mean_score(y_test, y_pred_bbc):.2f}\"\n)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n\nfig, axs = plt.subplots(ncols=2, figsize=(10, 5))\nplot_confusion_matrix(bagging, X_test, y_test, ax=axs[0], colorbar=False)\naxs[0].set_title(\"Bagging\")\n\nplot_confusion_matrix(balanced_bagging, X_test, y_test, ax=axs[1], colorbar=False)\naxs[1].set_title(\"Balanced Bagging\")\n\nfig.tight_layout()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Classification using random forest classifier with and without sampling\n\nRandom forest is another popular ensemble method and it is usually\noutperforming bagging. Here, we used a vanilla random forest and its balanced\ncounterpart in which each bootstrap sample is balanced.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.ensemble import RandomForestClassifier\nfrom imblearn.ensemble import BalancedRandomForestClassifier\n\nrf = RandomForestClassifier(n_estimators=50, random_state=0)\nbrf = BalancedRandomForestClassifier(n_estimators=50, random_state=0)\n\nrf.fit(X_train, y_train)\nbrf.fit(X_train, y_train)\n\ny_pred_rf = rf.predict(X_test)\ny_pred_brf = brf.predict(X_test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Similarly to the previous experiment, the balanced classifier outperform the\nclassifier which learn from imbalanced bootstrap samples. In addition, random\nforest outsperforms the bagging classifier.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "print(\"Random Forest classifier performance:\")\nprint(\n    f\"Balanced accuracy: {balanced_accuracy_score(y_test, y_pred_rf):.2f} - \"\n    f\"Geometric mean {geometric_mean_score(y_test, y_pred_rf):.2f}\"\n)\nprint(\"Balanced Random Forest classifier performance:\")\nprint(\n    f\"Balanced accuracy: {balanced_accuracy_score(y_test, y_pred_brf):.2f} - \"\n    f\"Geometric mean {geometric_mean_score(y_test, y_pred_brf):.2f}\"\n)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "fig, axs = plt.subplots(ncols=2, figsize=(10, 5))\nplot_confusion_matrix(rf, X_test, y_test, ax=axs[0], colorbar=False)\naxs[0].set_title(\"Random forest\")\n\nplot_confusion_matrix(brf, X_test, y_test, ax=axs[1], colorbar=False)\naxs[1].set_title(\"Balanced random forest\")\n\nfig.tight_layout()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Boosting classifier\n\nIn the same manner, easy ensemble classifier is a bag of balanced AdaBoost\nclassifier. However, it will be slower to train than random forest and will\nachieve worse performance.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.ensemble import AdaBoostClassifier\nfrom imblearn.ensemble import EasyEnsembleClassifier, RUSBoostClassifier\n\nbase_estimator = AdaBoostClassifier(n_estimators=10)\neec = EasyEnsembleClassifier(n_estimators=10, base_estimator=base_estimator)\neec.fit(X_train, y_train)\ny_pred_eec = eec.predict(X_test)\n\nrusboost = RUSBoostClassifier(n_estimators=10, base_estimator=base_estimator)\nrusboost.fit(X_train, y_train)\ny_pred_rusboost = rusboost.predict(X_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "print(\"Easy ensemble classifier performance:\")\nprint(\n    f\"Balanced accuracy: {balanced_accuracy_score(y_test, y_pred_eec):.2f} - \"\n    f\"Geometric mean {geometric_mean_score(y_test, y_pred_eec):.2f}\"\n)\nprint(\"RUSBoost classifier performance:\")\nprint(\n    f\"Balanced accuracy: {balanced_accuracy_score(y_test, y_pred_rusboost):.2f} - \"\n    f\"Geometric mean {geometric_mean_score(y_test, y_pred_rusboost):.2f}\"\n)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "fig, axs = plt.subplots(ncols=2, figsize=(10, 5))\n\nplot_confusion_matrix(eec, X_test, y_test, ax=axs[0], colorbar=False)\naxs[0].set_title(\"Easy Ensemble\")\nplot_confusion_matrix(rusboost, X_test, y_test, ax=axs[1], colorbar=False)\naxs[1].set_title(\"RUSBoost classifier\")\n\nfig.tight_layout()\nplt.show()"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.12"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}