{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Bagging classifiers using sampler\n\nIn this example, we show how\n:class:`~imblearn.ensemble.BalancedBaggingClassifier` can be used to create a\nlarge variety of classifiers by giving different samplers.\n\nWe will give several examples that have been published in the passed year.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Authors: Guillaume Lemaitre <g.lemaitre58@gmail.com>\n# License: MIT"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "print(__doc__)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Generate an imbalanced dataset\n\nFor this example, we will create a synthetic dataset using the function\n:func:`~sklearn.datasets.make_classification`. The problem will be a toy\nclassification problem with a ratio of 1:9 between the two classes.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.datasets import make_classification\n\nX, y = make_classification(\n    n_samples=10_000,\n    n_features=10,\n    weights=[0.1, 0.9],\n    class_sep=0.5,\n    random_state=0,\n)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n\npd.Series(y).value_counts(normalize=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In the following sections, we will show a couple of algorithms that have\nbeen proposed over the years. We intend to illustrate how one can reuse the\n:class:`~imblearn.ensemble.BalancedBaggingClassifier` by passing different\nsampler.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.model_selection import cross_validate\nfrom sklearn.ensemble import BaggingClassifier\n\nebb = BaggingClassifier()\ncv_results = cross_validate(ebb, X, y, scoring=\"balanced_accuracy\")\n\nprint(f\"{cv_results['test_score'].mean():.3f} +/- {cv_results['test_score'].std():.3f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Exactly Balanced Bagging and Over-Bagging\n\nThe :class:`~imblearn.ensemble.BalancedBaggingClassifier` can use in\nconjunction with a :class:`~imblearn.under_sampling.RandomUnderSampler` or\n:class:`~imblearn.over_sampling.RandomOverSampler`. These methods are\nreferred as Exactly Balanced Bagging and Over-Bagging, respectively and have\nbeen proposed first in [1]_.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from imblearn.ensemble import BalancedBaggingClassifier\nfrom imblearn.under_sampling import RandomUnderSampler\n\n# Exactly Balanced Bagging\nebb = BalancedBaggingClassifier(sampler=RandomUnderSampler())\ncv_results = cross_validate(ebb, X, y, scoring=\"balanced_accuracy\")\n\nprint(f\"{cv_results['test_score'].mean():.3f} +/- {cv_results['test_score'].std():.3f}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from imblearn.over_sampling import RandomOverSampler\n\n# Over-bagging\nover_bagging = BalancedBaggingClassifier(sampler=RandomOverSampler())\ncv_results = cross_validate(over_bagging, X, y, scoring=\"balanced_accuracy\")\n\nprint(f\"{cv_results['test_score'].mean():.3f} +/- {cv_results['test_score'].std():.3f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## SMOTE-Bagging\n\nInstead of using a :class:`~imblearn.over_sampling.RandomOverSampler` that\nmake a bootstrap, an alternative is to use\n:class:`~imblearn.over_sampling.SMOTE` as an over-sampler. This is known as\nSMOTE-Bagging [2]_.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from imblearn.over_sampling import SMOTE\n\n# SMOTE-Bagging\nsmote_bagging = BalancedBaggingClassifier(sampler=SMOTE())\ncv_results = cross_validate(smote_bagging, X, y, scoring=\"balanced_accuracy\")\n\nprint(f\"{cv_results['test_score'].mean():.3f} +/- {cv_results['test_score'].std():.3f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Roughly Balanced Bagging\nWhile using a :class:`~imblearn.under_sampling.RandomUnderSampler` or\n:class:`~imblearn.over_sampling.RandomOverSampler` will create exactly the\ndesired number of samples, it does not follow the statistical spirit wanted\nin the bagging framework. The authors in [3]_ proposes to use a negative\nbinomial distribution to compute the number of samples of the majority\nclass to be selected and then perform a random under-sampling.\n\nHere, we illustrate this method by implementing a function in charge of\nresampling and use the :class:`~imblearn.FunctionSampler` to integrate it\nwithin a :class:`~imblearn.pipeline.Pipeline` and\n:class:`~sklearn.model_selection.cross_validate`.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from collections import Counter\nimport numpy as np\nfrom imblearn import FunctionSampler\n\n\ndef roughly_balanced_bagging(X, y, replace=False):\n    \"\"\"Implementation of Roughly Balanced Bagging for binary problem.\"\"\"\n    # find the minority and majority classes\n    class_counts = Counter(y)\n    majority_class = max(class_counts, key=class_counts.get)\n    minority_class = min(class_counts, key=class_counts.get)\n\n    # compute the number of sample to draw from the majority class using\n    # a negative binomial distribution\n    n_minority_class = class_counts[minority_class]\n    n_majority_resampled = np.random.negative_binomial(n=n_minority_class, p=0.5)\n\n    # draw randomly with or without replacement\n    majority_indices = np.random.choice(\n        np.flatnonzero(y == majority_class),\n        size=n_majority_resampled,\n        replace=replace,\n    )\n    minority_indices = np.random.choice(\n        np.flatnonzero(y == minority_class),\n        size=n_minority_class,\n        replace=replace,\n    )\n    indices = np.hstack([majority_indices, minority_indices])\n\n    return X[indices], y[indices]\n\n\n# Roughly Balanced Bagging\nrbb = BalancedBaggingClassifier(\n    sampler=FunctionSampler(func=roughly_balanced_bagging, kw_args={\"replace\": True})\n)\ncv_results = cross_validate(rbb, X, y, scoring=\"balanced_accuracy\")\n\nprint(f\"{cv_results['test_score'].mean():.3f} +/- {cv_results['test_score'].std():.3f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        ".. topic:: References:\n\n   .. [1] R. Maclin, and D. Opitz. \"An empirical evaluation of bagging and\n          boosting.\" AAAI/IAAI 1997 (1997): 546-551.\n\n   .. [2] S. Wang, and X. Yao. \"Diversity analysis on imbalanced data sets by\n          using ensemble models.\" 2009 IEEE symposium on computational\n          intelligence and data mining. IEEE, 2009.\n\n   .. [3] S. Hido, H. Kashima, and Y. Takahashi. \"Roughly balanced bagging\n         for imbalanced data.\" Statistical Analysis and Data Mining: The ASA\n         Data Science Journal 2.5\u20106 (2009): 412-426.\n\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.12"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}