{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stats sub-module\n",
    "\n",
    "The ```acgc.stats``` sub-module is a collection of statistical methods. See ```help(acgc.stats)``` for complete list of methods.\n",
    "\n",
    "The most commonly used functions include BivariateStatistics, line fitting methods (sma, sen, york), and weighted statistics (wmean, wmedian, wcov, wcorr, etc.)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from acgc import stats as mystats\n",
    "from acgc import figstyle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BivariateStatistics\n",
    "This function quickly computes many statistical measures of the relationship between two populations.\n",
    "\n",
    "As an example, we will create samples of two populations with a linear relationship between them, which is obscured by noise added to both "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate two sample populations\n",
    "n = 100\n",
    "x_true = np.random.rand(n) + 2\n",
    "y_true = 2*x_true + 1 \n",
    "\n",
    "# Add noise with normal distribution\n",
    "x = x_true + np.random.randn(n)*0.1\n",
    "y = y_true + np.random.randn(n)*0.3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use `BivariateStatistics` to compute statistical relationships and `summary()` to display some of the most commonly used statistics. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MD    = 3.5359\n",
      "MAD   = 3.5359\n",
      "RMD   = 1.4353\n",
      "RMAD  = 1.4353\n",
      "MRD   = 1.4427\n",
      "R2    = 0.6839\n",
      "slope = 2.2886\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Compute bivariate statistics\n",
    "result = mystats.BivariateStatistics(x,y)\n",
    "\n",
    "# Display a table showing some common statistical measures of population differences\n",
    "print(result.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The abbreviations seen in the `summary()` output are defined in the documentation for the `result` variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show documentation including abbreviations\n",
    "help(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Users can specify which variables they want to include in the summary output and the type of line fitting that they want to use (SMA, OLS, WLS, Theil-Sen, York)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MD        = 3.5359\n",
      "MedD      = 3.5411\n",
      "slope     = 1.8926\n",
      "intercept = 1.3371\n",
      "R         = 0.8270\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Specify exactly which variables to include in the summary\n",
    "#   mean difference (MD)\n",
    "#   Median difference (MedD)\n",
    "#   slope (SMA method)\n",
    "#   intercept (SMA method)\n",
    "#   Pearson correlation (R)\n",
    "display_vars = ['MD','MedD','slope','intercept','R']\n",
    "\n",
    "# Summarize, using OLS line fitting\n",
    "print(result.summary(display_vars, \n",
    "    fitline_kw={'method':'OLS'}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Show a figure with summary statistics as an inset table\n",
    "\n",
    "The summary table can be written on a figure. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 64368 characters]",
      "text/plain": [
       "<Figure size 400x330 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 341,
       "width": 411
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot data\n",
    "ax = plt.subplot()\n",
    "ax.scatter(x,y)\n",
    "\n",
    "# Add text displaying slope and R2\n",
    "t=result.summary_fig_table(ax,\n",
    "                         ['MD','RMD','slope','R2'],\n",
    "                         floatformat='{:5.2f}',\n",
    "                         loc=(0.75,0.05), loc_units='axes',\n",
    "                         fontsize=9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Weighted statistics\n",
    "\n",
    "Weighted versions of mean, standard deviation, covariance, etc. are available. The functions names follow the numpy functions prefixed with 'w'. For example 'wmean' is the weighted mean and 'wstd' is the weighted standard deviation. When weights are equal to 1, the result is identical to the unweighted statistic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Weighted mean:  2.431814616080641\n",
      "Weighted std: 0.27380121568197974\n",
      "Weighted covariance: 0.13680414193311458\n",
      "Standard covariance: 0.14796233496639685\n"
     ]
    }
   ],
   "source": [
    "# Generate hypothetical weights. These should be positive\n",
    "w = (np.random.randn(n)+1)**2\n",
    "\n",
    "# Print some weighted statistics\n",
    "print('Weighted mean:', mystats.wmean(x,w) )\n",
    "print('Weighted std:',  mystats.wstd(x,w))\n",
    "print('Weighted covariance:', mystats.wcov(x,y,w))\n",
    "print('Standard covariance:', mystats.wcov(x,y,np.ones_like(x)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the `robust` keyword, the sample weights are computed internally to reduce the influence of extreme outliers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Standard mean: 0.7238700659114198\n",
      "Robust mean: 2.3264274483208176\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/cdholmes/anaconda3/envs/general/lib/python3.11/site-packages/sklearn/covariance/_robust_covariance.py:747: UserWarning: The covariance matrix associated to your dataset is not full rank\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "# Add some extreme outliers\n",
    "xoutliers = x + np.random.standard_cauchy(n)\n",
    "\n",
    "# The outliers affect the mean much more than the robust mean\n",
    "print('Standard mean:', np.mean(xoutliers))\n",
    "print('Robust mean:', mystats.wmean(xoutliers,robust=True))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "general",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
