{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Insights\n",
    "\n",
    "The Model Insights feature lets you monitor how the environment that your model operates within may be changing in ways that affect it's predictions so that you can intervene (retrain) in an efficient and timely manner. Changes in the inputs, **data drift**, can occur due to errors in the data processing pipeline or due to changes in the environment such as user preference or behavior. \n",
    "\n",
    "The [validation framework](../monitoring.html) performs per infererence range checks with count frequency based thresholds for alerts and is ideal for catching many errors in input and output data.\n",
    "\n",
    "In complement to the validation framework model insights focuses on the differences in the distributions of data in a time based window measured against a baseline for a given pipeline and can detect situations where values are still within the expected range but the distribution has shifted. For example, if your model predicts housing prices you might expect the predictions to be between \\\\$200,000 and \\\\$1,000,000 with a distribution centered around \\\\$400,000. If your model suddenly starts predicting prices centered around \\\\$250,000 or \\\\$750,000 the predictions may still be within the expected range but the shift may signal something has changed that should be investigated.\n",
    "\n",
    "Ideally we'd also monitor the _quality_ of the predictions, **concept drift**. However this can be difficult as true labels are often not available or are severely delayed in practice. That is there may be a signficant lag between the time the prediction is made and the true (sale price) value is observed.\n",
    "\n",
    "Consequently, model insights uses data drift detection techniques on both inputs and outputs to detect changes in the distributions of the data.\n",
    "\n",
    "There are many useful statistical tests for calculating the difference between distributions; however, they typically require assumptions about the underlying distributions or confusing and expensive calculations. We've implemented a data drift framework that is easy to understand, fast to compute, runs in an automated fashion and is extensible to many specific use cases.\n",
    "\n",
    "The methodology currently revolves around calculating the specific percentile-based bins of the baseline distribution and measuring how future distributions fall into these bins. This approach is both visually intuitive and supports an easy to calculate difference score between distributions. Users can tune the scoring mechanism to emphasize different regions of the distribution: for example, you may only care if there is a change in the top 20th percentile of the distribution, compared to the baseline.\n",
    "\n",
    "You can specify the inputs or outputs that you want to monitor and the data to use for your baselines. You can also specify how often you want to monitor distributions and set parameters to define what constitutes a meaningful change in a distribution for your application. \n",
    "\n",
    "Once you've set up a monitoring task, called an assay, comparisons against your baseline are then run automatically on a scheduled basis. You can be notified if the system notices any abnormally different behavior. The framework also allows you to quickly investigate the cause of any unexpected drifts in your predictions.\n",
    "\n",
    "The rest of this notebook will shows how to create assays to monitor your pipelines.\n",
    "\n",
    "**NOTE:** model insights operates over time and is difficult to demo in a notebook without pre-canned data. **We assume you have an active pipeline that has been running and making predictions over time and show you the code you may use to analyze your pipeline.**\n",
    "\n",
    "\n",
    "### Workflow\n",
    "\n",
    "Model Insights has the capability to perform interactive assays so that you can explore the data from a pipeline and learn how the data is behaving. With this information and the knowledge of your particular business use case you can then choose appropriate thresholds for persistent automatic assays as desired.\n",
    "\n",
    "To get started lets import some libraries we'll need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd \n",
    "\n",
    "import wallaroo\n",
    "from wallaroo.assay_config import BinMode, Aggregation, Metric\n",
    "\n",
    "\n",
    "plt.rcParams[\"figure.figsize\"] = (12,6)\n",
    "pd.options.display.float_format = '{:,.2f}'.format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then lets get a handle to your existing running pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = wallaroo.Client()\n",
    "\n",
    "pipeline_name = 'mypipeline'\n",
    "model_name = 'mymodel'\n",
    "\n",
    "pipeline = client.pipelines_by_name(pipeline_name)[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We assume the pipeline has been running for a while and there is a period of time that is free of errors that we'd like to use as the _baseline_. Lets note the start and end times. For this example we have 30 days of data from Jan 2022 and well use Jan 1 data as our baseline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "baseline_start = datetime.datetime.fromisoformat('2022-01-01T00:00:00+00:00')\n",
    "baseline_end = datetime.datetime.fromisoformat('2022-01-02T00:00:00+00:00')\n",
    "last_day = datetime.datetime.fromisoformat('2022-02-01T00:00:00+00:00')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets create an assay using that pipeline and the model in the pipeline. We also specify the start end end of the baseline. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "assay_name = \"example assay\"\n",
    "assay_builder = client.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We don't know much about our baseline data yet so lets examine the data and create a couple of visual representations. First lets get some basic stats on the baseline data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Baseline</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1813</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>11.88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>15.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>13.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>median</th>\n",
       "      <td>13.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>start</th>\n",
       "      <td>2022-01-01T00:00:00Z</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>end</th>\n",
       "      <td>2022-01-02T00:00:00Z</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    Baseline\n",
       "count                   1813\n",
       "min                    11.88\n",
       "max                    15.02\n",
       "mean                   13.02\n",
       "median                 13.00\n",
       "std                     0.46\n",
       "start   2022-01-01T00:00:00Z\n",
       "end     2022-01-02T00:00:00Z"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "baseline_run = assay_builder.build().interactive_baseline_run()\n",
    "baseline_run.baseline_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets look at a histogram, kernel density estimate (KDE), and Emperical Cumulative Distribution (ecdf) charts of the baseline data. These will give us insite into the distributions of the predictions and features that the assay is configured for."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 30748 characters]\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "assay_builder.baseline_histogram()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 51128 characters]\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "assay_builder.baseline_kde()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 39664 characters]\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "assay_builder.baseline_ecdf()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interactive Baseline Runs\n",
    "We can do an interactive run of just the baseline part to see how the baseline data will be put into bins. This assay uses quintiles so all 5 bins (not counting the outlier bins) have 20% of the predictions. We can see the bin boundaries along the x-axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baseline mean = 13.021036998645679\n",
      "baseline median = 13.004066467285156\n",
      "bin_mode = Quantile\n",
      "aggregation = Density\n",
      "metric = PSI\n",
      "weighted = False\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 47200 characters]\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "baseline_run.chart()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also get a dataframe with the bin/edge information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b_edges</th>\n",
       "      <th>b_edge_names</th>\n",
       "      <th>b_aggregated_values</th>\n",
       "      <th>b_aggregation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11.88</td>\n",
       "      <td>left_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12.63</td>\n",
       "      <td>q_20</td>\n",
       "      <td>0.20</td>\n",
       "      <td>Density</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>12.90</td>\n",
       "      <td>q_40</td>\n",
       "      <td>0.20</td>\n",
       "      <td>Density</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>13.11</td>\n",
       "      <td>q_60</td>\n",
       "      <td>0.20</td>\n",
       "      <td>Density</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>13.36</td>\n",
       "      <td>q_80</td>\n",
       "      <td>0.20</td>\n",
       "      <td>Density</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>15.02</td>\n",
       "      <td>q_100</td>\n",
       "      <td>0.20</td>\n",
       "      <td>Density</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>179,769,313,486,231,570,814,527,423,731,704,356...</td>\n",
       "      <td>right_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             b_edges   b_edge_names  \\\n",
       "0                                              11.88   left_outlier   \n",
       "1                                              12.63           q_20   \n",
       "2                                              12.90           q_40   \n",
       "3                                              13.11           q_60   \n",
       "4                                              13.36           q_80   \n",
       "5                                              15.02          q_100   \n",
       "6 179,769,313,486,231,570,814,527,423,731,704,356...  right_outlier   \n",
       "\n",
       "   b_aggregated_values b_aggregation  \n",
       "0                 0.00       Density  \n",
       "1                 0.20       Density  \n",
       "2                 0.20       Density  \n",
       "3                 0.20       Density  \n",
       "4                 0.20       Density  \n",
       "5                 0.20       Density  \n",
       "6                 0.00       Density  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "baseline_run.baseline_bins()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previous assay used quintiles so all of the bins had the same percentage/count of samples.  To get bins that are divided equaly along the range of values we can use `BinMode.EQUAL`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baseline mean = 13.021036998645679\n",
      "baseline median = 13.004066467285156\n",
      "bin_mode = Equal\n",
      "aggregation = Density\n",
      "metric = PSI\n",
      "weighted = False\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 40964 characters]\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "equal_bin_builder = client.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end)\n",
    "equal_bin_builder.summarizer_builder.add_bin_mode(BinMode.EQUAL)\n",
    "equal_baseline = equal_bin_builder.build().interactive_baseline_run()\n",
    "equal_baseline.chart()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now see very different bin edges and sample percentages per bin."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b_edges</th>\n",
       "      <th>b_edge_names</th>\n",
       "      <th>b_aggregated_values</th>\n",
       "      <th>b_aggregation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11.88</td>\n",
       "      <td>left_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12.51</td>\n",
       "      <td>p_1.25e1</td>\n",
       "      <td>0.12</td>\n",
       "      <td>Density</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13.13</td>\n",
       "      <td>p_1.31e1</td>\n",
       "      <td>0.50</td>\n",
       "      <td>Density</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>13.76</td>\n",
       "      <td>p_1.38e1</td>\n",
       "      <td>0.31</td>\n",
       "      <td>Density</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>14.39</td>\n",
       "      <td>p_1.44e1</td>\n",
       "      <td>0.06</td>\n",
       "      <td>Density</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>15.02</td>\n",
       "      <td>p_1.50e1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>Density</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>179,769,313,486,231,570,814,527,423,731,704,356...</td>\n",
       "      <td>right_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             b_edges   b_edge_names  \\\n",
       "0                                              11.88   left_outlier   \n",
       "1                                              12.51       p_1.25e1   \n",
       "2                                              13.13       p_1.31e1   \n",
       "3                                              13.76       p_1.38e1   \n",
       "4                                              14.39       p_1.44e1   \n",
       "5                                              15.02       p_1.50e1   \n",
       "6 179,769,313,486,231,570,814,527,423,731,704,356...  right_outlier   \n",
       "\n",
       "   b_aggregated_values b_aggregation  \n",
       "0                 0.00       Density  \n",
       "1                 0.12       Density  \n",
       "2                 0.50       Density  \n",
       "3                 0.31       Density  \n",
       "4                 0.06       Density  \n",
       "5                 0.01       Density  \n",
       "6                 0.00       Density  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "equal_baseline.baseline_bins()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interactive Assay Runs\n",
    "By default the assay builder creates an assay with some good starting parameters. In particular the assay is configured to run a new analysis for every 24 hours starting at the end of the baseline period. Additionally, it sets the number of bins to 5 so creates quintiles, and sets the target `iopath` to `\"outputs 0 0\"` which means we want to monitor the first column of the first output/prediction.\n",
    "\n",
    "We can do an interactive run of just the baseline part to see how the baseline data will be put into bins. This assay uses quintiles so all 5 bins (not counting the outlier bins) have 20% of the predictions. We can see the bin boundaries along the x-axis.\n",
    "\n",
    "We then run it with `interactive_run` and convert it to a dataframe for easy analysis with `to_dataframe`.\n",
    "\n",
    "Now lets do an interactive run of the first assay as it is configured.  Interactive runs, don't save the assay to the database (so they won't be scheduled in the future) nor do they save the assay results. Instead the results are returned after a short while for further analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assay_id</th>\n",
       "      <th>name</th>\n",
       "      <th>iopath</th>\n",
       "      <th>score</th>\n",
       "      <th>start</th>\n",
       "      <th>min</th>\n",
       "      <th>max</th>\n",
       "      <th>mean</th>\n",
       "      <th>median</th>\n",
       "      <th>std</th>\n",
       "      <th>warning_threshold</th>\n",
       "      <th>alert_threshold</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>0.02</td>\n",
       "      <td>2022-01-02T00:00:00Z</td>\n",
       "      <td>11.85</td>\n",
       "      <td>15.28</td>\n",
       "      <td>13.02</td>\n",
       "      <td>13.00</td>\n",
       "      <td>0.45</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>0.01</td>\n",
       "      <td>2022-01-03T00:00:00Z</td>\n",
       "      <td>11.89</td>\n",
       "      <td>15.46</td>\n",
       "      <td>13.01</td>\n",
       "      <td>12.99</td>\n",
       "      <td>0.45</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>0.01</td>\n",
       "      <td>2022-01-04T00:00:00Z</td>\n",
       "      <td>11.65</td>\n",
       "      <td>14.88</td>\n",
       "      <td>13.02</td>\n",
       "      <td>13.01</td>\n",
       "      <td>0.45</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>0.02</td>\n",
       "      <td>2022-01-05T00:00:00Z</td>\n",
       "      <td>11.78</td>\n",
       "      <td>15.37</td>\n",
       "      <td>13.02</td>\n",
       "      <td>13.00</td>\n",
       "      <td>0.45</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>0.02</td>\n",
       "      <td>2022-01-06T00:00:00Z</td>\n",
       "      <td>11.84</td>\n",
       "      <td>15.20</td>\n",
       "      <td>13.01</td>\n",
       "      <td>13.00</td>\n",
       "      <td>0.44</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>0.02</td>\n",
       "      <td>2022-01-07T00:00:00Z</td>\n",
       "      <td>11.93</td>\n",
       "      <td>15.06</td>\n",
       "      <td>13.02</td>\n",
       "      <td>13.01</td>\n",
       "      <td>0.45</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>0.01</td>\n",
       "      <td>2022-01-08T00:00:00Z</td>\n",
       "      <td>11.90</td>\n",
       "      <td>15.57</td>\n",
       "      <td>13.00</td>\n",
       "      <td>12.99</td>\n",
       "      <td>0.44</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>0.02</td>\n",
       "      <td>2022-01-09T00:00:00Z</td>\n",
       "      <td>11.87</td>\n",
       "      <td>15.27</td>\n",
       "      <td>13.02</td>\n",
       "      <td>13.00</td>\n",
       "      <td>0.45</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>2022-01-10T00:00:00Z</td>\n",
       "      <td>11.89</td>\n",
       "      <td>14.80</td>\n",
       "      <td>13.02</td>\n",
       "      <td>13.00</td>\n",
       "      <td>0.44</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>0.01</td>\n",
       "      <td>2022-01-11T00:00:00Z</td>\n",
       "      <td>11.84</td>\n",
       "      <td>14.93</td>\n",
       "      <td>13.02</td>\n",
       "      <td>13.01</td>\n",
       "      <td>0.44</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>0.01</td>\n",
       "      <td>2022-01-12T00:00:00Z</td>\n",
       "      <td>11.47</td>\n",
       "      <td>14.89</td>\n",
       "      <td>13.01</td>\n",
       "      <td>13.00</td>\n",
       "      <td>0.46</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>0.03</td>\n",
       "      <td>2022-01-13T00:00:00Z</td>\n",
       "      <td>8.20</td>\n",
       "      <td>14.82</td>\n",
       "      <td>12.95</td>\n",
       "      <td>12.95</td>\n",
       "      <td>0.50</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>0.99</td>\n",
       "      <td>2022-01-14T00:00:00Z</td>\n",
       "      <td>6.68</td>\n",
       "      <td>15.56</td>\n",
       "      <td>12.46</td>\n",
       "      <td>12.83</td>\n",
       "      <td>1.35</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Alert</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>4.56</td>\n",
       "      <td>2022-01-15T00:00:00Z</td>\n",
       "      <td>5.66</td>\n",
       "      <td>17.48</td>\n",
       "      <td>10.93</td>\n",
       "      <td>10.99</td>\n",
       "      <td>1.83</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Alert</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>3.93</td>\n",
       "      <td>2022-01-16T00:00:00Z</td>\n",
       "      <td>6.43</td>\n",
       "      <td>18.89</td>\n",
       "      <td>11.29</td>\n",
       "      <td>11.22</td>\n",
       "      <td>1.86</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Alert</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>3.49</td>\n",
       "      <td>2022-01-17T00:00:00Z</td>\n",
       "      <td>7.26</td>\n",
       "      <td>20.83</td>\n",
       "      <td>11.73</td>\n",
       "      <td>11.68</td>\n",
       "      <td>1.98</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Alert</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>3.16</td>\n",
       "      <td>2022-01-18T00:00:00Z</td>\n",
       "      <td>7.91</td>\n",
       "      <td>19.74</td>\n",
       "      <td>12.16</td>\n",
       "      <td>12.03</td>\n",
       "      <td>2.02</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Alert</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>2.81</td>\n",
       "      <td>2022-01-19T00:00:00Z</td>\n",
       "      <td>8.65</td>\n",
       "      <td>22.57</td>\n",
       "      <td>12.60</td>\n",
       "      <td>12.43</td>\n",
       "      <td>1.92</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Alert</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>2.01</td>\n",
       "      <td>2022-01-20T00:00:00Z</td>\n",
       "      <td>9.24</td>\n",
       "      <td>23.23</td>\n",
       "      <td>13.30</td>\n",
       "      <td>13.09</td>\n",
       "      <td>1.81</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Alert</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>2.00</td>\n",
       "      <td>2022-01-21T00:00:00Z</td>\n",
       "      <td>10.18</td>\n",
       "      <td>23.73</td>\n",
       "      <td>13.99</td>\n",
       "      <td>13.86</td>\n",
       "      <td>1.74</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Alert</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>2.92</td>\n",
       "      <td>2022-01-22T00:00:00Z</td>\n",
       "      <td>11.00</td>\n",
       "      <td>24.38</td>\n",
       "      <td>14.86</td>\n",
       "      <td>14.69</td>\n",
       "      <td>1.71</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Alert</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>5.02</td>\n",
       "      <td>2022-01-23T00:00:00Z</td>\n",
       "      <td>11.82</td>\n",
       "      <td>26.12</td>\n",
       "      <td>15.69</td>\n",
       "      <td>15.50</td>\n",
       "      <td>1.70</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Alert</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>7.95</td>\n",
       "      <td>2022-01-24T00:00:00Z</td>\n",
       "      <td>12.52</td>\n",
       "      <td>24.01</td>\n",
       "      <td>16.42</td>\n",
       "      <td>16.26</td>\n",
       "      <td>1.59</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Alert</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>7.88</td>\n",
       "      <td>2022-01-25T00:00:00Z</td>\n",
       "      <td>13.39</td>\n",
       "      <td>26.72</td>\n",
       "      <td>17.30</td>\n",
       "      <td>17.18</td>\n",
       "      <td>1.67</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Alert</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>9.03</td>\n",
       "      <td>2022-01-26T00:00:00Z</td>\n",
       "      <td>14.78</td>\n",
       "      <td>27.51</td>\n",
       "      <td>18.15</td>\n",
       "      <td>17.97</td>\n",
       "      <td>1.67</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Alert</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>3.65</td>\n",
       "      <td>2022-01-27T00:00:00Z</td>\n",
       "      <td>12.08</td>\n",
       "      <td>29.19</td>\n",
       "      <td>16.74</td>\n",
       "      <td>17.41</td>\n",
       "      <td>3.17</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Alert</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>0.01</td>\n",
       "      <td>2022-01-28T00:00:00Z</td>\n",
       "      <td>12.04</td>\n",
       "      <td>14.81</td>\n",
       "      <td>12.99</td>\n",
       "      <td>13.01</td>\n",
       "      <td>0.37</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>0.03</td>\n",
       "      <td>2022-01-29T00:00:00Z</td>\n",
       "      <td>12.01</td>\n",
       "      <td>14.45</td>\n",
       "      <td>13.00</td>\n",
       "      <td>13.01</td>\n",
       "      <td>0.36</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>0.02</td>\n",
       "      <td>2022-01-30T00:00:00Z</td>\n",
       "      <td>11.97</td>\n",
       "      <td>14.46</td>\n",
       "      <td>12.99</td>\n",
       "      <td>13.01</td>\n",
       "      <td>0.37</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>None</td>\n",
       "      <td>example assay</td>\n",
       "      <td>output 0 0</td>\n",
       "      <td>0.02</td>\n",
       "      <td>2022-01-31T00:00:00Z</td>\n",
       "      <td>12.05</td>\n",
       "      <td>14.80</td>\n",
       "      <td>12.98</td>\n",
       "      <td>13.01</td>\n",
       "      <td>0.38</td>\n",
       "      <td>None</td>\n",
       "      <td>0.25</td>\n",
       "      <td>Ok</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assay_id           name      iopath  score                 start   min  \\\n",
       "0      None  example assay  output 0 0   0.02  2022-01-02T00:00:00Z 11.85   \n",
       "1      None  example assay  output 0 0   0.01  2022-01-03T00:00:00Z 11.89   \n",
       "2      None  example assay  output 0 0   0.01  2022-01-04T00:00:00Z 11.65   \n",
       "3      None  example assay  output 0 0   0.02  2022-01-05T00:00:00Z 11.78   \n",
       "4      None  example assay  output 0 0   0.02  2022-01-06T00:00:00Z 11.84   \n",
       "5      None  example assay  output 0 0   0.02  2022-01-07T00:00:00Z 11.93   \n",
       "6      None  example assay  output 0 0   0.01  2022-01-08T00:00:00Z 11.90   \n",
       "7      None  example assay  output 0 0   0.02  2022-01-09T00:00:00Z 11.87   \n",
       "8      None  example assay  output 0 0   0.00  2022-01-10T00:00:00Z 11.89   \n",
       "9      None  example assay  output 0 0   0.01  2022-01-11T00:00:00Z 11.84   \n",
       "10     None  example assay  output 0 0   0.01  2022-01-12T00:00:00Z 11.47   \n",
       "11     None  example assay  output 0 0   0.03  2022-01-13T00:00:00Z  8.20   \n",
       "12     None  example assay  output 0 0   0.99  2022-01-14T00:00:00Z  6.68   \n",
       "13     None  example assay  output 0 0   4.56  2022-01-15T00:00:00Z  5.66   \n",
       "14     None  example assay  output 0 0   3.93  2022-01-16T00:00:00Z  6.43   \n",
       "15     None  example assay  output 0 0   3.49  2022-01-17T00:00:00Z  7.26   \n",
       "16     None  example assay  output 0 0   3.16  2022-01-18T00:00:00Z  7.91   \n",
       "17     None  example assay  output 0 0   2.81  2022-01-19T00:00:00Z  8.65   \n",
       "18     None  example assay  output 0 0   2.01  2022-01-20T00:00:00Z  9.24   \n",
       "19     None  example assay  output 0 0   2.00  2022-01-21T00:00:00Z 10.18   \n",
       "20     None  example assay  output 0 0   2.92  2022-01-22T00:00:00Z 11.00   \n",
       "21     None  example assay  output 0 0   5.02  2022-01-23T00:00:00Z 11.82   \n",
       "22     None  example assay  output 0 0   7.95  2022-01-24T00:00:00Z 12.52   \n",
       "23     None  example assay  output 0 0   7.88  2022-01-25T00:00:00Z 13.39   \n",
       "24     None  example assay  output 0 0   9.03  2022-01-26T00:00:00Z 14.78   \n",
       "25     None  example assay  output 0 0   3.65  2022-01-27T00:00:00Z 12.08   \n",
       "26     None  example assay  output 0 0   0.01  2022-01-28T00:00:00Z 12.04   \n",
       "27     None  example assay  output 0 0   0.03  2022-01-29T00:00:00Z 12.01   \n",
       "28     None  example assay  output 0 0   0.02  2022-01-30T00:00:00Z 11.97   \n",
       "29     None  example assay  output 0 0   0.02  2022-01-31T00:00:00Z 12.05   \n",
       "\n",
       "     max  mean  median  std warning_threshold  alert_threshold status  \n",
       "0  15.28 13.02   13.00 0.45              None             0.25     Ok  \n",
       "1  15.46 13.01   12.99 0.45              None             0.25     Ok  \n",
       "2  14.88 13.02   13.01 0.45              None             0.25     Ok  \n",
       "3  15.37 13.02   13.00 0.45              None             0.25     Ok  \n",
       "4  15.20 13.01   13.00 0.44              None             0.25     Ok  \n",
       "5  15.06 13.02   13.01 0.45              None             0.25     Ok  \n",
       "6  15.57 13.00   12.99 0.44              None             0.25     Ok  \n",
       "7  15.27 13.02   13.00 0.45              None             0.25     Ok  \n",
       "8  14.80 13.02   13.00 0.44              None             0.25     Ok  \n",
       "9  14.93 13.02   13.01 0.44              None             0.25     Ok  \n",
       "10 14.89 13.01   13.00 0.46              None             0.25     Ok  \n",
       "11 14.82 12.95   12.95 0.50              None             0.25     Ok  \n",
       "12 15.56 12.46   12.83 1.35              None             0.25  Alert  \n",
       "13 17.48 10.93   10.99 1.83              None             0.25  Alert  \n",
       "14 18.89 11.29   11.22 1.86              None             0.25  Alert  \n",
       "15 20.83 11.73   11.68 1.98              None             0.25  Alert  \n",
       "16 19.74 12.16   12.03 2.02              None             0.25  Alert  \n",
       "17 22.57 12.60   12.43 1.92              None             0.25  Alert  \n",
       "18 23.23 13.30   13.09 1.81              None             0.25  Alert  \n",
       "19 23.73 13.99   13.86 1.74              None             0.25  Alert  \n",
       "20 24.38 14.86   14.69 1.71              None             0.25  Alert  \n",
       "21 26.12 15.69   15.50 1.70              None             0.25  Alert  \n",
       "22 24.01 16.42   16.26 1.59              None             0.25  Alert  \n",
       "23 26.72 17.30   17.18 1.67              None             0.25  Alert  \n",
       "24 27.51 18.15   17.97 1.67              None             0.25  Alert  \n",
       "25 29.19 16.74   17.41 3.17              None             0.25  Alert  \n",
       "26 14.81 12.99   13.01 0.37              None             0.25     Ok  \n",
       "27 14.45 13.00   13.01 0.36              None             0.25     Ok  \n",
       "28 14.46 12.99   13.01 0.37              None             0.25     Ok  \n",
       "29 14.80 12.98   13.01 0.38              None             0.25     Ok  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "assay_builder = client.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end)\n",
    "assay_config = assay_builder.add_run_until(last_day).build()\n",
    "assay_results = assay_config.interactive_run()\n",
    "assay_df = assay_results.to_dataframe()\n",
    "assay_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basic functionality for creating quick charts is included."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 38908 characters]\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "assay_results.chart_scores()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the difference scores are low for a while and then jump up to indicate there is an issue. We can examine that particular window to help us decide if that threshold is set correctly or not.\n",
    "\n",
    "We can generate a quick chart of the results. This chart shows the 5 quantile bins (quintiles) derived from the baseline data plus one for left outliers and one for right outliers.  We also see that the data from the window falls within the baseline quintiles but in a different proportion and is skewing higher. Whether this is an issue or not is specific to your use case.\n",
    "\n",
    "First lets examine a day that is only slightly different than the baseline. We see that we do see some values that fall outside of the range from the baseline values, the left and right outliers, and that the bin values are different but similar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baseline mean = 13.021036998645679\n",
      "window mean = 13.01618284105465\n",
      "baseline median = 13.004066467285156\n",
      "window median = 13.004111289978027\n",
      "bin_mode = Quantile\n",
      "aggregation = Density\n",
      "metric = PSI\n",
      "weighted = False\n",
      "score = 0.019674378474700036\n",
      "scores = [0.00980308228898587, 0.0002603476402290938, 0.00022327892034488384, 0.0031099778889999417, 9.781216093661028e-07, 0.0003905674264324862, 0.005886146188098397]\n",
      "index = None\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 53056 characters]\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "assay_results[0].chart()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other days, however are significantly different."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baseline mean = 13.021036998645679\n",
      "window mean = 12.462097006128324\n",
      "baseline median = 13.004066467285156\n",
      "window median = 12.829922676086426\n",
      "bin_mode = Quantile\n",
      "aggregation = Density\n",
      "metric = PSI\n",
      "weighted = False\n",
      "score = 0.9897758782012277\n",
      "scores = [0.8594285651741624, 0.0008166837246366183, 0.039563316977030236, 0.05914513530289921, 0.02759016554769916, 0.001900118150881501, 0.0013318933239185415]\n",
      "index = None\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 51176 characters]\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "assay_results[12].chart()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to investigate further, we can run interactive assays on each of the inputs to see if any of them show anything abnormal. In this example we'll provide the feature labels to create more understandable titles.\n",
    "\n",
    "The current assay expects continous data. Sometimes categorical data is encoded as 1 or 0 in a feature and sometimes in a limited number of values such as 1, 2, 3. If one value has high a percentage the analysis emits a warning so that we know the scores for that feature may not behave as we expect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input column distinct_vals label           largest_pct\n",
      "    0     0             12 bedrooms        0.4567 \n",
      "    0     1             26 bathrooms       0.2421 \n",
      "    0     2           1492 lat             0.0022 \n",
      "    0     3            502 long            0.0077 \n",
      "    0     4              2 waterfront      0.9928 *** May not be continuous feature\n",
      "    0     5            425 sqft_living     0.0094 \n",
      "    0     6           1407 sqft_lot        0.0188 \n",
      "    0     7              6 floors          0.5036 \n",
      "    0     8              5 view            0.9068 *** May not be continuous feature\n",
      "    0     9              5 condition       0.6293 \n",
      "    0    10             10 grade           0.4242 \n",
      "    0    11            393 sqft_above      0.0121 \n",
      "    0    12            162 sqft_basement   0.6034 \n",
      "    0    13            116 yr_built        0.0237 \n",
      "    0    14             37 yr_renovated    0.9597 *** May not be continuous feature\n",
      "    0    15            340 sqft_living15   0.0132 \n",
      "    0    16           1347 sqft_lot15      0.0177 \n"
     ]
    }
   ],
   "source": [
    "labels = ['bedrooms', 'bathrooms', 'lat', 'long', 'waterfront', 'sqft_living', 'sqft_lot', 'floors', 'view', 'condition', 'grade', 'sqft_above', 'sqft_basement', 'yr_built', 'yr_renovated', 'sqft_living15', 'sqft_lot15']\n",
    "\n",
    "baseline_inferences = client.get_raw_pipeline_inference_logs(pipeline_name, baseline_start, baseline_end, model_name)\n",
    "\n",
    "assay_results = assay_config.interactive_input_run(baseline_inferences, labels)\n",
    "iadf = assay_results.to_dataframe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can chart each of the iopaths and do a visual inspection. From the charts we see that if any of the input features had significant differences in the first two days which we can choose to inspect further. Here we choose to show 3 charts just to save space in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 53284 characters]\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 50096 characters]\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 46232 characters]\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "assay_results.chart_iopaths(labels=labels, selected_labels=['bedrooms', 'lat', 'sqft_living'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we are comfortable with what alert threshold should be for our specific purposes we can create and save an assay that will be automatically run on a daily basis.\n",
    "\n",
    "In this example we're create an assay that runs everyday against the baseline and has an alert threshold of 0.5.\n",
    "\n",
    "Once we upload it it will be saved and scheduled for future data as well as run against past data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "alert_threshold = 0.5\n",
    "assay_builder = client.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end).add_alert_threshold(alert_threshold)\n",
    "assay_id = assay_builder.upload()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After a short while, we can get the assay results for further analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "assay_results = client.get_assay_results_logs(baseline_end,datetime.datetime.now(), assay_id=assay_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the assays analysis is similar to the interactive run we started with though the analysis for the third day does not exceed the new alert threshold we set. And since we called `upload` instead of `interactive_run` the assay was saved to the system and will continue to run automatically on schedule from now on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 38908 characters]\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "assay_results.chart_scores()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scheduling Assays\n",
    "\n",
    "By default assays are scheduled to run every 24 hours starting immediately after the baseline period ends.\n",
    "\n",
    "However, you can control the start time by setting `start` and the frequency by setting `interval` on the window.\n",
    "\n",
    "So to recap:\n",
    "\n",
    "* The window width is the size of the window. The default is 24 hours.\n",
    "* The interval is how often the analysis is run, how far the window is slid into the future based on the last run. The default is the window width.\n",
    "* The window start is when the analysis should start. The default is the end of the baseline period.\n",
    "\n",
    "For example to run an analysis every 12 hours on the previous 24 hours of data you'd set the window width to 24 (the default) and the interval to 12."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated 59 analyses\n"
     ]
    }
   ],
   "source": [
    "assay_builder = client.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end)\n",
    "assay_builder = assay_builder.add_run_until(last_day)\n",
    "\n",
    "assay_builder.window_builder().add_width(hours=24).add_interval(hours=12)\n",
    "\n",
    "assay_config = assay_builder.build()\n",
    "\n",
    "assay_results = assay_config.interactive_run()\n",
    "print(f\"Generated {len(assay_results)} analyses\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start a weekly analysis of the previous week on a specific day, set the start date (taking care to specify the desired timezone), and the width and interval to 1 week and of course an analysis won't be generated till a window is complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated 4 analyses\n"
     ]
    }
   ],
   "source": [
    "report_start = datetime.datetime.fromisoformat('2022-01-03T00:00:00+00:00')\n",
    "\n",
    "assay_builder = client.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end)\n",
    "assay_builder = assay_builder.add_run_until(last_day)\n",
    "\n",
    "assay_builder.window_builder().add_width(weeks=1).add_interval(weeks=1).add_start(report_start)\n",
    "\n",
    "assay_config = assay_builder.build()\n",
    "\n",
    "assay_results = assay_config.interactive_run()\n",
    "print(f\"Generated {len(assay_results)} analyses\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced Configuration\n",
    "\n",
    "The assay can be configured in a variety of ways to help customize it to your particular needs. Specifically you can:\n",
    "* change the BinMode to evenly spaced, quantile or user provided\n",
    "* change the number of bins to use\n",
    "* provide weights to use when scoring the bins\n",
    "* calculate the score using the sum of differences, maximum difference or population stability index\n",
    "* change the value aggregation for the bins to density, cumulative or edges\n",
    "\n",
    "Lets take a look at these in turn.\n",
    "\n",
    "### Default configuration\n",
    "\n",
    "First lets look at the default configuration. This is a lot of information but much of it is useful to know where it is available.\n",
    "\n",
    "We see that the assay is broken up into 4 sections. A top level meta data section, a section for the baseline specification, a section for the window specification and a section that specifies the summarization configuration.\n",
    "\n",
    "In the meta section we see the name of the assay, that it runs on the first column of the first output `\"outputs 0 0\"` and that there is a default threshold of 0.25.\n",
    "\n",
    "The summarizer section shows us the defaults of Quantile, Density and PSI on 5 bins.\n",
    "\n",
    "The baseline section shows us that it is configured as a fixed baseline with the specified start and end date times.\n",
    "\n",
    "And the window tells us what model in the pipeline we are analyzing and how often."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"name\": \"Test Assay\",\n",
      "    \"pipeline_id\": 1,\n",
      "    \"pipeline_name\": \"mypipeline\",\n",
      "    \"active\": true,\n",
      "    \"status\": \"created\",\n",
      "    \"iopath\": \"output 0 0\",\n",
      "    \"baseline\": {\n",
      "        \"Fixed\": {\n",
      "            \"pipeline\": \"mypipeline\",\n",
      "            \"model\": \"mymodel\",\n",
      "            \"start_at\": \"2022-01-01T00:00:00+00:00\",\n",
      "            \"end_at\": \"2022-01-02T00:00:00+00:00\"\n",
      "        }\n",
      "    },\n",
      "    \"window\": {\n",
      "        \"pipeline\": \"mypipeline\",\n",
      "        \"model\": \"mymodel\",\n",
      "        \"width\": \"24 hours\",\n",
      "        \"start\": null,\n",
      "        \"interval\": null\n",
      "    },\n",
      "    \"summarizer\": {\n",
      "        \"type\": \"UnivariateContinuous\",\n",
      "        \"bin_mode\": \"Quantile\",\n",
      "        \"aggregation\": \"Density\",\n",
      "        \"metric\": \"PSI\",\n",
      "        \"num_bins\": 5,\n",
      "        \"bin_weights\": null,\n",
      "        \"bin_width\": null,\n",
      "        \"provided_edges\": null,\n",
      "        \"add_outlier_edges\": true\n",
      "    },\n",
      "    \"warning_threshold\": null,\n",
      "    \"alert_threshold\": 0.25,\n",
      "    \"next_run\": \"2022-01-02T00:00:00+00:00\",\n",
      "    \"run_until\": \"2022-02-01T00:00:00+00:00\",\n",
      "    \"model_insights_url\": \"http://model-insights:5150\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "assay_builder = client.build_assay(\"Test Assay\", pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)\n",
    "print(assay_builder.build().to_json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defaults\n",
    "\n",
    "We can run the assay interactively and review the first analysis. The method `compare_basic_stats` gives us a dataframe with basic stats for the baseline and window data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Baseline</th>\n",
       "      <th>Window</th>\n",
       "      <th>diff</th>\n",
       "      <th>pct_diff</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1,813.00</td>\n",
       "      <td>1,812.00</td>\n",
       "      <td>-1.00</td>\n",
       "      <td>-0.06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>11.88</td>\n",
       "      <td>11.85</td>\n",
       "      <td>-0.03</td>\n",
       "      <td>-0.23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>15.02</td>\n",
       "      <td>15.28</td>\n",
       "      <td>0.26</td>\n",
       "      <td>1.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>13.02</td>\n",
       "      <td>13.02</td>\n",
       "      <td>-0.00</td>\n",
       "      <td>-0.04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>median</th>\n",
       "      <td>13.00</td>\n",
       "      <td>13.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.46</td>\n",
       "      <td>0.45</td>\n",
       "      <td>-0.00</td>\n",
       "      <td>-0.97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>start</th>\n",
       "      <td>2022-01-01T00:00:00Z</td>\n",
       "      <td>2022-01-02T00:00:00Z</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>end</th>\n",
       "      <td>2022-01-02T00:00:00Z</td>\n",
       "      <td>2022-01-03T00:00:00Z</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    Baseline                Window  diff  pct_diff\n",
       "count               1,813.00              1,812.00 -1.00     -0.06\n",
       "min                    11.88                 11.85 -0.03     -0.23\n",
       "max                    15.02                 15.28  0.26      1.75\n",
       "mean                   13.02                 13.02 -0.00     -0.04\n",
       "median                 13.00                 13.00  0.00      0.00\n",
       "std                     0.46                  0.45 -0.00     -0.97\n",
       "start   2022-01-01T00:00:00Z  2022-01-02T00:00:00Z   NaN       NaN\n",
       "end     2022-01-02T00:00:00Z  2022-01-03T00:00:00Z   NaN       NaN"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "assay_results = assay_builder.build().interactive_run()\n",
    "ar = assay_results[0]\n",
    "\n",
    "ar.compare_basic_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The method `compare_bins` gives us a dataframe with the bin information. Such as the number of bins, the right edges, suggested bin/edge names and the values for each bin in the baseline and the window."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b_edges</th>\n",
       "      <th>b_edge_names</th>\n",
       "      <th>b_aggregated_values</th>\n",
       "      <th>b_aggregation</th>\n",
       "      <th>w_edges</th>\n",
       "      <th>w_edge_names</th>\n",
       "      <th>w_aggregated_values</th>\n",
       "      <th>w_aggregation</th>\n",
       "      <th>diff_in_pcts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11.88</td>\n",
       "      <td>left_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>11.88</td>\n",
       "      <td>left_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12.63</td>\n",
       "      <td>q_20</td>\n",
       "      <td>0.20</td>\n",
       "      <td>Density</td>\n",
       "      <td>12.63</td>\n",
       "      <td>e_1.26e1</td>\n",
       "      <td>0.21</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>12.90</td>\n",
       "      <td>q_40</td>\n",
       "      <td>0.20</td>\n",
       "      <td>Density</td>\n",
       "      <td>12.90</td>\n",
       "      <td>e_1.29e1</td>\n",
       "      <td>0.21</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>13.11</td>\n",
       "      <td>q_60</td>\n",
       "      <td>0.20</td>\n",
       "      <td>Density</td>\n",
       "      <td>13.11</td>\n",
       "      <td>e_1.31e1</td>\n",
       "      <td>0.18</td>\n",
       "      <td>Density</td>\n",
       "      <td>-0.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>13.36</td>\n",
       "      <td>q_80</td>\n",
       "      <td>0.20</td>\n",
       "      <td>Density</td>\n",
       "      <td>13.36</td>\n",
       "      <td>e_1.34e1</td>\n",
       "      <td>0.20</td>\n",
       "      <td>Density</td>\n",
       "      <td>-0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>15.02</td>\n",
       "      <td>q_100</td>\n",
       "      <td>0.20</td>\n",
       "      <td>Density</td>\n",
       "      <td>15.02</td>\n",
       "      <td>e_1.50e1</td>\n",
       "      <td>0.21</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>179,769,313,486,231,570,814,527,423,731,704,356...</td>\n",
       "      <td>right_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>179,769,313,486,231,570,814,527,423,731,704,356...</td>\n",
       "      <td>right_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             b_edges   b_edge_names  \\\n",
       "0                                              11.88   left_outlier   \n",
       "1                                              12.63           q_20   \n",
       "2                                              12.90           q_40   \n",
       "3                                              13.11           q_60   \n",
       "4                                              13.36           q_80   \n",
       "5                                              15.02          q_100   \n",
       "6 179,769,313,486,231,570,814,527,423,731,704,356...  right_outlier   \n",
       "\n",
       "   b_aggregated_values b_aggregation  \\\n",
       "0                 0.00       Density   \n",
       "1                 0.20       Density   \n",
       "2                 0.20       Density   \n",
       "3                 0.20       Density   \n",
       "4                 0.20       Density   \n",
       "5                 0.20       Density   \n",
       "6                 0.00       Density   \n",
       "\n",
       "                                             w_edges   w_edge_names  \\\n",
       "0                                              11.88   left_outlier   \n",
       "1                                              12.63       e_1.26e1   \n",
       "2                                              12.90       e_1.29e1   \n",
       "3                                              13.11       e_1.31e1   \n",
       "4                                              13.36       e_1.34e1   \n",
       "5                                              15.02       e_1.50e1   \n",
       "6 179,769,313,486,231,570,814,527,423,731,704,356...  right_outlier   \n",
       "\n",
       "   w_aggregated_values w_aggregation  diff_in_pcts  \n",
       "0                 0.00       Density          0.00  \n",
       "1                 0.21       Density          0.01  \n",
       "2                 0.21       Density          0.01  \n",
       "3                 0.18       Density         -0.02  \n",
       "4                 0.20       Density         -0.00  \n",
       "5                 0.21       Density          0.01  \n",
       "6                 0.00       Density          0.00  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ar.compare_bins()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also plot the chart to visualize the values of the bins."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baseline mean = 13.021036998645679\n",
      "window mean = 13.01618284105465\n",
      "baseline median = 13.004066467285156\n",
      "window median = 13.004111289978027\n",
      "bin_mode = Quantile\n",
      "aggregation = Density\n",
      "metric = PSI\n",
      "weighted = False\n",
      "score = 0.019674378474700036\n",
      "scores = [0.00980308228898587, 0.0002603476402290938, 0.00022327892034488384, 0.0031099778889999417, 9.781216093661028e-07, 0.0003905674264324862, 0.005886146188098397]\n",
      "index = None\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 53056 characters]\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ar.chart()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Binning Mode\n",
    "\n",
    "We can change the bin mode algorithm to equal and see that the bins/edges are partitioned at different points and the bins have different values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b_edges</th>\n",
       "      <th>b_edge_names</th>\n",
       "      <th>b_aggregated_values</th>\n",
       "      <th>b_aggregation</th>\n",
       "      <th>w_edges</th>\n",
       "      <th>w_edge_names</th>\n",
       "      <th>w_aggregated_values</th>\n",
       "      <th>w_aggregation</th>\n",
       "      <th>diff_in_pcts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11.88</td>\n",
       "      <td>left_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>11.88</td>\n",
       "      <td>left_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12.51</td>\n",
       "      <td>p_1.25e1</td>\n",
       "      <td>0.12</td>\n",
       "      <td>Density</td>\n",
       "      <td>12.51</td>\n",
       "      <td>e_1.25e1</td>\n",
       "      <td>0.13</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13.13</td>\n",
       "      <td>p_1.31e1</td>\n",
       "      <td>0.50</td>\n",
       "      <td>Density</td>\n",
       "      <td>13.13</td>\n",
       "      <td>e_1.31e1</td>\n",
       "      <td>0.49</td>\n",
       "      <td>Density</td>\n",
       "      <td>-0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>13.76</td>\n",
       "      <td>p_1.38e1</td>\n",
       "      <td>0.31</td>\n",
       "      <td>Density</td>\n",
       "      <td>13.76</td>\n",
       "      <td>e_1.38e1</td>\n",
       "      <td>0.33</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>14.39</td>\n",
       "      <td>p_1.44e1</td>\n",
       "      <td>0.06</td>\n",
       "      <td>Density</td>\n",
       "      <td>14.39</td>\n",
       "      <td>e_1.44e1</td>\n",
       "      <td>0.05</td>\n",
       "      <td>Density</td>\n",
       "      <td>-0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>15.02</td>\n",
       "      <td>p_1.50e1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>Density</td>\n",
       "      <td>15.02</td>\n",
       "      <td>e_1.50e1</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>-0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>179,769,313,486,231,570,814,527,423,731,704,356...</td>\n",
       "      <td>right_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>179,769,313,486,231,570,814,527,423,731,704,356...</td>\n",
       "      <td>right_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             b_edges   b_edge_names  \\\n",
       "0                                              11.88   left_outlier   \n",
       "1                                              12.51       p_1.25e1   \n",
       "2                                              13.13       p_1.31e1   \n",
       "3                                              13.76       p_1.38e1   \n",
       "4                                              14.39       p_1.44e1   \n",
       "5                                              15.02       p_1.50e1   \n",
       "6 179,769,313,486,231,570,814,527,423,731,704,356...  right_outlier   \n",
       "\n",
       "   b_aggregated_values b_aggregation  \\\n",
       "0                 0.00       Density   \n",
       "1                 0.12       Density   \n",
       "2                 0.50       Density   \n",
       "3                 0.31       Density   \n",
       "4                 0.06       Density   \n",
       "5                 0.01       Density   \n",
       "6                 0.00       Density   \n",
       "\n",
       "                                             w_edges   w_edge_names  \\\n",
       "0                                              11.88   left_outlier   \n",
       "1                                              12.51       e_1.25e1   \n",
       "2                                              13.13       e_1.31e1   \n",
       "3                                              13.76       e_1.38e1   \n",
       "4                                              14.39       e_1.44e1   \n",
       "5                                              15.02       e_1.50e1   \n",
       "6 179,769,313,486,231,570,814,527,423,731,704,356...  right_outlier   \n",
       "\n",
       "   w_aggregated_values w_aggregation  diff_in_pcts  \n",
       "0                 0.00       Density          0.00  \n",
       "1                 0.13       Density          0.00  \n",
       "2                 0.49       Density         -0.01  \n",
       "3                 0.33       Density          0.02  \n",
       "4                 0.05       Density         -0.01  \n",
       "5                 0.00       Density         -0.00  \n",
       "6                 0.00       Density          0.00  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "None"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baseline mean = 13.021036998645679\n",
      "window mean = 13.01618284105465\n",
      "baseline median = 13.004066467285156\n",
      "window median = 13.004111289978027\n",
      "bin_mode = Equal\n",
      "aggregation = Density\n",
      "metric = PSI\n",
      "weighted = False\n",
      "score = 0.02174688998992665\n",
      "scores = [0.00980308228898587, 0.00012262223368267688, 0.00014769845960773225, 0.000993907207070759, 0.0023700328024471758, 0.002423400810034042, 0.005886146188098397]\n",
      "index = None\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 49236 characters]\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "assay_builder = client.build_assay(\"Test Assay\", pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)\n",
    "assay_builder.summarizer_builder.add_bin_mode(BinMode.EQUAL)\n",
    "assay_results = assay_builder.build().interactive_run()\n",
    "display(display(assay_results[0].compare_bins()))\n",
    "assay_results[0].chart()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## User Provided Bin Edges\n",
    "\n",
    "The values in this dataset run from ~11.6 to ~15.81. And lets say we had a business reason to use specific bin edges.  We can specify them with the BinMode.PROVIDED and specifying a list of floats with the right hand / upper edge of each bin and optionally the lower edge of the smallest bin. If the lowest edge is not specified the threshold for left outliers is taken from the smallest value in the baseline dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b_edges</th>\n",
       "      <th>b_edge_names</th>\n",
       "      <th>b_aggregated_values</th>\n",
       "      <th>b_aggregation</th>\n",
       "      <th>w_edges</th>\n",
       "      <th>w_edge_names</th>\n",
       "      <th>w_aggregated_values</th>\n",
       "      <th>w_aggregation</th>\n",
       "      <th>diff_in_pcts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11.00</td>\n",
       "      <td>left_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>11.00</td>\n",
       "      <td>left_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12.00</td>\n",
       "      <td>e_1.20e1</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>12.00</td>\n",
       "      <td>e_1.20e1</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13.00</td>\n",
       "      <td>e_1.30e1</td>\n",
       "      <td>0.49</td>\n",
       "      <td>Density</td>\n",
       "      <td>13.00</td>\n",
       "      <td>e_1.30e1</td>\n",
       "      <td>0.49</td>\n",
       "      <td>Density</td>\n",
       "      <td>-0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>14.00</td>\n",
       "      <td>e_1.40e1</td>\n",
       "      <td>0.48</td>\n",
       "      <td>Density</td>\n",
       "      <td>14.00</td>\n",
       "      <td>e_1.40e1</td>\n",
       "      <td>0.49</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>15.00</td>\n",
       "      <td>e_1.50e1</td>\n",
       "      <td>0.03</td>\n",
       "      <td>Density</td>\n",
       "      <td>15.00</td>\n",
       "      <td>e_1.50e1</td>\n",
       "      <td>0.02</td>\n",
       "      <td>Density</td>\n",
       "      <td>-0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>16.00</td>\n",
       "      <td>e_1.60e1</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>16.00</td>\n",
       "      <td>e_1.60e1</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>179,769,313,486,231,570,814,527,423,731,704,356...</td>\n",
       "      <td>right_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>179,769,313,486,231,570,814,527,423,731,704,356...</td>\n",
       "      <td>right_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             b_edges   b_edge_names  \\\n",
       "0                                              11.00   left_outlier   \n",
       "1                                              12.00       e_1.20e1   \n",
       "2                                              13.00       e_1.30e1   \n",
       "3                                              14.00       e_1.40e1   \n",
       "4                                              15.00       e_1.50e1   \n",
       "5                                              16.00       e_1.60e1   \n",
       "6 179,769,313,486,231,570,814,527,423,731,704,356...  right_outlier   \n",
       "\n",
       "   b_aggregated_values b_aggregation  \\\n",
       "0                 0.00       Density   \n",
       "1                 0.00       Density   \n",
       "2                 0.49       Density   \n",
       "3                 0.48       Density   \n",
       "4                 0.03       Density   \n",
       "5                 0.00       Density   \n",
       "6                 0.00       Density   \n",
       "\n",
       "                                             w_edges   w_edge_names  \\\n",
       "0                                              11.00   left_outlier   \n",
       "1                                              12.00       e_1.20e1   \n",
       "2                                              13.00       e_1.30e1   \n",
       "3                                              14.00       e_1.40e1   \n",
       "4                                              15.00       e_1.50e1   \n",
       "5                                              16.00       e_1.60e1   \n",
       "6 179,769,313,486,231,570,814,527,423,731,704,356...  right_outlier   \n",
       "\n",
       "   w_aggregated_values w_aggregation  diff_in_pcts  \n",
       "0                 0.00       Density          0.00  \n",
       "1                 0.00       Density          0.00  \n",
       "2                 0.49       Density         -0.00  \n",
       "3                 0.49       Density          0.01  \n",
       "4                 0.02       Density         -0.01  \n",
       "5                 0.00       Density          0.00  \n",
       "6                 0.00       Density          0.00  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "None"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baseline mean = 13.021036998645679\n",
      "window mean = 13.01618284105465\n",
      "baseline median = 13.004066467285156\n",
      "window median = 13.004111289978027\n",
      "bin_mode = Provided\n",
      "aggregation = Density\n",
      "metric = PSI\n",
      "weighted = False\n",
      "score = 0.0052011866476329915\n",
      "scores = [0.0, 0.0024270797943831413, 1.8763984174233916e-05, 0.00011480770647089651, 0.002257487960844032, 0.00038304720176068804, 0.0]\n",
      "index = None\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 46532 characters]\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "edges = [11.0, 12.0, 13.0, 14.0, 15.0, 16.0]\n",
    "assay_builder = client.build_assay(\"Test Assay\", pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)\n",
    "assay_builder.summarizer_builder.add_bin_mode(BinMode.PROVIDED, edges)\n",
    "assay_results = assay_builder.build().interactive_run()\n",
    "display(display(assay_results[0].compare_bins()))\n",
    "assay_results[0].chart()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Number of Bins\n",
    "\n",
    "We could also choose to a different number of bins, lets say 10, which can be evenly spaced or based on the quantiles (deciles)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b_edges</th>\n",
       "      <th>b_edge_names</th>\n",
       "      <th>b_aggregated_values</th>\n",
       "      <th>b_aggregation</th>\n",
       "      <th>w_edges</th>\n",
       "      <th>w_edge_names</th>\n",
       "      <th>w_aggregated_values</th>\n",
       "      <th>w_aggregation</th>\n",
       "      <th>diff_in_pcts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11.88</td>\n",
       "      <td>left_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>11.88</td>\n",
       "      <td>left_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12.46</td>\n",
       "      <td>q_10</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>12.46</td>\n",
       "      <td>e_1.25e1</td>\n",
       "      <td>0.11</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>12.63</td>\n",
       "      <td>q_20</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>12.63</td>\n",
       "      <td>e_1.26e1</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>-0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>12.78</td>\n",
       "      <td>q_30</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>12.78</td>\n",
       "      <td>e_1.28e1</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>12.90</td>\n",
       "      <td>q_40</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>12.90</td>\n",
       "      <td>e_1.29e1</td>\n",
       "      <td>0.11</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>13.00</td>\n",
       "      <td>q_50</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>13.00</td>\n",
       "      <td>e_1.30e1</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>13.11</td>\n",
       "      <td>q_60</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>13.11</td>\n",
       "      <td>e_1.31e1</td>\n",
       "      <td>0.09</td>\n",
       "      <td>Density</td>\n",
       "      <td>-0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>13.22</td>\n",
       "      <td>q_70</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>13.22</td>\n",
       "      <td>e_1.32e1</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>-0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>13.36</td>\n",
       "      <td>q_80</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>13.36</td>\n",
       "      <td>e_1.34e1</td>\n",
       "      <td>0.09</td>\n",
       "      <td>Density</td>\n",
       "      <td>-0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>13.60</td>\n",
       "      <td>q_90</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>13.60</td>\n",
       "      <td>e_1.36e1</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>15.02</td>\n",
       "      <td>q_100</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>15.02</td>\n",
       "      <td>e_1.50e1</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>-0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>179,769,313,486,231,570,814,527,423,731,704,356...</td>\n",
       "      <td>right_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>179,769,313,486,231,570,814,527,423,731,704,356...</td>\n",
       "      <td>right_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              b_edges   b_edge_names  \\\n",
       "0                                               11.88   left_outlier   \n",
       "1                                               12.46           q_10   \n",
       "2                                               12.63           q_20   \n",
       "3                                               12.78           q_30   \n",
       "4                                               12.90           q_40   \n",
       "5                                               13.00           q_50   \n",
       "6                                               13.11           q_60   \n",
       "7                                               13.22           q_70   \n",
       "8                                               13.36           q_80   \n",
       "9                                               13.60           q_90   \n",
       "10                                              15.02          q_100   \n",
       "11 179,769,313,486,231,570,814,527,423,731,704,356...  right_outlier   \n",
       "\n",
       "    b_aggregated_values b_aggregation  \\\n",
       "0                  0.00       Density   \n",
       "1                  0.10       Density   \n",
       "2                  0.10       Density   \n",
       "3                  0.10       Density   \n",
       "4                  0.10       Density   \n",
       "5                  0.10       Density   \n",
       "6                  0.10       Density   \n",
       "7                  0.10       Density   \n",
       "8                  0.10       Density   \n",
       "9                  0.10       Density   \n",
       "10                 0.10       Density   \n",
       "11                 0.00       Density   \n",
       "\n",
       "                                              w_edges   w_edge_names  \\\n",
       "0                                               11.88   left_outlier   \n",
       "1                                               12.46       e_1.25e1   \n",
       "2                                               12.63       e_1.26e1   \n",
       "3                                               12.78       e_1.28e1   \n",
       "4                                               12.90       e_1.29e1   \n",
       "5                                               13.00       e_1.30e1   \n",
       "6                                               13.11       e_1.31e1   \n",
       "7                                               13.22       e_1.32e1   \n",
       "8                                               13.36       e_1.34e1   \n",
       "9                                               13.60       e_1.36e1   \n",
       "10                                              15.02       e_1.50e1   \n",
       "11 179,769,313,486,231,570,814,527,423,731,704,356...  right_outlier   \n",
       "\n",
       "    w_aggregated_values w_aggregation  diff_in_pcts  \n",
       "0                  0.00       Density          0.00  \n",
       "1                  0.11       Density          0.01  \n",
       "2                  0.10       Density         -0.00  \n",
       "3                  0.10       Density          0.00  \n",
       "4                  0.11       Density          0.01  \n",
       "5                  0.10       Density          0.00  \n",
       "6                  0.09       Density         -0.01  \n",
       "7                  0.10       Density         -0.00  \n",
       "8                  0.09       Density         -0.00  \n",
       "9                  0.10       Density          0.00  \n",
       "10                 0.10       Density         -0.00  \n",
       "11                 0.00       Density          0.00  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "None"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baseline mean = 13.021036998645679\n",
      "window mean = 13.010907055789534\n",
      "baseline median = 13.004066467285156\n",
      "window median = 12.993888854980469\n",
      "bin_mode = Quantile\n",
      "aggregation = Density\n",
      "metric = PSI\n",
      "weighted = False\n",
      "score = 0.008209060319467209\n",
      "scores = [0.0, 0.0006680706132507951, 0.00019356523211058643, 1.3374098424755057e-05, 0.0004323412561987516, 3.6626035060501716e-06, 0.0007068329210884422, 2.478165590023963e-06, 0.000247803939212629, 2.8917879748960253e-05, 2.586742223781745e-05, 0.005886146188098397]\n",
      "index = None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jovyan/wallaroo-plus/sdk/wallaroo/assay.py:317: UserWarning: FixedFormatter should only be used together with FixedLocator\n",
      "  ax.set_xticklabels(labels=edge_names, rotation=45)\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 65228 characters]\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "assay_builder = client.build_assay(\"Test Assay\", pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)\n",
    "assay_builder.summarizer_builder.add_bin_mode(BinMode.QUANTILE).add_num_bins(10)\n",
    "assay_results = assay_builder.build().interactive_run()\n",
    "display(display(assay_results[1].compare_bins()))\n",
    "assay_results[1].chart()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bin Weights\n",
    "\n",
    "Now lets say we only care about differences at the higher end of the range. We can use weights to specify that difference in the lower bins should not be counted in the score. \n",
    "\n",
    "If we stick with 10 bins we can provide 10 a vector of 12 weights. One weight each for the original bins plus one at the front for the left outlier bin and one at the end for the right outlier bin.\n",
    "\n",
    "Note we still show the values for the bins but the scores for the lower 5 and left outlier are 0 and only the right half is counted and reflected in the score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using weights:  [0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b_edges</th>\n",
       "      <th>b_edge_names</th>\n",
       "      <th>b_aggregated_values</th>\n",
       "      <th>b_aggregation</th>\n",
       "      <th>w_edges</th>\n",
       "      <th>w_edge_names</th>\n",
       "      <th>w_aggregated_values</th>\n",
       "      <th>w_aggregation</th>\n",
       "      <th>diff_in_pcts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11.88</td>\n",
       "      <td>left_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>11.88</td>\n",
       "      <td>left_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12.46</td>\n",
       "      <td>q_10</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>12.46</td>\n",
       "      <td>e_1.25e1</td>\n",
       "      <td>0.11</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>12.63</td>\n",
       "      <td>q_20</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>12.63</td>\n",
       "      <td>e_1.26e1</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>-0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>12.78</td>\n",
       "      <td>q_30</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>12.78</td>\n",
       "      <td>e_1.28e1</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>12.90</td>\n",
       "      <td>q_40</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>12.90</td>\n",
       "      <td>e_1.29e1</td>\n",
       "      <td>0.11</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>13.00</td>\n",
       "      <td>q_50</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>13.00</td>\n",
       "      <td>e_1.30e1</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>13.11</td>\n",
       "      <td>q_60</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>13.11</td>\n",
       "      <td>e_1.31e1</td>\n",
       "      <td>0.09</td>\n",
       "      <td>Density</td>\n",
       "      <td>-0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>13.22</td>\n",
       "      <td>q_70</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>13.22</td>\n",
       "      <td>e_1.32e1</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>-0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>13.36</td>\n",
       "      <td>q_80</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>13.36</td>\n",
       "      <td>e_1.34e1</td>\n",
       "      <td>0.09</td>\n",
       "      <td>Density</td>\n",
       "      <td>-0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>13.60</td>\n",
       "      <td>q_90</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>13.60</td>\n",
       "      <td>e_1.36e1</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>15.02</td>\n",
       "      <td>q_100</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>15.02</td>\n",
       "      <td>e_1.50e1</td>\n",
       "      <td>0.10</td>\n",
       "      <td>Density</td>\n",
       "      <td>-0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>179,769,313,486,231,570,814,527,423,731,704,356...</td>\n",
       "      <td>right_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>179,769,313,486,231,570,814,527,423,731,704,356...</td>\n",
       "      <td>right_outlier</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Density</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              b_edges   b_edge_names  \\\n",
       "0                                               11.88   left_outlier   \n",
       "1                                               12.46           q_10   \n",
       "2                                               12.63           q_20   \n",
       "3                                               12.78           q_30   \n",
       "4                                               12.90           q_40   \n",
       "5                                               13.00           q_50   \n",
       "6                                               13.11           q_60   \n",
       "7                                               13.22           q_70   \n",
       "8                                               13.36           q_80   \n",
       "9                                               13.60           q_90   \n",
       "10                                              15.02          q_100   \n",
       "11 179,769,313,486,231,570,814,527,423,731,704,356...  right_outlier   \n",
       "\n",
       "    b_aggregated_values b_aggregation  \\\n",
       "0                  0.00       Density   \n",
       "1                  0.10       Density   \n",
       "2                  0.10       Density   \n",
       "3                  0.10       Density   \n",
       "4                  0.10       Density   \n",
       "5                  0.10       Density   \n",
       "6                  0.10       Density   \n",
       "7                  0.10       Density   \n",
       "8                  0.10       Density   \n",
       "9                  0.10       Density   \n",
       "10                 0.10       Density   \n",
       "11                 0.00       Density   \n",
       "\n",
       "                                              w_edges   w_edge_names  \\\n",
       "0                                               11.88   left_outlier   \n",
       "1                                               12.46       e_1.25e1   \n",
       "2                                               12.63       e_1.26e1   \n",
       "3                                               12.78       e_1.28e1   \n",
       "4                                               12.90       e_1.29e1   \n",
       "5                                               13.00       e_1.30e1   \n",
       "6                                               13.11       e_1.31e1   \n",
       "7                                               13.22       e_1.32e1   \n",
       "8                                               13.36       e_1.34e1   \n",
       "9                                               13.60       e_1.36e1   \n",
       "10                                              15.02       e_1.50e1   \n",
       "11 179,769,313,486,231,570,814,527,423,731,704,356...  right_outlier   \n",
       "\n",
       "    w_aggregated_values w_aggregation  diff_in_pcts  \n",
       "0                  0.00       Density          0.00  \n",
       "1                  0.11       Density          0.01  \n",
       "2                  0.10       Density         -0.00  \n",
       "3                  0.10       Density          0.00  \n",
       "4                  0.11       Density          0.01  \n",
       "5                  0.10       Density          0.00  \n",
       "6                  0.09       Density         -0.01  \n",
       "7                  0.10       Density         -0.00  \n",
       "8                  0.09       Density         -0.00  \n",
       "9                  0.10       Density          0.00  \n",
       "10                 0.10       Density         -0.00  \n",
       "11                 0.00       Density          0.00  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "None"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baseline mean = 13.021036998645679\n",
      "window mean = 13.010907055789534\n",
      "baseline median = 13.004066467285156\n",
      "window median = 12.993888854980469\n",
      "bin_mode = Quantile\n",
      "aggregation = Density\n",
      "metric = PSI\n",
      "weighted = True\n",
      "score = 0.0011496744193293783\n",
      "scores = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0001178054868480737, 4.130275983373272e-07, 4.1300656535438165e-05, 4.8196466248267085e-06, 4.311237039636242e-06, 0.0009810243646830661]\n",
      "index = None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jovyan/wallaroo-plus/sdk/wallaroo/assay.py:317: UserWarning: FixedFormatter should only be used together with FixedLocator\n",
      "  ax.set_xticklabels(labels=edge_names, rotation=45)\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 63408 characters]\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "weights = [0] * 6\n",
    "weights.extend([1] * 6)\n",
    "print(\"Using weights: \", weights)\n",
    "assay_builder = client.build_assay(\"Test Assay\", pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)\n",
    "assay_builder.summarizer_builder.add_bin_mode(BinMode.QUANTILE).add_num_bins(10).add_bin_weights(weights)\n",
    "assay_results = assay_builder.build().interactive_run()\n",
    "display(display(assay_results[1].compare_bins()))\n",
    "assay_results[1].chart()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Metrics\n",
    "\n",
    "The `score` is a distance or dis-similarity measure. The larger it is the less similar the two distributions are. We currently support\n",
    "summing the differences of each individual bin, taking the maximum difference and a modified Population Stability Index (PSI).\n",
    "\n",
    "The following three charts use each of the metrics. Note how the scores change. The best one will depend on your particular use case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baseline mean = 13.021036998645679\n",
      "window mean = 13.01618284105465\n",
      "baseline median = 13.004066467285156\n",
      "window median = 13.004111289978027\n",
      "bin_mode = Quantile\n",
      "aggregation = Density\n",
      "metric = PSI\n",
      "weighted = False\n",
      "score = 0.019674378474700036\n",
      "scores = [0.00980308228898587, 0.0002603476402290938, 0.00022327892034488384, 0.0031099778889999417, 9.781216093661028e-07, 0.0003905674264324862, 0.005886146188098397]\n",
      "index = None\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 53056 characters]\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "assay_builder = client.build_assay(\"Test Assay\", pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)\n",
    "assay_results = assay_builder.build().interactive_run()\n",
    "assay_results[0].chart()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baseline mean = 13.021036998645679\n",
      "window mean = 13.01618284105465\n",
      "baseline median = 13.004066467285156\n",
      "window median = 13.004111289978027\n",
      "bin_mode = Quantile\n",
      "aggregation = Density\n",
      "metric = SumDiff\n",
      "weighted = False\n",
      "score = 0.024613747414125847\n",
      "scores = [0.0005518763796909492, 0.007284889971739539, 0.006732709192501046, 0.02417206367064456, 0.00044168374348130257, 0.008940519110812395, 0.0011037527593818985]\n",
      "index = None\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 54452 characters]\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "assay_builder = client.build_assay(\"Test Assay\", pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)\n",
    "assay_builder.summarizer_builder.add_metric(Metric.SUMDIFF)\n",
    "assay_results = assay_builder.build().interactive_run()\n",
    "assay_results[0].chart()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baseline mean = 13.021036998645679\n",
      "window mean = 13.01618284105465\n",
      "baseline median = 13.004066467285156\n",
      "window median = 13.004111289978027\n",
      "bin_mode = Quantile\n",
      "aggregation = Density\n",
      "metric = MaxDiff\n",
      "weighted = False\n",
      "score = 0.02417206367064456\n",
      "scores = [0.0005518763796909492, 0.007284889971739539, 0.006732709192501046, 0.02417206367064456, 0.00044168374348130257, 0.008940519110812395, 0.0011037527593818985]\n",
      "index = 3\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 54416 characters]\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "assay_builder = client.build_assay(\"Test Assay\", pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)\n",
    "assay_builder.summarizer_builder.add_metric(Metric.MAXDIFF)\n",
    "assay_results = assay_builder.build().interactive_run()\n",
    "assay_results[0].chart()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aggregation Options\n",
    "\n",
    "Also, bin aggregation can be done in histogram `Aggregation.DENSITY` style (the default) where we count the number/percentage of values that fall in each bin or Empirical Cumulative Density Function style `Aggregation.CUMULATIVE` where we keep a cumulative count of the values/percentages that fall in each bin."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baseline mean = 13.021036998645679\n",
      "window mean = 13.01618284105465\n",
      "baseline median = 13.004066467285156\n",
      "window median = 13.004111289978027\n",
      "bin_mode = Quantile\n",
      "aggregation = Density\n",
      "metric = PSI\n",
      "weighted = False\n",
      "score = 0.019674378474700036\n",
      "scores = [0.00980308228898587, 0.0002603476402290938, 0.00022327892034488384, 0.0031099778889999417, 9.781216093661028e-07, 0.0003905674264324862, 0.005886146188098397]\n",
      "index = None\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 53056 characters]\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "assay_builder = client.build_assay(\"Test Assay\", pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)\n",
    "assay_builder.summarizer_builder.add_aggregation(Aggregation.DENSITY)\n",
    "assay_results = assay_builder.build().interactive_run()\n",
    "assay_results[0].chart()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baseline mean = 13.021036998645679\n",
      "window mean = 13.01618284105465\n",
      "baseline median = 13.004066467285156\n",
      "window median = 13.004111289978027\n",
      "bin_mode = Quantile\n",
      "aggregation = Cumulative\n",
      "metric = PSI\n",
      "weighted = False\n",
      "score = 0.04370873103134221\n",
      "scores = [0.0005518763796909492, 0.00783676635143049, 0.014569475543931565, 0.009602588126712996, 0.010044271870194299, 0.001103752759381904, 0.0]\n",
      "index = None\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 49456 characters]\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "assay_builder = client.build_assay(\"Test Assay\", pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)\n",
    "assay_builder.summarizer_builder.add_aggregation(Aggregation.CUMULATIVE)\n",
    "assay_results = assay_builder.build().interactive_run()\n",
    "assay_results[0].chart()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
