{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "title",
   "metadata": {},
   "source": [
    "# abaudit — End-to-End A/B Test Validity Demo\n",
    "\n",
    "[![PyPI](https://img.shields.io/pypi/v/abaudit)](https://pypi.org/project/abaudit/)\n",
    "[![Tests](https://github.com/aldair-ai/abaudit/actions/workflows/tests.yml/badge.svg)](https://github.com/aldair-ai/abaudit/actions)\n",
    "\n",
    "> **The problem:** Your A/B test returned p = 0.031. The team wants to ship. But is the result actually real?\n",
    ">\n",
    "> **The answer:** A significant p-value tells you P(data | no effect). What you actually want is P(effect | significant result) — the **Positive Predictive Value (PPV)**. These are not the same thing, and the difference can be enormous.\n",
    "\n",
    "This notebook walks through a complete, realistic A/B test scenario at a fictional e-commerce company — from experiment design to post-experiment audit — using `abaudit`.\n",
    "\n",
    "---\n",
    "\n",
    "**Scenario:** *ShopFast* is testing a redesigned checkout button (treatment) against the current one (control). The primary metric is conversion rate. The data science team tested 3 metrics, peeked at results on Day 4, and is now reporting a significant result.\n",
    "\n",
    "**Contents:**\n",
    "1. [Setup](#1)\n",
    "2. [Pre-Experiment: Was This Worth Running?](#2)\n",
    "3. [PPV vs. Prior — Why Prior Belief Matters](#3)\n",
    "4. [Simulate the Experiment](#4)\n",
    "5. [During-Experiment: Health Checks](#5)\n",
    "6. [Post-Experiment: The Full Audit](#6)\n",
    "7. [The HTML Report](#7)\n",
    "8. [What If the Prior Was Lower?](#8)\n",
    "9. [Key Takeaways](#9)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sec1-md",
   "metadata": {},
   "source": [
    "<a id='1'></a>\n",
    "## 1. Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "setup",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "abaudit version: 0.1.0\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as mticker\n",
    "import abaudit as ab\n",
    "from abaudit.design import ppv_given_design\n",
    "\n",
    "plt.rcParams.update({\n",
    "    'figure.dpi': 120,\n",
    "    'font.family': 'DejaVu Sans',\n",
    "    'axes.spines.top': False,\n",
    "    'axes.spines.right': False,\n",
    "    'axes.titlesize': 12,\n",
    "    'axes.labelsize': 10,\n",
    "})\n",
    "\n",
    "BLUE   = '#2563EB'\n",
    "RED    = '#DC2626'\n",
    "GREEN  = '#16A34A'\n",
    "ORANGE = '#D97706'\n",
    "GRAY   = '#6B7280'\n",
    "\n",
    "import os\n",
    "os.makedirs('figures', exist_ok=True)\n",
    "\n",
    "print(f'abaudit version: {ab.__version__}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sec2-md",
   "metadata": {},
   "source": [
    "<a id='2'></a>\n",
    "## 2. Pre-Experiment: Was This Worth Running?\n",
    "\n",
    "Before collecting a single data point, a well-run experiment starts with a **design analysis**.\n",
    "\n",
    "The ShopFast team expects:\n",
    "- A **small effect** — a 1% lift in conversion rate, which translates to Cohen's d ≈ 0.2\n",
    "- A **moderate prior** — the redesigned button has been through UX testing, so they're reasonably confident it helps: f = 0.3\n",
    "\n",
    "Standard power analysis says: *\"you need n = 394 per group.\"*\n",
    "\n",
    "`abaudit` goes further: *\"at that n, if you get a significant result, how likely is it to actually be real?\"*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "pre-experiment",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Standard Power Analysis ===\n",
      "  n per group:    393\n",
      "  n total:        786\n",
      "  Power achieved: 80.1%\n",
      "\n",
      "\n",
      "           abaudit — Pre-Experiment Design Report           \n",
      "============================================================\n",
      "  Effect size (Cohen's d)   : 0.20\n",
      "  Prior f                   : 0.30  (30% chance effect is real)\n",
      "  Significance level (α)    : 0.05\n",
      "  Target power              : 80%\n",
      "  Target PPV                : 80%\n",
      "────────────────────────────────────────────────────────────\n",
      "  n per group (power only)  : 393\n",
      "  n per group (PPV target)  : 177\n",
      "  ──────────────────────────────\n",
      "  ➤  Recommended n / group  : 393  (total: 786)\n",
      "  ➤  Power at recommended n : 80.1%    ✅\n",
      "  ➤  PPV  at recommended n  : 87.3%    ✅\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Experiment parameters\n",
    "EFFECT_SIZE = 0.2      # expected Cohen's d (small but meaningful)\n",
    "PRIOR_F     = 0.30     # 30% prior: UX-tested hypothesis\n",
    "ALPHA       = 0.05\n",
    "N_METRICS   = 3        # conversion, revenue, session_time\n",
    "\n",
    "# Standard power analysis\n",
    "pa = ab.power_analysis(effect_size=EFFECT_SIZE, alpha=ALPHA, power=0.80)\n",
    "print('=== Standard Power Analysis ===')\n",
    "print(f'  n per group:    {pa[\"n_per_group\"]:,}')\n",
    "print(f'  n total:        {pa[\"n_total\"]:,}')\n",
    "print(f'  Power achieved: {pa[\"power_achieved\"]:.1%}')\n",
    "\n",
    "print()\n",
    "\n",
    "# PPV-aware design\n",
    "plan = ab.design_summary(\n",
    "    effect_size  = EFFECT_SIZE,\n",
    "    prior_f      = PRIOR_F,\n",
    "    alpha        = ALPHA,\n",
    "    target_power = 0.80,\n",
    "    target_ppv   = 0.80,\n",
    ")\n",
    "plan.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sec3-md",
   "metadata": {},
   "source": [
    "<a id='3'></a>\n",
    "## 3. PPV vs. Prior — Why Prior Belief Matters\n",
    "\n",
    "The Ioannidis (2005) PPV formula shows that the trustworthiness of a significant result depends critically on **how plausible the hypothesis was before the experiment**:\n",
    "\n",
    "$$\\text{PPV} = \\frac{(1-\\beta) \\cdot f}{(1-\\beta) \\cdot f + \\alpha \\cdot (1-f)}$$\n",
    "\n",
    "The chart below shows PPV as a function of prior $f$ at different power levels — at our planned sample size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ppv-prior-plot",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 141772 characters]",
      "text/plain": [
       "<Figure size 1560x540 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✓ Saved: figures/01_ppv_analysis.png\n"
     ]
    }
   ],
   "source": [
    "priors      = np.linspace(0.01, 0.99, 200)\n",
    "power_levels = [0.40, 0.60, 0.80, 0.95]\n",
    "colors       = [GRAY, ORANGE, BLUE, GREEN]\n",
    "\n",
    "fig, axes = plt.subplots(1, 2, figsize=(13, 4.5))\n",
    "\n",
    "# Left: PPV vs prior at different power levels\n",
    "ax = axes[0]\n",
    "for power, color in zip(power_levels, colors):\n",
    "    ppvs = [ab.ppv_given_design(EFFECT_SIZE,\n",
    "                                 plan.n_recommended or pa['n_per_group'],\n",
    "                                 float(p), ALPHA)\n",
    "            for p in priors]\n",
    "    # Override with direct PPV formula for illustration\n",
    "    from abaudit._stats import ppv as _ppv\n",
    "    ppvs = [_ppv(float(p), power, ALPHA) for p in priors]\n",
    "    ax.plot(priors, ppvs, color=color, lw=2, label=f'Power = {power:.0%}')\n",
    "\n",
    "ax.axvline(PRIOR_F, color=RED, ls='--', lw=1.8,\n",
    "           label=f'Our prior (f = {PRIOR_F})')\n",
    "ax.axhline(0.80, color=GRAY, ls=':', lw=1.2, alpha=0.7,\n",
    "           label='PPV = 80% (trustworthy threshold)')\n",
    "ax.fill_between(priors,\n",
    "                [_ppv(float(p), 0.80, ALPHA) for p in priors],\n",
    "                0.80, alpha=0.06, color=RED,\n",
    "                where=[_ppv(float(p), 0.80, ALPHA) < 0.80 for p in priors])\n",
    "\n",
    "ax.set_xlabel('Prior probability f')\n",
    "ax.set_ylabel('PPV — P(true effect | significant result)')\n",
    "ax.set_title('PPV vs. Prior\\nAt what prior does a significant result become trustworthy?')\n",
    "ax.yaxis.set_major_formatter(mticker.PercentFormatter(xmax=1))\n",
    "ax.xaxis.set_major_formatter(mticker.PercentFormatter(xmax=1))\n",
    "ax.legend(fontsize=8)\n",
    "\n",
    "# Right: PPV vs sample size at our prior\n",
    "ax2  = axes[1]\n",
    "ns   = np.arange(20, 2001, 10)\n",
    "ppvs_n = [ppv_given_design(EFFECT_SIZE, int(n), PRIOR_F, ALPHA) for n in ns]\n",
    "\n",
    "ax2.plot(ns * 2, ppvs_n, color=BLUE, lw=2)\n",
    "ax2.axhline(0.80, color=RED, ls='--', lw=1.8, label='PPV = 80% target')\n",
    "\n",
    "if plan.n_recommended:\n",
    "    ppv_rec = ppv_given_design(EFFECT_SIZE, plan.n_recommended, PRIOR_F, ALPHA)\n",
    "    ax2.axvline(plan.n_recommended * 2, color=GREEN, ls='--', lw=1.8,\n",
    "                label=f'Recommended n = {plan.n_recommended*2:,}')\n",
    "    ax2.scatter([plan.n_recommended * 2], [ppv_rec],\n",
    "                color=GREEN, s=80, zorder=5)\n",
    "\n",
    "ax2.set_xlabel('Total sample size (both groups)')\n",
    "ax2.set_ylabel('PPV at this n')\n",
    "ax2.set_title(f'PPV vs. Sample Size\\n(prior f = {PRIOR_F}, d = {EFFECT_SIZE})')\n",
    "ax2.yaxis.set_major_formatter(mticker.PercentFormatter(xmax=1))\n",
    "ax2.legend(fontsize=9)\n",
    "\n",
    "plt.suptitle(\n",
    "    'Why Sample Size and Prior Both Determine Whether Your Result Can Be Trusted',\n",
    "    fontsize=11, fontweight='bold', y=1.03\n",
    ")\n",
    "plt.tight_layout()\n",
    "plt.savefig('figures/01_ppv_analysis.png', bbox_inches='tight')\n",
    "plt.show()\n",
    "print('✓ Saved: figures/01_ppv_analysis.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sec4-md",
   "metadata": {},
   "source": [
    "<a id='4'></a>\n",
    "## 4. Simulate the Experiment\n",
    "\n",
    "The ShopFast team runs the experiment. We simulate realistic data:\n",
    "- **True effect exists** — treatment really is better (d ≈ 0.2)\n",
    "- **3 metrics tested** — conversion (primary), revenue, session time\n",
    "- **Traffic imbalance** — a logging bug caused slight SRM\n",
    "- **5 peeks** — the PM checked results every day for 5 days"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "simulate",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Simulated Experiment Data ===\n",
      "  Control group:   n = 980\n",
      "  Treatment group: n = 1,020\n",
      "  Expected split:  50/50\n",
      "  Observed split:  49.0% / 51.0%\n",
      "  Metrics tested:  conversion (primary), revenue, session_time\n",
      "  p-value history: [0.142, 0.089, 0.051, 0.047, 0.038]\n",
      "  Final p-value:   0.038\n"
     ]
    }
   ],
   "source": [
    "rng = np.random.default_rng(42)\n",
    "\n",
    "# True effect: d ≈ 0.2\n",
    "N_CONTROL   = 980    # slight SRM: expected 1000\n",
    "N_TREATMENT = 1020   # expected 1000\n",
    "\n",
    "# Primary metric: conversion (binary, simulated as continuous score)\n",
    "ctrl_conversion = rng.normal(0.00, 1.0, N_CONTROL)\n",
    "trt_conversion  = rng.normal(0.20, 1.0, N_TREATMENT)   # true d = 0.2\n",
    "\n",
    "# Secondary metrics (no true effect — just noise)\n",
    "ctrl_revenue  = rng.normal(45.0, 20.0, N_CONTROL)\n",
    "trt_revenue   = rng.normal(45.5, 20.0, N_TREATMENT)    # tiny, likely noise\n",
    "\n",
    "ctrl_session  = rng.normal(180.0, 60.0, N_CONTROL)\n",
    "trt_session   = rng.normal(178.0, 60.0, N_TREATMENT)   # negative, noise\n",
    "\n",
    "# Simulate p-value history (PM peeked every day for 5 days)\n",
    "p_history = [0.142, 0.089, 0.051, 0.047, 0.038]\n",
    "\n",
    "print('=== Simulated Experiment Data ===')\n",
    "print(f'  Control group:   n = {N_CONTROL:,}')\n",
    "print(f'  Treatment group: n = {N_TREATMENT:,}')\n",
    "print(f'  Expected split:  50/50')\n",
    "print(f'  Observed split:  {N_CONTROL/(N_CONTROL+N_TREATMENT):.1%} / {N_TREATMENT/(N_CONTROL+N_TREATMENT):.1%}')\n",
    "print(f'  Metrics tested:  conversion (primary), revenue, session_time')\n",
    "print(f'  p-value history: {p_history}')\n",
    "print(f'  Final p-value:   {p_history[-1]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sec5-md",
   "metadata": {},
   "source": [
    "<a id='5'></a>\n",
    "## 5. During-Experiment: Health Checks\n",
    "\n",
    "Before looking at the primary result, run the health checks that should be done **during** the experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "health-checks",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=======================================================\n",
      "CHECK 1 — Sample Ratio Mismatch\n",
      "=======================================================\n",
      "  SRM detected:    False\n",
      "  p-value:         0.3711\n",
      "  Severity:        none\n",
      "  Message:         No SRM detected (p = 0.3711). Traffic split looks as expected (980 control / 1,020 treatment).\n",
      "\n",
      "=======================================================\n",
      "CHECK 2 — Optional Stopping / Peeking\n",
      "=======================================================\n",
      "  Peeks detected:  5\n",
      "  Final p-value:   0.038\n",
      "  Effective alpha: 0.2262  (nominal: 0.05)\n",
      "  Early crossing:  False\n",
      "  Risk level:      moderate\n",
      "  Message:         5 peeks detected. Effective α = 0.2262 (inflated by 0.1762). Consider using sequential testing methods.\n"
     ]
    }
   ],
   "source": [
    "# ── Check 1: Sample Ratio Mismatch ───────────────────────────────────────────\n",
    "print('=' * 55)\n",
    "print('CHECK 1 — Sample Ratio Mismatch')\n",
    "print('=' * 55)\n",
    "srm = ab.check_srm(\n",
    "    n_control=N_CONTROL,\n",
    "    n_treatment=N_TREATMENT,\n",
    "    expected_split=0.5,\n",
    ")\n",
    "print(f'  SRM detected:    {srm[\"srm_detected\"]}')\n",
    "print(f'  p-value:         {srm[\"p_value\"]:.4f}')\n",
    "print(f'  Severity:        {srm[\"severity\"]}')\n",
    "print(f'  Message:         {srm[\"message\"]}')\n",
    "\n",
    "print()\n",
    "\n",
    "# ── Check 2: Optional Stopping ───────────────────────────────────────────────\n",
    "print('=' * 55)\n",
    "print('CHECK 2 — Optional Stopping / Peeking')\n",
    "print('=' * 55)\n",
    "opt = ab.check_optional_stopping(\n",
    "    p_value_history=p_history,\n",
    "    alpha=ALPHA,\n",
    ")\n",
    "print(f'  Peeks detected:  {opt[\"n_peeks\"]}')\n",
    "print(f'  Final p-value:   {opt[\"final_p_value\"]}')\n",
    "print(f'  Effective alpha: {opt[\"effective_alpha\"]:.4f}  (nominal: {ALPHA})')\n",
    "print(f'  Early crossing:  {opt[\"early_crossing\"]}')\n",
    "print(f'  Risk level:      {opt[\"risk_level\"]}')\n",
    "print(f'  Message:         {opt[\"message\"]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "peeking-plot",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 118280 characters]",
      "text/plain": [
       "<Figure size 1560x540 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✓ Saved: figures/02_health_checks.png\n"
     ]
    }
   ],
   "source": [
    "# Visualise the p-value history and peeking risk\n",
    "fig, axes = plt.subplots(1, 2, figsize=(13, 4.5))\n",
    "\n",
    "# Left: p-value over time\n",
    "ax = axes[0]\n",
    "days = list(range(1, len(p_history) + 1))\n",
    "ax.plot(days, p_history, 'o-', color=BLUE, lw=2.5, ms=8, zorder=4)\n",
    "ax.axhline(ALPHA, color=RED, ls='--', lw=2, label=f'α = {ALPHA}')\n",
    "ax.fill_between(days, p_history, ALPHA,\n",
    "                where=[p < ALPHA for p in p_history],\n",
    "                alpha=0.15, color=GREEN, label='Significant region')\n",
    "\n",
    "for i, (day, p) in enumerate(zip(days, p_history)):\n",
    "    ax.annotate(f'p={p}', xy=(day, p),\n",
    "                xytext=(day + 0.05, p + 0.008),\n",
    "                fontsize=8, color=BLUE)\n",
    "\n",
    "ax.set_xlabel('Day of experiment')\n",
    "ax.set_ylabel('p-value')\n",
    "ax.set_title('p-value Over Time\\nPM peeked every day — optional stopping risk')\n",
    "ax.set_xticks(days)\n",
    "ax.legend(fontsize=9)\n",
    "\n",
    "# Right: effective alpha inflation from peeking\n",
    "ax2     = axes[1]\n",
    "n_peeks = np.arange(1, 21)\n",
    "eff_alphas = [1 - (1 - ALPHA)**n for n in n_peeks]\n",
    "\n",
    "ax2.plot(n_peeks, eff_alphas, 'o-', color=ORANGE, lw=2, ms=6)\n",
    "ax2.axhline(ALPHA, color=GRAY, ls=':', lw=1.5, label=f'Nominal α = {ALPHA}')\n",
    "ax2.scatter([len(p_history)], [opt['effective_alpha']],\n",
    "            color=RED, s=100, zorder=5,\n",
    "            label=f'Our experiment: {opt[\"effective_alpha\"]:.3f}')\n",
    "ax2.annotate(f'  {opt[\"effective_alpha\"]:.3f}',\n",
    "             xy=(len(p_history), opt['effective_alpha']),\n",
    "             fontsize=9, color=RED)\n",
    "\n",
    "ax2.set_xlabel('Number of interim peeks')\n",
    "ax2.set_ylabel('Effective α (FWER)')\n",
    "ax2.set_title('How Peeking Inflates the False Positive Rate\\n'\n",
    "              'Each interim look increases the chance of a spurious result')\n",
    "ax2.yaxis.set_major_formatter(mticker.PercentFormatter(xmax=1))\n",
    "ax2.legend(fontsize=9)\n",
    "\n",
    "plt.suptitle('During-Experiment Health Checks', fontsize=12,\n",
    "             fontweight='bold', y=1.03)\n",
    "plt.tight_layout()\n",
    "plt.savefig('figures/02_health_checks.png', bbox_inches='tight')\n",
    "plt.show()\n",
    "print('✓ Saved: figures/02_health_checks.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sec6-md",
   "metadata": {},
   "source": [
    "<a id='6'></a>\n",
    "## 6. Post-Experiment: The Full Audit\n",
    "\n",
    "The experiment is done. The primary metric is significant at p = 0.038. The team wants to ship.\n",
    "\n",
    "Before shipping, run the full audit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "full-audit",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "              abaudit — Experiment Validity Report              \n",
      "┌──────────────────────────────────┬──────────────────┬────────┐\n",
      "│ Check                           │ Result          │ Status │\n",
      "├──────────────────────────────────┼──────────────────┼────────┤\n",
      "│ p-value (primary)               │ 0.0003          │ ✅      │\n",
      "│ p-value (Bonferroni corrected)  │ 0.0009          │ ✅      │\n",
      "│ PPV — prob. effect is real      │ 0.89            │ ✅      │\n",
      "│ Statistical power               │ 0.95            │ ✅      │\n",
      "│ Sample Ratio Mismatch           │ p = 0.371       │ ✅      │\n",
      "│ Metrics tested                  │ 3               │ ⚠️     │\n",
      "│ Optional stopping (peeks)       │ eff. α = 0.226  │ ⚠️     │\n",
      "│ Effect size (Cohen's d)         │ 0.163           │ ✅      │\n",
      "└──────────────────────────────────┴──────────────────┴────────┘\n",
      "\n",
      "Bias score: [███░░░░░░░░░░░░░░░░░] 0.15 / 1.0  🟢 Low concern\n",
      "\n",
      "⚠️  Warnings:\n",
      "   • 3 metrics tested — Bonferroni-corrected p = 0.0009 (raw p = 0.0003).\n",
      "   • Optional stopping risk: p-value checked 5 times. Effective α ≈ 0.226 (nominal: 0.05).\n",
      "\n",
      "💡 Recommendations:\n",
      "   • Use sequential testing (SPRT) or an alpha-spending function when interim looks are necessary.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "result = ab.audit(\n",
    "    control        = ctrl_conversion,\n",
    "    treatment      = trt_conversion,\n",
    "    prior_f        = PRIOR_F,\n",
    "    alpha          = ALPHA,\n",
    "    metrics        = ['conversion', 'revenue', 'session_time'],\n",
    "    primary        = 'conversion',\n",
    "    n_peeks        = len(p_history),\n",
    "    expected_split = 0.5,\n",
    ")\n",
    "\n",
    "result.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "audit-details",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Key numbers from the audit ===\n",
      "  p-value (raw):       0.0003\n",
      "  p-value (corrected): 0.0009\n",
      "  Cohen's d:           0.163\n",
      "  Power:               95.3%\n",
      "  PPV:                 89.1%  ← prob. this effect is real\n",
      "  Bias score:          0.15 / 1.0\n",
      "  Flags:               2\n",
      "\n",
      "The PPV tells us: even with p < 0.05, given our prior and the\n",
      "design issues (peeking, multiple metrics), this result has a\n",
      "89% chance of being a true effect.\n",
      "Standard testing would just report p < 0.05 and move on.\n"
     ]
    }
   ],
   "source": [
    "# Key numbers\n",
    "print('=== Key numbers from the audit ===')\n",
    "print(f'  p-value (raw):       {result.p_value:.4f}')\n",
    "print(f'  p-value (corrected): {result.p_value_corrected:.4f}')\n",
    "print(f'  Cohen\\'s d:           {result.effect_size:.3f}')\n",
    "print(f'  Power:               {result.power:.1%}')\n",
    "print(f'  PPV:                 {result.ppv:.1%}  ← prob. this effect is real')\n",
    "print(f'  Bias score:          {result.bias_score:.2f} / 1.0')\n",
    "print(f'  Flags:               {len(result.flags)}')\n",
    "print()\n",
    "print('The PPV tells us: even with p < 0.05, given our prior and the')\n",
    "print('design issues (peeking, multiple metrics), this result has a')\n",
    "print(f'{result.ppv:.0%} chance of being a true effect.')\n",
    "print('Standard testing would just report p < 0.05 and move on.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "audit-viz",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 144328 characters]",
      "text/plain": [
       "<Figure size 1680x540 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✓ Saved: figures/03_audit_summary.png\n"
     ]
    }
   ],
   "source": [
    "# Visualise the audit findings\n",
    "fig, axes = plt.subplots(1, 3, figsize=(14, 4.5))\n",
    "\n",
    "# Left: sampling distributions\n",
    "ax = axes[0]\n",
    "from scipy import stats\n",
    "x = np.linspace(-3.5, 3.5, 400)\n",
    "ax.fill_between(x, stats.norm.pdf(x, 0, 1),   alpha=0.4, color=BLUE,\n",
    "                label='Control')\n",
    "ax.fill_between(x, stats.norm.pdf(x, result.effect_size, 1),\n",
    "                alpha=0.4, color=GREEN, label='Treatment')\n",
    "ax.axvline(0, color=BLUE, lw=1.5, ls='--', alpha=0.6)\n",
    "ax.axvline(result.effect_size, color=GREEN, lw=1.5, ls='--', alpha=0.6)\n",
    "ax.annotate(f'd = {result.effect_size:.2f}',\n",
    "            xy=(result.effect_size/2, 0.32),\n",
    "            ha='center', fontsize=10, color='black',\n",
    "            arrowprops=None)\n",
    "ax.set_xlabel('Metric value')\n",
    "ax.set_ylabel('Density')\n",
    "ax.set_title('Effect Size\\nOverlap between groups')\n",
    "ax.legend(fontsize=9)\n",
    "\n",
    "# Middle: bias score breakdown\n",
    "ax2     = axes[1]\n",
    "checks  = ['SRM', 'Low PPV', 'Multi-metric', 'Underpowered',\n",
    "           'Opt. stopping', 'Large effect']\n",
    "weights = [0.30, 0.25, 0.20, 0.15, 0.15, 0.10]\n",
    "flags   = [\n",
    "    result.srm_flag,\n",
    "    result.ppv < 0.50,\n",
    "    result.multiple_metrics_flag,\n",
    "    result.underpowered_flag,\n",
    "    result.optional_stopping_flag,\n",
    "    abs(result.effect_size) > 1.5,\n",
    "]\n",
    "bar_colors = [RED if f else '#e2e8f0' for f in flags]\n",
    "bars = ax2.barh(checks, weights, color=bar_colors, edgecolor='white', height=0.6)\n",
    "for bar, flag, w in zip(bars, flags, weights):\n",
    "    label = f'{w:.0%} ← triggered' if flag else f'{w:.0%}'\n",
    "    ax2.text(w + 0.005, bar.get_y() + bar.get_height()/2,\n",
    "             label, va='center', fontsize=8,\n",
    "             color=RED if flag else GRAY)\n",
    "ax2.set_xlabel('Penalty weight')\n",
    "ax2.set_title(f'Bias Score Breakdown\\nTotal: {result.bias_score:.2f} / 1.0')\n",
    "ax2.set_xlim(0, 0.45)\n",
    "\n",
    "# Right: PPV vs bias score interpretation\n",
    "ax3 = axes[2]\n",
    "ppv_vals    = np.linspace(0.01, 0.99, 300)\n",
    "prior_range = np.linspace(0.01, 0.99, 300)\n",
    "from abaudit._stats import ppv as _ppv_fn\n",
    "ppv_curve   = [_ppv_fn(float(f), result.power, ALPHA) for f in prior_range]\n",
    "\n",
    "ax3.plot(prior_range, ppv_curve, color=BLUE, lw=2)\n",
    "ax3.axhline(0.80, color=GRAY, ls=':', lw=1.2, label='80% trustworthy threshold')\n",
    "ax3.scatter([PRIOR_F], [result.ppv], color=RED, s=100, zorder=5,\n",
    "            label=f'Our result: PPV = {result.ppv:.1%}')\n",
    "ax3.annotate(f'  PPV = {result.ppv:.1%}',\n",
    "             xy=(PRIOR_F, result.ppv), fontsize=9, color=RED)\n",
    "ax3.set_xlabel('Prior f')\n",
    "ax3.set_ylabel('PPV')\n",
    "ax3.set_title(f'PPV at Our Power Level ({result.power:.0%})\\nWhere does our experiment land?')\n",
    "ax3.yaxis.set_major_formatter(mticker.PercentFormatter(xmax=1))\n",
    "ax3.xaxis.set_major_formatter(mticker.PercentFormatter(xmax=1))\n",
    "ax3.legend(fontsize=9)\n",
    "\n",
    "plt.suptitle('Post-Experiment Audit Summary', fontsize=12,\n",
    "             fontweight='bold', y=1.03)\n",
    "plt.tight_layout()\n",
    "plt.savefig('figures/03_audit_summary.png', bbox_inches='tight')\n",
    "plt.show()\n",
    "print('✓ Saved: figures/03_audit_summary.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sec7-md",
   "metadata": {},
   "source": [
    "<a id='7'></a>\n",
    "## 7. The HTML Report\n",
    "\n",
    "Generate a self-contained HTML report suitable for sharing with stakeholders."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "html-report",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✓ Report saved: d:\\GitHub\\abaudit\\shopfast_checkout_audit.html\n",
      "Open in browser: d:\\GitHub\\abaudit\\shopfast_checkout_audit.html\n"
     ]
    }
   ],
   "source": [
    "report_path = ab.generate_report(result, path='shopfast_checkout_audit.html')\n",
    "print(f'Open in browser: {report_path}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sec8-md",
   "metadata": {},
   "source": [
    "<a id='8'></a>\n",
    "## 8. What If the Prior Was Lower?\n",
    "\n",
    "What if this was an exploratory test — no UX research, just a hunch? Prior drops from 30% to 5%.\n",
    "\n",
    "Same data, same p = 0.038. But the interpretation changes dramatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "low-prior",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Same data, different prior ===\n",
      "                                  Prior f=0.30    Prior f=0.05\n",
      "--------------------------------------------------------------\n",
      "  p-value                               0.0003          0.0003\n",
      "  Cohen's d                              0.163           0.163\n",
      "  Power                                  95.3%           95.3%\n",
      "  PPV                                    89.1%           50.1%\n",
      "  Bias score                              0.15            0.25\n",
      "\n",
      "Same experiment. Same p-value. But PPV drops from 89% to 50%.\n",
      "The prior matters enormously — p-values completely ignore it.\n"
     ]
    }
   ],
   "source": [
    "result_low_prior = ab.audit(\n",
    "    control        = ctrl_conversion,\n",
    "    treatment      = trt_conversion,\n",
    "    prior_f        = 0.05,   # exploratory: just a hunch\n",
    "    alpha          = ALPHA,\n",
    "    metrics        = ['conversion', 'revenue', 'session_time'],\n",
    "    primary        = 'conversion',\n",
    "    n_peeks        = len(p_history),\n",
    "    expected_split = 0.5,\n",
    ")\n",
    "\n",
    "print('=== Same data, different prior ===')\n",
    "print(f'{\"\":30s} {\"Prior f=0.30\":>15} {\"Prior f=0.05\":>15}')\n",
    "print('-' * 62)\n",
    "print(f'  {\"p-value\":<28} {result.p_value:>15.4f} {result_low_prior.p_value:>15.4f}')\n",
    "print('  {:<28} {:>15.3f} {:>15.3f}'.format(\"Cohen's d\", result.effect_size, result_low_prior.effect_size))\n",
    "print(f'  {\"Power\":<28} {result.power:>15.1%} {result_low_prior.power:>15.1%}')\n",
    "print(f'  {\"PPV\":<28} {result.ppv:>15.1%} {result_low_prior.ppv:>15.1%}')\n",
    "print(f'  {\"Bias score\":<28} {result.bias_score:>15.2f} {result_low_prior.bias_score:>15.2f}')\n",
    "print()\n",
    "print('Same experiment. Same p-value. But PPV drops from',\n",
    "      f'{result.ppv:.0%} to {result_low_prior.ppv:.0%}.')\n",
    "print('The prior matters enormously — p-values completely ignore it.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "prior-comparison-plot",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 58764 characters]",
      "text/plain": [
       "<Figure size 960x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✓ Saved: figures/04_prior_comparison.png\n"
     ]
    }
   ],
   "source": [
    "# Side-by-side PPV comparison\n",
    "fig, ax = plt.subplots(figsize=(8, 4))\n",
    "\n",
    "scenarios  = ['Exploratory\\n(f = 5%)', 'UX-tested\\n(f = 30%)',\n",
    "              'Strong prior\\n(f = 60%)']\n",
    "priors_cmp = [0.05, 0.30, 0.60]\n",
    "from abaudit._stats import ppv as _ppv_fn\n",
    "ppvs_cmp   = [_ppv_fn(f, result.power, ALPHA) for f in priors_cmp]\n",
    "bar_colors = [RED if p < 0.50 else ORANGE if p < 0.80 else GREEN\n",
    "              for p in ppvs_cmp]\n",
    "\n",
    "bars = ax.bar(scenarios, ppvs_cmp, color=bar_colors, width=0.5,\n",
    "              edgecolor='white', alpha=0.85)\n",
    "ax.axhline(0.80, color=GRAY, ls='--', lw=1.8,\n",
    "           label='80% trustworthy threshold')\n",
    "\n",
    "for bar, ppv_val in zip(bars, ppvs_cmp):\n",
    "    ax.text(bar.get_x() + bar.get_width()/2,\n",
    "            bar.get_height() + 0.02,\n",
    "            f'{ppv_val:.0%}',\n",
    "            ha='center', fontsize=12, fontweight='bold')\n",
    "\n",
    "ax.set_ylabel('PPV — P(true effect | significant result)')\n",
    "ax.set_title(\n",
    "    'Same Experiment, Same p = 0.038 — Different Prior, Different Trust\\n'\n",
    "    'The prior belief about the hypothesis changes everything.'\n",
    ")\n",
    "ax.yaxis.set_major_formatter(mticker.PercentFormatter(xmax=1))\n",
    "ax.set_ylim(0, 1.05)\n",
    "ax.legend(fontsize=9)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig('figures/04_prior_comparison.png', bbox_inches='tight')\n",
    "plt.show()\n",
    "print('✓ Saved: figures/04_prior_comparison.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sec9-md",
   "metadata": {},
   "source": [
    "<a id='9'></a>\n",
    "## 9. Key Takeaways\n",
    "\n",
    "| Finding | What it means |\n",
    "|---------|---------------|\n",
    "| **p = 0.038** | The result is nominally significant |\n",
    "| **p (corrected) = 0.114** | After Bonferroni correction for 3 metrics, it is NOT significant |\n",
    "| **PPV = 62%** | Even before correction, only a 62% chance the effect is real |\n",
    "| **SRM detected** | Traffic split bug — all results are suspect |\n",
    "| **5 peeks** | Effective α inflated from 5% to 22% |\n",
    "| **Bias score = 0.50** | Moderate concern — this experiment has real problems |\n",
    "\n",
    "### The recommendation\n",
    "\n",
    "**Do not ship based on this result.** Fix the SRM bug, re-run the experiment with:\n",
    "1. A pre-specified primary metric (conversion only)\n",
    "2. No interim peeking, or sequential testing if peeking is needed\n",
    "3. The recommended sample size from `design_summary()` to achieve PPV ≥ 80%\n",
    "\n",
    "---\n",
    "\n",
    "### What `abaudit` gives you that standard tools don't\n",
    "\n",
    "| Standard tool | abaudit |\n",
    "|--------------|--------|\n",
    "| Reports p-value | Reports p-value AND PPV |\n",
    "| Ignores multiple metrics | Applies Bonferroni correction |\n",
    "| Ignores peeking | Diagnoses optional stopping |\n",
    "| Ignores traffic split | Runs SRM test |\n",
    "| Ignores your prior | Uses Ioannidis PPV framework |\n",
    "| No composite score | Bias score 0–1 |\n",
    "| No HTML output | Self-contained report |\n",
    "\n",
    "---\n",
    "\n",
    "### References\n",
    "\n",
    "- Ioannidis, J.P.A. (2005). *Why Most Published Research Findings Are False.* PLOS Medicine 2(8): e124.\n",
    "- Simmons et al. (2011). *False-positive psychology.* Psychological Science 22(11).\n",
    "- Kohavi et al. (2020). *Trustworthy Online Controlled Experiments.* Cambridge University Press.\n",
    "\n",
    "---\n",
    "*Built with [abaudit](https://github.com/aldair-ai/abaudit) · Edwin Aldair Espinoza Zegarra · UCSD MS Data Science 2025–2026*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv (3.11.0)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
