{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.0"
  },
  "colab": {
   "provenance": []
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-0",
   "metadata": {},
   "source": [
    "# insurance-frequency-severity\n",
    "\n",
    "**Sarmanov copula joint frequency-severity modelling for UK personal lines.**\n",
    "\n",
    "This notebook challenges the independence assumption that underlies every\n",
    "standard two-part GLM pricing model: `Pure premium = E[N|x] × E[S|x]`.\n",
    "\n",
    "That multiplication assumes claim count and average severity are independent\n",
    "given rating factors. In UK motor, NCD suppresses borderline claims — policyholders\n",
    "near the NCD threshold do not report near-misses. The result is a systematic\n",
    "negative correlation between frequency and severity.\n",
    "\n",
    "This library measures that correlation and corrects the premium for it.\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/burning-cost/insurance-frequency-severity/blob/main/notebooks/quickstart.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-1",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q insurance-frequency-severity statsmodels numpy pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-2",
   "metadata": {},
   "source": [
    "## Synthetic motor book\n",
    "\n",
    "We generate 10,000 motor policies with a negative frequency-severity correlation\n",
    "(omega < 0). This mimics the NCD suppression effect: policyholders who claim\n",
    "frequently tend to have smaller individual claims (they report everything);\n",
    "low-frequency policyholders report only large events."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import statsmodels.api as sm\n",
    "\n",
    "rng = np.random.default_rng(42)\n",
    "n = 10_000\n",
    "\n",
    "# Rating factors\n",
    "age    = rng.integers(18, 70, n).astype(float)\n",
    "ncd    = rng.integers(0, 9, n).astype(float)\n",
    "area   = rng.integers(0, 5, n).astype(float)\n",
    "\n",
    "# True frequency and severity parameters\n",
    "log_mu_n = -2.5 + 0.3 * (age < 25) - 0.08 * ncd + 0.1 * area\n",
    "log_mu_s = 6.5  - 0.03 * ncd  # severity slightly lower for high NCD\n",
    "\n",
    "mu_n = np.exp(log_mu_n)   # claim frequency (mean ~0.08)\n",
    "mu_s = np.exp(log_mu_s)   # mean severity (~£600)\n",
    "\n",
    "# Simulate with negative correlation: high-frequency policies get smaller claims\n",
    "# True omega (Sarmanov dependence) = -0.25\n",
    "claim_count = rng.poisson(mu_n)\n",
    "# Severity: reduce for policies with >1 claims (NCD suppression effect)\n",
    "sev_scale = mu_s / 3.0 * np.exp(-0.2 * claim_count)  # negative dependence\n",
    "avg_severity = np.where(\n",
    "    claim_count > 0,\n",
    "    rng.gamma(shape=3.0, scale=sev_scale),\n",
    "    np.nan\n",
    ")\n",
    "\n",
    "claims_df = pd.DataFrame({\n",
    "    \"age\": age, \"ncd\": ncd, \"area\": area,\n",
    "    \"claim_count\": claim_count,\n",
    "    \"avg_severity\": avg_severity,\n",
    "})\n",
    "\n",
    "print(f\"Policies: {n:,}\")\n",
    "print(f\"Claims:   {(claim_count > 0).sum():,} with at least one claim ({(claim_count > 0).mean():.1%})\")\n",
    "print(f\"Mean severity (when claimed): £{np.nanmean(avg_severity):.0f}\")\n",
    "claims_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-4",
   "metadata": {},
   "source": [
    "## Step 1: Test for dependence\n",
    "\n",
    "`DependenceTest` runs three tests for frequency-severity dependence:\n",
    "Spearman rank correlation, a chi-square test on sorted deciles,\n",
    "and a concordance test on the copula pseudo-observations.\n",
    "\n",
    "If all three are non-significant, the independence assumption holds and\n",
    "the standard two-part GLM is fine. If they are significant — proceed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from insurance_frequency_severity import DependenceTest\n",
    "\n",
    "dt = DependenceTest()\n",
    "test_result = dt.test(claims_df, n_col=\"claim_count\", s_col=\"avg_severity\")\n",
    "\n",
    "print(f\"Spearman rho: {test_result.spearman_rho:.3f}  p={test_result.spearman_p:.4f}\")\n",
    "print(f\"Decile test:  chi2={test_result.chi2:.2f}  p={test_result.chi2_p:.4f}\")\n",
    "print()\n",
    "print(f\"Conclusion: {test_result.conclusion}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-6",
   "metadata": {},
   "source": [
    "## Step 2: Fit marginal GLMs\n",
    "\n",
    "The Sarmanov copula uses IFM estimation (Inference Functions for Margins):\n",
    "fit the marginal GLMs first, then estimate the copula parameter omega\n",
    "from the margins. You plug in your existing GLM objects — no refitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-7",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = claims_df[[\"age\", \"ncd\", \"area\"]]\n",
    "X_const = sm.add_constant(X)\n",
    "\n",
    "# Frequency GLM: Negative Binomial\n",
    "freq_glm = sm.GLM(\n",
    "    claims_df[\"claim_count\"],\n",
    "    X_const,\n",
    "    family=sm.families.NegativeBinomial(alpha=0.5)\n",
    ").fit(disp=False)\n",
    "\n",
    "# Severity GLM: Gamma (fitted on claims-only rows)\n",
    "mask = claims_df[\"claim_count\"] > 0\n",
    "sev_glm = sm.GLM(\n",
    "    claims_df.loc[mask, \"avg_severity\"],\n",
    "    X_const[mask],\n",
    "    family=sm.families.Gamma(link=sm.families.links.Log())\n",
    ").fit(disp=False)\n",
    "\n",
    "print(f\"Freq GLM deviance:   {freq_glm.deviance:.1f}\")\n",
    "print(f\"Sev GLM deviance:    {sev_glm.deviance:.1f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-8",
   "metadata": {},
   "source": [
    "## Step 3: Fit Sarmanov copula and compute premium correction\n",
    "\n",
    "`JointFreqSev` wraps the two GLMs and estimates omega (the Sarmanov\n",
    "dependence parameter). `premium_correction()` returns a multiplicative\n",
    "factor — multiply your standard `E[N] × E[S]` by this to get the\n",
    "copula-corrected pure premium."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from insurance_frequency_severity import JointFreqSev\n",
    "\n",
    "joint = JointFreqSev(freq_glm=freq_glm, sev_glm=sev_glm)\n",
    "joint.fit(claims_df, n_col=\"claim_count\", s_col=\"avg_severity\")\n",
    "\n",
    "print(f\"Omega (Sarmanov): {joint.omega_:.4f}\")\n",
    "print(f\"  Negative omega confirms NCD suppression: high freq = low severity\")\n",
    "print()\n",
    "\n",
    "# Correction factors for first 5 policies\n",
    "corrections = joint.premium_correction(X_const[:5])\n",
    "\n",
    "# Compare standard vs corrected premium\n",
    "freq_pred = freq_glm.predict(X_const[:5])\n",
    "sev_pred  = sev_glm.predict(X_const[:5])\n",
    "standard_pp   = freq_pred * sev_pred\n",
    "corrected_pp  = standard_pp * corrections\n",
    "\n",
    "comparison = pd.DataFrame({\n",
    "    \"standard_pp\":  np.round(standard_pp, 2),\n",
    "    \"correction\":   np.round(corrections, 4),\n",
    "    \"corrected_pp\": np.round(corrected_pp, 2),\n",
    "})\n",
    "print(\"Premium correction (first 5 policies):\")\n",
    "print(comparison)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-10",
   "metadata": {},
   "source": [
    "## What you should see\n",
    "\n",
    "- `DependenceTest` should detect significant dependence (p < 0.05 on at least\n",
    "  two tests). We built it in.\n",
    "- Omega should be negative (~-0.1 to -0.3). The exact value depends on\n",
    "  how tightly the IFM fitting converges on this synthetic dataset.\n",
    "- `premium_correction()` returns values below 1.0 on average — the independence\n",
    "  assumption overstates the pure premium when frequency and severity are\n",
    "  negatively correlated.\n",
    "\n",
    "## Next steps\n",
    "\n",
    "- **`GaussianCopulaMixed`** — standard Gaussian copula with PIT approximation\n",
    "  for discrete margins, for presenting rho in familiar terms\n",
    "- **`ConditionalFreqSev`** — simplest approach: add N as a covariate in severity GLM\n",
    "- **`insurance_frequency_severity.dependent`** — neural two-part model with shared\n",
    "  encoder trunk for large books\n",
    "\n",
    "**GitHub:** https://github.com/burning-cost/insurance-frequency-severity  \n",
    "**PyPI:** https://pypi.org/project/insurance-frequency-severity/"
   ]
  }
 ]
}
