{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Error computation with HESSE and MINOS\n",
    "\n",
    "This is a topical tutorial. If you are new to iminuit, you should go through the basic tutorial first. \n",
    "\n",
    "iminuit (and C++ MINUIT) offers two major ways of computing parameter uncertainties from the cost function (which must be of least-squares form or negative log-likelihood), HESSE and MINOS. These have different pros and cons, on which we already touched in the basic tutorial. Here we want to go a bit deeper into the pros and cons and also try to reveal a bit of the math behind the two approaches.\n",
    "\n",
    "There are several sources which explain what HESSE and MINOS do, in particular the MINUIT User's Guide (linked from the [iminuit documentation](https://iminuit.readthedocs.io/en/latest/about.html)). The mathematical details are covered in F. James, \"Statistical Methods in Experimental Physics\", 2nd edition, World Scientific (2006)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Definitions\n",
    "\n",
    "We need to use a couple of technical terms from mathematical statistics when talk about proven properties. We also use recurring variable names with the same meaning throughout.\n",
    "\n",
    "* $n$ number of [observations](https://en.wikipedia.org/wiki/Sample_%28statistics%29)\n",
    "* $x$ observation from data distribution $f(x; \\theta)$ with parameter $\\theta$ (not an angle!)\n",
    "* $\\hat \\theta$ estimate of $\\theta$ obtained from a sample (the parameter value obtained from the fit)\n",
    "* $\\mathcal L$ likelihood function\n",
    "* $\\mathcal I$ [Fisher information](https://en.wikipedia.org/wiki/Fisher_information)\n",
    "* $E[\\dots]$ expectation value of $\\dots$ over distribution of $x$\n",
    "* $V[\\dots]$ variance of $\\dots$\n",
    "\n",
    "The terms **asymptotic** and **asymptotic limit** refer to inference from an infinite data sample. Mathematical properties of statistical methods are most commonly computed in this limit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HESSE\n",
    "\n",
    "HESSE in a nutshell:\n",
    "* Computes approximate covariance matrix $C_{ij}$ for all fitted parameters\n",
    "* Square-root of diagonal elements of $C_{ij}$ corresponds to one standard deviation; to obtain $k$ standard deviations, multiply covariance matrix $C$ by $k^2$\n",
    "* Can be used to form confidence intervals $\\hat \\theta_i \\pm C_{ii}^{1/2}$\n",
    "* Only way to obtain parameter correlations\n",
    "* Comparably fast, requires:\n",
    "    * Numerical computation of all second derivatives of the cost function (Hessian matrix)\n",
    "    * Inverting this matrix\n",
    "* Cannot compute uncertainty for only one parameter if there are several\n",
    "* Approximately computed as by-product of MIGRAD algorithm (usually accurate enough when `strategy` >= 1 is used)\n",
    "\n",
    "The algorithm derives its name from the Hessian matrix of second derivatives which is computed to obtain the uncertainties.\n",
    "\n",
    "### How does it work?\n",
    "\n",
    "The matrix of second derivatives of the cost function is computed for all free parameters and multiplied by 2. The result is inverted and multiplied by `errordef`.\n",
    "\n",
    "MINUIT implements parameter limits by applying a variable transformation and therefore distinguishes internal and external parameter space. The Hessian matrix is computed in the internal parameter space and transformed using the chain rule to external space.\n",
    "\n",
    "### Why does it work?\n",
    "\n",
    "One can prove under general conditions in the asymptotic limit that a parameter estimate $\\hat\\theta$ obtained from the least-squares and the maximum-likelihood methods is normally distributed with minimum variance, given by the [Cramer-Rao lower bound](https://en.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93Rao_bound), which is the minimum for any unbiased estimator (these methods are asymptotically unbiased).\n",
    "\n",
    "$$\n",
    "V(\\hat \\theta) \\underset{n\\rightarrow \\infty}{\\longrightarrow} \\left\\{ n E\\left[\\left( \\frac{\\partial \\ln\\! f(x;\\theta)}{\\partial \\theta} \\right)^2 \\right]_{\\theta = \\hat\\theta} \\right\\}^{-1} = \\left\\{ -n E\\left[\\frac{\\partial^2 \\ln\\! f(x;\\theta)}{\\partial \\theta^2} \\right]_{\\theta = \\hat\\theta} \\right\\}^{-1}\n",
    "$$\n",
    "\n",
    "The expectation here is taken over the data distribution. Since the expectation value is constant, we see that the variance of $\\hat\\theta$ scales goes down with $n^{-1}$ in the asymptotic limit.\n",
    "\n",
    "If the data range is independent of $\\theta$, which we usually assume (but see F. James book for a counter example), we can swap integration over $x$ and differentiation with respect to $\\theta$. Doing this and replacing the expectation with its plug-in estimate, the arithmetic average, we obtain:\n",
    "\n",
    "$$\n",
    "-n E\\left[\\frac{\\partial^2 \\ln\\! f(x;\\theta)}{\\partial \\theta^2} \\right]_{\\theta = \\hat \\theta} = -n \\frac{1}{n} \\sum_i \\frac{\\partial^2 \\ln\\! f(x_i; \\theta)}{\\partial \\theta^2}\\Big\\vert_{\\theta = \\hat \\theta} = \\frac{\\partial^2 \\big(-\\sum_i \\ln\\! f(x_i; \\theta)\\big)}{\\partial \\theta^2}\\Big\\vert_{\\theta = \\hat \\theta}\n",
    "$$\n",
    "\n",
    "We now see that the numerator contains the negative log-likelihood  function that we often plug into iminuit. If there is a vector of parameters $\\hat{\\vec \\theta}$, then this turns into the Hessian matrix of second derivatives.\n",
    "\n",
    "#### A bit of history\n",
    "\n",
    "So what about these factors of $2$ and `errordef` in MINUIT that were mentioned above? These don't appear here. There are historical reasons for those. In the asymptotic limit, the least-squares cost function that corresponds to the log-likelihood is $Q = -2 \\ln\\! \\mathcal{L} - \\text{constants}$. MINUIT was originally developed with least-squares fits in mind, therefore its internal math assumes the $Q$ form. If the second derivatives are computed from $Q$, the constants are removed but the Hessian must be multiplied by a factor of two to get the right variance. Correspondingly, if the user puts in a negative log-likelihood function, the same procedure now introduces an extra factor of two $2$, which must be compensated by the `errordef` value of 0.5 for the negative log-likelihood.\n",
    "\n",
    "### Why is HESSE approximate\n",
    "\n",
    "We started out from Cramer-Rao bound, the asymptotic limit for the parameter variance. How fast the finite sample approaches the limit depends on the problem at hand. For normal distributed data, the bound is exact.\n",
    "\n",
    "We further approximated the computation of the bound by replacing the expectation over the likelihood with the sample mean of the likelihood. \n",
    "\n",
    "## MINOS\n",
    "\n",
    "MINOS in a nutshell:\n",
    "\n",
    "* Approximate confidence intervals (intervals are wrongly claimed to be \"exact\" in some sources, including the MINUIT paper from 1975)\n",
    "* Cannot compute parameter correlations\n",
    "* Some (unverified) sources claim better coverage probability than intervals based on HESSE\n",
    "* In general slower than HESSE (requiring more function evaluations):\n",
    "    * Iteratively computes the value pair $\\theta^d, \\theta^u$ which increases the cost function by `errordef` over the minimum\n",
    "    * If the cost function has several parameters, it is minimised with respect to all other parameters during this scan\n",
    "* Can be used to compute uncertainty for only one parameter - but this is not more efficient than HESSE, since the computation requires at least one evaluation of HESSE\n",
    "\n",
    "The MINOS algorithm computes [profile-likelihood based](https://en.wikipedia.org/wiki/Likelihood_function#Profile_likelihood) confidence intervals.\n",
    "\n",
    "### How does it work?\n",
    "\n",
    "MINOS scans the likelihood along one parameter $\\theta_i$,  while minimizing the likelihood with respect to all other parameters $\\theta_k$ with $k \\ne i$. This is effectively the same as expressing the other parameter estimates as a function of $\\theta_i$, $\\hat \\theta_k(\\theta_i)$, and scanning the now one-dimensional negative log-likelihood $-\\ln \\mathcal{L}(\\theta_i; \\theta_k = \\hat \\theta_k(\\theta_i) , k\\ne i)$. This is called the [profile likelihood](https://en.wikipedia.org/wiki/Likelihood_function#Profile_likelihood) for parameter $\\theta_i$.\n",
    "\n",
    "One follows this curve until $-\\ln \\mathcal{L}$ increases by `errordef` with respect to its minimum and stores the two corresponding values $\\theta^d_i$ and $\\theta^u_i$. The interval $(\\theta^d_i, \\theta^u_i)$ has $68\\,\\%$ coverage probability in the asymptotic limit. In multi-dimensional parameter space, the computation is comparably expensive due to the iterative steps of scanning and minimization.\n",
    "\n",
    "An efficient algorithm to compute the interval is described by Venzon and Moolgavkar in *A Method for Computing Profile-Likelihood-Based Confidence Intervals*, Journal of the Royal Statistical Society C37 (1988) 87–94 [DOI](https://doi.org/10.2307%2F2347496).\n",
    "\n",
    "### Why does it work?\n",
    "\n",
    "We define the likelihood ratio $\\ell(\\vec\\theta) = \\mathcal{L}(\\vec\\theta) / \\mathcal{L}(\\hat{\\vec\\theta})$. In the asymptotic limit, $-2 \\ln \\ell(\\vec\\theta)$ is $\\chi^2(k)$ distributed, where $k$ is the number of fitted parameters. For $k = 1$, the $\\chi^2$-interval $[0, 1)$ contains $68\\,\\%$ probability. For a single parameter fit therefore a corresponding interval is found by the two values $\\theta^d$ and $\\theta^u$ which solve\n",
    "\n",
    "$$\n",
    "-2\\ln\\ell(\\theta) = 1 \\Leftrightarrow -\\ln\\ell(\\theta) = 1/2\n",
    "$$\n",
    "\n",
    "We recognize the difference in the negative log-likelihood on the left-hand side and our `errordef = 0.5` on the right-hand side. Confidence intervals with other coverage probability can be constructed by finding the corresponding upper value of the $\\chi^2$-interval with that integrated probability. In a least-squares fit, we have\n",
    "\n",
    "$$\n",
    "\\Delta Q(\\theta) = -2 \\ln \\ell(\\theta) = 1\n",
    "$$\n",
    "\n",
    "therefore `errordef = 1` is the crossing value for a least-squares cost function.\n",
    "\n",
    "In the multi-parameter case, when we search for the interval for $\\theta_i$, the other parameters are effectively fixed to their best-fit values for the current value $\\theta_i$ in the scan, $\\theta_k = \\hat\\theta_k(\\theta_i)$. Therefore, during the scan they are not free. The profile likelihood is effectively a likelihood for a single parameter. Thus, $68\\,\\%$-intervals are also here obtained when the negative log-likelihood crosses the value $1/2$ above the minimum. \n",
    "\n",
    "### Why is MINOS approximate\n",
    "\n",
    "In some sources, the MINOS intervals are wrongly described as *exact*. They are not exact, because for finite samples the intervals do not necessarily have $68\\,\\%$ coverage probability, only in the asymptotic limit.\n",
    "\n",
    "Some sources claim that two approximations are involved in the HESSE calculation of an interval, but only one in the MINOS calculation and conclude that MINOS intervals therefore approach the asymptotic limit more rapidly. This claim is disputed by others."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Coverage probability of HESSE and MINOS intervals in toy experiments\n",
    "\n",
    "## Poisson distributed data\n",
    "\n",
    "We construct HESSE and MINOS intervals for a counting experiment. We consider the extreme case of a single observation $k$ drawn from a Poisson distribution $P(k;\\lambda)$. We use the maximum-likelihood method to find the best estimate for $\\lambda$ for each $k$ under the constraint $\\lambda > 0$, which is trivially just $\\hat \\lambda = k$, and construct intervals with the HESSE and MINOS algorithms to check their coverage.\n",
    "\n",
    "This case can be fully handled analytically, but here we use iminuit's HESSE and MINOS algorithms to compute the intervals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/site-packages/ipykernel_launcher.py:16: RuntimeWarning: invalid value encountered in log\n",
      "  app.launch_new_instance()\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 45792 characters]\n",
      "text/plain": [
       "<Figure size 720x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from iminuit import Minuit\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "class MemoizedMinuit:\n",
    "    \"\"\"This class avoids repeated computation of the same intervals, which only depend on k.\"\"\"\n",
    "\n",
    "    def __init__(self):\n",
    "        self._results = {}\n",
    "\n",
    "    def __call__(self, k):\n",
    "        r = self._results.get(k, None)\n",
    "        if r is None:\n",
    "\n",
    "            def nll(lambd):\n",
    "                return lambd - k * np.log(lambd)\n",
    "\n",
    "            m = Minuit(nll, lambd=k+1, limit_lambd=(0, None), errordef=0.5, pedantic=False)\n",
    "            m.migrad()\n",
    "            m.hesse()\n",
    "            m.minos()\n",
    "            assert m.valid\n",
    "\n",
    "            p = m.values[\"lambd\"]\n",
    "            dp = m.errors[\"lambd\"]\n",
    "            pm = max(p + m.merrors[\"lambd\"].lower, 0.0), p + m.merrors[\"lambd\"].upper\n",
    "            \n",
    "            r = p, dp, *pm\n",
    "            self._results[k] = r\n",
    "\n",
    "        return r\n",
    "\n",
    "rng = np.random.default_rng(seed=1)\n",
    "nmc = 5000\n",
    "mu = 10 ** np.linspace(-1, 2, 100)\n",
    "    \n",
    "m = MemoizedMinuit()\n",
    "\n",
    "pcov = {\n",
    "    \"HESSE\": np.empty_like(mu),\n",
    "    \"MINOS\": np.empty_like(mu),\n",
    "}\n",
    "\n",
    "for i, mui in enumerate(mu):\n",
    "\n",
    "    nh = 0\n",
    "    nm = 0\n",
    "    for imc in range(nmc):\n",
    "        k = rng.poisson(mui)\n",
    "        \n",
    "        p, dp, pd, pu = m(k)\n",
    "\n",
    "        if p - dp < mui < p + dp:\n",
    "            nh += 1\n",
    "        if pd < mui < pu:\n",
    "            nm += 1\n",
    "\n",
    "    pcov[\"HESSE\"][i] = nh / nmc\n",
    "    pcov[\"MINOS\"][i] = nm / nmc\n",
    "    \n",
    "fig, ax = plt.subplots(1, 2, figsize=(10, 4))\n",
    "\n",
    "plt.sca(ax[0])\n",
    "n = np.arange(101)\n",
    "interval = {\n",
    "    \"HESSE\": np.empty((len(n), 2)),\n",
    "    \"MINOS\": np.empty((len(n), 2)),\n",
    "}\n",
    "for i, k in enumerate(n):\n",
    "    p, dp, pd, pu = m(k)\n",
    "    interval[\"HESSE\"][i] = (p - dp, p + dp)\n",
    "    interval[\"MINOS\"][i] = (pd, pu)\n",
    "\n",
    "for algo, vals in interval.items():\n",
    "    plt.plot(n, vals[:, 0] - n, color=\"C0\" if algo == \"HESSE\" else \"C1\", label=algo)\n",
    "    plt.plot(n, vals[:, 1] - n, color=\"C0\" if algo == \"HESSE\" else \"C1\", label=algo)\n",
    "plt.xlabel(\"$k$\")\n",
    "plt.ylabel(r\"$\\lambda^d - \\hat\\lambda$, $\\lambda^u - \\hat\\lambda$\")\n",
    "    \n",
    "plt.sca(ax[1])\n",
    "for algo, vals in pcov.items():\n",
    "    plt.plot(mu, vals, label=algo)\n",
    "    \n",
    "plt.axhline(0.68, ls=\":\", color=\"0.5\", zorder=0)\n",
    "plt.xlabel(r\"$\\lambda$\")\n",
    "plt.ylabel(\"coverage probability\")\n",
    "plt.legend()\n",
    "plt.semilogx();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this special case, the intervals found by both methods are very close. The MINOS interval is identical to the HESSE interval up to a small almost constant shift. Visually, the rate of converge to the asymptotic coverage probability of 68% seems to be equal for both methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can speak about the rate of convergence although we have drawn only a single observation from the Poisson pdf. The log-likelihood for $n$ observations with the same expectation is identical to the log-likelihood for one observation with an $n$-times higher expectation up to additive constants.\n",
    "\n",
    "$$\n",
    "\\ln L = \\sum_i^n (\\lambda_i - k_i \\ln \\lambda_i)\n",
    "= \\lambda - \\sum_i k_i (\\ln \\lambda - \\ln n)\n",
    "= \\lambda - k \\ln \\lambda + k \\ln n\n",
    "$$\n",
    "\n",
    "with $\\sum_i^n \\lambda_i = \\lambda$, $\\sum_i^n k_i = k$, and $\\lambda_i = \\lambda / n$. Therefore, the test cases with large values of $\\lambda$ correspond to large observation samples with a small constant $\\lambda$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fit of transformed normally distributed data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr>\n",
       "<td colspan=\"2\" title=\"Minimum value of function\">\n",
       "FCN = 3.122\n",
       "</td>\n",
       "<td align=\"center\" colspan=\"3\" title=\"No. of calls in last algorithm and total number of calls\">\n",
       "Ncalls = 58 (168 total)\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td colspan=\"2\" title=\"Estimated distance to minimum and target threshold\">\n",
       "EDM = 2.02e-05 (Goal: 0.0001)\n",
       "</td>\n",
       "<td align=\"center\" colspan=\"3\" title=\"Increase in FCN which corresponds to 1 standard deviation\">\n",
       "up = 0.5\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td align=\"center\" title=\"Validity of the migrad call\">\n",
       "Valid Min.\n",
       "</td>\n",
       "<td align=\"center\" title=\"Validity of parameters\">\n",
       "Valid Param.\n",
       "</td>\n",
       "<td align=\"center\" title=\"Is EDM above goal EDM?\">\n",
       "Above EDM\n",
       "</td>\n",
       "<td align=\"center\" colspan=\"2\" title=\"Did last migrad call reach max call limit?\">\n",
       "Reached call limit\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td align=\"center\" style=\"background-color:#92CCA6;\">\n",
       "True\n",
       "</td>\n",
       "<td align=\"center\" style=\"background-color:#92CCA6;\">\n",
       "True\n",
       "</td>\n",
       "<td align=\"center\" style=\"background-color:#92CCA6;\">\n",
       "False\n",
       "</td>\n",
       "<td align=\"center\" colspan=\"2\" style=\"background-color:#92CCA6;\">\n",
       "False\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td align=\"center\" title=\"Did Hesse fail?\">\n",
       "Hesse failed\n",
       "</td>\n",
       "<td align=\"center\" title=\"Has covariance matrix\">\n",
       "Has cov.\n",
       "</td>\n",
       "<td align=\"center\" title=\"Is covariance matrix accurate?\">\n",
       "Accurate\n",
       "</td>\n",
       "<td align=\"center\" title=\"Is covariance matrix positive definite?\">\n",
       "Pos. def.\n",
       "</td>\n",
       "<td align=\"center\" title=\"Was positive definiteness enforced by Minuit?\">\n",
       "Forced\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td align=\"center\" style=\"background-color:#92CCA6;\">\n",
       "False\n",
       "</td>\n",
       "<td align=\"center\" style=\"background-color:#92CCA6;\">\n",
       "True\n",
       "</td>\n",
       "<td align=\"center\" style=\"background-color:#92CCA6;\">\n",
       "True\n",
       "</td>\n",
       "<td align=\"center\" style=\"background-color:#92CCA6;\">\n",
       "True\n",
       "</td>\n",
       "<td align=\"center\" style=\"background-color:#92CCA6;\">\n",
       "False\n",
       "</td>\n",
       "</tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "------------------------------------------------------------------\n",
       "| FCN = 3.122                   |     Ncalls=58 (168 total)      |\n",
       "| EDM = 2.02e-05 (Goal: 0.0001) |            up = 0.5            |\n",
       "------------------------------------------------------------------\n",
       "|  Valid Min.   | Valid Param.  | Above EDM | Reached call limit |\n",
       "------------------------------------------------------------------\n",
       "|     True      |     True      |   False   |       False        |\n",
       "------------------------------------------------------------------\n",
       "| Hesse failed  |   Has cov.    | Accurate  | Pos. def. | Forced |\n",
       "------------------------------------------------------------------\n",
       "|     False     |     True      |   True    |   True    | False  |\n",
       "------------------------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr style=\"background-color:#F4F4F4;\">\n",
       "<td/>\n",
       "<th title=\"Variable name\">\n",
       "Name\n",
       "</th>\n",
       "<th title=\"Value of parameter\">\n",
       "Value\n",
       "</th>\n",
       "<th title=\"Hesse error\">\n",
       "Hesse Error\n",
       "</th>\n",
       "<th title=\"Minos lower error\">\n",
       "Minos Error-\n",
       "</th>\n",
       "<th title=\"Minos upper error\">\n",
       "Minos Error+\n",
       "</th>\n",
       "<th title=\"Lower limit of the parameter\">\n",
       "Limit-\n",
       "</th>\n",
       "<th title=\"Upper limit of the parameter\">\n",
       "Limit+\n",
       "</th>\n",
       "<th title=\"Is the parameter fixed in the fit\">\n",
       "Fixed\n",
       "</th>\n",
       "</tr>\n",
       "<tr style=\"background-color:#FFFFFF;\">\n",
       "<td>\n",
       "0\n",
       "</td>\n",
       "<td>\n",
       "cx\n",
       "</td>\n",
       "<td>\n",
       "0.11\n",
       "</td>\n",
       "<td>\n",
       "0.06\n",
       "</td>\n",
       "<td>\n",
       "-0.08\n",
       "</td>\n",
       "<td>\n",
       "0.05\n",
       "</td>\n",
       "<td>\n",
       "\n",
       "</td>\n",
       "<td>\n",
       "\n",
       "</td>\n",
       "<td>\n",
       "\n",
       "</td>\n",
       "</tr>\n",
       "<tr style=\"background-color:#F4F4F4;\">\n",
       "<td>\n",
       "1\n",
       "</td>\n",
       "<td>\n",
       "cy\n",
       "</td>\n",
       "<td>\n",
       "0.05\n",
       "</td>\n",
       "<td>\n",
       "0.10\n",
       "</td>\n",
       "<td>\n",
       "-0.11\n",
       "</td>\n",
       "<td>\n",
       "0.08\n",
       "</td>\n",
       "<td>\n",
       "\n",
       "</td>\n",
       "<td>\n",
       "\n",
       "</td>\n",
       "<td>\n",
       "\n",
       "</td>\n",
       "</tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "------------------------------------------------------------------------------------------\n",
       "|   | Name |   Value   | Hesse Err | Minos Err- | Minos Err+ | Limit-  | Limit+  | Fixed |\n",
       "------------------------------------------------------------------------------------------\n",
       "| 0 | cx   |   0.11    |   0.06    |   -0.08    |    0.05    |         |         |       |\n",
       "| 1 | cy   |   0.05    |   0.10    |   -0.11    |    0.08    |         |         |       |\n",
       "------------------------------------------------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr>\n",
       "<th title=\"Parameter name\">\n",
       "cx\n",
       "</th>\n",
       "<td align=\"center\" colspan=\"2\" style=\"background-color:#92CCA6;\">\n",
       "Valid\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td title=\"Lower and upper minos error of the parameter\">\n",
       "Error\n",
       "</td>\n",
       "<td>\n",
       "-0.08\n",
       "</td>\n",
       "<td>\n",
       "0.05\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td title=\"Validity of lower/upper minos error\">\n",
       "Valid\n",
       "</td>\n",
       "<td style=\"background-color:#92CCA6;\">\n",
       "True\n",
       "</td>\n",
       "<td style=\"background-color:#92CCA6;\">\n",
       "True\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td title=\"Did scan hit limit of any parameter?\">\n",
       "At Limit\n",
       "</td>\n",
       "<td style=\"background-color:#92CCA6;\">\n",
       "False\n",
       "</td>\n",
       "<td style=\"background-color:#92CCA6;\">\n",
       "False\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td title=\"Did scan hit function call limit?\">\n",
       "Max FCN\n",
       "</td>\n",
       "<td style=\"background-color:#92CCA6;\">\n",
       "False\n",
       "</td>\n",
       "<td style=\"background-color:#92CCA6;\">\n",
       "False\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td title=\"New minimum found when doing scan?\">\n",
       "New Min\n",
       "</td>\n",
       "<td style=\"background-color:#92CCA6;\">\n",
       "False\n",
       "</td>\n",
       "<td style=\"background-color:#92CCA6;\">\n",
       "False\n",
       "</td>\n",
       "</tr>\n",
       "</table>\n",
       "\n",
       "<table>\n",
       "<tr>\n",
       "<th title=\"Parameter name\">\n",
       "cy\n",
       "</th>\n",
       "<td align=\"center\" colspan=\"2\" style=\"background-color:#92CCA6;\">\n",
       "Valid\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td title=\"Lower and upper minos error of the parameter\">\n",
       "Error\n",
       "</td>\n",
       "<td>\n",
       "-0.11\n",
       "</td>\n",
       "<td>\n",
       "0.08\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td title=\"Validity of lower/upper minos error\">\n",
       "Valid\n",
       "</td>\n",
       "<td style=\"background-color:#92CCA6;\">\n",
       "True\n",
       "</td>\n",
       "<td style=\"background-color:#92CCA6;\">\n",
       "True\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td title=\"Did scan hit limit of any parameter?\">\n",
       "At Limit\n",
       "</td>\n",
       "<td style=\"background-color:#92CCA6;\">\n",
       "False\n",
       "</td>\n",
       "<td style=\"background-color:#92CCA6;\">\n",
       "False\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td title=\"Did scan hit function call limit?\">\n",
       "Max FCN\n",
       "</td>\n",
       "<td style=\"background-color:#92CCA6;\">\n",
       "False\n",
       "</td>\n",
       "<td style=\"background-color:#92CCA6;\">\n",
       "False\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td title=\"New minimum found when doing scan?\">\n",
       "New Min\n",
       "</td>\n",
       "<td style=\"background-color:#92CCA6;\">\n",
       "False\n",
       "</td>\n",
       "<td style=\"background-color:#92CCA6;\">\n",
       "False\n",
       "</td>\n",
       "</tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "-------------------------------------------------\n",
       "|       cx        |            Valid            |\n",
       "-------------------------------------------------\n",
       "|      Error      |    -0.08     |     0.05     |\n",
       "|      Valid      |     True     |     True     |\n",
       "|    At Limit     |    False     |    False     |\n",
       "|     Max FCN     |    False     |    False     |\n",
       "|     New Min     |    False     |    False     |\n",
       "-------------------------------------------------\n",
       "-------------------------------------------------\n",
       "|       cy        |            Valid            |\n",
       "-------------------------------------------------\n",
       "|      Error      |    -0.11     |     0.08     |\n",
       "|      Valid      |     True     |     True     |\n",
       "|    At Limit     |    False     |    False     |\n",
       "|     Max FCN     |    False     |    False     |\n",
       "|     New Min     |    False     |    False     |\n",
       "-------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "(-0.1, 0.2)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 49092 characters]\n",
      "text/plain": [
       "<Figure size 1008x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from iminuit import Minuit\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "from scipy.stats import multivariate_normal\n",
    "from argparse import Namespace\n",
    "\n",
    "rng = np.random.default_rng(1)\n",
    "\n",
    "truth = Namespace(cr=0.1, cphi=0, sr=0.1, sphi=2)\n",
    "truth.cx = truth.cr * np.cos(truth.cphi)\n",
    "truth.cy = truth.cr * np.sin(truth.cphi)\n",
    "\n",
    "d_r = rng.normal(truth.cr, truth.sr, size=5)\n",
    "d_phi = rng.normal(truth.cphi, truth.sphi, size=d_r.shape)\n",
    "\n",
    "cov = np.eye(2)\n",
    "cov[0,0] = truth.sr ** 2\n",
    "cov[1,1] = truth.sphi ** 2\n",
    "\n",
    "def nll(cx, cy):\n",
    "    cr = np.linalg.norm((cx, cy))\n",
    "    cphi = np.arctan2(cy, cx)\n",
    "    return -np.sum(multivariate_normal((cr, cphi), cov).logpdf(np.transpose((d_r, d_phi))))\n",
    "\n",
    "m = Minuit(nll, cx=0.1, cy=0, errordef=Minuit.LIKELIHOOD)\n",
    "m.migrad()\n",
    "m.hesse()\n",
    "m.minos()\n",
    "display(m.fmin, m.params)\n",
    "display(m.merrors)\n",
    "\n",
    "plt.figure(figsize=(14, 4))\n",
    "\n",
    "plt.subplot(121, polar=True)\n",
    "plt.plot(d_phi, d_r, \".\")\n",
    "\n",
    "plt.subplot(122, aspect=\"equal\")\n",
    "m.draw_mncontour(\"cx\", \"cy\", nsigma=1, numpoints=100)\n",
    "plt.plot(m.values[\"cx\"], m.values[\"cy\"], \"+k\")\n",
    "plt.xlim(-0.1, 0.2)\n",
    "plt.ylim(-0.1, 0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "from scipy.stats import multivariate_normal\n",
    "from argparse import Namespace\n",
    "from multiprocessing.pool import Pool\n",
    "from iminuit import Minuit\n",
    "\n",
    "truth = Namespace(cr=0.1, cphi=0, sr=0.1, sphi=2)\n",
    "truth.cx = truth.cr * np.cos(truth.cphi)\n",
    "truth.cy = truth.cr * np.sin(truth.cphi)\n",
    "truth.cov = np.eye(2)\n",
    "truth.cov[0,0] = truth.sr ** 2\n",
    "truth.cov[1,1] = truth.sphi ** 2\n",
    "\n",
    "n_tries = 50 # increase this to 500 get smoother curves (running will take a while)\n",
    "n_data = np.unique(np.geomspace(5, 100, 20, dtype=int))\n",
    "\n",
    "\n",
    "def compute(n):\n",
    "    rng = np.random.default_rng(seed=n)\n",
    "\n",
    "    n_h = 0\n",
    "    n_m = 0\n",
    "    h_lus = []\n",
    "    m_lus = []\n",
    "    xs = []\n",
    "    for i_try in range(n_tries):\n",
    "        while True:\n",
    "            d_r = rng.normal(truth.cr, truth.sr, size=n)\n",
    "            d_phi = rng.normal(truth.cphi, truth.sphi, size=n)\n",
    "\n",
    "            def nll(cx, cy):\n",
    "                cr = np.linalg.norm((cx, cy))\n",
    "                cphi = np.arctan2(cy, cx)\n",
    "                return -np.sum(multivariate_normal((cr, cphi), truth.cov).logpdf(np.transpose((d_r, d_phi))))\n",
    "\n",
    "            m = Minuit(nll, cx=0.1, cy=0, errordef=Minuit.LIKELIHOOD)\n",
    "            try:\n",
    "                m.migrad()\n",
    "                if not m.valid:\n",
    "                    continue\n",
    "\n",
    "                m.hesse()\n",
    "\n",
    "                if not m.accurate:\n",
    "                    continue\n",
    "\n",
    "                m.minos(\"cx\")\n",
    "                if m.merrors[\"cx\"].is_valid:\n",
    "                    break\n",
    "\n",
    "            except Exception as e:\n",
    "                print(f\"exception in n={n} i_try={i_try}\")\n",
    "                print(e)\n",
    "\n",
    "        x = m.values[\"cx\"]\n",
    "        dx = m.errors[\"cx\"]\n",
    "        me = m.merrors[\"cx\"]\n",
    "        h_lu = x - dx, x + dx \n",
    "        m_lu = x + me.lower, x + me.upper\n",
    "        if h_lu[0] < truth.cx < h_lu[1]:\n",
    "            n_h += 1\n",
    "        if m_lu[0] < truth.cx < m_lu[1]:\n",
    "            n_m += 1\n",
    "        xs.append(x)\n",
    "        h_lus.append(h_lu)\n",
    "        m_lus.append(m_lu)\n",
    "\n",
    "    x = np.mean(xs)\n",
    "    h_l, h_u = np.mean(h_lus, axis=0)\n",
    "    m_l, m_u = np.mean(m_lus, axis=0)\n",
    "    return n_h, n_m, x, h_l, h_u, m_l, m_u\n",
    "\n",
    "with Pool() as p:\n",
    "    n_h, n_m, x, hl, hu, ml, mu = np.transpose([x for x in p.map(compute, n_data)])\n",
    "\n",
    "h_pcov = n_h.astype(float) / n_tries\n",
    "m_pcov = n_m.astype(float) / n_tries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 27440 characters]\n",
      "text/plain": [
       "<Figure size 720x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(1, 2, figsize=(10, 4))\n",
    "plt.sca(ax[0])\n",
    "plt.fill_between(n_data, hl, hu, alpha=0.5, label=\"HESSE\")\n",
    "plt.fill_between(n_data, ml, mu, alpha=0.5, label=\"MINOS\")\n",
    "plt.plot(n_data, x, \"-k\")\n",
    "plt.legend()\n",
    "plt.xlabel(\"n\")\n",
    "plt.ylabel(\"cx\")\n",
    "plt.semilogx();\n",
    "plt.sca(ax[1])\n",
    "plt.plot(n_data, h_pcov, label=\"HESSE\")\n",
    "plt.plot(n_data, m_pcov, label=\"MINOS\")\n",
    "plt.axhline(0.68, ls=\":\", color=\"0.5\", zorder=0)\n",
    "plt.xlabel(r\"cx\")\n",
    "plt.ylabel(\"coverage probability\")\n",
    "plt.legend()\n",
    "plt.ylim(0, 1)\n",
    "plt.semilogx();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
