{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Results Object Tutorial\n",
    "This tutorial explains the structure and usage of the \"model estimate results\" objects (derived from `pygsti.protocols.ProtocolResults`).  These objects are returned from a `Protocol` object's `.run(...)` method and thus constitute the main way results are communicated back to the user in pyGSTi.  A `ProtocolResults` object is used to store results corresponding to a given *experiment design* and its accompanying data (packaged together into a `ProtocolData` object).  A result object's `.data` attribute holds the `ProtocolData` associated with it, and its `.protocol` attribute holds the `Protocol` that created the results. \n",
    "\n",
    "\n",
    "## `pygsti.protocols.ModelEstimateResults`\n",
    "A `ModelEstimateResults` object stores the output of protocols that contain `Model` objects and accompanying data.  The results object stores references to the protocol that generated it and the experiment design and data set it was generated from.\n",
    "\n",
    "As a concrete example, we'll explore one of the `ModelEstimateResults` objects generated by the [GST protocols tutorial](../../algorithms/GST-Protocols.ipynb) (so **if you haven't run this tutorial, go do it now**)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pygsti"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------\n",
      "----------- pyGSTi ModelEstimateResults Object -----------\n",
      "----------------------------------------------------------\n",
      "\n",
      "How to access my contents:\n",
      "\n",
      " .dataset    -- the DataSet used to generate these results\n",
      "\n",
      " .circuit_lists   -- a dict of Circuit lists w/keys:\n",
      " ---------------------------------------------------------\n",
      "  iteration\n",
      "  prep fiducials\n",
      "  meas fiducials\n",
      "  germs\n",
      "  final\n",
      "\n",
      " .estimates   -- a dictionary of Estimate objects:\n",
      " ---------------------------------------------------------\n",
      "  GateSetTomography\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "results = pygsti.io.read_results_from_dir(\"../../tutorial_files/Example_GST_Data\",\"GateSetTomography\")\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, printing a `ModelEstimateResults` object gives you a summary of its structure and what you can do with it.  The *single* `DataSet` can be accessed via the `.dataset` member, and the estimated `Model` objects can be found within the `pygsti.report.Estimate` objects contained within the `.estimates` member.  As the summary states, `.estimates` is a *dictionary* of `Estimate` objects, and **can contain as multiple estimates of the data *with the caveat* that all of these estimates must use the same experiment design** (roughly, the same `Circuit` lists and/or `CircuitStructure`s for each algorithm iteration, and the same number of iterations).\n",
    "\n",
    "## `pygsti.report.Estimate`\n",
    "The `Estimate` objects represent different *gauge-unfixed* or *\"up-to-gauge\"* estimates, and each holds one or more `Model` and associated `ConfidenceRegion` objects, and dictionaries containing the parameters used to generate the estimate.  They also may contain **multiple gauge-optimized \"versions\" of their single gauge-unfixed estimate**.  They can be printed to display a summary of their contents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------\n",
      "---------------- pyGSTi Estimate Object ------------------\n",
      "----------------------------------------------------------\n",
      "\n",
      "How to access my contents:\n",
      "\n",
      " .models   -- a dictionary of Model objects w/keys:\n",
      " ---------------------------------------------------------\n",
      "  target\n",
      "  seed\n",
      "  iteration 0 estimate\n",
      "  iteration 1 estimate\n",
      "  iteration 2 estimate\n",
      "  iteration 3 estimate\n",
      "  iteration 4 estimate\n",
      "  iteration 5 estimate\n",
      "  iteration 6 estimate\n",
      "  iteration 7 estimate\n",
      "  final iteration estimate\n",
      "  stdgaugeopt\n",
      "\n",
      " .goparameters   -- a dictionary of gauge-optimization parameter dictionaries:\n",
      " ---------------------------------------------------------\n",
      "  stdgaugeopt\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(results.estimates['GateSetTomography'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Estimate` objects do *not* store the operation sequences - rather, since these must be the same for all the estimates of a `Results` object, the `Results` object holds them separately in its `.circuit_lists` member (a dictionaries like `.estimates`).  Furthermore, since varying the gauge optimization parameters is such a common variation, a single `Estimate` may hold *multiple* dictionaries of gauge-optimization parameters as the elements of its `.goparameters` dictionary.  **The keys of `goparameters` will and must correspond to keys within the `.models` member** (the `Model` estimate of that gauge optimization).  The `.confidence_regions` dictionary (empty in the above example and so not included in the summary) holds `ConfidenceRegion` objects, each associated with 1) one the `Model`s in `.models`, 2) one of the `Circuit` lists in the parent `Results` object's `.circuit_lists`, and 3) a confidence level.  The keys of `.confidence_regions` are complicated because they must include all three of these associations, and so the `.confidence_region(...)` method is preferred to directly accessing `.confidence_regions`.  Different `Estimate` objects typically hold estimates for different model parameterizations and/or algorithm parameters.\n",
    "\n",
    "In our example, `results` contains only a single `Estimate` called `GateSetTomography` (the default estimate label created by a `pygsti.protocols.GateSetTomography` protocol).  This `Estimate` contains the raw un-gauge-optimized `Model` labeled `\"final iteration estimate\"`, as well as a single gauge-optimized `Model` labeled `stdgaugeopt` (again, the default created by a GST protocol).\n",
    "\n",
    "## Rolling your own...\n",
    "\n",
    "Creating your own `ModelEstimateResults` object is typically done within a custom `Protocol` object's `.run(...)` method.  In this context, you create and populate the results object as follows:\n",
    "1. call `pygsti.protocol.ModelEstimateResults(data, my_protocol)` to create the empty object.  Here `data` is almost always the data object passed to the protocol's `.run` method, and `my_protocol` is the protocol object itself (i.e., `self`).\n",
    "2. create contained `Estimate` objects by calling `add_estimate` with the essential components of a new gauge-unfixed estimate (the target model, the starting model, the estimated models by iteration, and the parameter dictionary).\n",
    "\n",
    "This is demonstrated with dummy parameters below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyProtocol(pygsti.protocols.Protocol):\n",
    "    def __init__(self, depol_amt, name=None):\n",
    "        super().__init__(name)\n",
    "        self.depol_amt = depol_amt\n",
    "    \n",
    "    def run(self, data, target_model):\n",
    "        assert(isinstance(data.edesign, pygsti.protocols.GateSetTomographyDesign)) # a GST-like protocol\n",
    "        edesign = data.edesign\n",
    "        nIters = len(edesign.maxlengths)  # number of GST iterations\n",
    "        my_estimate = target_model.depolarize(op_noise=self.depol_amt)  # for example...\n",
    "        my_estimate_by_iter = [my_estimate]*nIters  # same estimate for each iteration\n",
    "        \n",
    "        res = pygsti.protocols.ModelEstimateResults(data, self)\n",
    "        my_parameters = {'depol_amt': self.depol_amt }\n",
    "        est = pygsti.protocols.estimate.Estimate.create_gst_estimate(res, target_model, my_estimate,\n",
    "                                                 my_estimate_by_iter, my_parameters)\n",
    "        res.add_estimate(est, estimate_key=\"myTestEstimate\")\n",
    "        return res\n",
    "\n",
    "data = results.data  # use same data as results loaded in above \n",
    "my_results = MyProtocol(0.01).run(data, target_model= results.estimates['GateSetTomography'].models['target'])\n",
    "print(my_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a new gauge-optimized estimate\n",
    "In many circumstances, one may want to perform a new gauge optimization on an existing gauge-unfixed `Estimate`, `est`, creating a new gauge-optimized `Model` to be stored in `est`.  This is accomplished using the `est.add_gaugeoptimized` which lightly wraps a call to `pygsti.gaugeopt_to_target`.  You specify the arguments to `gaugeopt_to_target` as a dictionary to `add_gaugeoptimized`, but you're allowed to leave out the first two: the `Model` to be optimized, (`model`, taken to be the `est.models['final iteration estimate']`) and the model to optimize toward (`target_model`, taken to be the `est.models['target']`).  Note that these arguments *can* still be specified to override their defaults.  In particular, **setting `target_model` in the dictionary of parameters allows one to independently specify the model to optimize toward** (*and this need not be a perfect, ideal model!*).\n",
    "\n",
    "The optional `label` argument of `add_gaugeoptimized` specifies the key within `est.goparameters` and `est.models` where the gauge optimization argument dictionary and resulting gauge-optimized `Model` will be stored.  If the label given already exists, that gauge-optimized estimate is replaced with the new one.  If `label` is left as `None`, then \"go*X*\" is used as the label, where *X* is the next available integer.\n",
    "\n",
    "If the `model` argument of `add_gaugeoptimized` is supplied, then this is taken to be the result of the described gauge optimization and no call to `gaugeopt_to_target` is made.  (In this case, one could simply pass an empty dictionary of as `goparams`.)\n",
    "\n",
    "Below we demonstrate how to add gauge-optimized models to an `Estimate` in several ways.  Please refer to the previous tutorial on low-level algorithms for an explanation of the various arguments to `gaugeopt_to_target`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "est = results.estimates['GateSetTomography']\n",
    "est.add_gaugeoptimized({'item_weights': {'gates': 1.0, 'spam': 1.0}}, label=\"equal_footing\")\n",
    "est.add_gaugeoptimized({'item_weights': {'gates': 1.0, 'spam': 1.0, 'Gx': 10.0}}, label=\"Gx_heavy\")\n",
    "\n",
    "mdl_target = est.models['target'].copy()\n",
    "mdl_target.set_all_parameterizations('full') # So that we can change the ops\n",
    "mdl_guess = mdl_target.depolarize(op_noise=0.05, spam_noise=0.02) # a guess at what gates should be...\n",
    "est.add_gaugeoptimized({'target_model': mdl_guess, 'item_weights': {'spam': 0.01}}, label=\"imperfect gopt target\")\n",
    "\n",
    "print(est)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Confidence Region Factories\n",
    "In order to compute confidence regions and intervals within reports (see later tutorials), an `Estimate` object must be equipped with one or more \"confidence region factory\" objects.  These factories are instances of `pygsti.objects.ConfidenceRegionFactory` (suprise, suprise).  Their purpose is to generate confidence regions and intervals (for *any* confidence level) for quantities computed from a particular `Model` that in turn resulted from optimizing the likelihood function corresponding to a particular set of circuits.  Thus, a confidence region factory has associated with it three things: 1) a `Model`, 2) a list of `Circuit`s, and 3) a `DataSet`.  A dictionary of factories is held as the `.confidence_region_factories` member of an `Estimate` object.  Each factory within this dictionary is associated with the one-and-only `DataSet` of the `Estimate`'s parent `ModelEstimateResults` object, and the associated `Model` and `Circuit` list are given by the keys of `.confidence_region_factories` (*model-key*, *circuit-list-key* tuples).  Here *model-key* is the key of a `Model` within the `Estimate`'s `.models` member and *circuit-list key* is the key of a list within the parent `ModelEstimateResults` object's `.circuit_lists` member.\n",
    "\n",
    "Thankfully, you won't usually need to deal with the `.confidence_region_factories` member directly. To create a new factory for a given `Model`, `Circuit`-list pair you can simply call the `add_confidence_region_factory` with the appropriate key labels.  Once a factory is created, it must be initialized for computing confidence regions.  The only non-experimental way to do this currently is to compute the Hessian of the log-likelihood (often computationally intensive) and then projecting the inverse of this Hessian onto the non-gauge space of the model.  These two steps are performed via the `compute_hessian` and `project_hessian` member functions of a `ConfidenceRegionFactory` object.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_label = \"stdgaugeopt\"\n",
    "clist_label = \"final\"\n",
    "crfactory = results.estimates['GateSetTomography'].add_confidence_region_factory(model_label, clist_label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that there are different ways of projecting the Hessian which have different strengths and weakenesses.  The \"optimal gate CIs\" method is the most robust method for giving the smallest error bars possible, but it takes significant computation time.  The \"intrinsic error\" method is fast and usually reliable, but may not always give the smallest possible error bars."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "crfactory.compute_hessian(comm=None) #could use lots of processors here...\n",
    "inv_proj_H = crfactory.project_hessian('intrinsic error')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Alternate way**: In the special case of constructing factories for `Model`s which are gauge-equivalent to one another, one can skip the `compute_hessian` step for all but the first `Model`, so long as the gauge optimization parameters *and* the final gauge-tranformation element are stored in the `Estimate`s `.goparameters` dictionary (automatically populated when adding a gauge optimization via `add_gaugeoptimized`).  Instead, one must *gauge-propagate* the Hessian from the first `Model` to the others using the `gauge_propagate_confidence_region_factory` method of the `Estimate` object.\n",
    "\n",
    "Below, we show how this might usually be done: first a confidence region factory for the \"final iteration estimate\" `Model` and 'final' operation sequence list (the defaults) is created and a Hessian is computed.  Then, when a factory is needed for the gauge-equivalent `Model` \"go0\", the Hessian is propagated from the \"final iteration estimate\" `Model`.  Note that the propagated Hessian must still be projected for the \"go0\" model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "crfact_final = results.estimates['GateSetTomography'].add_confidence_region_factory() #default 'final iteration estimate'\n",
    "crfact_final.compute_hessian(comm=None)\n",
    "\n",
    "crfact_stdgo = results.estimates['GateSetTomography'].gauge_propagate_confidence_region_factory('stdgaugeopt', verbosity=1) #instead of computing one\n",
    "inv_proj_H = crfact_stdgo.project_hessian('intrinsic error')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "In summary, when thinking about `ProtocolResults`, `ModelEstimateResults`, and `Estimate` objects, remember:\n",
    "- each **`ProtocolResults`** object represents the **results for a single set of data (experiment design)**.\n",
    "- each contained **`Estimate`** object represents a **single *gauge-unfixed* estimate** based on the data.  An `Estimate` may also contain **one or more *gauge-optimized* versions** of the gauge-invariant estimate.  \n",
    "- an `Estimate` can construct confidence intervals only after a **`ConfidenceRegionFactory`** object is created and initialized using a multi-step process.  Because it may be computationally expensive, these steps are *not* performed automatically when reports are generated.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
