{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Defining a New Matrix\n",
    "\n",
    "This is a tutorial for Brightway2, an open source framework for Life Cycle Assessment. This tutorial will cover defining new matrices to be used in the LCA calculation - specifically, new matrices for weighting and normalization. \n",
    "\n",
    "The code from this example has **already been included** in Brightway2, and therefore for this tutorial you **don't** need to download this notebook, but can just read through it at your leisure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adding Weighting and Normalization to Brightway2\n",
    "\n",
    "The default impact assessment methods used in Brightway2 are processed by the ecoinvent centre, and already include weighting and normalization. That is why we have method names like `('eco-indicator 99, (I,I)', 'ecosystem quality', 'total')`.\n",
    "\n",
    "We want to define weighting and normalization separately from the characterization factors themselves so we can try using different weighting or normalization scenarios, and so we can apply uncertainty distributions.\n",
    "\n",
    "Brightway2 is designed to be easily extended, and we can add some weighting and normalization without much trouble. Lets start by defining some metadata classes for our weighting and normalization, similar to the [methods](https://bw2data.readthedocs.org/en/latest/metadata.html#bw2data.meta.Methods) class. The metadata classes store a list of all available weightings and normalizations, plus some additional data about each weighting and normalization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from bw2data.meta import Methods\n",
    "\n",
    "class WeightingMeta(Methods):\n",
    "    \"\"\"A dictionary for weighting metadata. File data is saved in ``methods.json``.\"\"\"\n",
    "    _filename = \"tutorial-weightings.json\"\n",
    "\n",
    "class NormalizationMeta(Methods):\n",
    "    \"\"\"A dictionary for normalization metadata. File data is saved in ``methods.json``.\"\"\"\n",
    "    _filename = \"tutorial-normalizations.json\"\n",
    "\n",
    "weightings = WeightingMeta()\n",
    "normalizations = NormalizationMeta()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we define the classes that will store the actual weighting and normalization data itself. These are based on the [Impact Assessment data store](http://bw2data.readthedocs.org/en/latest/data.html#impactassessmentdatastore) class.\n",
    "\n",
    "## Weighting\n",
    "\n",
    "Weighting is used to combine different impact categories - each category is weighted relative to the others. [Here is a good review of weighting methods](http://lct.jrc.ec.europa.eu/pdf-directory/ReqNo-JRC67215-LB-NA-24997-EN-N.pdf). As such, a weighting is just a number with an uncertainty distribution. The data format is simple:\n",
    "\n",
    "    {\n",
    "        \"amount\": float,  \n",
    "        \"uncertainty_type\": integer uncertainty type from stats_toolkit.uncertainty_choices (optional),\n",
    "        # .. plus some fields specific to the uncertainty distribution\n",
    "    }\n",
    "\n",
    "If you paid attention reading the documentation, you will recognize that this is simply an [uncertainty dictionary](http://brightway2.readthedocs.org/en/latest/key-concepts/data-formats.html#uncertainty-types-and-uncertainty-dictionaries).\n",
    "\n",
    "We define the data storage class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from bw2data.ia_data_store import ImpactAssessmentDataStore\n",
    "from bw2data.meta import weightings, mapping, normalizations\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "class Weighting(ImpactAssessmentDataStore):\n",
    "    \"\"\"LCIA weighting data - used to combine or compare different impact categories.\"\"\"\n",
    "    metadata = weightings\n",
    "    dtype_fields = []\n",
    "\n",
    "    def write(self, data):\n",
    "        \"\"\"Because of DataStore assumptions, need a one-element list\"\"\"\n",
    "        if not isinstance(data, list) or not len(data) == 1:\n",
    "            raise ValueError(\"Weighting data must be one-element list\")\n",
    "        super(Weighting, self).write(data)\n",
    "\n",
    "    def process_data(self, row):\n",
    "        \"\"\"Return a tuple of length two:\n",
    "        \n",
    "          * additional values for `dtype_fields` (we have none, so this is empty)\n",
    "          * a fixed number *or* an uncertainty dictionary, which is the input, so return it unchanged\n",
    "        \n",
    "        \"\"\"\n",
    "        return (\n",
    "            (), # don't know much,\n",
    "            row # but I know I love you\n",
    "        )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks confusingly simple, because the underlying [DataStore](http://bw2data.readthedocs.org/en/latest/data.html#datastore) and [ImpactAssessmentDataStore](http://bw2data.readthedocs.org/en/latest/data.html#impactassessmentdatastore) classes are doing much of the work for us.\n",
    "\n",
    "First, we specify the `metadata`: this tells us *where* to register each instance of `Weighting`. We defined the `weightings` metadata store above.\n",
    "\n",
    "Next, we specify what additional fields need to be added to our [processed array](http://brightway2.readthedocs.org/en/latest/key-concepts/processed.html). Usually we would have somehting like a biosphere flow, which would then get mapped to rows in the biosphere matrix. But as weightings are a single number, we don't need to know where they go - this will be a static value, not a matrix at all. Aside from our uncertainty dictionary, we don't need anything else, and the values for the uncertainty field are [added automatically](https://bitbucket.org/cmutel/brightway2-data/src/a6b58d9e440851db0cada2d804976e9793ddb3f0/bw2data/data_store.py?at=default#cl-98).\n",
    "\n",
    "Similary, the definition of `.process_data(row)` doesn't seem to do anything, because in this case we don't need to do anything. We will have a more complicated `process_data` method below when defining normalization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example weighting\n",
    "\n",
    "Let's make a simple weighting - the API is the same as for normal LCIA method datasets. For the distribution-specific uncertainty fields, see the [stats_arrays documentation](http://stats-arrays.readthedocs.org/en/latest/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from stats_arrays import NormalUncertainty\n",
    "\n",
    "really_important_data = [{\n",
    "    \"uncertainty_type\": NormalUncertainty.id,\n",
    "    \"amount\": 100,\n",
    "    \"sigma\": 10,\n",
    "    \"minimum\": 0\n",
    "}]\n",
    "\n",
    "ri = Weighting((\"really important\",))\n",
    "if (\"really important\",) not in weightings.list:\n",
    "    ri.register(description=\"Something we care about\")\n",
    "ri.write(really_important_data)\n",
    "ri.process()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Weightings [(u'really important',)]\n",
      "Metadata {u'abbreviation': u'reallyi-Wa2r00Hn', u'description': u'Something we care about'}\n",
      "Data {'amount': 100, 'minimum': 0, 'sigma': 10, 'uncertainty_type': 3}\n"
     ]
    }
   ],
   "source": [
    "print \"Weightings\", weightings.list\n",
    "print \"Metadata\", weightings[('really important',)]\n",
    "print \"Data\", ri.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalization\n",
    "\n",
    "Normalization is trickier, and to be perfectly honest, I don't always understand the motivation some types of normalization. The ISO 14042 says that normalization is calculating the \"magnitude of indicator results relative to reference information\". A lot of normalization steps are country-specific, e.g [Development of the U.S. Normalization Database](http://pubs.acs.org/doi/full/10.1021/es052494b). In any case, normalization is different from weighting, in that the values are **biosphere flow**-specific.\n",
    "\n",
    "The data format is therefore a list of flows, plus their (potentially uncertain) normalization factors, \n",
    "\n",
    "    [\n",
    "        [flow tuple, e.g. (\"biosphere\", \"cadmium\"), uncertainty dictionary],\n",
    "    ]\n",
    "\n",
    "As opposed to weighting, with normalization we will have a list with many different elements, one for each normalized flow.\n",
    "\n",
    "We define the data storage class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from bw2data.utils import MAX_INT_32\n",
    "\n",
    "\n",
    "class Normalization(ImpactAssessmentDataStore):\n",
    "    \"\"\"\n",
    "    LCIA normalization data - used to transform meaningful units, like mass or damage, into \"person-equivalents\" or some such thing.\n",
    "\n",
    "    The data schema for IA normalization is:\n",
    "\n",
    "    .. code-block:: python\n",
    "\n",
    "            Schema([\n",
    "                [valid_tuple, maybe_uncertainty]\n",
    "            ])\n",
    "\n",
    "    where:\n",
    "        * ``valid_tuple`` is a dataset identifier, like ``(\"biosphere\", \"CO2\")``\n",
    "        * ``maybe_uncertainty`` is either a number or an uncertainty dictionary\n",
    "\n",
    "    \"\"\"\n",
    "    metadata = normalizations\n",
    "    dtype_fields = [\n",
    "        ('flow', np.uint32),  # 32 bit unsigned integer\n",
    "        ('index', np.uint32),\n",
    "    ]\n",
    "\n",
    "    def add_mappings(self, data):\n",
    "        \"\"\"Add each normalization flow (should be biosphere flows) to global mapping\"\"\"\n",
    "        mapping.add({obj[0] for obj in data})\n",
    "\n",
    "    def process_data(self, row):\n",
    "        \"\"\"Return values that match `dtype_fields`, as well as number or uncertainty dictionary\"\"\"\n",
    "        return (\n",
    "            mapping[row[0]],  # Integer number corresponding to biosphere flow\n",
    "            MAX_INT_32,       # Will be replaced with matrix row number\n",
    "            ), row[1]         # Actual certain/uncertain value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of these fields are similar to weighting, and indeed most new matrices will be similar, as we are doing the same thing over and over - storing data, and then putting it into matrices in a reasonable fashion. Let's look at the new elements:\n",
    "\n",
    "First, `dtype_fields` is no longer empty. Instead, there are two fields, `flow` and `index`. Flow in this case means biosphere flows. Because of the way structured arrays work, this needs to be a integer, so we can't insert something like `(\"biosphere\", \"CO2\")`. Instead, we keep a global mapping of biosphere and technosphere flows to integers - this is what `mapping` is, a dictionary from flows to integers, which is in a sense just a giant counter. Each new flow inserted into `mapping` gets the next biggest integer. `index` will be the index in the matrix, e.g. row 0 or row 42.\n",
    "\n",
    "You can now guess what `add_mappings` does - you could define normalization factors for completely new biosphere flows. `mapping` needs to know about all flows, so we add all flows in each normalization method (flows already existing are ignored by `mapping`).\n",
    "\n",
    "For each `dtype_field`, we need to define what kind of number to use - in this case, we choose a 32 bit unsigned (i.e. implicitly positive) integer.\n",
    "\n",
    "In `process_data`, we take as an input a single row from the normalization dataset, and return the values that go into the `dtype_fields` we define above, as well as the actual data value. In this case, for `flow` we in the integer number given by looking our biosphere flow up in `mapping`. For `index`, we insert a dummy value, as this will be determined dynamically as we build the matrix (we don't know how big the normalization matrix is yet).\n",
    "\n",
    "Details of structured arrays, and how matrices are built, are covered more in the [brightway2-calc](https://brightway2-calc.readthedocs.org/en/latest/#building-matrices) documentation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example normalization\n",
    "\n",
    "Again, we make a simple normalization dataset, with made up numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "greenhouse_gases = [\n",
    "    {\"amount\": 4000, \"flow\": (u'biosphere', u'6382dd23b5ac86860bdc9951ab449777')},  # Carbon dioxide, fossil\n",
    "    {\"amount\": 0.25, \"flow\": (u'biosphere', u'34abd8a0c832e8bc96ef5e560b574a05')}   # Dinitrogen monoxide\n",
    "]\n",
    "\n",
    "gg = Normalization((\"some greenhouse gases\",))  # Names have to be a tuple, just like IA methods\n",
    "if (\"some greenhouse gases\",) not in normalizations.list:\n",
    "    gg.register(description=\"Some like it hot\")\n",
    "gg.write(greenhouse_gases)\n",
    "gg.process()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normalizations [(u'some gases',), (u'some greenhouse gases',)]\n",
      "Metadata {u'abbreviation': u'somegg-o83vbiDK', u'description': u'Some like it hot'}\n",
      "Data [{'amount': 4000, 'flow': (u'biosphere', u'6382dd23b5ac86860bdc9951ab449777')}, {'amount': 0.25, 'flow': (u'biosphere', u'34abd8a0c832e8bc96ef5e560b574a05')}]\n"
     ]
    }
   ],
   "source": [
    "print \"Normalizations\", normalizations.list\n",
    "print \"Metadata\", normalizations[('some greenhouse gases',)]\n",
    "print \"Data:\"\n",
    "gg.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Applying to LCA\n",
    "\n",
    "Now we need to use all the data managers we just defined in an actual LCA. We do this by creating a subclass of `LCA`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from bw2calc.lca import LCA\n",
    "from bw2calc.matrices import MatrixBuilder\n",
    "from bw2calc.utils import load_arrays\n",
    "\n",
    "class ComplicatedLCA(LCA):\n",
    "    def __init__(self, demand, method=None, weighting=None,\n",
    "                 normalization=None, config=None):\n",
    "        super(ComplicatedLCA, self).__init__(  # Do all the other initialization stuff\n",
    "            demand, \n",
    "            method=method, \n",
    "            config=config\n",
    "        )\n",
    "        self.weighting = weighting             # Our weighting method\n",
    "        self.normalization = normalization     # Our normalization method\n",
    "\n",
    "    def load_normalization_data(self, builder=MatrixBuilder):\n",
    "        \"\"\"Load normalization data.\"\"\"\n",
    "        self.normalization_params, _, _, self.normalization_matrix = \\  # _ is common python shorthand for an ignored value\n",
    "            builder.build(\n",
    "                self.dirpath,\n",
    "                [Normalization(self.normalization).filename],  # Filename of processed array\n",
    "                \"amount\",\n",
    "                \"flow\",\n",
    "                \"index\",\n",
    "                row_dict=self.biosphere_dict,\n",
    "                one_d=True\n",
    "            )\n",
    "\n",
    "    def load_weighting_data(self):\n",
    "        \"\"\"Load weighting data, a 1-element array.\"\"\"\n",
    "        self.weighting_params = load_arrays(\n",
    "            self.dirpath,\n",
    "            [Weighting(self.weighting).filename]\n",
    "        )\n",
    "        self.weighting_value = self.weighting_params['amount']\n",
    "\n",
    "    def normalize(self):\n",
    "        assert hasattr(self, \"characterized_inventory\"), \\\n",
    "            \"Must do LCI and LCIA before normalization\"\n",
    "        self.load_normalization_data()\n",
    "        self.normalized_inventory = \\\n",
    "            self.normalization_matrix * self.characterized_inventory\n",
    "\n",
    "    def weight(self):\n",
    "        assert hasattr(self, \"normalized_inventory\"), \\\n",
    "            \"Must do LCI, LCIA, and normalization before weighting\"\n",
    "        self.load_weighting_data()\n",
    "        self.weighted_inventory = \\\n",
    "            self.weighting_data['amount'][0] * self.normalized_inventory  # Just multiplying by a weighting value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hopefully the comments in the source code are self-explanatory. See the above mention brightway2-calc documentation, and especially  [MatrixBuilder](https://brightway2-calc.readthedocs.org/en/latest/matrix.html#matrix-builder), for details on how to turn processed arrays into matrices.\n",
    "\n",
    "We can then apply our weighting and normalization to a inventory dataset from ecoinvent 2.2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normalized and weighted score: 400025.0\n"
     ]
    }
   ],
   "source": [
    "lca = ComplicatedLCA(\n",
    "    demand={Database('ecoinvent 2.2').random(): 1}, \n",
    "    method=(u'IPCC 2001', u'climate change', u'GWP 500a'),\n",
    "    weighting=(\"really important\",),\n",
    "    normalization=(\"some greenhouse gases\",)\n",
    ")\n",
    "lca.lci()\n",
    "lca.lcia()\n",
    "lca.normalize()\n",
    "lca.weight()\n",
    "print \"Normalized and weighted score:\", lca.weighted_inventory.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, this result doesn't mean anything, because our normalization and weighting figures were made up."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusions\n",
    "\n",
    "In a relatively few lines of code, we added the normalization and weighting steps, including the ability to give these values uncertainty distributions. Because Brightway2 is open source and flexible, you can add similar functionality in ways that meet your specific calculation needs - you have the freedom to add or remove any functionality you want.\n",
    "\n",
    "* Brightway2 makes it relatively easy to add extra steps like weighting and normalization to impact assessment.\n",
    "* Brightway2 base classes like `ImpactAssessmentDataStore` and `MatrixBuilder` are made to be easily extended.\n",
    "* You can add whatever additional steps or transformations you want - you have the freedom to define and use new data stores in creative ways."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
