{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Importing Ecoinvent\n",
    "\n",
    "This noteboook shows you how to import the various flavors and versions of ecoinvent, and how to resolve any problems that occur."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from brightway2 import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic setup\n",
    "\n",
    "Start a new project, and install base data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "projects.set_current(\"ecoinvent-import2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating default biosphere\n",
      "\n",
      "Applying strategy: drop_unspecified_subcategories"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Writing activities to SQLite3 database:\n",
      "0%                          100%\n",
      "[##############################] | ETA[sec]: 0.000 \n",
      "Total time elapsed: 0.341 sec\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Title: Writing activities to SQLite3 database:\n",
      "  Started: 05/21/2015 12:30:27\n",
      "  Finished: 05/21/2015 12:30:28\n",
      "  Total time elapsed: 0.341 sec\n",
      "  CPU %: 96.900000\n",
      "  Memory %: 0.798309\n",
      "Created database: biosphere3\n",
      "Creating default LCIA methods\n",
      "\n",
      "Applying strategy: set_biosphere_type\n",
      "Applying strategy: drop_unspecified_subcategories\n",
      "Applying strategy: link_iterable_by_fields\n",
      "Wrote 692 LCIA methods with 170915 characterization factors\n",
      "Creating core data migrations\n",
      "\n"
     ]
    }
   ],
   "source": [
    "bw2setup()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ecoinvent 2.2\n",
    "\n",
    "Importing ecoinvent 2.2 is easy and relatively fast."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ei22 = SingleOutputEcospold1Importer(\n",
    "    \"/Users/cmutel/Documents/LCA Documents/Ecoinvent/2.2/processes\",\n",
    "    \"ecoinvent 2.2\"\n",
    ")\n",
    "ei22.apply_strategies()\n",
    "ei22.statistics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All exchanges are linked, so we can write the database:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ei22.write_database()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To reduce memory use, we remove the stored copy by setting the importer to `None`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ei22 = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ecoinvent 3.1 cutoff\n",
    "\n",
    "The cutoff system model is similar to the ecoinvent 1.x and 2.x system model, and its easy linking algorithm means that there are no problems when importing it:\n",
    "\n",
    "**When downloading files from ecoinvent.org, take one of the following** (from the 'Files' tab):\n",
    "\n",
    "* current_Version_3.1_allocation_default_ecoSpold02.7z\n",
    "* current_Version_3.1_consequential_longterm_ecoSpold02.7z\n",
    "* current_Version_3.1_cutoff_ecoSpold02.7z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ei31cutoff = SingleOutputEcospold2Importer(\n",
    "    \"/Users/cmutel/Documents/LCA Documents/Ecoinvent/3.1/cutoff/datasets\",\n",
    "    \"ecoinvent 3.1 cutoff\"\n",
    ")\n",
    "ei31cutoff.apply_strategies()\n",
    "ei31cutoff.statistics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, no linking errors, so we can write the database:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ei31cutoff.write_database()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ecoinvent 3.1 default\n",
    "\n",
    "We can do the same thing with the default system model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Extracting ecospold2 files:\n",
      "0%                          100%\n",
      "[##############################] | ETA[sec]: 0.000 | Item ID: ffed8e5b-8ecb-4\n",
      "Total time elapsed: 104.858 sec\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Title: Extracting ecospold2 files:\n",
      "  Started: 05/18/2015 11:15:39\n",
      "  Finished: 05/18/2015 11:17:24\n",
      "  Total time elapsed: 104.858 sec\n",
      "  CPU %: 87.300000\n",
      "  Memory %: 7.371628\n",
      "Extracted 11329 datasets in 105.83 seconds\n",
      "Applying strategy: remove_zero_amount_coproducts\n",
      "Applying strategy: remove_zero_amount_inputs_with_no_activity\n",
      "Applying strategy: es2_assign_only_product_with_amount_as_reference_product\n",
      "Applying strategy: assign_single_product_as_activity\n",
      "Applying strategy: create_composite_code\n",
      "Applying strategy: drop_unspecified_subcategories\n",
      "Applying strategy: link_biosphere_by_flow_uuid\n",
      "Applying strategy: link_internal_technosphere_by_composite_code\n",
      "Applying strategy: delete_exchanges_missing_activity\n",
      "Applying strategy: delete_ghost_exchanges\n",
      "116 exchanges couldn't be linked and were deleted. See the logfile for details:\n",
      "\t/Users/cmutel/Library/Application Support/Brightway3/ecoinvent-import.99d620c13d8ce5ab521afdd7a1c6ab75/logs/Ecospold2-import-error.bVBLjP.log\n",
      "11329 datasets\n",
      "806149 exchanges\n",
      "0 unlinked exchanges\n",
      "  \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(11329, 806149, 0)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ei31default = SingleOutputEcospold2Importer(\n",
    "    \"/Users/cmutel/Documents/LCA Documents/Ecoinvent/3.1/default/datasets\",\n",
    "    \"ecoinvent 3.1 default\"\n",
    ")\n",
    "ei31default.apply_strategies()\n",
    "ei31default.statistics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, we have no linking errors are applying the default strategies, but we do have some complaints. Let's look at the logfile:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Purging unlinked exchange:\n",
      "Filename: 064b8ab0-3be5-4461-aeda-fa8cdc91d150_96f36f31-8f6a-48f8-921f-1d781b7a545f.spold\n",
      "{ 'activity': '22265c57-a1cf-4aab-add0-3bd6c4ca0110',\n",
      "'amount': 0.00790582069651522,\n",
      "'comment': 'Calculated value, acording to paper mass ratio at factory. '\n",
      "'Biogas (from sanitary landfill) burning provides most of the '\n",
      "'heat on site. A direct link to the dataset \"heat and power '\n",
      "'co-generation, biogas, gas engine, Quebec\" is made to ensure '\n",
      "'this fuel specificity. This dataset does not exactly reflect '\n",
      "'this specificity, as there is no co-generation but only heat '\n",
      "'production, but it is the most suitable is the current version '\n",
      "'of the database. ',\n",
      "'flow': 'dd80f0f2-f4d5-40f0-9035-09c1a7f3f07b',\n",
      "'loc': 9.07,\n",
      "'name': 'heat, central or small-scale, other than natural gas',\n",
      "'pedigree': { 'completeness': 1,\n",
      "'further technological correlation': 1,\n",
      "'geographical correlation': 5,\n",
      "'reliability': 1,\n",
      "'temporal correlation': 3},\n",
      "'production volume': 0.0,\n",
      "'scale': 0.0046,\n",
      "'scale without pedigree': 0.0006,\n",
      "'type': 'technosphere',\n",
      "'uncertainty type': 2,\n",
      "'unit': 'megajoule'}\n"
     ]
    }
   ],
   "source": [
    "fp = \"/Users/cmutel/Library/Application Support/Brightway3/ecoinvent-import.99d620c13d8ce5ab521afdd7a1c6ab75/logs/Ecospold2-import-error.bVBLjP.log\"\n",
    "\n",
    "with open(fp) as f:\n",
    "    lines = [x.strip() for x in f]\n",
    "\n",
    "for line in lines[:26]:\n",
    "    print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, so in file `064b8ab0-3be5-4461-aeda-fa8cdc91d150_96f36f31-8f6a-48f8-921f-1d781b7a545f.spold` there is a link to a flow `dd80f0f2-f4d5-40f0-9035-09c1a7f3f07b` from activity `22265c57-a1cf-4aab-add0-3bd6c4ca0110`. The problem is that activity `22265c57-a1cf-4aab-add0-3bd6c4ca0110` doesn't produce this flow. We can see the flows it produces from the filenames, which are `{activity}-{flow}.spold`. Here are the available flows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22265c57-a1cf-4aab-add0-3bd6c4ca0110_ad551fe0-84c5-471a-9acf-b7d2204fdb65.spold\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "for filename in os.listdir(\"/Users/cmutel/Documents/LCA Documents/Ecoinvent/3.1/default/datasets\"):\n",
    "    if \"22265c57-a1cf-4aab-add0-3bd6c4ca0110\" in filename:\n",
    "        print(filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So activity `22265c57-a1cf-4aab-add0-3bd6c4ca0110` only produces flow `ad551fe0-84c5-471a-9acf-b7d2204fdb65`.\n",
    "\n",
    "If we look at the spold file, we can see details on this activity:\n",
    "\n",
    "      <activity \n",
    "        id=\"22265c57-a1cf-4aab-add0-3bd6c4ca0110\"\n",
    "        activityNameId=\"982cad19-2f9d-4792-b78c-98b1bf2e107a\"\n",
    "        parentActivityId=\"46bffdb4-30de-47b3-91f8-1568c9bb1fdf\" \n",
    "        inheritanceDepth=\"-1\" \n",
    "        type=\"1\"\n",
    "        specialActivityType=\"0\">\n",
    "        <activityName xml:lang=\"en\">treatment of manure and biowaste by anaerobic\n",
    "          digestion</activityName>\n",
    "        <includedActivitiesStart xml:lang=\"en\">Input of livestock manure (cattle slurry, pig slurry,\n",
    "          cattle manure) and biowaste (agroindustrial waste, municipal biowaste), used edible oil\n",
    "          and glycerine) to incoming storage at the biogas plant. </includedActivitiesStart>\n",
    "        <includedActivitiesEnd xml:lang=\"en\">The treatment includes storage (and 10% of the total\n",
    "          pre-treatment storage emissions) of the livestock manure and cosubstrates, anerobic\n",
    "          fermentation, as well as the storage after fermentation. The activity ends with the biogas\n",
    "          and digestate being available at the biogas plant. Due to the absence of reliable\n",
    "          references and pertinent data, H2S emissions during substrate storage are not taken into\n",
    "          account in the present study.</includedActivitiesEnd>\n",
    "        <generalComment>\n",
    "          <text xml:lang=\"en\" index=\"1\">This multi-output activity produces biogas and digestate\n",
    "            from manure and biowaste. The methane content of the biogas is calculated depending on\n",
    "            the substrate input.</text>\n",
    "        </generalComment>\n",
    "        <tag>combined production</tag>\n",
    "        <tag>combined production</tag>\n",
    "      </activity>\n",
    "\n",
    "Combined production is a tricky business, and treatment datasets are also known to be difficult to link correctly.\n",
    "\n",
    "We also have two interesting exchanges:\n",
    "\n",
    "First, the reference product:\n",
    "\n",
    "      <intermediateExchange \n",
    "        id=\"096d7d5f-4c25-4607-bb8f-91f84152af2f\"\n",
    "        unitId=\"487df68b-4994-4027-8fdc-a4dc298257b7\" \n",
    "        variableName=\"swine_slurry\" \n",
    "        amount=\"0\"  (This can't be correct - you can't produce zero of your reference product)\n",
    "        sourceId=\"f816fdd3-6033-449b-937b-d8de44d687a5\" \n",
    "        sourceYear=\"2011\"\n",
    "        sourceFirstAuthor=\"Dauriat A.\" \n",
    "        intermediateExchangeId=\"6c50f204-1e98-422e-b20c-70088087f505\"\n",
    "        productionVolumeAmount=\"0\" \n",
    "        productionVolumeMathematicalRelation=\"\">\n",
    "        <name xml:lang=\"en\">manure, liquid, swine</name>\n",
    "        <unitName xml:lang=\"en\">kg</unitName>\n",
    "        <comment xml:lang=\"en\">Shares of substrates as in Switzerland 2009</comment>\n",
    "        <synonym/>\n",
    "        (... deleted properties)\n",
    "        <productionVolumeComment xml:lang=\"en\">Calculated from production volume of biogas using the\n",
    "          relative outputs.</productionVolumeComment>\n",
    "        <classification classificationId=\"a20a9e2f-b3f0-40fd-8358-f8adf7f419c4\">\n",
    "          <classificationSystem xml:lang=\"en\">By-product classification</classificationSystem>\n",
    "          <classificationValue xml:lang=\"en\">Recyclable</classificationValue>\n",
    "        </classification>\n",
    "        <outputGroup>0</outputGroup>  (outputGroup 0 is \"Reference product\")\n",
    "      </intermediateExchange>\n",
    "\n",
    "Second, our missing flow (`dd80f0f2-f4d5-40f0-9035-09c1a7f3f07b`) is in the file:\n",
    "\n",
    "       <intermediateExchange \n",
    "            id=\"ff13e968-0bb7-4c92-9be3-3568c472dc5d\"\n",
    "            unitId=\"980b811e-3905-4797-82a5-173f5568bc7e\" \n",
    "            amount=\"0.0891321897\"\n",
    "            isCalculatedAmount=\"true\" \n",
    "            mathematicalRelation=\"(biogas*-1)* factor_process_heat\"\n",
    "            sourceId=\"f816fdd3-6033-449b-937b-d8de44d687a5\" \n",
    "            sourceYear=\"2011\"\n",
    "            sourceFirstAuthor=\"Dauriat A.\" \n",
    "            intermediateExchangeId=\"dd80f0f2-f4d5-40f0-9035-09c1a7f3f07b\"  (Our flow!)\n",
    "            activityLinkId=\"61e83ed0-8b91-426c-af94-62b189e8c098\">         (Wrong activity)\n",
    "        <name xml:lang=\"en\">heat, central or small-scale, other than natural gas</name>\n",
    "        <unitName xml:lang=\"en\">MJ</unitName>\n",
    "        <comment xml:lang=\"en\">Calculated with a fixed factor per m3 biogas produced</comment>\n",
    "        <classification classificationId=\"39b0f0ab-1a2f-401b-9f4d-6e39400760a4\">\n",
    "          <classificationSystem xml:lang=\"en\">By-product classification</classificationSystem>\n",
    "          <classificationValue xml:lang=\"en\">allocatable product</classificationValue>\n",
    "        </classification>\n",
    "        <inputGroup>5</inputGroup>  (inputGroup 5 is \"FromTechnosphere\")\n",
    "       </intermediateExchange>\n",
    "\n",
    "In this case, we are requesting an input in the original activity of something which is not produced by the linked activity, but rather consumed (it is an input!) of the linked activity. Here, the Brightway2-io library gives up and just deletes the exchange.\n",
    "\n",
    "# If you have ideas on how to handle this better, please get in touch\n",
    "\n",
    "We don't want to make manual fixes to individual files (it is not our job, and I consider the files produced by the ecoinvent center to be the definitive versions - the last thing we need is for each software developer to develop their own database versions. SimaPro making changes left and right is already frustrating enough). But if we could define a strategy that would apply cleanly to all datasets that would be great."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
