{
 "metadata": {
  "name": "",
  "signature": "sha256:96cba20548457712eba77b55d594e43075b6ad08095858554d4de0a6bf7c6b75"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from brightway2 import *\n",
      "from time import time"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Disable cache to get fair timings of different approaches"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "config.p[\"use_cache\"] = False"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Copy ``ecoinvent 2.2`` to a new database using the JSON backend."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ei = Database(\"ecoinvent 2.2\")\n",
      "\n",
      "# Can also do \n",
      "# from bw2data.backends import JSONDatabase\n",
      "# db = JSONDatabase(\"ecoinvent 2.2 json\")\n",
      "# But you can't change backend after instantiation of database object\n",
      "db = Database(\"ecoinvent 2.2 json\", backend=\"json\")\n",
      "db.register()\n",
      "\n",
      "# Writing ~4000 files takes a little while\n",
      "start = time()\n",
      "db.write(ei.relabel_data(ei.load(), \"ecoinvent 2.2 json\"))\n",
      "print time() - start\n",
      "\n",
      "# So does loading everything again to process it\n",
      "db.process()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "47.1706380844\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "/Users/cmutel/local/bw2dev/lib/python2.7/site-packages/bw2data/data_store.py:46: UserWarning: \n",
        "\tBrightway2 JSONDatabase: ecoinvent 2.2 json is not registered\n",
        "  warnings.warn(u\"\\n\\t%s is not registered\" % self, UserWarning)\n"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "There are two main beenfits to JSON databases. \n",
      "\n",
      "First, because each file is stored in a separate text file, with indentation and line breaks, it integrates perfectly in version control.\n",
      "\n",
      "Second, faster random access to datasets.\n",
      "\n",
      "The correct backend is created based on the metadata key ``backend``. The default backend is a single file database."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print databases[\"ecoinvent 2.2 json\"]\n",
      "print databases[\"ecoinvent 2.2\"]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "{u'depends': [u'biosphere'], u'backend': u'json'}\n",
        "{u'depends': [u'biosphere'], u'version': 9}\n"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "db = Database(\"ecoinvent 2.2 json\")\n",
      "type(db)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 6,
       "text": [
        "bw2data.backends.json.database.JSONDatabase"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ei = Database(\"ecoinvent 2.2\")\n",
      "type(ei)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 8,
       "text": [
        "bw2data.backends.default.database.SingleFileDatabase"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Getting a single dataset, or even a single dataset key, from ``ecoinvent 2.2`` requires loading the whole database. On the otherhand, once it is loaded, it will be stored in the cache, so further access is quick."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "config.p[\"use_cache\"] = True"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 9
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "start = time()\n",
      "print \"First time slow\"\n",
      "print ei.random()\n",
      "print time() - start\n",
      "\n",
      "start = time()\n",
      "print \"Next time fast\"\n",
      "print ei.random()\n",
      "print time() - start"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "First time slow\n",
        "(u'ecoinvent 2.2', u'42bd150f219fde7cabe2b2541dc29548')"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "2.67773008347\n",
        "Next time fast\n",
        "(u'ecoinvent 2.2', u'8d0bc3cfb14fe7a5747eacd38cc4cb17')\n",
        "0.000772953033447\n"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The JSON database has access to all keys without loading a single file."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "start = time()\n",
      "print \"First time fast\"\n",
      "print db.random()\n",
      "print time() - start\n",
      "\n",
      "start = time()\n",
      "print \"Next time fast\"\n",
      "print db.random()\n",
      "print time() - start"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "First time fast\n",
        "('ecoinvent 2.2 json', u'bf5d6693ffb27e88dc021b3d17285a2c')\n",
        "0.00043511390686\n",
        "Next time fast\n",
        "('ecoinvent 2.2 json', u'9474e77916f97e16425ecf2e34133990')\n",
        "0.000362157821655\n"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Getting a single dataset is also fast"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "start = time()\n",
      "ds = db.load()[db.random()]\n",
      "print time() - start"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.00414896011353\n"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ds.keys()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 14,
       "text": [
        "[u'code',\n",
        " u'name',\n",
        " u'unit',\n",
        " u'key',\n",
        " u'exchanges',\n",
        " u'type',\n",
        " u'categories',\n",
        " u'location']"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "However, calling ``db.load()`` doesn't give us the whole database:\n",
      "(See https://bitbucket.org/cmutel/brightway2-data/src/default/bw2data/backends/json/sync_json_dict.py for the dirty details)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "type(db.load())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 15,
       "text": [
        "bw2data.backends.json.sync_json_dict.SynchronousJSONDict"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "However, it is pretty good at pretending to be the same as a database loaded into memory:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print len(db.load())\n",
      "print db.random() in db.load()\n",
      "for index, x in enumerate(db.load()):\n",
      "    print x\n",
      "    if index > 5:\n",
      "        break"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "4087\n",
        "True\n",
        "('ecoinvent 2.2 json', u'31be4cd55f5f227d0cb087637d32f14f')\n",
        "('ecoinvent 2.2 json', u'a143e52d90b9eba20e43a96a2c89193a')\n",
        "('ecoinvent 2.2 json', u'ab6f4f2d9765cd42d4e041c5596a580d')\n",
        "('ecoinvent 2.2 json', u'0363effc128adc4042b0dd249a1ac32e')\n",
        "('ecoinvent 2.2 json', u'1e81d6df5644b1c07cb902c0515358f6')\n",
        "('ecoinvent 2.2 json', u'4741952ad13b80b4ecb1519b28481f59')\n",
        "('ecoinvent 2.2 json', u'0ba6d475a1cc836e64010b1c3c8a2741')\n"
       ]
      }
     ],
     "prompt_number": 18
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "One important difference is in how you update data. A dataset return from a JSON database is not a normal dictionary, and you can't change it:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ds = db.load()[db.random()]\n",
      "print type(ds)\n",
      "ds['foo'] = 'bar'"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "<class 'bw2data.backends.json.sync_json_dict.frozendict'>\n"
       ]
      },
      {
       "ename": "AttributeError",
       "evalue": "A frozendict cannot be modified",
       "output_type": "pyerr",
       "traceback": [
        "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
        "\u001b[0;32m<ipython-input-19-2b99ba6ce1ff>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mds\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdb\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mdb\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrandom\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;32mprint\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mds\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'foo'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'bar'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
        "\u001b[0;32m/Users/cmutel/local/bw2dev/lib/python2.7/site-packages/bw2data/backends/json/sync_json_dict.pyc\u001b[0m in \u001b[0;36m_blocked_attribute\u001b[0;34m(obj)\u001b[0m\n\u001b[1;32m     12\u001b[0m     From http://code.activestate.com/recipes/414283-frozen-dictionaries/\"\"\"\n\u001b[1;32m     13\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_blocked_attribute\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 14\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0mAttributeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"A frozendict cannot be modified\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     15\u001b[0m     \u001b[0m_blocked_attribute\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mproperty\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_blocked_attribute\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     16\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
        "\u001b[0;31mAttributeError\u001b[0m: A frozendict cannot be modified"
       ]
      }
     ],
     "prompt_number": 19
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Instead, you must make a copy, make your changes, and then write the changed dictionary to the database:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "key = db.random()\n",
      "db_data = db.load()\n",
      "ds = db_data[key]\n",
      "new_ds = dict(ds)\n",
      "new_ds['foo'] = 'bar'\n",
      "db_data[key] = new_ds"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 20
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This limitation exists because changes are instantly saved to disk, and Brightway2 doesn't know if you make a change in a nested dictionary, so any such changes aren't allowed.\n",
      "\n",
      "Instant syncing means you don't need to call ``.write()`` anymore - indeed, it doesn't do anything (https://bitbucket.org/cmutel/brightway2-data/src/default/bw2data/backends/json/database.py?at=default#cl-65). Note, however, that you still need to call ``.process()`` at the end of your changes."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}