{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n===============================\nImproving I/O using compressors\n===============================\n\nThis example compares the compressors available in Joblib. In the example,\nZlib, LZMA and LZ4 compression only are used but Joblib also supports BZ2 and\nGZip compression methods.\nFor each compared compression method, this example dumps and reloads a\ndataset fetched from an online machine-learning database. This gives 3\ninformations: the size on disk of the compressed data, the time spent to dump\nand the time spent to reload the data from disk.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import os\nimport os.path\nimport time"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Get some data from real-world use cases\n---------------------------------------\n\nFirst fetch the benchmark dataset from an online machine-learning database\nand load it in a pandas dataframe.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n\nurl = (\"https://archive.ics.uci.edu/ml/machine-learning-databases/\"\n       \"kddcup99-mld/kddcup.data.gz\")\nnames = (\"duration, protocol_type, service, flag, src_bytes, \"\n         \"dst_bytes, land, wrong_fragment, urgent, hot, \"\n         \"num_failed_logins, logged_in, num_compromised, \"\n         \"root_shell, su_attempted, num_root, \"\n         \"num_file_creations, \").split(', ')\n\ndata = pd.read_csv(url, names=names, nrows=1e6)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Dump and load the dataset without compression\n---------------------------------------------\n\nThis gives reference values for later comparison.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from joblib import dump, load\n\npickle_file = './pickle_data.joblib'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Start by measuring the time spent for dumping the raw data:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "start = time.time()\nwith open(pickle_file, 'wb') as f:\n    dump(data, f)\nraw_dump_duration = time.time() - start\nprint(\"Raw dump duration: %0.3fs\" % raw_dump_duration)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Then measure the size of the raw dumped data on disk:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "raw_file_size = os.stat(pickle_file).st_size / 1e6\nprint(\"Raw dump file size: %0.3fMB\" % raw_file_size)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Finally measure the time spent for loading the raw data:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "start = time.time()\nwith open(pickle_file, 'rb') as f:\n    load(f)\nraw_load_duration = time.time() - start\nprint(\"Raw load duration: %0.3fs\" % raw_load_duration)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Dump and load the dataset using the Zlib compression method\n-----------------------------------------------------------\n\nThe compression level is using the default value, 3, which is, in general, a\ngood compromise between compression and speed.\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Start by measuring the time spent for dumping of the zlib data:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "start = time.time()\nwith open(pickle_file, 'wb') as f:\n    dump(data, f, compress='zlib')\nzlib_dump_duration = time.time() - start\nprint(\"Zlib dump duration: %0.3fs\" % zlib_dump_duration)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Then measure the size of the zlib dump data on disk:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "zlib_file_size = os.stat(pickle_file).st_size / 1e6\nprint(\"Zlib file size: %0.3fMB\" % zlib_file_size)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Finally measure the time spent for loading the compressed dataset:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "start = time.time()\nwith open(pickle_file, 'rb') as f:\n    load(f)\nzlib_load_duration = time.time() - start\nprint(\"Zlib load duration: %0.3fs\" % zlib_load_duration)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<div class=\"alert alert-info\"><h4>Note</h4><p>The compression format is detected automatically by Joblib.\n   The compression format is identified by the standard magic number present\n   at the beginning of the file. Joblib uses this information to determine\n   the compression method used.\n   This is the case for all compression methods supported by Joblib.</p></div>\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Dump and load the dataset using the LZMA compression method\n-----------------------------------------------------------\n\nLZMA compression method has a very good compression rate but at the cost\nof being very slow.\nIn this example, a light compression level, e.g. 3, is used to speed up a\nbit the dump/load cycle.\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Start by measuring the time spent for dumping the lzma data:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "start = time.time()\nwith open(pickle_file, 'wb') as f:\n    dump(data, f, compress=('lzma', 3))\nlzma_dump_duration = time.time() - start\nprint(\"LZMA dump duration: %0.3fs\" % lzma_dump_duration)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Then measure the size of the lzma dump data on disk:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "lzma_file_size = os.stat(pickle_file).st_size / 1e6\nprint(\"LZMA file size: %0.3fMB\" % lzma_file_size)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Finally measure the time spent for loading the lzma data:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "start = time.time()\nwith open(pickle_file, 'rb') as f:\n    load(f)\nlzma_load_duration = time.time() - start\nprint(\"LZMA load duration: %0.3fs\" % lzma_load_duration)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Dump and load the dataset using the LZ4 compression method\n----------------------------------------------------------\n\nLZ4 compression method is known to be one of the fastest available\ncompression method but with a compression rate a bit lower than Zlib. In\nmost of the cases, this method is a good choice.\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<div class=\"alert alert-info\"><h4>Note</h4><p>In order to use LZ4 compression with Joblib, the\n   `lz4 <http://https://pypi.python.org/pypi/lz4>`_ package must be installed\n   on the system.</p></div>\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Start by measuring the time spent for dumping the lz4 data:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "start = time.time()\nwith open(pickle_file, 'wb') as f:\n    dump(data, f, compress='lz4')\nlz4_dump_duration = time.time() - start\nprint(\"LZ4 dump duration: %0.3fs\" % lz4_dump_duration)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Then measure the size of the lz4 dump data on disk:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "lz4_file_size = os.stat(pickle_file).st_size / 1e6\nprint(\"LZ4 file size: %0.3fMB\" % lz4_file_size)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Finally measure the time spent for loading the lz4 data:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "start = time.time()\nwith open(pickle_file, 'rb') as f:\n    load(f)\nlz4_load_duration = time.time() - start\nprint(\"LZ4 load duration: %0.3fs\" % lz4_load_duration)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Comparing the results\n---------------------\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import numpy as np\nimport matplotlib.pyplot as plt\n\nN = 4\nload_durations = (raw_load_duration, lz4_load_duration, zlib_load_duration,\n                  lzma_load_duration)\ndump_durations = (raw_dump_duration, lz4_dump_duration, zlib_dump_duration,\n                  lzma_dump_duration)\nfile_sizes = (raw_file_size, lz4_file_size, zlib_file_size, lzma_file_size)\nind = np.arange(N)\nwidth = 0.5\n\nplt.figure(1, figsize=(5, 4))\np1 = plt.bar(ind, dump_durations, width)\np2 = plt.bar(ind, load_durations, width, bottom=dump_durations)\nplt.ylabel('Time in seconds')\nplt.title('Dump and load durations')\nplt.xticks(ind, ('Raw', 'LZ4', 'Zlib', 'LZMA'))\nplt.yticks(np.arange(0, lzma_load_duration + lzma_dump_duration))\nplt.legend((p1[0], p2[0]), ('Dump duration', 'Load duration'))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Compared with other compressors, LZ4 is clearly the fastest, especially for\ndumping compressed data on disk. In this particular case, it can even be\nfaster than the raw dump.\nAlso note that dump and load durations depend on the I/O speed of the\nunderlying storage: for example, with SSD hard drives the LZ4 compression\nwill be slightly slower than raw dump/load, whereas with spinning hard disk\ndrives (HDD) or remote storage (NFS), LZ4 is faster in general.\n\nLZMA and Zlib, even if always slower for dumping data, are quite fast when\nre-loading compressed data from disk.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "plt.figure(2, figsize=(5, 4))\nplt.bar(ind, file_sizes, width, log=True)\nplt.ylabel('File size in MB')\nplt.xticks(ind, ('Raw', 'LZ4', 'Zlib', 'LZMA'))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Compressed data obviously takes a lot less space on disk than raw data. LZMA\nis the best compression method in terms of compression rate. Zlib also has a\nbetter compression rate than LZ4.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Clear the pickle file\n---------------------\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import os\nos.remove(pickle_file)"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.6"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}