{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n==================================================\nCheckpoint using joblib.Memory and joblib.Parallel\n==================================================\n\nThis example illustrates how to cache intermediate computing results using\n:class:`joblib.Memory` within :class:`joblib.Parallel`.\n\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Embed caching within parallel processing\n##############################################################################\n\n It is possible to cache a computationally expensive function executed during\n a parallel process. ``costly_compute`` emulates such time consuming function.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import time\n\n\ndef costly_compute(data, column):\n    \"\"\"Emulate a costly function by sleeping and returning a column.\"\"\"\n    time.sleep(2)\n    return data[column]\n\n\ndef data_processing_mean(data, column):\n    \"\"\"Compute the mean of a column.\"\"\"\n    return costly_compute(data, column).mean()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Create some data. The random seed is fixed to generate deterministic data\nacross Python session. Note that this is not necessary for this specific\nexample since the memory cache is cleared at the end of the session.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import numpy as np\nrng = np.random.RandomState(42)\ndata = rng.randn(int(1e4), 4)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "It is first possible to make the processing without caching or parallel\nprocessing.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "start = time.time()\nresults = [data_processing_mean(data, col) for col in range(data.shape[1])]\nstop = time.time()\n\nprint('\\nSequential processing')\nprint('Elapsed time for the entire processing: {:.2f} s'\n      .format(stop - start))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "``costly_compute`` is expensive to compute and it is used as an intermediate\nstep in ``data_processing_mean``. Therefore, it is interesting to store the\nintermediate results from ``costly_compute`` using :class:`joblib.Memory`.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from joblib import Memory\n\nlocation = './cachedir'\nmemory = Memory(location, verbose=0)\ncostly_compute_cached = memory.cache(costly_compute)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now, we define ``data_processing_mean_using_cache`` which benefits from the\ncache by calling ``costly_compute_cached``\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "def data_processing_mean_using_cache(data, column):\n    \"\"\"Compute the mean of a column.\"\"\"\n    return costly_compute_cached(data, column).mean()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Then, we execute the same processing in parallel and caching the intermediate\nresults.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from joblib import Parallel, delayed\n\nstart = time.time()\nresults = Parallel(n_jobs=2)(\n    delayed(data_processing_mean_using_cache)(data, col)\n    for col in range(data.shape[1]))\nstop = time.time()\n\nprint('\\nFirst round - caching the data')\nprint('Elapsed time for the entire processing: {:.2f} s'\n      .format(stop - start))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "By using 2 workers, the parallel processing gives a x2 speed-up compared to\nthe sequential case. By executing again the same process, the intermediate\nresults obtained by calling ``costly_compute_cached`` will be loaded from the\ncache instead of executing the function.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "start = time.time()\nresults = Parallel(n_jobs=2)(\n    delayed(data_processing_mean_using_cache)(data, col)\n    for col in range(data.shape[1]))\nstop = time.time()\n\nprint('\\nSecond round - reloading from the cache')\nprint('Elapsed time for the entire processing: {:.2f} s'\n      .format(stop - start))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Reuse intermediate checkpoints\n##############################################################################\n\n Having cached the intermediate results of the ``costly_compute_cached``\n function, they are reusable by calling the function. We define a new\n processing which will take the maximum of the array returned by\n ``costly_compute_cached`` instead of previously the mean.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "def data_processing_max_using_cache(data, column):\n    \"\"\"Compute the max of a column.\"\"\"\n    return costly_compute_cached(data, column).max()\n\n\nstart = time.time()\nresults = Parallel(n_jobs=2)(\n    delayed(data_processing_max_using_cache)(data, col)\n    for col in range(data.shape[1]))\nstop = time.time()\n\nprint('\\nReusing intermediate checkpoints')\nprint('Elapsed time for the entire processing: {:.2f} s'\n      .format(stop - start))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The processing time only corresponds to the execution of the ``max``\nfunction. The internal call to ``costly_compute_cached`` is reloading the\nresults from the cache.\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Clean-up the cache folder\n##############################################################################\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "memory.clear(warn=False)"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.1"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}