{
  "nbformat_minor": 0, 
  "nbformat": 4, 
  "cells": [
    {
      "execution_count": null, 
      "cell_type": "code", 
      "source": [
        "%matplotlib inline"
      ], 
      "outputs": [], 
      "metadata": {
        "collapsed": false
      }
    }, 
    {
      "source": [
        "\n===============================\nNumPy memmap in joblib.Parallel\n===============================\n\nThis example illustrates some features enabled by using a memory map\n(:class:`numpy.memmap`) within :class:`joblib.Parallel`. First, we show that\ndumping a huge data array ahead of passing it to :class:`joblib.Parallel`\nspeeds up computation. Then, we show the possibility to provide write access to\noriginal data.\n\n\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "source": [
        "Speed up processing of a large data array\n#############################################################################\n\n We create a large data array for which the average is computed for several\n slices.\n\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": null, 
      "cell_type": "code", 
      "source": [
        "import numpy as np\n\ndata = np.random.random((int(1e7),))\nwindow_size = int(5e5)\nslices = [slice(start, start + window_size)\n          for start in range(0, data.size - window_size, int(1e5))]"
      ], 
      "outputs": [], 
      "metadata": {
        "collapsed": false
      }
    }, 
    {
      "source": [
        "The ``slow_mean`` function introduces a :func:`time.sleep` call to simulate a\nmore expensive computation cost for which parallel computing is beneficial.\nParallel may not be beneficial for very fast operation, due to extra overhead\n(workers creations, communication, etc.).\n\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": null, 
      "cell_type": "code", 
      "source": [
        "import time\n\n\ndef slow_mean(data, sl):\n    \"\"\"Simulate a time consuming processing.\"\"\"\n    time.sleep(0.01)\n    return data[sl].mean()"
      ], 
      "outputs": [], 
      "metadata": {
        "collapsed": false
      }
    }, 
    {
      "source": [
        "First, we will evaluate the sequential computing on our problem.\n\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": null, 
      "cell_type": "code", 
      "source": [
        "tic = time.time()\nresults = [slow_mean(data, sl) for sl in slices]\ntoc = time.time()\nprint('\\nElapsed time computing the average of couple of slices {:.2f} s'\n      .format(toc - tic))"
      ], 
      "outputs": [], 
      "metadata": {
        "collapsed": false
      }
    }, 
    {
      "source": [
        ":class:`joblib.Parallel` is used to compute in parallel the average of all\nslices using 2 workers.\n\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": null, 
      "cell_type": "code", 
      "source": [
        "from joblib import Parallel, delayed\n\n\ntic = time.time()\nresults = Parallel(n_jobs=2)(delayed(slow_mean)(data, sl) for sl in slices)\ntoc = time.time()\nprint('\\nElapsed time computing the average of couple of slices {:.2f} s'\n      .format(toc - tic))"
      ], 
      "outputs": [], 
      "metadata": {
        "collapsed": false
      }
    }, 
    {
      "source": [
        "Parallel processing is already faster than the sequential processing. It is\nalso possible to remove a bit of overhead by dumping the ``data`` array to a\nmemmap and pass the memmap to :class:`joblib.Parallel`.\n\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": null, 
      "cell_type": "code", 
      "source": [
        "import os\nfrom joblib import dump, load\n\nfolder = './joblib_memmap'\ntry:\n    os.mkdir(folder)\nexcept FileExistsError:\n    pass\n\ndata_filename_memmap = os.path.join(folder, 'data_memmap')\ndump(data, data_filename_memmap)\ndata = load(data_filename_memmap, mmap_mode='r')\n\ntic = time.time()\nresults = Parallel(n_jobs=2)(delayed(slow_mean)(data, sl) for sl in slices)\ntoc = time.time()\nprint('\\nElapsed time computing the average of couple of slices {:.2f} s\\n'\n      .format(toc - tic))"
      ], 
      "outputs": [], 
      "metadata": {
        "collapsed": false
      }
    }, 
    {
      "source": [
        "Therefore, dumping large ``data`` array ahead of calling\n:class:`joblib.Parallel` can speed up the processing by removing some\noverhead.\n\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "source": [
        "Writable memmap for shared memory :class:`joblib.Parallel`\n##############################################################################\n\n ``slow_mean_write_output`` will compute the mean for some given slices as in\n the previous example. However, the resulting mean will be directly written on\n the output array.\n\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": null, 
      "cell_type": "code", 
      "source": [
        "def slow_mean_write_output(data, sl, output, idx):\n    \"\"\"Simulate a time consuming processing.\"\"\"\n    time.sleep(0.005)\n    res_ = data[sl].mean()\n    print(\"[Worker %d] Mean for slice %d is %f\" % (os.getpid(), idx, res_))\n    output[idx] = res_"
      ], 
      "outputs": [], 
      "metadata": {
        "collapsed": false
      }
    }, 
    {
      "source": [
        "Prepare the folder where the memmap will be dumped.\n\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": null, 
      "cell_type": "code", 
      "source": [
        "output_filename_memmap = os.path.join(folder, 'output_memmap')"
      ], 
      "outputs": [], 
      "metadata": {
        "collapsed": false
      }
    }, 
    {
      "source": [
        "Pre-allocate a writable shared memory map as a container for the results of\nthe parallel computation.\n\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": null, 
      "cell_type": "code", 
      "source": [
        "output = np.memmap(output_filename_memmap, dtype=data.dtype,\n                   shape=len(slices), mode='w+')"
      ], 
      "outputs": [], 
      "metadata": {
        "collapsed": false
      }
    }, 
    {
      "source": [
        "``data`` is replaced by its memory mapped version. Note that the buffer as\nalready been dumped in the previous section.\n\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": null, 
      "cell_type": "code", 
      "source": [
        "data = load(data_filename_memmap, mmap_mode='r')"
      ], 
      "outputs": [], 
      "metadata": {
        "collapsed": false
      }
    }, 
    {
      "source": [
        "Fork the worker processes to perform computation concurrently\n\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": null, 
      "cell_type": "code", 
      "source": [
        "Parallel(n_jobs=2)(delayed(slow_mean_write_output)(data, sl, output, idx)\n                   for idx, sl in enumerate(slices))"
      ], 
      "outputs": [], 
      "metadata": {
        "collapsed": false
      }
    }, 
    {
      "source": [
        "Compare the results from the output buffer with the expected results\n\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": null, 
      "cell_type": "code", 
      "source": [
        "print(\"\\nExpected means computed in the parent process:\\n {}\"\n      .format(np.array(results)))\nprint(\"\\nActual means computed by the worker processes:\\n {}\"\n      .format(output))"
      ], 
      "outputs": [], 
      "metadata": {
        "collapsed": false
      }
    }, 
    {
      "source": [
        "Clean-up the memmap\n##############################################################################\n\n Remove the different memmap that we created. It might fail in Windows due\n to file permissions.\n\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": null, 
      "cell_type": "code", 
      "source": [
        "import shutil\n\ntry:\n    shutil.rmtree(folder)\nexcept:  # noqa\n    print('Could not clean-up automatically.')"
      ], 
      "outputs": [], 
      "metadata": {
        "collapsed": false
      }
    }
  ], 
  "metadata": {
    "kernelspec": {
      "display_name": "Python 2", 
      "name": "python2", 
      "language": "python"
    }, 
    "language_info": {
      "mimetype": "text/x-python", 
      "nbconvert_exporter": "python", 
      "name": "python", 
      "file_extension": ".py", 
      "version": "2.7.14", 
      "pygments_lexer": "ipython2", 
      "codemirror_mode": {
        "version": 2, 
        "name": "ipython"
      }
    }
  }
}