{
  "nbformat_minor": 0, 
  "nbformat": 4, 
  "cells": [
    {
      "execution_count": null, 
      "cell_type": "code", 
      "source": [
        "%matplotlib inline"
      ], 
      "outputs": [], 
      "metadata": {
        "collapsed": false
      }
    }, 
    {
      "source": [
        "\nUsing dask distributed for single-machine parallel computing\n=============================================================\n\nThis example shows the simplest usage of the dask `distributed\n<https://distributed.readthedocs.io>`__ backend, on the local computer.\n\nThis is useful for prototyping a solution, to later be run on a truly\ndistributed cluster, as the only change to be made is the address of the\nscheduler.\n\nAnother realistic usage scenario: combining dask code with joblib code,\nfor instance using dask for preprocessing data, and scikit-learn for\nmachine learning. In such a setting, it may be interesting to use\ndistributed as a backend scheduler for both dask and joblib, to\norchestrate well the computation.\n\n\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "source": [
        "Setup the distributed client\n##############################################################################\n\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": null, 
      "cell_type": "code", 
      "source": [
        "from distributed import Client\n# Typically, to execute on a remote machine, the address of the scheduler\n# would go there\nclient = Client()\n\n# Recover the address\naddress = client.scheduler_info()['address']\n\n# This import registers the dask.distributed backend for joblib\nimport distributed.joblib  # noqa"
      ], 
      "outputs": [], 
      "metadata": {
        "collapsed": false
      }
    }, 
    {
      "source": [
        "Run parallel computation using dask.distributed\n##############################################################################\n\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": null, 
      "cell_type": "code", 
      "source": [
        "import time\nimport joblib\n\n\ndef long_running_function(i):\n    time.sleep(.1)\n    return i"
      ], 
      "outputs": [], 
      "metadata": {
        "collapsed": false
      }
    }, 
    {
      "source": [
        "The verbose messages below show that the backend is indeed the\ndask.distributed one\n\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": null, 
      "cell_type": "code", 
      "source": [
        "with joblib.parallel_backend('dask.distributed', scheduler_host=address):\n    joblib.Parallel(n_jobs=2, verbose=100)(\n        joblib.delayed(long_running_function)(i)\n        for i in range(10))"
      ], 
      "outputs": [], 
      "metadata": {
        "collapsed": false
      }
    }, 
    {
      "source": [
        "Progress in computation can be followed on the distributed web\ninterface, see http://distributed.readthedocs.io/en/latest/web.html\n\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }
  ], 
  "metadata": {
    "kernelspec": {
      "display_name": "Python 2", 
      "name": "python2", 
      "language": "python"
    }, 
    "language_info": {
      "mimetype": "text/x-python", 
      "nbconvert_exporter": "python", 
      "name": "python", 
      "file_extension": ".py", 
      "version": "2.7.14", 
      "pygments_lexer": "ipython2", 
      "codemirror_mode": {
        "version": 2, 
        "name": "ipython"
      }
    }
  }
}