{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\nUsing dask distributed for single-machine parallel computing\n=============================================================\n\nThis example shows the simplest usage of the dask `distributed\n<https://distributed.readthedocs.io>`__ backend, on the local computer.\n\nThis is useful for prototyping a solution, to later be run on a truly\ndistributed cluster, as the only change to be made is the address of the\nscheduler.\n\nAnother realistic usage scenario: combining dask code with joblib code,\nfor instance using dask for preprocessing data, and scikit-learn for\nmachine learning. In such a setting, it may be interesting to use\ndistributed as a backend scheduler for both dask and joblib, to\norchestrate well the computation.\n\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Setup the distributed client\n##############################################################################\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from dask.distributed import Client\n\n# If you have a remote cluster running Dask\n# client = Client('tcp://scheduler-address:8786')\n\n# If you want Dask to set itself up on your personal computer\nclient = Client(processes=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Run parallel computation using dask.distributed\n##############################################################################\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import time\nimport joblib\n\n\ndef long_running_function(i):\n    time.sleep(.1)\n    return i"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The verbose messages below show that the backend is indeed the\ndask.distributed one\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "with joblib.parallel_backend('dask'):\n    joblib.Parallel(verbose=100)(\n        joblib.delayed(long_running_function)(i)\n        for i in range(10))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Progress in computation can be followed on the distributed web\ninterface, see http://dask.pydata.org/en/latest/diagnostics-distributed.html\n\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.6"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}