{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Serialization of un-picklable objects\n\nThis example highlights the options for tempering with joblib serialization\nprocess.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Code source: Thomas Moreau\n# License: BSD 3 clause\n\nimport sys\nimport time\nimport traceback\nfrom joblib.externals.loky import set_loky_pickler\nfrom joblib import parallel_config\nfrom joblib import Parallel, delayed\nfrom joblib import wrap_non_picklable_objects"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "First, define functions which cannot be pickled with the standard ``pickle``\nprotocol. They cannot be serialized with ``pickle`` because they are defined\nin the ``__main__`` module. They can however be serialized with\n``cloudpickle``. With the default behavior, ``loky`` is to use\n``cloudpickle`` to serialize the objects that are sent to the workers.\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "def func_async(i, *args):\n    return 2 * i\n\n\nprint(Parallel(n_jobs=2)(delayed(func_async)(21) for _ in range(1))[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "For most use-cases, using ``cloudpickle`` is efficient enough. However, this\nsolution can be very slow to serialize large python objects, such as dict or\nlist, compared to the standard ``pickle`` serialization.\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "def func_async(i, *args):\n    return 2 * i\n\n\n# We have to pass an extra argument with a large list (or another large python\n# object).\nlarge_list = list(range(1000000))\n\nt_start = time.time()\nParallel(n_jobs=2)(delayed(func_async)(21, large_list) for _ in range(1))\nprint(\"With loky backend and cloudpickle serialization: {:.3f}s\"\n      .format(time.time() - t_start))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If you are on a UNIX system, it is possible to fallback to the old\n``multiprocessing`` backend, which can pickle interactively defined functions\nwith the default pickle module, which is faster for such large objects.\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "if sys.platform != 'win32':\n    def func_async(i, *args):\n        return 2 * i\n\n    with parallel_config('multiprocessing'):\n        t_start = time.time()\n        Parallel(n_jobs=2)(\n            delayed(func_async)(21, large_list) for _ in range(1))\n        print(\"With multiprocessing backend and pickle serialization: {:.3f}s\"\n              .format(time.time() - t_start))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "However, using ``fork`` to start new processes can cause violation of the\nPOSIX specification and can have bad interaction with compiled extensions\nthat use ``openmp``. Also, it is not possible to start processes with\n``fork`` on windows where only ``spawn`` is available. The ``loky`` backend\nhas been developed to mitigate these issues.\n\nTo have fast pickling with ``loky``, it is possible to rely on ``pickle`` to\nserialize all communications between the main process and the workers with\nthe ``loky`` backend. This can be done by setting the environment variable\n``LOKY_PICKLER=pickle`` before the script is launched. Here we use an\ninternal programmatic switch ``loky.set_loky_pickler`` for demonstration\npurposes but it has the same effect as setting ``LOKY_PICKLER``. Note that\nthis switch should not be used as it has some side effects with the workers.\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Now set the `loky_pickler` to use the pickle serialization from stdlib. Here,\n# we do not pass the desired function ``func_async`` as it is not picklable\n# but it is replaced by ``id`` for demonstration purposes.\n\nset_loky_pickler('pickle')\nt_start = time.time()\nParallel(n_jobs=2)(delayed(id)(large_list) for _ in range(1))\nprint(\"With pickle serialization: {:.3f}s\".format(time.time() - t_start))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "However, the function and objects defined in ``__main__`` are not\nserializable anymore using ``pickle`` and it is not possible to call\n``func_async`` using this pickler.\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "def func_async(i, *args):\n    return 2 * i\n\n\ntry:\n    Parallel(n_jobs=2)(delayed(func_async)(21, large_list) for _ in range(1))\nexcept Exception:\n    traceback.print_exc(file=sys.stdout)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "To have both fast pickling, safe process creation and serialization of\ninteractive functions, ``joblib`` provides a wrapper function\n:func:`~joblib.wrap_non_picklable_objects` to wrap the non-picklable function\nand indicate to the serialization process that this specific function should\nbe serialized using ``cloudpickle``. This changes the serialization behavior\nonly for this function and keeps using ``pickle`` for all other objects. The\ndrawback of this solution is that it modifies the object. This should not\ncause many issues with functions but can have side effects with object\ninstances.\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "@delayed\n@wrap_non_picklable_objects\ndef func_async_wrapped(i, *args):\n    return 2 * i\n\n\nt_start = time.time()\nParallel(n_jobs=2)(func_async_wrapped(21, large_list) for _ in range(1))\nprint(\"With pickle from stdlib and wrapper: {:.3f}s\"\n      .format(time.time() - t_start))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The same wrapper can also be used for non-picklable classes. Note that the\nside effects of ``wrap_non_picklable_objects`` on objects can break magic\nmethods such as ``__add__`` and can mess up the ``isinstance`` and\n``issubclass`` functions. Some improvements will be considered if use-cases\nare reported.\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Reset the loky_pickler to avoid border effects with other examples in\n# sphinx-gallery.\nset_loky_pickler()"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.9"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}