# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['minionize',
 'minionize.reporter',
 'minionize.tests.functionnal.pulsar',
 'minionize.tests.unit']

package_data = \
{'': ['*']}

install_requires = \
['asciichartpy>=1.5.25,<2.0.0',
 'pandas>=1.1.3,<2.0.0',
 'python-dotenv>=0.14.0,<0.15.0',
 'tox>=3.20.0,<4.0.0']

extras_require = \
{'execo': ['execo>=2.6.4,<3.0.0'],
 'google': ['google-cloud-pubsub>=2.0.0,<3.0.0'],
 'pulsar': ['pulsar-client>=2.6.1,<3.0.0']}

entry_points = \
{'console_scripts': ['minion-status = minionize.cli:status',
                     'minionize = minionize.cli:run']}

setup_kwargs = {
    'name': 'minionize',
    'version': '0.1.21',
    'description': 'Massively Parallel operations made easy',
    'long_description': '\nRationale\n---------\n\n- You wrote a program ``a.out`` with some parameters\n- You need to explore the space of parameters (or at least a large subset of it)\n\n**Minionize** is a solution to spawn a legion of ``a.out`` in a massively\nparallel manner. By *minionizing* your program, your *minions* will take\ntheir inputs from various sources (e.g filesystem, pub/sub) and run it. Also\ninputs can be acked or redelivered to another minions.\n\nThat makes **Minionize** a good choice for your `besteffort and idempotent\n<http://oar.imag.fr/docs/latest/user/mechanisms.html#besteffort-jobs>`_\ncomputations on scientific testbeds (Igrida, Grid\'5000) but also if you need\na quick way to turn you binary into a micro-service (e.g for Kubernetes).\n\n**Minionize** provides some observability features out-of-the box:\nit exposes an api to retrieve the stats and a basic cli to display some of\nthem (to be honest, I dream of an ``mtop``: ``htop`` for minions). But\ncurrently it looks more or less like this:\n\n.. image:: https://gitlab.inria.fr/msimonin/minionize/-/raw/master/images/stats.png\n   :height: 500\n\nHow does it work\n----------------\n\nA classical pattern to do the above is to apply the master/worker pattern\nwhere a master gives tasks to workers. Workers repeatedly fetch a new task\nfrom a queue , run it and report back to the master its status (e.g success,\nfailure). **Minionize** applies somehow this pattern but is *masterless*\nout-of-the box. Indeed modern queue implementations expose APIs to\nacknowledge/requeue messages.\n\nCurrently we support:\n\n- ``execo`` based queue: the queue is stored in a shared file system in your cluster (no external process involved)\n- ``Google pub/sub`` based queue: the queue is hosted by Google\n- ``Apache pulsar``, a pub/sub system you can self-host\n\n\nSome examples\n-------------\n\nSimplest use case\n~~~~~~~~~~~~~~~~~\n\nIn this case the received params are appended to the\nminionized program. If you need more control on the params you\'ll need to\nwrite your own Callback (see below).\n\n- with `Execo` engine:\n\n    .. code-block:: bash\n\n        # Install the execo minionizer\n        pip install minionize[execo]\n\n        # Create the queue of params\n        # You\'ll have to run this prior to launching your minions (adapt to\n        # your need / make a regular script)\n        $) python -c "from execo_engine.sweep import ParamSweeper; ParamSweeper(\'sweeps\', sweeps=range(10), save_sweeps=True)"\n\n        # start your minions\n        $) MINION_ENGINE=execo minionize echo hello\n        hello 0\n        hello 1\n        hello 2\n        hello 3\n        hello 4\n        hello 5\n        hello 6\n        hello 7\n        hello 8\n        hello 9\n\n    .. note::\n\n        In other words the ``minionize`` wrapper lets you populate the queue\n        with strings representing the parameter of your command line\n\n- Record some stats: you need to setup a ``Reporter`` to report your stats.\n\n    .. code-block:: bash\n\n        # Install the execo minionizer\n        pip install minionize[execo]\n\n        # Create the queue of params\n        # You\'ll have to run this prior to launching your minions (adapt to\n        # your need / make a regular script)\n        $) python -c "from execo_engine.sweep import ParamSweeper; ParamSweeper(\'sweeps\', sweeps=range(10), save_sweeps=True)"\n\n        # start your minions\n        MINION_ENGINE=execo MINION_REPORTER=json minionize sleep\n\n        # read the stats (while running or no)\n        MINION_REPORTER=json minion-status\n\n- On a OAR cluster (Igrida/Grid5000):\n\n  - Generate the queue for example with Execo\n\n    .. code-block:: bash\n\n        python -c "from execo_engine.sweep import ParamSweeper; ParamSweeper(\'sweeps\', sweeps=range(1000), save_sweeps=True)"\n\n    - Create your oar scan script:\n\n    .. code-block:: bash\n\n        #!/usr/bin/env bash\n\n        #OAR -n kpd\n        #OAR -l nodes=1,walltime=1:0:0\n        #OAR -t besteffort\n        #OAR -t idempotent\n\n        # oarsub --array 10 -S ./oar.sh\n\n        set -eux\n\n        pip install minionize\n\n        minionize echo "hello from $OAR_JOB_ID"\n\n    - Start your minions\n\n    .. code-block:: bash\n\n        echo "MINION_ENGINE=execo" > .env\n        oarsub --array 10 -S ./oar.sh\n\n    .. note::\n\n        ``.env`` file is read when minionizing starts so the scan script can\n        remain the same whatever engine is used.\n\n    - Example of output:\n\n    .. code-block:: bash\n\n        $) cat OAR.1287856.stdout\n        [...]\n        hello from 1287856 135\n        hello from 1287856 139\n        hello from 1287856 143\n        hello from 1287856 147\n        hello from 1287856 151\n        hello from 1287856 155\n        hello from 1287856 159\n        hello from 1287856 163\n        hello from 1287856 167\n        [...]\n\n    .. note::\n\n        As expected params have been distributed to different minions\n\nCustom Callbacks\n~~~~~~~~~~~~~~~~\n\nThe params sent to your program can be anything (e.g a python dict). In\nsome cases (many actually), you\'ll need to transform these params to\nsomething that you program can understand. **So you\'ll need to tell\nminionize how to minionize**. This is achieved using specific callbacks.\n\nThe easiest way to write a custom callbacks is to inherit from\n``ProcessCallback`` or ``FuncCallback``. With these Callbacks you don\'t\nhave to worry about the acknowledgement logic.\n\n.. code-block:: python\n\n    #   a.out is invoked like this: a.out --arg1 varg1 varg2\n    #   but the queue holds json like object:\n    #   {"arg1": varg11, "arg2": varg21}, {"arg1": varg12, "arg2": varg22} ...\n    # we can write a custom ProcessCallback which overrides the to_cmd method\n\n    class MyProcessCallBack(ProcessCallback):\n        def to_cmd(param: Param):\n            return f"a.out --arg1 {param[\'arg1\']} {param[\'arg2\']}"\n\n    m = minionizer(MyProcessCallback())\n    m.run()\n\n.. code-block:: python\n\n    #   you want to minionize a python function `my_function`\n    #   but the queue holds json like object:\n    #   {"arg1": varg11, "arg2": varg21}, {"arg1": varg12, "arg2": varg22} ...\n    # we can use the FuncCallback for this purpose\n\n    def myfunc(...)\n        # this is your function\n\n    def _myfunc(param: Param)\n        # this is the wrapper which invokes myfunc based on the params\n        return myfunc(param["arg1"], param["arg2"])\n\n    m = minionizer(FuncCallback(_myfunc))\n    m.run()\n\n\nEnvironment variables\n---------------------\n\n*Minionize* is configured using environment variables.\nBy default it reads a ``.env`` file in the current directory but doesn\'t\noverride existing system environment variables.\n\nDefault values\n~~~~~~~~~~~~~~\n\n.. code-block:: bash\n\n    --------------------------------------------\n\n    # which engine (queue implementation) to use\n    MINION_ENGINE=execo # google, pulsar\n\n    # Execo\n    EXECO_PERSISTENCE_DIR=sweeps\n\n    # Google\n    GOOGLE_PROJECT_ID=/mandatory/\n    GOOGLE_TOPIC_ID=/mandatory/\n    GOOGLE_SUBSCRIPTIOn=/mandatory/\n    GOOGLE_APPLICATION_CREDENTIALS=/mandatory/\n    GOOGLE_DECODER=identity\n\n\n    # Pulsar\n    PULSAR_CONNECTION=pulsar://localhost:6650\n    PULSAR_TOPIC=/mandatory/\n    PULSAR_DECODER=identity\n\n    ---------------------------------------------\n\n    # Stat reporter\n    MINION_REPORTER=null # json, stdout\n\n    # Json\n    REPORTER_JSON_DIRECTORY=minion-report\n\n\nRoadmap\n-------\n\n- Easy integration as docker entrypoint\n- Minionize python function (e.g @minionize decorator)\n- Support new queues (Apache pulsar, Redis stream, RabbitMQ, Kakfa ...)\n- Support new abstractions to run container based application (docker, singularity...)\n- Automatic encapsulation using a .minionize.yml\n- Minions statistics\n- Keep in touch (matthieu dot simonin at inria dot fr)',
    'author': 'msimonin',
    'author_email': 'matthieu.simonin@inria.fr',
    'maintainer': None,
    'maintainer_email': None,
    'url': 'https://gitlab.inria.fr/msimonin/minionize',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'extras_require': extras_require,
    'entry_points': entry_points,
    'python_requires': '>=3.6.1,<4.0.0',
}


setup(**setup_kwargs)
