# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['pnp',
 'pnp.config',
 'pnp.engines',
 'pnp.plugins',
 'pnp.plugins.pull',
 'pnp.plugins.push',
 'pnp.plugins.udf',
 'pnp.runner',
 'pnp.shared']

package_data = \
{'': ['*']}

install_requires = \
['Jinja2>=2.11.2,<3.0.0',
 'argresolver>=0.3.3,<0.4.0',
 'asyncio>=3.4.3,<4.0.0',
 'asyncws>=0.1,<0.2',
 'attrs>=19.3.0,<20.0.0',
 'binaryornot>=0.4.4,<0.5.0',
 'cachetools>=4.1.0,<5.0.0',
 'cronex>=0.1.3,<0.2.0',
 'dictmentor>=0.2.2,<0.3.0',
 'docopt>=0.6.2,<0.7.0',
 'glom>=19.10.0,<20.0.0',
 'influxdb>=5.3.0,<6.0.0',
 'paho-mqtt>=1.5.0,<2.0.0',
 'pathspec>=0.8.0,<0.9.0',
 'psutil>=5.7.0,<6.0.0',
 'python-box<=3.4.6',
 'pytz>=2020.1,<2021.0',
 'requests>=2.23.0,<3.0.0',
 'ruamel.yaml>=0.16.10,<0.17.0',
 'schedule>=0.6.0,<0.7.0',
 'schema>=0.7.2,<0.8.0',
 'schiene>=0.23,<0.24',
 'slacker>=0.14.0,<0.15.0',
 'syncasync>=20180812,<20180813',
 'typing-extensions>=3.7.4,<4.0.0',
 'tzlocal>=2.0.0,<3.0.0']

extras_require = \
{'dht': ['Adafruit_DHT>=1.3.2,<2.0.0'],
 'dropbox': ['dropbox>=9.0.0,<10.0.0', 'urllib3>=1.20,<2.0'],
 'faceR': ['face-recognition>=1.2.2,<2.0.0', 'image>=1.5.24,<2.0.0'],
 'fitbit': ['fitbit>=0.3.0,<0.4.0'],
 'fritz:python_version >= "3.6" and python_version < "4.0"': ['fritzconnection>=1.2.0,<2.0.0'],
 'fswatcher': ['watchdog>=0.8.3,<0.9.0'],
 'ftp': ['pyftpdlib>=1.5.0,<2.0.0'],
 'gmail': ['google-api-python-client>=1.7.7,<2.0.0',
           'google-auth-httplib2>=0.0.3,<0.0.4',
           'google-auth-oauthlib>=0.2.0,<0.3.0'],
 'gpio': ['RPi.GPIO>=0.6.5,<0.7.0'],
 'http-server': ['Flask>=1.0.2,<2.0.0',
                 'gevent>=1.3.4,<2.0.0',
                 'sanic>=18.12.0,<19.0.0'],
 'miflora': ['miflora>=0.4.0,<0.5.0'],
 'pushbullet': ['pushbullet.py>=0.10.0,<0.11.0', 'urllib3>=1.20,<2.0'],
 'sound': ['numpy>=1.16.0,<2.0.0',
           'PyAudio>=0.2.11,<0.3.0',
           'scipy>=1.2.0,<2.0.0']}

entry_points = \
{'console_scripts': ['pnp = pnp.runner.pnp:main',
                     'pnp_gmail_tokens = pnp.runner.pnp_gmail_tokens:main',
                     'pnp_record_sound = pnp.runner.pnp_record_sound:main']}

setup_kwargs = {
    'name': 'pnp',
    'version': '0.23.0',
    'description': "Pull 'n' Push",
    'long_description': '# Pull \'n\' Push\n\n[![Python](https://img.shields.io/badge/Python-3.5%20%7C%203.6%20%7C%203.7-green.svg)](https://www.python.org/)\n[![PyPI version](https://badge.fury.io/py/pnp.svg)](https://badge.fury.io/py/pnp)\n[![Build Status](https://travis-ci.org/HazardDede/pnp.svg?branch=master)](https://travis-ci.org/HazardDede/pnp)\n[![Coverage Status](https://coveralls.io/repos/github/HazardDede/pnp/badge.svg?branch=master)](https://coveralls.io/github/HazardDede/pnp?branch=master)\n[![Docker: hub](https://img.shields.io/badge/docker-hub-brightgreen.svg)](https://cloud.docker.com/u/hazard/repository/docker/hazard/pnp)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n\n> Pulls data from sources and pushes it to sinks with optional transformations in between.\n\n1\\.  [Installation](#installation)  \n2\\.  [Getting started](#gettingstarted)  \n3\\.  [Console runner](#consolerunner)  \n4\\.  [Building Blocks](#buildingblocks)  \n4.1\\.  [Pull](#pull)  \n4.2\\.  [Push](#push)  \n4.3\\.  [Selector](#selector)  \n4.4\\.  [Dependencies](#dependencies)  \n4.5\\.  [Envelope (0.7.1+)](#envelope0.7.1+)  \n4.6\\.  [Payload unwrapping](#payloadunwrapping)  \n5\\.  [Useful hints](#usefulhints)  \n5.1\\.  [Configuration checking](#configurationchecking)  \n5.2\\.  [Logging (0.11.0+)](#logging0.11.0+)  \n5.3\\.  [dictmentor (0.11.0+)](#dictmentor0.11.0+)  \n5.4\\.  [Advanced selector expressions (0.12.0+)](#advancedselectorexpressions0.12.0+)  \n5.5\\.  [UDF Throttle (0.15.0+)](#udfthrottle0.15.0+)  \n5.6\\.  [Docker images](#dockerimages)  \n6\\.  [Plugins](#plugins)  \n\n<a name="installation"></a>\n\n## 1\\. Installation\n\n    pip install pnp\n\nInstallation with extras:\n    \n    pip install pnp[fswatcher,faceR]\n\nPlease consult the component documentation to see if a component requires an extra or not.\n\n<a name="gettingstarted"></a>\n\n## 2\\. Getting started\n\nDefine `pulls` to fetch / pull data from source systems.\nDefine one `push` or multiple `pushes` per pull to transfer the pulled data anywhere else (you only need a plugin that \nknows how to handle the target). You configure your pipeline in `yaml`:\n\n```yaml\n- name: hello-world\n  pull:\n    plugin: pnp.plugins.pull.simple.Repeat\n    args:\n      interval: 1s\n      repeat: "Hello World"\n  push:\n    - plugin: pnp.plugins.push.simple.Echo\n\n```\n        \nCopy this configuration and create the file `helloworld.yaml`. Run it:\n\n    pnp helloworld.yaml\n\nThis example yields the string \'Hello World\' every second.\n\nTip: You can validate your config without actually executing it with\n\n```yaml\n   pnp --check helloworld.yaml\n```\n\n<a name="consolerunner"></a>\n\n## 3\\. Console runner\n\n```\n> pnp --help\nPull \'n\' Push\n\nUsage:\n  pnp [(--engine=<engine>)] [(-v | --verbose)] [--log=<log_conf>] <configuration>\n  pnp (-c | --check) <configuration>\n  pnp (-h | --help)\n  pnp --version\n\nOptions:\n  -c --check        Only check configuration and do not run it.\n  --engine=<engine> Override engine from configuration file (thread, process, sequential, async).\n  -v --verbose      Switches log level to debug.\n  --log=<log_conf>  Specify logging configuration to load.\n  -h --help         Show this screen.\n  --version         Show version.\n```\n\n<a name="buildingblocks"></a>\n\n## 4\\. Building Blocks\n\nBelow the basic building blocks of pull \'n\' push are explained in more detail\n\n\n<a name="pull"></a>\n\n### 4.1\\. Pull\n\nAs stated before pulls fetch data from various source systems and/or apis. Please see the section plugins for shipped\nimplemented pulls. To instantiate a pull by configuration file you only have to provide it\'s fully qualified name\nand the argument that should be passed.\n\n```yaml\n- name: example\n  pull:\n    plugin: pnp.plugins.pull.mqtt.Subscribe\n    args:\n      host: localhost\n      port: 1883\n      topic: test/#\n\n```\n        \nThe above snippet will create a pull that listens on the topic test/# on a mqtt broker. The output of the pull\nis a dictionary that contains the topic, levels and the actual payload.\n\n    # When the message \'Here i am\' arrives on the topic \'test/foo/bar\' then the output will look like that:\n    {\'topic\': \'test/foo/bar\', \'levels\': [\'test\', \'foo\', \'bar\'], \'payload\': \'Here i am\'}\n\n\n<a name="push"></a>\n\n### 4.2\\. Push\n\nA pull passes its data to multiple pushes to transfer/transform the data. For example a push might save sensor data\nto influx or dump a file to the file system.\n\n```yaml\n- name: example\n  pull:\n    plugin: pnp.plugins.pull.mqtt.Subscribe\n    args:\n      host: localhost\n      port: 1883\n      topic: test/#\n  push:\n    - plugin: pnp.plugins.push.fs.FileDump\n      args:\n        directory: "/tmp"\n        binary_mode: false\n    - plugin: pnp.plugins.push.simple.Echo\n\n```\n      \nThe above snippet adds two pushes to the already known pull. The first push takes the incoming data and dumps it into\nthe specified directory as a textfile. The second push just prints out the incoming data.\n\n\n<a name="selector"></a>\n\n### 4.3\\. Selector\n\nSometimes the output of a pull needs to be transformed before the specified push can handle it. `Selectors` to the \nrescue. Given our input we decide to just dump the payload and print out the first level of the topic.\n\n```yaml\n- name: example\n  pull:\n    plugin: pnp.plugins.pull.mqtt.Subscribe\n    args:\n      host: localhost\n      port: 1883\n      topic: test/#\n  push:\n    - plugin: pnp.plugins.push.fs.FileDump\n      selector: data.payload\n      args:\n        directory: "/tmp"\n        binary_mode: false\n    - plugin: pnp.plugins.push.simple.Echo\n      selector: data.levels[0]\n\n```\n\nEasy as that. We can reference our incoming data via `data` or `payload`.\n\n\n<a name="dependencies"></a>\n\n### 4.4\\. Dependencies\n\nBy default any pushes will execute in parallel (not completely true) when new incoming data is available.\nBut now it would be nice if we could chain pushes together. So that the output of one push becomes the\ninput of the next push. The good thing is: Yes we can.\n\nBack to our example let\'s assume we want to print out the path to the created file dump after the dump is created.\n\n```yaml\n- name: example\n  pull:\n    plugin: pnp.plugins.pull.mqtt.Subscribe\n    args:\n      host: localhost\n      port: 1883\n      topic: test/#\n  push:\n    - plugin: pnp.plugins.push.fs.FileDump\n      selector: data.payload\n      args:\n        directory: "/tmp"\n        binary_mode: false\n      deps:\n        - plugin: pnp.plugins.push.simple.Echo\n    - plugin: pnp.plugins.push.simple.Echo\n      selector: data.levels[0]\n\n```\n        \nAs you can see we just add a dependant push to the previous one.\n\n\n<a name="envelope0.7.1+"></a>\n\n### 4.5\\. Envelope (0.7.1+)\n\nUsing envelopes it is possible to change the behaviour of `pushes` during runtime.\nBest examples are the `pnp.plugins.push.fs.FileDump` and `pnp.plugins.push.mqtt.Publish` plugins, where\nyou can override the actual `file_name` and `extension` of the file to dump\nresp. the `topic` where the message should be published.\n\nEasier to understand by a practical example:\n\n```yaml\n- name: envelope\n  pull:\n    plugin: pnp.plugins.pull.simple.Count\n    args:\n      interval: 1s\n  push:\n    plugin: pnp.plugins.push.fs.FileDump\n    selector:\n      file_name: "lambda data: str(data)"\n      extension: ".cnt"\n      data: "lambda data: data"\n    args:\n      directory: "/tmp/counter"\n      file_name: "counter"  # Overridden by envelope\n      extension: ".txt"  #  Overridden by envelope\n      binary_mode: false  # text mode\n\n```\n          \nThe push dumps multiple files (0.cnt, 1.cnt, 2.cnt, ...) for each counter value emitted by the pull,\ninstead of dumping one file \'counter.txt\' which is overridden each time a new counter is emitted.\n\nHow does this work: If the emitted or transformed payload (via selector) contains the key `data` or\n`payload` the pipeline assumes that this is the actual payload and all other keys\nrepresent the so called `envelope`.\n\n<a name="payloadunwrapping"></a>\n\n### 4.6\\. Payload unwrapping\n\nBy default any payload that is provided to a push will be "as-is". If the payload is an iterable, it is possible\nto `unwrap` each individual item of the iterable and provide that single item to the push instead of the whole list.\nYes, now you can perform for each loops with pushes.\n\n```yaml\n- name: unwrapping\n  pull:\n    plugin: pnp.plugins.pull.simple.Repeat\n    args:\n      interval: 1s\n      repeat:\n        - 1\n        - 2\n        - 3\n  push:\n    - plugin: pnp.plugins.push.simple.Echo\n      unwrap: true\n\n```\n\nHint: Selector expressions are applied after unwrapping. So the selector is applied to each individual item.\nIf you need the selector to augment your list, use a `push.simple.Nop` with `unwrap = False` and a dependent push.\n\n\n```yaml\n- name: unwrapping\n  pull:\n    plugin: pnp.plugins.pull.simple.Repeat\n    args:\n      interval: 1s\n      repeat:\n        - 1\n        - 2\n        - 3\n  push:\n    - plugin: pnp.plugins.push.simple.Nop\n      selector: "data + [4, 5, 6]"\n      unwrap: false  # Which is the default\n      deps:\n        - plugin: pnp.plugins.push.simple.Echo\n          unwrap: true\n\n```\n\n\n<a name="usefulhints"></a>\n\n## 5\\. Useful hints\n\n<a name="configurationchecking"></a>\n\n### 5.1\\. Configuration checking\n\nYou can check your pnp configuration file by starting pnp with the `-c | --check` flag set. This will only run\nthe initializer but not execute the configuration.\n\n```bash\npnp --check <pnp_configuration>\n```\n\n<a name="logging0.11.0+"></a>\n\n### 5.2\\. Logging (0.11.0+)\n\nYou can use different logging configurations in two ways:\n\n```bash\n# Specify when starting pnp\npnp --log=<logging_configuration> <pnp_configuration>\n# Specify by environment variable\nexport PNP_LOG_CONF=<logging_configuration>\npnp <pnp_configuration>\n```\n\nA simple logging configuration that will log severe errors to a separate rotating log file looks like this:\n\n```yaml\nversion: 1\ndisable_existing_loggers: False\n\nformatters:\n    simple:\n        format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"\n\nhandlers:\n    console:\n        class: logging.StreamHandler\n        level: DEBUG\n        formatter: simple\n        stream: ext://sys.stdout\n\n    error_file_handler:\n        class: logging.handlers.RotatingFileHandler\n        level: ERROR\n        formatter: simple\n        filename: errors.log\n        maxBytes: 10485760 # 10MB\n        backupCount: 20\n        encoding: utf8\n\nroot:\n    level: INFO\n    handlers: [console, error_file_handler]\n```\n\nA simple slack logging confiuration that will log warnings and errors to a slack channel looks like this;\n\n```yaml\nversion: 1\ndisable_existing_loggers: False\n\nformatters:\n    simple:\n        format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"\n\nhandlers:\n    console:\n        class: logging.StreamHandler\n        formatter: simple\n        stream: ext://sys.stdout\n\n    slack:\n        level: WARNING  # Do not use DEBUG - This will result in a recursion (cause slacker is using urllib which uses logging)\n        api_key: \'<your_api_key>\'  # Retrieve from api.slack.com\n        class: pnp.logging.SlackHandler  # Do not change\n        channel: \'#alerts\'  # The channel to use\n        ping_level: ERROR  # Ping users when the message has this severity\n        ping_users:  # Ping these users (can be real name, display name, internal name, ...)\n          - dede\n\nroot:\n    level: INFO\n    handlers:\n        - slack\n        - console\n```\n\n\n<a name="dictmentor0.11.0+"></a>\n\n### 5.3\\. dictmentor (0.11.0+)\n\nYou can augment the configuration by extensions from the `dictmentor` package.\nPlease see [DictMentor](https://github.com/HazardDede/dictmentor) for further reference.\n\nThe `DictMentor` instance will be instantiated with the following code and thus the following extensions:\n\n```yaml\nfrom dictmentor import DictMentor, ext\nreturn DictMentor(\n    ext.Environment(fail_on_unset=True),\n    ext.ExternalResource(base_path=os.path.dirname(config_path)),\n    ext.ExternalYamlResource(base_path=os.path.dirname(config_path))\n)\n```\n\nExample:\n\n```yaml\n---\n# Uses the dictmentor package to augment the configuration by dictmentor extensions.\n# Make sure to export the environment variable to echo:\n# export MESSAGE="Hello World"\n\n- name: dictmentor\n  pull:\n    plugin: pnp.plugins.pull.simple.Repeat\n    args:\n      wait: 1\n      repeat: "{{env::MESSAGE}}"\n  push:\n    - external: echo.pull\n    - external: nop.pull\n\n```\n\n```yaml\n# Contents of echo.pull\nplugin: pnp.plugins.push.simple.Echo\n```\n\n```yaml\n# Contents of nop.pull\nplugin: pnp.plugins.push.simple.Nop\n```\n\n<a name="advancedselectorexpressions0.12.0+"></a>\n\n### 5.4\\. Advanced selector expressions (0.12.0+)\n\nInstead of string-only selector expressions, you may now use complex dictionary and/or list constructs in your yaml\nto define a selector expression. If you use a dictionary or a list make sure to provide "real" selectors as a\nlambda expression, so the evaluator can decide if this is a string literal or an expression to evaluate.\n\nThe configuration below will repeat `{\'hello\': \'Hello\', \'words\': [\'World\', \'Moon\', \'Mars\']}`.\n\n```yaml\n- name: selector\n  pull:\n    plugin: pnp.plugins.pull.simple.Repeat\n    args:\n      wait: 1\n      repeat: "Hello World Moon Mars"\n  push:\n    - plugin: pnp.plugins.push.simple.Echo\n      selector:\n        hello: "lambda payload: payload.split(\' \')[0]"\n        words:\n          - "lambda payload: payload.split(\' \')[1]"\n          - "lambda payload: payload.split(\' \')[2]"\n          - "lambda payload: payload.split(\' \')[3]"\n```\n\nBefore the implementation of the advanced selector feature your epxressions would have probably looked similiar to this:\n```\ndict(hello=payload.split(\' \')[0], words=[payload.split(\' \')[1], payload.split(\' \')[2], payload.split(\' \')[3]])\n```\nThe first one is more readable, isn\'t it?\n\nAdditional example:\n\n```yaml\n---\n- name: selector\n  pull:\n    plugin: pnp.plugins.pull.simple.Repeat\n    args:\n      interval: 1s\n      repeat: "Hello World"\n  push:\n    - plugin: pnp.plugins.push.simple.Echo\n      # Returns: \'World\'\n      selector: "str(payload.split(\' \')[0])"  # no complex structure. Evaluator assumes that this is an expression -> you do not need a lambda\n    - plugin: pnp.plugins.push.simple.Echo\n      selector:  # Returns {\'header\': \'this is a header\', \'data\': \'World\', \'Hello\': \'World\'}\n        header: this is a header  # Just string literals\n        data: "lambda data: data.split(\' \')[1]"  # Value is lambda and therefore evaluated\n        "lambda data: str(data.split(\' \')[0])": "lambda data: data.split(\' \')[1]"  # Both are lambdas and therefore evaluated\n    - plugin: pnp.plugins.push.simple.Echo\n      selector:  # Returns [\'foo\', \'bar\', \'Hello\', \'World\']\n        - foo  # String literal\n        - bar  # String literal\n        - "lambda d: d.split(\' \')[0]"  # Lambda -> evaluate the expression\n        - "lambda d: d.split(\' \')[1]"  # Lambda -> evaluate the expression\n\n```\n\n<a name="udfthrottle0.15.0+"></a>\n\n### 5.5\\. UDF Throttle (0.15.0+)\n\nConsider the following situation: You have a selector that uses a udf to fetch a state from an external system.\nThe state won\'t change so often, but your selector will fetch the state every time a pull transports a payload to\nthe push. You want to decrease the load on the external system and you want to increase throughput. `Throttle` to the\nrescue. Specifying `throttle` when instantiating your `udf` will manage that your `udf` will only call the external\nsystem once and cache the result. Subsequent calls will either return the cached result or call the external system again\nwhen a specified time has passed since the last call that actually fetched a result from the external system.\n\nExample:\n\n```yaml\nudfs:\n  - name: count  # Instantiate a Counter user defined function\n    plugin: pnp.plugins.udf.simple.Counter\n    args:  # The presence of args tells pnp to instantiate a Counter - important because it has a state (the actual count)\n      init: 1\n      # Will only call the counter if 10 seconds passed between current call and last call.\n      # In the meantime a cached result will be returned.\n      throttle: 5s\ntasks:\n  - name: hello-world\n    pull:\n      plugin: pnp.plugins.pull.simple.Repeat\n      args:\n        interval: 1s\n        repeat: "Hello World"\n    push:\n      - plugin: pnp.plugins.push.simple.Echo\n        selector:\n          counter: "lambda d: count()"\n\n```\n\n\n<a name="dockerimages"></a>\n\n### 5.6\\. Docker images\n\n```bash\n# Mount the task and logging configuration when starting up the container\ndocker run --rm \\\n    -v /path/to/pnp/config/01_hello_world.yaml:/config/config.yaml \\\n    -v /path/to/logging/config/file.logging:/config/logging.yaml \\\n    hazard/pnp:latest\n```\n\n<a name="plugins"></a>\n\n## 6\\. Plugins\n\nA complete list of plugins is available [here](https://github.com/HazardDede/pnp/blob/master/docs/plugins/README.md)\n',
    'author': 'Dennis Muth',
    'author_email': 'd.muth@gmx.net',
    'maintainer': 'Dennis Muth',
    'maintainer_email': 'd.muth@gmx.net',
    'url': 'https://github.com/HazardDede/pnp',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'extras_require': extras_require,
    'entry_points': entry_points,
    'python_requires': '>=3.5,<4.0',
}


setup(**setup_kwargs)
