# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['flowmancer',
 'flowmancer.checkpointer',
 'flowmancer.eventbus',
 'flowmancer.extensions',
 'flowmancer.extensions.notifications',
 'flowmancer.jobdefinition',
 'flowmancer.loggers']

package_data = \
{'': ['*']}

install_requires = \
['PyYAML>=6.0.1,<7.0.0',
 'pydantic>=2.6.3,<3.0.0',
 'requests>=2.31.0,<3.0.0',
 'rich>=13.7.1,<14.0.0',
 'types-pyyaml>=6.0.12.12,<7.0.0.0',
 'urllib3>=2.2.1,<3.0.0']

setup_kwargs = {
    'name': 'flowmancer',
    'version': '0.11.1',
    'description': 'The Python Thing-Doer',
    'long_description': '# Flowmancer\n\n[![pypi-version](https://img.shields.io/pypi/v/flowmancer?style=flat-square)](https://pypi.org/project/flowmancer)\n[![python-version](https://img.shields.io/badge/dynamic/json?query=info.requires_python&label=python&url=https%3A%2F%2Fpypi.org%2Fpypi%2Fflowmancer%2Fjson&style=flat-square)](https://pypi.org/project/flowmancer)\n[![license](https://img.shields.io/github/license/natsunlee/flowmancer?style=flat-square)](LICENSE)\n[![circle-ci](https://img.shields.io/circleci/build/github/natsunlee/flowmancer?style=flat-square)](https://app.circleci.com/pipelines/github/natsunlee/flowmancer)\n[![coveralls](https://img.shields.io/coveralls/github/natsunlee/flowmancer?style=flat-square)](https://coveralls.io/github/natsunlee/flowmancer?branch=main)\n[![pypi-downloads](https://img.shields.io/pypi/dm/flowmancer?style=flat-square)](https://pypistats.org/packages/flowmancer)\n[![Ko-Fi](https://img.shields.io/badge/Support%20Me%20On%20Ko--fi-F16061?style=flat-square&logo=ko-fi&logoColor=white)](https://ko-fi.com/natsunlee)\n\nFlowmancer aims to help you do *things* in a sequential or parallel manner. It enables you to write tasks in Python, describe their order, then execute them with as little effort as possible.\n\nBut why do I need this? Couldn\'t I just write my own Python code to do *stuff*?\n\nYou certainly could!\n\nThough Flowmancer provides gives you a head-start to building your custom processes with optional add-ons for logging, checkpoint/restarts in the event of failures, or even custom task observers to do...things while your things do things!\n\n## Installation\nSimply install the `flowmancer` package with:\n```bash\npip3 install flowmancer\n```\n\nNOTE: `flowmancer` supports only Python 3.7 and higher.\n\n## Basic Usage\nLet\'s assume you have a new project with a basic structure like so:\n```\nmy_project\n├─ job.yaml\n├─ main.py\n└─ tasks/\n   ├─ __init__.py\n   └─ mytasks.py\n```\n\nTo use `flowmancer`, you\'ll need to provide a few things:\n* `Task` implementations (`mytasks.py`)\n* A job YAML file (`job.yaml`)\n* Your main/driver code (`main.py`)\n\n### Tasks\nBy default, Flowmancer recursively searches in the `./tasks` directory (relative to where `Flowmancer()` is initialized - in this case, `main.py`) for `Task` implementations decorated with `@task`. See the Advanced Usage section for details on how to add other directories or packages that contain `Task` implementations.\n\nA `flowmancer` task is simply a class that extends the `Task` abstract class, which, at minimum requires that the `run` method be implemented:\n```python\nimport time\nfrom flowmancer.task import Task, task\n\n@task\nclass WaitAndSucceed(Task):\n    # All variables should be given type hints and optional vars should be given default values.\n    my_required_string_var: str\n    my_optional_int_var: int = 5\n\n    def run(self):\n        # Store string input var in the shared dictionary accessible by other tasks.\n        self.shared_dict["my_var"] = f"Hello from: {self.my_required_string_var}!"\n\n        # Sleep for seconds defined by input var (using default of 5).\n        print(f"Starting up and sleeping for {self.my_optional_int_var} seconds!")\n        time.sleep(self.my_optional_int_var)\n        print("Done!")\n\n@task\nclass ImmediatelySucceed(Task):\n    def run(self):\n        # Print statements will automatically be sent to configured loggers.\n        print("Success!")\n\n@task\nclass FailImmediately(Task):\n    def run(self):\n        print(f"Printing `my_var` value: {self.shared_dict[\'my_var\']}")\n        # Raise errors to cause tasks to fail and additionally block dependent tasks, if any.\n        raise RuntimeError("Let this be caught by Flowmancer")\n```\n\nAny `print()` or exceptions will write log messages to any configured loggers (zero or more loggers may be defined).\n\n### Job Definition YAML File\nThis file describes what code to run, in what order, as well as additional add-ons to supplement the job during execution:\n```yaml\nversion: 0.1\n\n# This entire config block is currently optional, however, it is recommended to at least provide a unique name for each\n# Job Definition YAML file, as this name is used for checkpointing jobs in the event of failures.\nconfig:\n  name: \'my-flowmancer-job\'\n\ntasks:\n  # No dependency - run right away\n  # Add `parameters` key-value pairs for any required and optional task variables.\n  succeed-task-a:\n    task: WaitAndSucceed\n    parameters:\n      my_required_string_var: "My First Task!"\n\n  # No dependency - run right away\n  succeed-task-b:\n    task: ImmediatelySucceed\n\n  # Only run if prior 2 tasks complete successfully\n  final-fail-task:\n    task: FailImmediately\n    max_attempts: 3  # Retry uup to 2 times upon failure (1 initial exec + 2 retries = 3 attempts)\n    dependencies:\n      - succeed-task-a\n      - succeed-task-b\n```\n\n### Driver\nThe driver is super simple and simply requires running an instance of `Flowmancer`\n```python\n# main.py\nimport sys\nfrom flowmancer import Flowmancer\n\nif __name__ == \'__main__\':\n    # The `start()` method will return a non-zero integer on failure, typically equal to the number of failed tasks.\n    ret = Flowmancer().start()\n\n    # Exceptions from tasks will be captured and logged, rather than being raised up to this level. To cause this\n    # driver program to fail, either explicitly raise your own error OR call sys.exit.\n    if ret:\n      raise RuntimeError(\'Flowmancer job has failed!\')\n\n    # Alternatively, instead of crashing w/ an exception, simply exit with a non-zero value.\n    # sys.exit(ret)\n```\n\n### Executing the Job\n```bash\npython3 main.py -j ./path/to/job.yaml\n```\n\nTo run from point-of-failure (if any):\n```bash\npython3 main.py -j ./path/to/job.yaml -r\n```\nIf no prior failure is detected, the job will start as if no `-r` flag were given.\n\nNote that the job definition must still be provided with the `-r` flag.\n\n## Advanced Usage\n\n### Optional Configurations\nIn the `config` block of the Job Definition, the following optional parameters may be given:\n|Parameter|Type|Default Value|Description|\n|---|---|---|---|\n|name|str|\'flowmancer\'|Name/identifier for Job Definition. Used for saving checkpoints used for job restarts in the event of a failure.|\n|max_concurrency|int|0|Maximum number tasks that can run in parallel. If 0 or less, then there is no limit.|\n|extension_directories|List[str]|[]|List of paths, either absolute or relative to driver `.py` file, that contain any `@task`, `@logger`, or `@extension` decorated classes to make accessible to Flowmancer. The `./task`, `./extensions`, and `./loggers` directories are ALWAYS checked by default.|\n|extension_packages|List[str]|[]|List of installed Python packages that contain `@task`, `@logger`, or `@extension` decorated classes to make accessible to Flowmancer.|\n|synchro_interval_seconds|float|0.25|Core execution loop interval for waking and checking status of tasks and whether loggers/extensions/checkpointer should trigger.|\n|loggers_interval_seconds|float|0.25|Interval in seconds to wait before emitting log messages to configured `Logger` instances.|\n|extensions_interval_seconds|float|0.25|Interval in seconds to wait before emitting state change information to configured `Extension` instances.|\n|checkpointer_interval_seconds|float|10.0|Interval in seconds to wait before writing checkpoint information to the configured `Checkpointer`.|\n\nFor example:\n```yaml\nconfig:\n  name: \'most-important-job\'\n  max_concurrency: 20\n  extension_directories:\n    - ./client_implementations\n    - /opt/flowmancer/tasks\n  extension_packages:\n    - internal_flowmancer_package\n```\n\n### Include YAML Files\nAn optional `include` block may be defined in the Job Definition in order to merge multiple Job Definition YAML files.\nYAML files are provided in a list and processed in the order given, with the containing YAML being processed last.\n\nFor example:\n```yaml\n# <app_root_dir>/jobdefs/template.yaml\nconfig:\n  name: generic-template\n\ntasks:\n  do-something:\n    task: DoSomething\n    parameters:\n      some_required_param: I am a required string parameter\n```\n\n```yaml\n# <app_root_dir>/jobdefs/cleanup_addon.yaml\ninclude:\n  - ./jobdefs/template.yaml\n\ntasks:\n  cleanup:\n    task: Cleanup\n    dependencies:\n      - do-something\n```\n\n```yaml\n# <app_root_dir>/jobdefs/complete.yaml\nconfig:\n  name: complete-job\n\ninclude:\n  - ./jobdefs/cleanup_addon.yaml\n\ntasks:\n  do-something:\n    task: Do Something\n    parameters:\n      added_optional_param: 99\n```\n\nLoading the `complete.yaml` job definition will result in a YAML equivalent to:\n```yaml\nconfig:\n  name: complete-job\n\ntasks:\n  do-something:\n    task: Do Something\n    parameters:\n      some_required_param: I am a required string parameter\n      added_optional_param: 99\n```\n\n> :warning: Array values are **NOT** merged like dictionaries are. Any array values (and therfore any nested structures) within them will be replaced if modified in a later YAML.\n\nAdditionally, the above example could have all `include` values in the `complete.yaml` file and the `include` block removed from `cleanup_addon.yaml`:\n```yaml\n# <app_root_dir>/jobdefs/complete.yaml\nconfig:\n  name: complete-job\n\n# As with most paths in Job Definition, paths to `include` YAML files are relative to `.py` file where the `.start()`\n# method for Flowmancer is invoked.\ninclude:\n  - ./jobdefs/template.yaml\n  - ./jobdefs/cleanup_addon.yaml\n\ntasks:\n  do-something:\n    task: Do Something\n    parameters:\n      added_optional_param: 99\n```\n\nThe `include` values are processed in order and results in the same outcome as the original example.\n\n### Changing Default File Logger Directory\nThe Job Definition accepts an optional `loggers` section, which if left empty will default to using a `FileLogger` with default settings.\nTo utilize the default `FileLogger`, but with a different configuration, explicitly provide the `loggers` block:\n```yaml\nloggers:\n  my-file-logger:\n    logger: FileLogger\n    parameters:\n      # NOTE: this path is relative to the `.py` file where `Flowmancer().start()` is invoked.\n      base_log_dir: ./my_custom_log_dir  # ./logs is the default, if omitted.\n      retention_days: 3  # 10 is the default, if omitted.\n```\n\n### Complex Parameters\nWhile this is mostly used for `Task` implementations, the details outlined here apply for any built-in and custom `Extension` and `Logger` implementations.\n\nFlowmancer makes heavy use of [Pydantic](https://docs.pydantic.dev/latest/) to validate parameters and ensure that values loaded from the Job Definition are of the appropriate type.\n\nThis means that a `Task` can have complex types (including custom models) like:\n```python\nfrom enum import Enum\nfrom flowmancer.task import Task, task\nfrom pydantic import BaseModel\nfrom typing import Dict, List\n\nclass Protocol(str, Enum):\n    HTTP: \'HTTP\'\n    HTTPS: \'HTTPS\'\n\nclass APIDetails(BaseModel):\n    protocol: Protocol = Protocol.HTTPS\n    base_url: str\n    endpoint: str\n\n@task\nclass DownloadDataFromRestApi(Task):\n    api_details: APIDetails\n    target_dir: str\n    target_filename: str = \'data.json\'\n\n    def run(self) -> None:\n        url = f\'{self.api_details.protocol}://{self.api_details.base_url}/{self.api_details.endpoint}\'\n        # Continued implementation...\n```\n\nAnd the Job Definition snippet for this task might be:\n```yaml\ntasks:\n  download-file-one:\n    task: DownloadDataFromRestApi\n    parameters:\n      api_details:\n        # We leave out `protocol` because we want to just use the default `HTTPS` value.\n        base_url: www.some_data_api.com\n        endpoint: /v1/data/weather/today\n      target_dir: /data/todays_weather\n      # Override the default `target_filename` value given in the class implementation.\n      target_filename: weather.json\n```\n\n### Task Lifecycle Methods\nIn addition to the required `run` method, an implementation of `Task` may optionally include the following methods:\n|Method|Required|Order|Description|\n|---|---|---|---|\n|on_create|No|1|First method executed when a task is released for execution. Note that a task is not considered "created" until it enters the `RUNNING` state.|\n|on_restart|No|2|Executed only if a task is running from the result of a recovery from `FAILED` state. If a task was failed in `DEFAULTED` state, this method will not be executed.|\n|run|Yes|3|Always required and always executed once task is in `RUNNING` state, unless prior lifecycle methods have failed.|\n|on_success|No|4|Executed only if `run` method ends in success.|\n|on_failure|No|5|Executed only if `run` method ends in failure/exception.|\n|on_destroy|No|6|Always executed after all other lifecycle methods.|\n|on_abort|No|-|Executed when `SIGINT` signal is sent to tasks/Flowmancer.|\n\nJust as with `run`, all lifecycle methods have access to `self.shared_dict` and any parameters.\n\n### Custom Loggers\nCustom implementations of the `Logger` may be provided to Flowmancer to either replace OR write to in addition to the default `FileLogger`.\n\nA custom implementation must extend the `Logger` class, be decorated with the `logger` decorator, and implement the async `update` method at minimum:\n```python\n@logger\nimport json\nimport requests\nfrom flowmancer.loggers.logger import Logger, logger\nfrom flowmancer.eventbus.log import LogEndEvent, LogStartEvent, LogWriteEvent, SerializableLogEvent\n\nclass SlackMessageLogger(Logger):\n    webhook: str\n\n    def _post_to_slack(self, msg: str) -> None:\n        requests.post(\n            self.webhook,\n            data=json.dumps({\'text\': title, \'attachments\': [{\'text\': msg}]}),\n            headers={\'Content-Type\': \'application/json\'},\n        )\n\n    async def update(self, evt: SerializableLogEvent) -> None:\n        # The `LogStartEvent` and `LogEndEvent` events only have a `name` property.\n        if isinstance(evt, LogStartEvent):\n            self._post_to_slack(f\'[{evt.name}] START: Logging is beginning\')\n        elif isinstance(evt, LogEndEvent):\n            self._post_to_slack(f\'[{evt.name}] END: Logging is ending\')\n        # The `LogWriteEvent` additionally has `severity` and `message` properties.\n        elif isinstance(evt, LogWriteEvent):\n            self._post_to_slack(f\'[{evt.name}] {evt.severity.value}: {evt.message}\')\n```\n\nThe `Logger` implementation may also have the following optional `async` lifecycle methods:\n* `on_create`\n* `on_restart`\n* `on_success`\n* `on_failure`\n* `on_destroy`\n* `on_abort`\n\nTo incorporate your custom `Logger` into Flowmancer, ensure that it exists in a module either in `./loggers` or in a module listed in `config.extension_directories` in the Job Definition.\n\nThis allows it to be provided in the `loggers` section of the Job Definition.\n> :warning: Providing the `loggers` section will remove the default logger (`FileLogger`) from your job\'s configuration.\n> If you want to add your custom logger alongside the default logger, the `FileLogger` must explicitly be configured.\n\n```yaml\nloggers:\n  # Load the default logger with default parameters\n  default-logger:\n    logger: FileLogger\n\n  # Custom logger implementation\n  slack-logger:\n    logger: SlackMessageLogger\n    parameters:\n      webhook: https://some.webhook.url\n```\n\n### Custom Extensions\nComing soon.\n\n### Custom Checkpointers\nCustom implementations of the `Checkpointer` may be provided to Flowmancer to replace the default `FileCheckpointer`.\n> :warning: Unlike loggers and extensions, only one checkpointer can be configured per Job Definition.\n\nA custom implementation must extend the `Checkpointer` class, be decorated with the `checkpointer` decorator, and implement the async `write_checkpoint`, `read_checkpoint`, and `clear_checkpoints` methods at minimum. It may also optinoally implement async lifecycle methods, similar to [Custom Loggers](#custom-loggers):\n```python\nfrom .checkpointer import CheckpointContents, Checkpointer, NoCheckpointAvailableError, checkpointer\n\n@checkpointer\nclass DatabaseCheckpointer(Checkpointer):\n    host: str\n    port: int\n    username: str\n    password: str\n\n    def write_checkpoint(self, name: str, content: CheckpointContents) -> None:\n        # Store checkpoint state - must be able to store contents of\n        # `CheckpointContents` in a way that it can be reconstructed later.\n\n    def read_checkpoint(self, name: str) -> CheckpointContents:\n        # Recall checkpoint state - reconstruct and return `CheckpointContents`\n        # if exists for `name`. Otherwise raise `NoCheckpointAvailableError`\n        # to indicate no valid checkpoint exists to restart from.\n\n    def clear_checkpoint(self, name: str) -> None:\n        # Remove checkpoint state for `name`.\n```\n\nTo incorporate your custom `Checkpointer` into Flowmancer, ensure that it exists in a module either in `./extensions` or in a module listed in `config.extension_directories` in the Job Definition.\n\nThis allows it to be provided in the `checkpointer` section of the Job Definition:\n```yaml\ncheckpointer:\n  checkpointer: DatabaseCheckpointer\n  parameters:\n    host: something\n    port: 9999\n    username: user\n    password: 1234\n```\n',
    'author': 'Nathan Lee',
    'author_email': 'lee.nathan.sh@outlook.com',
    'maintainer': 'None',
    'maintainer_email': 'None',
    'url': 'None',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'python_requires': '>=3.8.1,<4.0.0',
}


setup(**setup_kwargs)
