Metadata-Version: 2.1
Name: prefect-saturn
Version: 0.5.1
Summary: Client library for running Prefect Cloud flows in Saturn Cloud
Home-page: https://saturncloud.io/
Maintainer: Saturn Cloud Developers
Maintainer-email: open-source@saturncloud.io
License: BSD 3-clause
Project-URL: Issue Tracker, https://github.com/saturncloud/prefect-saturn/issues
Project-URL: Source, https://github.com/saturncloud/prefect-saturn
Project-URL: Documentation, http://docs.saturncloud.io
Keywords: dask,data-engineering,prefect,prefect-cloud,saturn-cloud,task-scheduling
Platform: UNKNOWN
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: BSD License
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: System :: Distributed Computing
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Requires-Python: >=3.6
Description-Content-Type: text/markdown
Requires-Dist: cloudpickle
Requires-Dist: dask-saturn (>=0.0.4)
Requires-Dist: packaging
Requires-Dist: prefect (>0.13.0)
Requires-Dist: requests
Requires-Dist: ruamel.yaml
Provides-Extra: dev
Requires-Dist: cloudpickle ; extra == 'dev'
Requires-Dist: dask-saturn (>=0.0.4) ; extra == 'dev'
Requires-Dist: packaging ; extra == 'dev'
Requires-Dist: prefect (>0.13.0) ; extra == 'dev'
Requires-Dist: requests ; extra == 'dev'
Requires-Dist: ruamel.yaml ; extra == 'dev'
Requires-Dist: pytest ; extra == 'dev'
Requires-Dist: pytest-cov ; extra == 'dev'
Requires-Dist: responses ; extra == 'dev'
Provides-Extra: test
Requires-Dist: pytest ; extra == 'test'
Requires-Dist: pytest-cov ; extra == 'test'
Requires-Dist: responses ; extra == 'test'

# prefect-saturn

[![GitHub Actions](https://github.com/saturncloud/prefect-saturn/workflows/GitHub%20Actions/badge.svg)](https://github.com/saturncloud/prefect-saturn/actions/workflows/main.yml?query=branch%3Amain)
[![PyPI Version](https://img.shields.io/pypi/v/prefect-saturn.svg)](https://pypi.org/project/prefect-saturn)

`prefect-saturn` is a Python package that makes it easy to run [Prefect Cloud](https://www.prefect.io/cloud/) flows on a Dask cluster with [Saturn Cloud](https://www.saturncloud.io/). For a detailed tutorial, see ["Fault-Tolerant Data Pipelines with Prefect Cloud
"](https://www.saturncloud.io/docs/tutorials/prefect-cloud/).

## Installation

`prefect-saturn` is available on PyPi.

```shell
pip install prefect-saturn
```

`prefect-saturn` can be installed directly from GitHub

```shell
pip install git+https://github.com/saturncloud/prefect-saturn.git@main
```

## Getting Started

`prefect-saturn` is intended for use inside a [Saturn Cloud](https://www.saturncloud.io/) environment, such as a Jupyter notebook.

```python
import prefect
from prefect import Flow, task
from prefect_saturn import PrefectCloudIntegration


@task
def hello_task():
    logger = prefect.context.get("logger")
    logger.info("hello prefect-saturn")


flow = Flow("sample-flow", tasks=[hello_task])

project_name = "sample-project"
integration = PrefectCloudIntegration(
    prefect_cloud_project_name=project_name
)
flow = integration.register_flow_with_saturn(flow)

flow.register(
    project_name=project_name,
    labels=["saturn-cloud"]
)
```

### Customize Dask

You can customize the size and behavior of the Dask cluster used to run prefect flows. `prefect_saturn.PrefectCloudIntegration.register_flow_with_saturn()` accepts to arguments to accomplish this:

* `dask_cluster_kwargs`: keyword arguments to pass to the constructor [`dask_saturn.SaturnCluster`](https://github.com/saturncloud/dask-saturn/blob/936c91d54964f578b7224fa9c6fea7ea812e47d7/dask_saturn/core.py#L68-L94).
* `dask_adapt_kwargs`: keyword arguments used to configure ["Adaptive Scaling"](https://docs.dask.org/en/latest/setup/adaptive.html)

For example, the code below tells Saturn that this flow should run on a Dask cluster with 3 xlarge workers, and that prefect should shut down the cluster once the flow run has finished.

```python
flow = integration.register_flow_with_saturn(
    flow=flow,
    dask_cluster_kwargs={
        "n_workers": 3,
        "worker_size": "xlarge",
        "autoclose": True
    }
)

flow.register(
    project_name=project_name,
    labels=["saturn-cloud"]
)
```

## Contributing

See [`CONTRIBUTING.md`](./CONTRIBUTING.md) for documentation on how to test and contribute to `prefect-saturn`.


