---
name: datatailr-deploy-workflow
description: Deploy a DAG workflow on the Datatailr platform using the Python SDK. Workflows are directed acyclic graphs of tasks that run in parallel when possible. Use when the user asks to deploy a workflow, data pipeline, batch job, or DAG on Datatailr / dt, or to make existing Python functions deployable as a workflow.
---

# Deploy a Workflow on Datatailr

## Overview

A Datatailr workflow is a DAG (directed acyclic graph) of tasks. Tasks are Python functions decorated with `@task()`. A workflow function decorated with `@workflow()` composes tasks -- dependencies are inferred automatically from the data flow between calls. Tasks with no dependencies run in parallel. The `datatailr` Python SDK handles packaging, image building, and scheduling. The SDK is already installed in the workspace -- do not look for it.

## Project layout

Do not use hyphens in directory names -- use underscores.

```
my_project/
├── tasks/
│   ├── __init__.py
│   └── processing.py    # @task-decorated functions
└── deploy.py            # @workflow function + invocation
```

## Task functions

Define tasks in a separate module. Each task is a regular Python function decorated with `@task()`:

```python
# tasks/processing.py
from datatailr import task

@task()
def get_data() -> dict:
    return {"a": 1, "b": 2, "c": 3}

@task()
def process_data(data: dict) -> dict:
    data["sum"] = sum(data.values())
    return data

@task(memory="512m", cpu=2)
def heavy_computation(x, y):
    return x ** y
```

`@task()` accepts optional `memory` and `cpu` to override the workflow defaults for that task.

## Workflow function

The workflow function composes tasks. Call tasks like normal functions -- the SDK builds the DAG from the data flow:

```python
# deploy.py
from datatailr import workflow, task
from tasks.processing import get_data, process_data

@task()
def add(a, b):
    return a + b

@workflow(name="My Data Pipeline", python_requirements=["requests"])
def my_pipeline():
    data = get_data()
    process_data(data)

    a = add(1, 2).alias("add_1_2")
    b = add(3, 4).alias("add_3_4")
    add(a, b).alias("sum_both")
```

Key rules:
- **Data passing is automatic.** Each task runs in its own container. When you pass the return value of one task as an argument to another, the platform automatically serializes, transfers, and deserializes it. You do not need to handle files, queues, or any inter-task communication -- just use normal function call syntax.
- **Dependencies are inferred from data flow.** If `process_data(data)` receives the output of `get_data()`, the platform ensures `get_data` finishes before `process_data` starts. No explicit ordering is needed.
- **Independent tasks run in parallel.** Tasks with no data dependency between them are scheduled concurrently.
- To share data between separate jobs (not tasks in the same DAG), use blob storage (see the blob-storage skill).
- When the same task is called multiple times, use `.alias("unique_name")` on each call to avoid name collisions.

## `@workflow()` parameters

- `name`: display name on the platform. Defaults to the function name, title-cased.
- `python_requirements`: pip packages needed by the tasks.
- `resources`: default `Resources(memory=..., cpu=...)` for all tasks. Individual `@task()` overrides take precedence.
- `schedule`: a `Schedule` for recurring execution (see below).
- `env_vars`: dict of environment variables passed to every task container.
- `python_version`: Python version for containers (default `"auto"`).
- `run_as` / `acl`: leave unset to use default permissions. Only if the user explicitly requests access restrictions, see [datatailr-acl.md](../datatailr-acl.md). If their request is vague (e.g., "restrict access"), ask who should be able to see, edit, and operate the job before proceeding.
- `fail_after`: max duration before marked failed (e.g., `"2h"` or `timedelta(hours=2)`).
- `expire_after`: duration after which completed workflow resources are cleaned up.
- `build_script_pre` / `build_script_post`: shell commands for the Docker image build. If the user needs to install OS packages or system libraries, see [datatailr-build-scripts.md](../datatailr-build-scripts.md) for formatting details.

## Scheduling

Use `Schedule` for recurring workflows:

```python
from datatailr import workflow, Schedule

@workflow(
    name="Daily Pipeline",
    schedule=Schedule(at_hours=[8, 16], weekdays=["mon","tue","wed","thu","fri"]),
)
def daily_pipeline():
    ...
```

`Schedule` accepts either a raw `cron_expression` or human-readable helpers: `at_hours`, `at_minutes`, `every_hour`, `every_minute`, `weekdays`, `day_of_month`, `in_month`, `every_month`, `timezone`.

## Deploying

Call the workflow function to deploy:

```python
if __name__ == "__main__":
    my_pipeline()
```

```bash
python deploy.py
```

### Other invocation modes

```python
my_pipeline(local_run=True)   # execute locally for testing
my_pipeline(save_only=True)   # save without starting
my_pipeline(to_json=True)     # return JSON representation without deploying
```

## Making existing code deployable

If the user has existing Python functions and wants to turn them into a Datatailr workflow, see [existing-code.md](existing-code.md).

## Using KV and Secrets in tasks

Each task runs in its own container. If a task reads from KV or Secrets:

- `KV().get()` **raises** if the key does not exist. Ensure keys are created before the workflow runs (ask the user to create it in the UI if required). Alternatively, wrap in try/except with a default.
- `KV().get()` may return a string or an already-parsed object. Always handle both (see the [kv-and-secrets skill](../kv-and-secrets/SKILL.md) for the pattern).

## Checklist

- [ ] Task functions are decorated with `@task()`
- [ ] Workflow function is decorated with `@workflow(name=...)`
- [ ] `.alias()` is used when the same task is called multiple times
- [ ] `python_requirements` includes all task dependencies
- [ ] Deploy script is outside the tasks module directory
- [ ] No hyphens in directory names (use underscores)

## Platform environment variables

The platform sets these environment variables in every running task container. Use them when tasks need to know their runtime context:

- `DATATAILR_JOB_NAME`: name of the running task.
- `DATATAILR_USER`: username running the workflow.
- `DATATAILR_JOB_TYPE`: always `batch` for workflow tasks.
- `DATATAILR_BATCH_RUN_ID`: unique ID for this workflow run.
- `DATATAILR_DOMAIN`: platform domain.
