---
name: datatailr-job-observability
description: Monitor and troubleshoot Datatailr jobs using the CLI. Check job statuses, read logs, list runs, inspect versions, and manage running jobs. Use when the user asks about job status, wants to see logs, check if something is running, debug a failed job, or manage jobs on Datatailr / dt.
---

# Job Observability on Datatailr

All observability commands use the `dt` CLI. Always add `--json` when you need to parse the output programmatically. Human-readable (table) output is the default.

## Safety: confirm before mutating

Commands are split into **read-only** and **mutating**. Always confirm with the user before running a mutating command.

| Category | Commands | Confirmation required? |
|---|---|---|
| Read-only | `job ls`, `job get`, `job runs`, `job versions`, `log read` | No |
| Mutating | `job stop`, `job start`, `job rm`, `job restore`, `job promote` | **Yes -- always ask the user first** |

When the user asks to stop, remove, or promote a job, state exactly what you are about to do (job name, environment, version) and wait for explicit approval before running the command. This is especially important for `job rm` and `job promote -e prod`.

## List jobs

```bash
dt job ls                              # all jobs
dt job ls -f type=app                  # only apps
dt job ls -f type=batch                # only workflows/batches
dt job ls -f type=service              # only services
dt job ls -f type=excel                # only Excel add-ins
dt job ls -f name=my_job               # by exact name
dt job ls -f prefix=my_                # by name prefix
dt job ls --json                       # JSON output
dt job ls --json --pretty              # pretty-printed JSON
```

### `job ls` filter keys

Filters are passed as `-f key=value`. Multiple filters can be comma-separated: `-f type=app,name=my_app`.

| Key | Description | Example values |
|---|---|---|
| `type` | Job type | `app`, `batch`, `service`, `excel`, `workspace` |
| `name` | Exact job name | `my_app` |
| `prefix` | Name prefix | `my_` |
| `id` | Job ID | `123` |
| `environment` | Environment | `dev`, `pre`, `prod` |
| `run-as` | User who owns the job | `alice` |

## Get job details

```bash
dt job get my_job                      # default (dev environment)
dt job get my_job -e prod              # specific environment
dt job get my_job -v 3                 # specific version
dt job get my_job --json               # JSON output
dt job get my_job -c                   # show child jobs (for workflows)
dt job get my_job -b stdout            # read build logs (stdout or stderr)
```

## List runs

Shows execution history for all jobs or filtered by type/state.

```bash
dt job runs                            # all runs
dt job runs -f type=batch              # only workflow runs
dt job runs -f state=running           # only currently running
dt job runs -f state=failed            # only failed
dt job runs -f name=my_job             # runs for a specific job
dt job runs --json                     # JSON output
```

### `job runs` filter keys

| Key | Description | Example values |
|---|---|---|
| `type` | Job type | `app`, `batch`, `service`, `excel` |
| `name` | Exact job name | `my_job` |
| `prefix` | Name prefix | `my_` |
| `job-id` | Job ID | `123` |
| `state` | Run state | `running`, `completed`, `failed`, `building`, `pending`, `stopped` |
| `environment` | Environment | `dev`, `pre`, `prod` |
| `run-id` | Specific run ID | `456` |
| `exit-code` | Exit code | `0`, `1` |

## Read logs

```bash
dt log read my_job                     # last 10 lines of stdout
dt log read my_job -l 100              # last 100 lines
dt log read my_job -f                  # follow (tail -f style)
dt log read my_job -r                  # read stderr instead
dt log read my_job -e prod             # specific environment
dt log read my_job -i 42               # specific run ID
dt log read my_job -c 3                # child job #3 (for workflows)
```

## Job states

| State | Meaning |
|---|---|
| `created` | Job definition saved, not yet built |
| `building` | Container image is being built |
| `ready` | Built and ready to start |
| `scheduled` | Waiting for its scheduled time |
| `pending` | Waiting for resources |
| `starting` | Container is starting up |
| `running` | Actively running |
| `completed` | Finished successfully |
| `failed` | Exited with an error |
| `failed_after` | Exceeded `fail_after` timeout |
| `out_of_memory` | Killed due to OOM |
| `stopped` | Manually stopped |
| `expired` | Cleaned up after `expire_after` |

## Manage jobs

```bash
dt job start my_job                    # start a stopped job
dt job stop my_job                     # stop a running job
dt job start my_job -e prod            # start in specific environment
dt job stop my_job my_other_job        # stop multiple jobs
```

## Versions and promotion

```bash
dt job versions my_job                 # list all versions
dt job versions my_job -e prod         # versions in prod
dt job promote my_job 5 dev            # promote version 5 from dev to next env
```

## Remove and restore

```bash
dt job rm my_job                       # remove job (soft delete)
dt job rm my_job -v 3                  # remove specific version
dt job restore my_job                  # restore a removed job
```

## Troubleshooting workflow

When a user reports a job issue, follow this sequence:

1. **Check status**: `dt job get <name> --json` -- look at `state` and `exit_code`.
2. **Check build logs** (if state is `building` or `failed` right after deploy): `dt job get <name> -b stderr`.
3. **Read runtime logs**: `dt log read <name> -l 50` for stdout, add `-r` for stderr.
4. **Check run history**: `dt job runs -f name=<name> --json` to see if it has been failing repeatedly.
5. **Check OOM**: if state is `out_of_memory`, the job needs more memory in its `Resources`.
6. **Follow live**: `dt log read <name> -f` to watch output in real time.
