# Monitoring

Workflow monitoring gives you live progress, durable history, and external event delivery without turning the workflow engine into a web-only system.

## Monitoring Surfaces

Workflows currently expose state through four channels:

1. CLI progress output during execution
2. durable SQLite records for runs, steps, and events
3. read-only REST endpoints
4. SSE and outbound notification hooks

These surfaces all come from the same engine events.

## CLI Progress

`aroom workflow run` and `aroom workflow resume` emit live progress while the run is executing. The run ID is printed immediately after creation, before execution begins, so you can start monitoring in a second terminal while the run is in progress.

For background execution, use `aroom workflow run --detach` to create the run and return immediately. Then use `aroom workflow watch <run_id>` to observe progress from any terminal.

### Timeline Display

`aroom workflow watch`, `status`, and `history` use a shared timeline visual vocabulary:

- **Status icons**: `✅` completed, `●` running, `○` pending, `❌` failed, `⏭` skipped, `⏸` paused, `⏳` waiting, `🚫` blocked
- **Friendly durations**: `5m 12s`, `< 1s`, `1h 3m` instead of raw milliseconds
- **Live elapsed**: running steps show updating elapsed time
- **Loop nesting**: steps inside loops are indented under their parent
- **Annotations**: `(cached)`, `(fallback: model)`, `(skipped)`, `(compensation)`, `(no progress)`

When `NO_COLOR` is set, icons degrade to ASCII: `[OK]`, `[..]`, `[  ]`, `[XX]`, `[>>]`, etc.

### Run/Resume Progress

Typical output from `aroom workflow run`:

```text
Starting workflow: issue_delivery v0.1.0
Target: issue:123

  ● gate_issue_current
  ✅ gate_issue_current  < 1s
  ● start_work
    [stdout] Cloning repository...
    [stdout] Checking out branch issue-123-add-widget
  ✅ start_work  18s
  ● fast_checks
    [stdout] Running ruff check src/...
    [stderr] src/foo.py:12: E501 line too long
  ❌ fast_checks  Exit code 1
```

This comes from engine event callbacks using the shared timeline formatters (`status_icon`, `format_duration_ms`). Transcript events (stdout, stderr, assistant messages, tool calls) are emitted during execution and rendered inline under the active step. Tool calls and tool results are shown as compact summaries so workflows that read many files or run verbose shell commands stay readable. Use `--no-transcript` to suppress inline transcript output.

After the live stream, the CLI prints a timeline overview of all steps, a diagnosis hint for non-success runs, and the run id. Non-success runs (including `waiting_for_approval` and `waiting_for_input`) also display **recovery actions** — concrete CLI commands for the next step (approve, deny, respond, resume, cancel). The same recovery actions appear in the web UI run detail panel and are available via the `recovery_actions` field in the `GET /api/workflow-runs/{run_id}` API response.

## Durable Records

The monitoring model starts with SQLite tables:

| Table | Purpose |
|---|---|
| `workflow_runs` | run-level state |
| `workflow_steps` | step execution history |
| `workflow_events` | durable event log |
| `workflow_locks` | target concurrency guard |

That durable state powers:

- `workflow status`
- `workflow history`
- recovery and resume
- the read-only API

`workflow status` output includes:

- **Definition hash:** — the content hash of the workflow definition at the time the run was created; used for drift detection on resume
- **Budget:** — current usage vs. configured limits, e.g. `7/50 steps, 23K/100K tokens, 4:05/60:00`

## Event Bus

Workflow events are also published to Anteroom's existing event bus on:

```text
workflow:{run_id}
```

This matters because the workflow command and the web app may be separate processes. The event bus persists events into `change_log`, and the web app's poller reads them back out for SSE delivery.

That is the bridge that makes CLI-started workflows visible to the web side.

## Event Types

The engine emits workflow events such as:

- `run_started`
- `run_resumed`
- `run_paused`
- `run_completed`
- `run_failed`
- `run_cancelled` — emitted when a run is cancelled via API or CLI (#890)
- `step_started` — includes `attempt` field (1-based) when retry is configured; for steps inside a `parallel` branch, the `step_id` is a qualified ID in the form `branch_id/step_id` (e.g., `fetch_api/call_api`)
- `step_finished`
- `step_failed`
- `step_continued` — emitted when a step with `continue_on: [failed]` fails but the workflow proceeds (#1126); payload includes `result_status`, `summary`, and `continue_on`. Distinct from `step_failed`, which means the run aborted
- `step_skipped` — emitted when a `when` clause evaluates to false; payload includes `reason`
- `step_retry` — emitted before a retry delay; payload includes `attempt`, `max_attempts`, `backoff_seconds`, `error`, and `retry_reason` (`exception`, `failed_status`, or `result_condition`) (#1129)
- `failure_branch_started` — emitted when an `on_failure` branch begins execution (#1127); payload includes `branch_steps` (list of step IDs)
- `failure_branch_completed` — emitted when all `on_failure` branch steps succeed (#1127)
- `failure_branch_failed` — emitted when an `on_failure` branch step fails (#1127); payload includes `failing_branch_step`
- `definition_drift_overridden` — emitted when `--force` overrides a definition hash mismatch on resume; the event payload includes the stored hash and the current hash
- `compensation_started` — emitted when the engine enters compensation phase after a step failure (#978)
- `compensation_step_completed` — emitted when a compensation step succeeds
- `compensation_step_failed` — emitted when a compensation step fails (engine continues best-effort)
- `run_compensated` — emitted when all compensation steps succeed (terminal)
- `run_compensation_failed` — emitted when one or more compensation steps fail (terminal)
- `llm_fallback` — emitted when an LLM step falls back to an alternative model in the fallback chain (#986); payload includes `step_id`, `failed_model`, `error`, `next_index`, and `remaining`
- `step_emitted` — emitted by `emit` steps (#1150); payload includes `level` (info/warning/success/error) and `message`

`run_failed` events may carry a `reason` field of `budget_exceeded:duration`, `budget_exceeded:steps`, or `budget_exceeded:tokens` when a run is stopped by budget enforcement.

## Transcript Events

Workflow steps record durable transcripts of what happened inside each step. Agent runners emit transcript events per-turn during execution. Opaque runners (shell, python_script) emit `transcript_stdout` and `transcript_stderr` per-line during execution, not after process exit. Transcript events are stored in the same `workflow_events` table as all other workflow events and are queryable via the same API.

### Transcript Event Types

| Event type | When emitted | Key payload fields |
|---|---|---|
| `transcript_assistant` | The agent produces a text response (per turn) | `content` (truncated to `max_assistant_chars`), `tokens`, `model` |
| `transcript_tool_call` | The agent invokes a tool | `tool_name`, `arguments` |
| `transcript_tool_result` | A tool returns output | `tool_name`, `status`, `output` (truncated to `max_tool_output_chars`) |
| `transcript_stdout` | A runner step emits stdout | `content` (truncated to `max_stdout_chars`) |
| `transcript_stderr` | A runner step emits stderr | `content` (truncated to `max_stderr_chars`) |
| `transcript_usage` | Cumulative token counts after an API call | `prompt_tokens`, `completion_tokens`, `total_tokens` |

### Storage and Filtering

Transcript events share the `workflow_events` table with all other workflow events. To retrieve only transcript events for a run, filter by event type prefix:

```bash
GET /api/workflow-runs/{run_id}/events?event_type=transcript
```

To retrieve the transcript for a specific step:

```bash
GET /api/workflow-runs/{run_id}/steps/{step_id}/transcript
```

### Configuration

Transcript recording is controlled via the `workflow.transcript` config section. It is enabled by default. See the [config reference](../configuration/config-file.md#workflowtranscript) for all knobs.

Content is truncated per-entry at the configured character limits before storage. Truncation is applied to individual entries, not to the aggregate transcript.

### Run-Level Replay

`aroom workflow replay <run_id>` stitches step-level transcript entries into a full run-level conversation view. It merges transcript events with lifecycle events (`step_started`, `step_finished`, `step_failed`), ordered by insertion id, and renders them with step boundary headers, status indicators, and a summary footer showing total steps and tokens. This is the best command for reviewing what an agent actually said and did across all steps.

### Run Statuses

| Status | Active/Terminal | Description |
|---|---|---|
| `pending` | Active | Created but not yet claimed |
| `claimed` | Active | Claimed by executor, not yet running |
| `running` | Active | Executing steps |
| `paused` | Active | Interrupted, can be resumed |
| `waiting_for_approval` | Active | Paused for tool approval |
| `waiting_for_input` | Active | Paused for human decision |
| `compensating` | Active | Executing rollback steps (#978) |
| `blocked` | Resumable | Gate blocked the run — resume after fixing the condition |
| `completed` | Terminal | All steps succeeded |
| `failed` | Terminal | Step failed, no compensation attempted |
| `cancelled` | Terminal | Operator cancelled |
| `compensated` | Terminal | Step failed, all rollback succeeded (#978) |
| `compensation_failed` | Terminal | Step failed, some rollback also failed (#978) |

On the SSE channel, these are published with `workflow_` prefixes, for example:

- `workflow_run_started`
- `workflow_step_finished`

## Notification Hooks

Workflows can optionally notify external consumers.

Supported transports:

- `webhook`
- `unix_socket`

### Webhooks

Webhook delivery is:

- HTTP `POST`
- JSON payload
- best-effort
- bounded by timeout

Webhook URLs are validated against Anteroom's egress policy before execution:

- `ai.allowed_domains`
- `ai.block_localhost_api`

### Unix Socket Hooks

Unix socket hooks send JSON datagrams to a local Unix socket path.

This is useful when you want local observability or integration without opening network listeners.

### Delivery Guarantees

Hooks are intentionally lightweight:

- hook failures are logged
- hook failures do not fail the workflow
- pending hook deliveries are drained with a bounded timeout before process exit

!!! warning
    Hooks are best-effort notifications, not a transactional outbox. If you need a guaranteed audit source, use the durable workflow tables.

## Token and Cost Data in `budget_usage`

Every run tracks token consumption in `budget_usage`, regardless of whether a budget policy is configured. The `budget_usage` object includes:

- `prompt_tokens` — total prompt tokens across all agent steps
- `completion_tokens` — total completion tokens across all agent steps
- `total_tokens` — sum of prompt + completion
- `estimated_cost_usd` — cost estimate based on configured `cli.usage.model_costs` rates
- `steps_completed` — number of steps that finished
- `elapsed_seconds` — wall-clock duration of the run

`workflow status` displays token and cost data when available. `workflow history` includes a Tokens column per step showing per-step token consumption from `result_artifacts`.

When response caching is enabled on an LLM step, cached responses report `prompt_tokens: 0`, `completion_tokens: 0`, and `total_tokens: 0` — they do not count toward budget enforcement. The original token count is preserved in `result_artifacts.original_tokens` for cost reporting. Steps served from cache show a `(cached)` badge in `workflow status` and `workflow history` output.

## Checkpoints

Checkpoints are durable snapshots of workflow state captured at key boundaries during execution. They record step results, budget usage, and run status at the moment they are created.

### When Checkpoints Are Created

| Event | Label format | Description |
|---|---|---|
| Step completes successfully | `after:<step_id>` | Captured after each successful step |
| All steps complete | `completed` | Final checkpoint for the run |
| Step fails (final attempt) | `failure:<step_id>` | Includes the failing step's error data |
| Gate blocks execution | `blocked:<step_id>` | Includes the blocked step's result |
| Approval pause | `pause:approval:<step_id>` | Run transitions to `waiting_for_approval` |
| Human gate pause | `pause:human_gate:<step_id>` | Run transitions to `waiting_for_input` |
| Compensation starts | `compensation_queue` | Stores the ordered queue of compensate steps, plus completed/failed lists. Resume from `compensating` reloads this checkpoint to skip already-processed entries (#978) |

Failure, block, and pause checkpoints include the current step's result data alongside all prior step results. This means a `failure:step_two` checkpoint contains both `step_one`'s success data and `step_two`'s failure data.

### Checkpoint Contents

Each checkpoint stores:

- **`step_results`** — sanitized map of step ID to result data (status, summary, safe artifact keys). Summaries are truncated to 500 characters. Artifact keys are allowlisted to prevent credential leakage
- **`budget_usage`** — token counts, step count, elapsed time, estimated cost at the time of the checkpoint
- **`run_status`** — the run's status when the checkpoint was created

### Checkpoint API

```
GET /api/workflow-runs/{run_id}/checkpoints
```

Returns a JSON object with a `checkpoints` array. Each entry includes `id`, `label`, `step_id`, `step_results`, `budget_usage`, `run_status`, and `created_at`.

### Checkpoint Visibility in CLI

- **`aroom workflow status <run_id>`** — displays the total checkpoint count for the run
- **`aroom workflow history <run_id>`** — lists all checkpoints with their labels, step IDs, run status, and creation timestamps

### Idempotency Keys in Step History

When a publish step has an `idempotency_key`, the resolved key is persisted on the step record before execution. The `aroom workflow history` command displays the idempotency key in the step details table when present.

## Publish Step Results

Publish steps record their outcome in `result_artifacts`:

- `destination` — adapter-prefixed destination (e.g., `file:/tmp/report.txt`, `webhook:https://example.com/hook`)
- `bytes_written` — for file adapter, the number of bytes written
- `status_code` — for webhook adapter, the HTTP response status code

These artifacts appear in `workflow history` step details.

## Failure Triage Artifacts (#1151)

When a step fails and its output contains parseable failure signals (pytest FAILED lines, Python tracebacks, ruff errors), the engine automatically extracts a `failure_triage` object into `result_artifacts`. This provides normalized, machine-readable fields for repair targeting:

| Field | Type | Description |
|---|---|---|
| `exit_code` | integer | Process exit code (from runner artifacts) |
| `first_failing_test` | string | First FAILED test path (pytest) or first error location (ruff) |
| `first_assertion` | string | Assertion message or error description |
| `first_traceback_file` | string | File path from first traceback frame |
| `first_traceback_line` | integer | Line number from first traceback frame |
| `failure_count` | integer | Total number of failures reported |
| `failure_signature` | string | Stable 16-char SHA-256 prefix for deduplication |

Triage extraction is best-effort — if no structured signal is found, the `failure_triage` key is omitted. The `failure_triage` key is included in `_SAFE_ARTIFACT_KEYS` and survives checkpoint sanitization.

## Structured Outputs in Step History (#1131)

Steps that declare `outputs` have their resolved output values stored in `result_outputs_json` on the step record. These appear in:

- **`workflow history`**: an "Outputs" column in the detail table showing `key=value` pairs
- **`workflow status`** timeline: a `[N outputs]` suffix on the step line
- **Web API**: `result_outputs` field in the step response object at `GET /workflow-runs/{run_id}`
- **Checkpoints**: `result_outputs` is preserved in checkpoint data for resume

## Transcript Rendering

Workflow runs that involve agent steps produce transcript events — assistant messages, tool calls, tool results, stdout/stderr, and usage summaries. These are stored as `transcript_*` event types in the workflow events table.

**CLI:** Use `aroom workflow transcript <run_id> [step_id]` to view transcript events, or pass `--transcript` to `aroom workflow history` to append them after the step and event tables.

**Web UI:** The run detail view in the workflows panel automatically fetches and renders transcript events below the step timeline. Transcript entries use chat-style card formatting consistent with the main chat interface.

## Current Web Scope

The web layer supports run monitoring via REST/SSE and run control via API.

You have:

- REST endpoints for run state, steps, events, and checkpoints
- `POST /api/workflow-runs` to enqueue runs for background execution
- Action endpoints for cancel, resume, approve, deny, respond (#890, #986)
- SSE streaming for real-time progress
- a browser workflows panel with run list, detail view, and live SSE updates

## Typical Monitoring Flows

### Local Developer Flow

1. Start a workflow from the CLI
2. Watch live CLI progress
3. Use `workflow status` or `workflow history` for follow-up inspection
4. Optionally subscribe to SSE from the web side

### External Observer Flow

1. Configure hooks in the workflow definition
2. Start the workflow from the CLI
3. Receive step/run events in your webhook consumer or local socket listener

### Web Observer Flow

1. Query `/api/workflow-runs`
2. Open `/api/workflow-runs/{run_id}`
3. Subscribe to `/api/events?workflow_run_id=<run_id>`

## What Monitoring Is For

Monitoring is not just UI polish. It supports:

- operator awareness
- recovery after interruption
- audit-friendly reconstruction of what happened
- external automation integrations
- CLI/web parity on the same underlying run state

## See Also

- [Workflow Commands](commands.md)
- [Workflow API Reference](api-reference.md)
- [Workflow Definitions](definitions.md)
