# Job Metrics Reference

This document defines the job-level metrics written by Geneva `JobTracker` and
surfaced in the Console/API job detail payload.

## Metric record schema

Each metric is stored as a JSON object in `geneva_jobs.metrics` with:

- `name`: metric identifier
- `n`: current value
- `total`: expected total for progress metrics (`0` means not set)
- `done`: whether the metric is considered complete
- `desc`: human-readable description shown in UIs

## Where to read metrics

- Console API list endpoint:
  - `GET /api/v1/jobs?table_name=<table>&db_uri_encoded=<...>`
- Console API detail endpoint:
  - `GET /api/v1/jobs/<job_id>?db_uri_encoded=<...>`

## Metric catalog

### Progress and capacity metrics

| Metric | Unit | Meaning | Notes |
| --- | --- | --- | --- |
| `fragments` | read tasks | Number of read tasks scheduled/started. | This is task count, not destination fragment count. |
| `workers` | workers | Current applier worker count in the actor pool. | Can move up/down with actor restarts/shutdown. |
| `writer_fragments` | fragments | Number of destination fragments recorded by writer stage. | `total` is initialized to destination fragment count at job start. |
| `cnt_ray_nodes` | nodes | Number of live Ray nodes. | From periodic cluster status snapshots. |
| `cnt_geneva_workers_active` | workers | Number of parallel UDF executors currently active (current parallelism). | Derived from active applier actors in Ray. |
| `cnt_geneva_workers_pending` | workers | Deficit from the desired parallelism level (workers still pending). | Derived from pending/restarting applier actors in Ray. |
| `rows_checkpointed` | rows | Rows that were checkpointed by read/applier stage. | Initialized with skipped rows from prior checkpoints when applicable. |
| `rows_ready_for_commit` | rows | Rows attached to fragment outputs and ready for atomic commit (to make changes visible to other DB connections). | Usually tracks close to `rows_checkpointed` at steady state. |
| `rows_committed` | rows | Rows already committed and visible to other DB connections. | May lag `rows_ready_for_commit` during commit retries. |
| `udf_values_computed` | rows | Rows actually computed by UDF code path. | Excludes rows skipped due to existing checkpoints. |

### Read/UDF path timing metrics

All metrics in this section are cumulative and measured in milliseconds.

| Metric | Meaning |
| --- | --- |
| `plan_read_time_ms` | Time spent planning read tasks. |
| `read_task_total_time_ms` | End-to-end wall time spent in read tasks (overlaps with sub-metrics below). |
| `read_io_time_ms` | Time spent reading source data. |
| `udf_processing_time` | Time spent inside UDF execution. |
| `batch_checkpointing_time` | Time spent writing per-batch checkpoints. |
| `checkpoint_load_time_ms` | Time spent loading checkpoint payloads. |
| `checkpoint_exists_time_ms` | Time spent checking task-level checkpoint existence. |
| `checkpoint_list_time_ms` | Time spent listing per-fragment checkpoints. |

### Writer path metrics

| Metric | Unit | Meaning |
| --- | --- | --- |
| `fragment_checkpointing_time` | ms | Time spent writing fragment-level checkpoint metadata. |
| `writer_align_time` | ms | Time spent aligning rows for writer output. |
| `writer_write_time` | ms | Time spent writing fragment output files. |
| `writer_queue_wait_time_ms` | ms | Time writers waited for checkpoint queue input. |
| `writer_checkpoint_read_time_ms` | ms | Time writers spent reading checkpoint payloads. |
| `writer_buffer_sort_time` | ms | Deprecated compatibility metric; expected to remain `0`. |
| `avg_batch_num_rows` | rows | Average checkpoint batch row count (unweighted by fragment). |
| `avg_batch_size` | bytes | Average checkpoint batch size (unweighted by fragment). |

### Commit metrics

| Metric | Unit | Meaning |
| --- | --- | --- |
| `commit_time_ms` | ms | Total commit attempt time. |
| `commit_backoff_time_ms` | ms | Total backoff wait time during robust commit retries. |
| `commit_conflict_retries` | count | Number of version-conflict retries. |
| `commit_concurrent_writer_retries` | count | Number of retries due to "Too many concurrent writers". |

## How to use these metrics for "job is slow"

1. Check `rows_checkpointed` vs `rows_ready_for_commit`.
   - If `rows_checkpointed` is high but `rows_ready_for_commit` is low, the
     fragment writer is typically the bottleneck.
   - In practice this usually points to object storage read/write pressure
     (for example S3 throughput/latency bottlenecks).
2. Compare read, UDF, and checkpoint timings.
   - High `read_io_time_ms`: storage or scan bottleneck.
   - High `udf_processing_time`: UDF compute bottleneck.
   - High checkpoint timings (`batch_checkpointing_time`, `checkpoint_*`):
     checkpoint overhead dominates.
   - For checkpoint bottlenecks, common mitigations are increasing
     `checkpoint_size`, increasing `max_checkpoint_size`, or compacting the table
     to produce larger fragments.
3. Check writer timings.
   - High `writer_queue_wait_time_ms`: writer starvation/backpressure.
   - High `writer_write_time`: usually object storage (for example S3)
     throttling / throughput bottleneck.
   - Mitigations include using higher-network-bandwidth node types and keeping
     object storage and compute nodes in the same region.
4. Check commit pressure.
   - High `commit_conflict_retries` / `commit_backoff_time_ms` /
     `commit_concurrent_writer_retries` indicates commit contention.
5. Check cluster capacity indicators.
   - Persistently high `cnt_geneva_workers_pending` means the job is running
     below desired parallelism (parallelism deficit), usually due to
     scheduling/resource constraints.

## Interpretation caveats

- Time metrics are cumulative and may overlap. Do not sum them to estimate wall time.
- Final row counters are reconciled at job end, so completed jobs should show
  stable final `rows_*` values even if incremental tracker updates were delayed.
- `avg_batch_num_rows` and `avg_batch_size` are unweighted by fragment.
- Older jobs may not contain newly introduced metrics.
