# Geneva Benchmark Guide (Large Image Embedding)

> We should update this document after each performance-improving change so we can continuously track new performance limitations and bottlenecks.

This document summarizes benchmark findings for Geneva's large image embedding pipeline and gives practical guidance for running and tuning benchmarks.

## Scope and Current Baseline

- Benchmark dataset size: approximately 4TB image dataset. Raw runs are tracked in the internal benchmark spreadsheet ([link](https://docs.google.com/spreadsheets/d/1EmYvFPjaDWAy32gAb2z67kzbN6zErVJX8_Ckxhn4gPk/edit?usp=sharing)).
- Full pipeline UDFs:
  - `decode` (CPU)
  - `preprocess` (CPU)
  - `infer`/`vit_logits` (GPU, `num_gpus=1.0`)
- These numbers are expected to change as optimizations and fixes land.
- This benchmark is based on Ray's `large_image_embedding` benchmark ([link](https://github.com/ray-project/ray/tree/master/release/nightly_tests/multimodal_inference_benchmarks/large_image_embedding)).

### Latest validated snapshot (jobtracker metrics)

| Field                                                        | Value                                                                      |
| ------------------------------------------------------------ | -------------------------------------------------------------------------- |
| Date (UTC)                                                   | 2026-03-06                                                                 |
| Workflow                                                     | `.github/workflows/e2e-large-image-embedding.yml`                          |
| `csp`                                                        | `aws`                                                                      |
| `num_images`                                                 | `50000`                                                                    |
| `dataset_path`                                               | `s3://geneva-integ-test-devland-us-east-1/yang/data/large_image_embedding` |
| `num_cpu_workers`                                            | `50`                                                                       |
| `num_gpu_workers`                                            | `10`                                                                       |
| `cpu_worker_memory`                                          | `29Gi`                                                                     |
| `cpu_udf_num_cpus`                                           | `4.0`                                                                      |
| `intra_applier_concurrency`                                  | `1`                                                                        |
| `min_checkpoint_size` / `max_checkpoint_size`                | `256` / `4096`                                                             |
| Pipeline wall time (`decoded -> preprocessed -> vit_logits`) | `1359.25s` (`22m39s`)                                                      |

## Best Practices for Performance Tuning

1. Worker scaling:
   - Treat CPU and GPU pools independently: `num_cpu_workers` and `num_gpu_workers`.
   - Recent runs show useful scaling at least through `num_cpu_workers=50` for this benchmark profile, provided the CPU worker node group can actually scale that far.
   - On the current AWS setup, `CPU_WORKER_CPUS=6` and `CPU_WORKER_MEMORY=29Gi` on `m5.2xlarge` means each CPU node effectively hosts one CPU worker, so the node group `maxSize` must be greater than or equal to the requested CPU worker count plus headroom for head pods.
   - Start from `num_gpu_workers=10` for this benchmark profile and tune with `vit_logits` stage metrics.
2. Applier/concurrency sizing:
   - CPU columns use `cpu_backfill_concurrency = num_cpu_workers`.
   - GPU columns use `gpu_backfill_concurrency = num_gpu_workers`.
   - Keep `intra_applier_concurrency=1` unless there is a measured reason to increase it.
   - Reserve headroom on CPU workers for runtime overhead. For this benchmark profile, set UDF CPU to `cpu_worker_cpus - 2` so each CPU worker tends to run one actor and reduces OOM risk.
3. Replica behavior:
   - Use fixed worker replicas during benchmark runs for repeatability.
   - Scale CPU and GPU groups explicitly to reduce run-to-run variance.
4. Data locality:
   - Prefer AWS for this benchmark because we already have a prepared S3 dataset.
   - Reuse existing dataset paths instead of rebuilding datasets when possible.
5. Isolation:
   - Do not run multiple benchmark workflows simultaneously against the same dataset.
6. Observability:
   - For performance analysis, use `jobtracker` metrics from job records (`/api/v1/jobs` on Geneva Console API).
   - Logs are still useful for failures, but not as reliable as jobtracker totals for throughput/bottleneck attribution.

## System Limits and Key Parameters

Operational limits below are based on observed benchmark behavior (not hard-coded product limits).

| Parameter                   |          Recommended value | Notes                                                         |
| --------------------------- | -------------------------: | ------------------------------------------------------------- |
| `num_cpu_workers`           |                   10 to 50 | Requires CPU node group capacity to match requested workers.  |
| `num_gpu_workers`           |                  around 10 | Tune based on `vit_logits` throughput and cluster capacity.   |
| CPU backfill concurrency    |          `num_cpu_workers` | Used for CPU columns (`decoded`, `preprocessed`).             |
| GPU backfill concurrency    |          `num_gpu_workers` | Used for GPU column (`vit_logits`).                           |
| `intra_applier_concurrency` |                        `1` | Higher values can increase per-worker memory pressure.        |
| `min_checkpoint_size`       |                      `256` | Current validated lower bound for adaptive checkpoint sizing. |
| `max_checkpoint_size`       |                     `4096` | Current validated upper bound for adaptive checkpoint sizing. |
| `num_images`                |         workload-dependent | Latest validated point: `50000` input images in `22m39s`.     |
| `dataset_path`              | set to existing table path | Avoids slow dataset download + conversion path.               |

## How to Make Jobs Faster

1. Use prepared dataset path:
   - `s3://geneva-integ-test-devland-us-east-1/yang/data/large_image_embedding`
2. Run on AWS unless you specifically need GCP comparison.
3. Tune CPU and GPU pools separately:
   - `num_cpu_workers` for `decoded` and `preprocessed`.
   - `num_gpu_workers` for `vit_logits`.
4. Keep checkpoint sizing consistent across comparisons. Current validated profile is `min_checkpoint_size=256` and `max_checkpoint_size=4096`.
5. Avoid concurrent workflows on the same dataset/table.
6. Re-run with the same prepared table when comparing code changes (to isolate system changes from data prep cost).

## Benchmark Dataset and Run Steps

### Dataset options

1. Preferred (fast path): existing S3 table  
   `s3://geneva-integ-test-devland-us-east-1/yang/data/large_image_embedding`
2. Slow path: leave `dataset_path` empty to download source data and convert to Lance table format.
3. Source dataset used by loader script:  
   `s3://anonymous@ray-example-data/image-datasets/10TiB-b64encoded-images-in-parquet-v3/`

### Run steps (GitHub Actions)

1. Open workflow: `.github/workflows/e2e-large-image-embedding.yml`.
2. Trigger `workflow_dispatch` with:
   - `csp`: `aws` (preferred) or `gcp`
   - `num_images`: e.g. `50000`
   - `min_checkpoint_size` / `max_checkpoint_size`: e.g. `256` / `4096`
   - `dataset_path`: existing table path above
   - `fresh_run`: usually `true` for clean comparison
   - `num_cpu_workers`: e.g. `10`
   - `num_gpu_workers`: e.g. `10`
   - `intra_applier_concurrency`: e.g. `1`
3. Wait for the benchmark step (`Run benchmark`) to finish.
4. Check timing in logs:
   - Search for `Runtime:` to get full pipeline runtime (3 columns).
   - Search for `backfill time` entries to inspect per-column backfill timing.
5. Collect jobtracker metrics from Geneva Console API:
   - `GET /api/v1/jobs?table_name=large_image_embedding&db_uri_encoded=...`
   - `GET /api/v1/jobs/{job_id}?db_uri_encoded=...`

## Bottlenecks

1. Fragment writer contention:
   - Issue: too many concurrent fragment writers can trigger S3 throttling.
   - Mitigation: pool fragment writers (WIP/roadmap as of v0.9.1).
2. Single fragment write latency:
   - Issue: each fragment write can still be slow.
   - Status: under investigation.
3. `decoded` remains IO-heavy even after checkpoint-path improvements:
   - End-to-end totals are now more balanced, but `decoded` is still dominated by checkpoint/write/read paths.

### Jobtracker metric breakdown (2026-03-06, `num_images=50000`)

Per-stage runtime from job records:

| Column         | Job ID                                 | Duration (s) | `rows_committed` | Throughput (rows/s) |
| -------------- | -------------------------------------- | -----------: | ---------------: | ------------------: |
| `decoded`      | `288136e5-4046-48dd-9c05-4692d13b1c73` |       577.32 |          201,301 |              348.68 |
| `preprocessed` | `0e5e3951-d54c-47d6-9bfe-15ac9f141ceb` |       426.49 |          201,301 |              472.00 |
| `vit_logits`   | `ea957e27-ff0d-4b57-9ba5-55bb2eb1d50a` |       355.44 |          201,301 |              566.34 |

Per-stage key metrics from `jobtracker.py`:

`decoded`:

| Metric                           |                              Value |
| -------------------------------- | ---------------------------------: |
| `udf_processing_time`            |                       1,781,012 ms |
| `batch_checkpointing_time`       |                       1,994,104 ms |
| `writer_write_time`              |                       1,881,863 ms |
| `writer_checkpoint_read_time_ms` |                       2,006,046 ms |
| `read_io_time_ms`                |                       1,974,143 ms |
| `read_task_total_time_ms`        |                       6,868,381 ms |
| `commit_time_ms`                 |                           4,379 ms |
| `avg_batch_num_rows`             |                                805 |
| `avg_batch_size`                 | 2,515,074,326 bytes (2,398.56 MiB) |

`preprocessed`:

| Metric                           |                          Value |
| -------------------------------- | -----------------------------: |
| `udf_processing_time`            |                   3,227,422 ms |
| `batch_checkpointing_time`       |                     756,536 ms |
| `writer_write_time`              |                     622,815 ms |
| `writer_checkpoint_read_time_ms` |                     555,137 ms |
| `read_io_time_ms`                |                   1,702,178 ms |
| `read_task_total_time_ms`        |                   5,761,763 ms |
| `commit_time_ms`                 |                       4,458 ms |
| `avg_batch_num_rows`             |                            502 |
| `avg_batch_size`                 | 302,065,540 bytes (288.07 MiB) |

`vit_logits`:

| Metric                           |                      Value |
| -------------------------------- | -------------------------: |
| `udf_processing_time`            |               1,148,311 ms |
| `batch_checkpointing_time`       |                  31,192 ms |
| `writer_write_time`              |                  49,472 ms |
| `writer_checkpoint_read_time_ms` |                  75,282 ms |
| `read_io_time_ms`                |                 406,235 ms |
| `read_task_total_time_ms`        |               1,649,406 ms |
| `commit_time_ms`                 |                  21,095 ms |
| `avg_batch_num_rows`             |                        805 |
| `avg_batch_size`                 | 3,227,258 bytes (3.08 MiB) |

Key findings:

1. CPU-side scaling reduced end-to-end stage wall time materially:
   - Compared with the previous successful 2026-03-05 run, total stage wall time fell from `1502.31s` to `1359.25s` (`-9.5%`).
   - The main gains came from `decoded` (`348.68 rows/s`, `+20.7%`) and `preprocessed` (`472.00 rows/s`, `+10.9%`).
2. `decoded` remains the slowest stage, and it is still IO/checkpoint dominated:
   - It has the longest duration (`577.32s`) and the largest `avg_batch_size` (`2,398.56 MiB`).
   - Within `decoded`, `batch_checkpointing_time + writer_write_time + writer_checkpoint_read_time_ms` still exceeds `udf_processing_time` by a wide margin.
3. More CPU parallelism did not reduce checkpoint overhead:
   - `preprocessed` got faster overall, but `batch_checkpointing_time` rose to `756,536ms` and `writer_checkpoint_read_time_ms` rose to `555,137ms`.
   - `vit_logits` stayed roughly flat-to-slightly-slower (`566.34 rows/s`, `-6.6%` vs the previous run), suggesting the latest gains are primarily on the CPU stages rather than the GPU stage.

Metric caveats:

1. `workers` can be noisy/inconsistent (for example negative values were observed in one run), so do not use it as a strict capacity metric.
2. `writer_fragments` total reflects table fragment count, not necessarily only the subset processed in a bounded benchmark run.
3. `rows_committed` can differ across columns when reusing an existing table with partially completed prior work (for example `decoded` vs downstream columns in this run).
