# Scale Stress Testing Design

## Table of Contents

1. [Problem](#problem)
   - [Current Scale and Actor Multiplier](#current-scale-and-actor-multiplier)
   - [Known Constraints](#known-constraints)
   - [Bottleneck Categories](#bottleneck-categories)
   - [Pipeline Bottleneck Map](#pipeline-bottleneck-map)
2. [Bottleneck Analysis](#bottleneck-analysis)
   - [Storage I/O (IOPS & Throughput)](#storage-io-iops--throughput)
   - [Actor Coordination & Fan-in](#actor-coordination--fan-in)
   - [Resource Scaling](#resource-scaling)
   - [Data Volume & Error Handling](#data-volume--error-handling)
3. [Mitigations Summary](#mitigations-summary)
4. [Considerations](#considerations)
   - [1. KubeRay Cluster Isolation](#1-kuberay-cluster-isolation)
   - [2. 10,000 Actor Limit](#2-the-10000-actor-limit-is-a-reporting-limit-not-a-hard-limit)
   - [3. Checkpoint Throttling Knobs](#3-checkpoint-throttling-knobs)
   - [4. Hierarchical JobTracker](#4-hierarchical-jobtracker-tree-design)
   - [5. AIMD for Commit Coordination](#5-aimd-for-commit-coordination)
   - [6. Unbounded Queues](#6-unbounded-queues)

**See also**: [Scale Stress Tests](scale_stress_tests.md) — test specifications, implementation plan, and prioritization.

---

## Problem

Geneva production jobs run at up to ~1,000 concurrent workers in real workloads today. We want a systematic understanding of where the system breaks as worker count increases beyond this. The stress test suite in `src/stress_tests/` sweeps up to 2,000 actors across subsystem-targeted tests (jobtracker saturation, actor pool ramp, fragment writer fanout, high fragment count, object store IOPS) with regression thresholds at lower scales and exploratory runs beyond.

These tests find the breaking points of each subsystem so we can (a) set safe concurrency limits, (b) prioritize scalability work, and (c) catch regressions before they hit production.

### Current Scale and Actor Multiplier

A single Geneva job at `concurrency=N` creates roughly `3N + 2` Ray actors:

```
N ApplierActors     (run UDFs)
N FragmentWriters   (write output)
N Queues            (buffer between applier and writer)
1 JobTracker        (centralized metrics)
1 Driver            (pipeline orchestrator)
```

At 1,000 workers that's ~3,002 actors per job. Multiple concurrent jobs compound this — three 1,000-worker jobs approach the Ray State API's 10,000-actor listing cap.

### Known Constraints

| Resource | Limit | Source |
|----------|-------|--------|
| Ray State API actor listing | 10,000 | `RayStateApiException` in `test_actor_overflow.py` |
| JobTracker `max_concurrency` | 1 (serial) | `jobtracker.py` |
| JobTracker resources | 0.1 CPU, 128 MiB | `admission.py` |
| FragmentWriter resources | 0.1 CPU, 1 GiB each | `admission.py` |
| Lance commit retries | 12 | `pipeline.py` |
| Version conflict retries | 10 | `pipeline.py` |
| Pipeline stall timeout | 600s | `pipeline.py` |
| Commit backoff cap | 30s exponential | `pipeline.py` |

### Bottleneck Categories

We group bottlenecks into four categories that share root causes and mitigations:

| Category | Bottlenecks | Core Problem |
|----------|------------|-------------|
| **Storage I/O** | Checkpoint contention, high fragment count, object store IOPS | Checkpoint pipeline generates many small object store ops; providers throttle per-prefix/account |
| **Actor Coordination** | JobTracker mailbox, commit coordination, stall detection | Single-actor fan-in points become latency bottlenecks at high worker count |
| **Resource Scaling** | Ray actor limits, pool ramp-up, writer memory | Each worker needs ~1.1 CPU + 1 GiB; cluster resources and Ray scheduling hit limits |
| **Data Volume & Errors** | Error table write storm | Mass UDF failures flood the error table with unbounded writes |

### Pipeline Bottleneck Map

```mermaid
flowchart TB
    Driver["Driver\n(orchestrator)"]

    subgraph pool["ActorPool — C: Resource Scaling"]
        A0["Applier 0"]
        A1["Applier 1"]
        AN["Applier N"]
    end

    subgraph objstore["Object Store — A: Storage I/O"]
        SRC[("Source\ndata")]
        CKP[("Check-\npoints")]
        MAN[("Manifest")]
    end

    JT["JobTracker\nmax_concurrency=1"]

    subgraph fwm["FragmentWriterManager"]
        S0["Session 0\nQueue + Writer"]
        SN["Session N\nQueue + Writer"]
    end

    COMMIT["Lance commit()"]
    ERR[("geneva_errors\ntable")]

    Driver -- "dispatch tasks" --> pool
    A0 -- "read source" --> SRC
    AN -- "read source" --> SRC
    A0 -- "write ckpts" --> CKP
    AN -- "write ckpts" --> CKP
    A0 -. "metrics" .-> JT
    AN -. "metrics" .-> JT
    A0 -. "errors" .-> ERR
    AN -. "errors" .-> ERR

    Driver -- "ingest results" --> fwm
    S0 -- "read ckpts" --> CKP
    SN -- "read ckpts" --> CKP
    fwm -- "batch commit" --> COMMIT
    COMMIT -- "manifest I/O" --> MAN

    style JT stroke:#c00,stroke-width:2px
    style CKP stroke:#c00,stroke-width:2px
    style MAN stroke:#c00,stroke-width:2px
    style COMMIT stroke:#c00,stroke-width:2px
    style ERR stroke:#c00,stroke-width:2px
```

> **Legend**: Red nodes are bottleneck points. **A: Storage I/O** — checkpoint and manifest IOPS exceed provider limits. **B: Actor Coordination** — JobTracker mailbox and commit serialization. **C: Resource Scaling** — actor count, memory, ramp-up. **D: Data Volume** — error table write storms.

---

## Bottleneck Analysis

### Storage I/O (IOPS & Throughput)

The dominant scaling wall at 80+ nodes. All three bottlenecks in this group share a root cause: Geneva's checkpoint-based pipeline generates many small object store operations, and all major cloud providers throttle per-prefix or per-account.

```mermaid
flowchart LR
    subgraph appliers["N Appliers"]
        A["Applier\n(× N workers)"]
    end

    subgraph objstore["Object Store (gs://bucket/prefix)"]
        SRC[("Source data\n(fragments)")]
        CKP[("Checkpoints\nN × batches")]
        MAN[("Manifest\ngrows with\nfragment count")]
    end

    subgraph fragwriters["N FragmentWriters"]
        FW["FragWriter\n(× N workers)"]
    end

    COMMIT["commit()"]

    A -- "source read\n2+ ops/batch\n(blob doubles)" --> SRC
    A -- "checkpoint write\n~100 PUTs/task" --> CKP
    FW -- "checkpoint read\n~100 GETs/task" --> CKP
    COMMIT -- "manifest read/write\nO(total fragments)" --> MAN
    FW -- "Lance file write" --> objstore

    IOPS["At 800 workers:\n200 ops/task × N\n= 160K+ concurrent ops\nvs. 5K/prefix limit"]

    style CKP stroke:#c00,stroke-width:2px
    style MAN stroke:#c00,stroke-width:2px
    style IOPS stroke:#856404,stroke-width:2px
```

#### Workload-Dependent I/O Profile

Which I/O dimension hits the wall first depends on the UDF workload. Different workloads stress different axes of the object store:

| Workload | Dominant Bottleneck | Why |
|----------|-------------------|-----|
| **Image classification — CPU (phash, aesthetics, NSFW)** | **Read bandwidth** | Each row reads 100KB–1MB of image data but writes only a few bytes of output (8-byte hash or 4-byte score). Workers are almost entirely blocked on object store reads of source images. Checkpoint and output writes are negligible. |
| **Image embedding (CLIP/ViT)** | **Read and write bandwidth** | Reads the same large source images as hashing, but writes high-dimensional embedding vectors (e.g., 768 floats = ~3KB per row). Both source reads and output/checkpoint writes are significant. GPU inference time partially masks I/O, but at scale both read and write bandwidth saturate. |
| **Media download (URL→blob)** | **Write bandwidth** | Source data is a small URL string (~100 bytes). The UDF fetches from an external service and writes the downloaded media (100KB–10MB) into a Lance blob column. Reads from the source table are trivial; writes to object store dominate. External service rate limits may also gate throughput. |
| **Record linkage / dedup** | **IOPS** | Each row requires multiple small random point lookups against a large reference index or table (e.g., checking near-duplicate candidates via a precomputed index). Individual I/Os are small (<1KB) but there are millions of them. Neither read nor write bandwidth is the constraint — the sheer number of small random operations exhausts the object store's per-prefix IOPS budget. |

Understanding the workload's I/O profile determines which mitigations matter most: read-heavy workloads benefit from source data sharding across prefixes/buckets, write-heavy workloads benefit from checkpoint coalescing and separate checkpoint storage, and IOPS-heavy workloads need prefix distribution and request batching.

#### Object Store Throttle Limits

| Dimension | GCS | AWS S3 | Azure Blob Storage (Standard GPv2) |
|---|---|---|---|
| **Read ops/sec** | ~5,000/bucket initial (auto-scales to ~80K with ramp-up) | 5,500/prefix (unlimited prefixes) | 20,000–40,000/account |
| **Write ops/sec** | ~1,000/bucket initial (auto-scales to ~16K) | 3,500/prefix (unlimited prefixes) | 20,000–40,000/account (shared with reads) |
| **Max egress bandwidth** | 200 Gbps/region (~25 GB/s) | No S3-side limit (100 Gbps/instance) | 50–200 Gbps/account (region-dependent) |
| **Max ingress bandwidth** | 200 Gbps/region | No S3-side limit | 25–60 Gbps/account (region-dependent) |
| **Scaling model** | Auto-scales per bucket; double rate every 20 min with gradual ramp-up | Auto-partitions per prefix; scale horizontally by distributing keys | Per-account fixed; request increase via support |
| **Throttle signal** | HTTP 429 | HTTP 503 | HTTP 503 / 500 |

**Key differences**: S3 is most flexible — limits are per-prefix with no cap on prefix count, so distributing keys across prefixes scales linearly. GCS starts conservatively but auto-scales; burst workloads must ramp up gradually or use hierarchical namespace buckets (8× initial QPS). Azure has explicit per-account caps; scaling beyond them requires support tickets or sharding across storage accounts. Azure Premium block blob accounts offer "hundreds of thousands" of requests/sec but have no published fixed numbers.

#### Checkpoint Store Contention

**Architecture**: Workers read and write checkpoint state to track progress. File-based checkpointing is known to fail under thundering herd conditions — `test_metadata_stampede.py` marks the file-based path as `xfail(strict=True)`. Session-mode checkpointing passes at tested scales (10 actors regression, 40 actors exploratory) but is untested at 800+.

**GCS auth token metadata server**: The original motivation for this test was a production incident where many Ray workers simultaneously hitting the GCS auth token metadata server caused throttling and cascading failures. The thundering herd on credential refresh, not checkpoint I/O itself, was the root cause. Workers needed workarounds (credential caching, staggered startup) to avoid saturating this service.

**Key code paths**:
- `stress_tests/test_metadata_stampede.py` — thundering herd test at multiple scales (10 actors × 500 ops regression, 40 actors × 10K ops exploratory)
- Tests both `session` and `file` checkpointer modes; file mode expected to fail under stampede

**Risk**: Checkpoint reads/writes becoming a bottleneck at scale, causing timeouts or data loss. GCS auth token metadata server saturation during mass worker startup or credential refresh.

#### High Fragment Count (150k+ fragments)

**Architecture**: Lance datasets store data in fragments. Geneva discovers fragments via `dataset.get_fragments()`, which loads ALL fragment metadata into memory at once — no pagination, no lazy evaluation. This call appears in multiple critical paths: planning, commit, and metadata operations.

**Known issue**: A table with 150k fragments has been observed appearing "hung" — likely due to manifest I/O or planning phase taking excessive time.

**Recent planning optimizations**: The planning phase (`_plan_read()` in `apply/__init__.py`) has been significantly optimized:

- **In-memory checkpoint index**: `_index_checkpoint_ranges()` builds a set of checkpoint keys once upfront, eliminating per-fragment `checkpoint_store.__contains__()` calls that triggered blob I/O. Previously ~2,000 blob storage calls for 1,000 fragments; now zero.
- **`physical_rows` fast path**: Uses `frag.physical_rows` (in-memory from manifest) instead of `frag.count_rows()` (which does I/O). Only falls back to `count_rows()` for fragments with deletion vectors (`frag.num_deletions > 0`).
- **Pre-computed field IDs and source files**: `relevant_field_ids` and `get_source_data_files` are computed once before the fragment loop instead of per-fragment.
- **Fragment batching via `skip_frags`**: `plan_read()` accepts `skip_frags` + `num_frags` to process slices of a dataset without loading all fragments at once.
- **Dry-run plan API**: `plan_backfill()` and `plan_refresh()` return plan objects (with `has_work`, `total_tasks`, `total_rows_pending`) without launching a distributed pipeline, enabling cheap "has work?" checks.

**Benchmark results** (local filesystem, 1 row/fragment):

| Fragments | Plan time (optimized) | Checkpoint store I/O calls |
|-----------|----------------------|---------------------------|
| 100 (all checkpointed) | 0.021s | 0 |
| 1,000 (all checkpointed) | 0.199s | 0 |
| 10,000 (all checkpointed) | 4.848s | 0 |
| 100 (fresh backfill) | 0.004s | 0 |
| 1,000 (fresh backfill) | 0.052s | 0 |
| 10,000 (fresh backfill) | 2.970s | 0 |

**Remaining scaling issues**:

1. **`get_fragments()` is still O(N)** — the call itself loads all fragment metadata into memory. Used in planning (`apply/__init__.py`), missing column detection (`pipeline.py`), writer tracking (`pipeline.py`), and commit (`pipeline.py`). The planning loop is now fast, but the enumeration cost remains.
2. **ReadTask count = fragment count** — 150k fragments produces 150k tasks, all tracked in the checkpoint store.
3. **Merge fallback enumerates ALL fragments** — when `DataReplacement` fails post-compaction, `_commit_with_merge_fallback()` (`pipeline.py`) iterates every fragment to build the Merge operation. This must reconstruct metadata for all 150k fragments, not just the modified ones.
4. **Manifest file I/O grows linearly** — every commit reads and writes the full manifest. At 150k fragments, manifest I/O itself becomes a bottleneck.
5. **FragmentWriterSession memory** — each active fragment holds a queue, writer actor reference, checkpoint state, and alignment state. At 150k fragments (only a subset active at a time), session creation/teardown overhead accumulates.

**Key code paths**:
- `apply/__init__.py` — `_plan_read()` with in-memory checkpoint index and `skip_frags` batching
- `pipeline.py` — `_get_fragments_missing_column_data()`, iterates all fragments twice
- `pipeline.py` — `_commit_with_merge_fallback()`, iterates ALL fragments
- `pipeline.py` — `_get_fragment_row_counts()` cache (mitigates but doesn't eliminate)

**Risk**: Planning phase is no longer the bottleneck for datasets up to ~10K fragments. Beyond that, `get_fragments()` enumeration cost and commit-phase manifest I/O dominate. Memory pressure from loading all fragment metadata simultaneously remains at 150K+ fragments.

#### Object Store IOPS and Throughput

**Architecture**: Geneva's checkpoint-based pipeline generates many small I/O operations rather than few large ones. Each batch checkpoint is a separate object store object, and the fragment writer reads them all back during assembly.

**Fundamental tension**: Small checkpoints (e.g., `checkpoint_size=100`) mean minimal recompute on failure — only the last 100 rows need to be re-processed. But they generate 10–100× more object store operations than large checkpoints (`checkpoint_size=1000` or `10000`). Large checkpoints reduce IOPS but increase the blast radius of a failure: if a worker dies mid-batch, up to `checkpoint_size` rows of work are lost and must be recomputed. For cheap CPU UDFs this tradeoff favors large checkpoints; for expensive GPU UDFs (minutes per batch), smaller checkpoints are worth the IOPS cost. This is why Geneva includes `AdaptiveCheckpointSizer` (`apply/task.py`) — it dynamically adjusts `checkpoint_size` to target ~10s of processing per batch, balancing IOPS against recompute cost based on observed UDF throughput.

**I/O pattern per ReadTask** (10K rows, `checkpoint_size=100`):

| Phase | Operation | Count |
|-------|-----------|-------|
| Checkpoint writes | Object store Put (one per batch) | ~100 |
| Checkpoint reads | Object store Get (writer collecting) | ~100 |
| Metadata reads | Object store Get (fragment info, manifest) | 2–3 |
| Output write | Lance file write | 1 |
| **Total** | | **~200 ops per task** |

**At scale**: 800 workers × 200 ops/task × multiple tasks/worker = potentially 100K+ concurrent object store operations. This exceeds the per-prefix/account limits of all major providers (see [Object Store Throttle Limits](#object-store-throttle-limits) above).

**Retry configuration** (`object_store_retry.py`):
- Retryable status codes: `{408, 429, 500, 502, 503, 504}`
- Retryable markers: "too many requests", "rate limit", "throttle", etc.
- `GENEVA_APPLIER_TRANSIENT_RETRIES=3` (default), base backoff 1s, max backoff 8s
- Only 3 retries with 8s max backoff is insufficient for sustained object store throttling, which can last minutes.

**No connection pooling or request batching** at the application level — relies on Lance/PyArrow internals.

**Key code paths**:
- `object_store_retry.py` — `GENEVA_APPLIER_TRANSIENT_RETRIES=3`
- `object_store_retry.py` — retryable error detection
- `apply/__init__.py` — `DEFAULT_BATCH_CHECKPOINT_FLUSH_INTERVAL_SECONDS = 10.0`

**Risk**: Object store throttling cascading into stall detection timeouts; insufficient retry budget for sustained throttling; checkpoint I/O becoming the dominant cost at high concurrency.

#### Mitigations: Today (user-tunable)

All environment variables are read at module import time — set before importing Geneva.

| Mitigation | How | Impact |
|-----------|-----|--------|
| Increase checkpoint size | `@udf(checkpoint_size=1000)` or `10000` | Reduces checkpoint file count by 10–100× |
| Increase flush interval | `batch_checkpoint_flush_interval_seconds=60` | Batches multiple checkpoints into fewer object store PUTs |
| Increase commit granularity | `JOB__COMMIT_GRANULARITY=256` (default 64) | Fewer manifest read/write cycles |
| Increase object store retries | `GENEVA_APPLIER_TRANSIENT_RETRIES=10`, `GENEVA_APPLIER_RETRY_MAX_BACKOFF_SECONDS=30` | Survives longer object store throttling bursts |
| Separate checkpoint storage | `JOB__CHECKPOINT__OBJECT_STORE__PATH=gs://other-bucket/ckpts` | Decouples checkpoint IOPS from data IOPS — separate prefix budget |
| Compact before backfill | `table.compact_files()` — target < 10K fragments | Reduces manifest size and planning/commit time |
| Increase commit retries | `GENEVA_COMMIT_MAX_RETRIES=20` | Headroom for Merge fallback on high-fragment datasets |
| Reduce concurrency | Lower `concurrency` to stay within prefix limits | Fewer concurrent writers = less IOPS pressure |
| Amortize reads across columns | Compute multiple outputs (phash, aesthetics, NSFW) in a single UDF pass | For read-amplified workloads, avoids re-reading the same source images N times for N columns. Today requires struct output with manual unnesting; see [multi-column UDF design](multi_column_udf_design.md) for first-class support. |

#### Mitigations: Future (code changes)

**Shard dataset storage across buckets/accounts** (Lance/LanceDB level): A single Lance dataset's data files all live under one object store prefix. If Lance supported writing different fragments to different buckets or prefixes (e.g., fragment 0–999 → `gs://bucket-a/table/`, fragment 1000–1999 → `gs://bucket-b/table/`), each shard would get its own IOPS budget. This is the same idea as writing different data files to different buckets — "striping" at the fragment level. This is a Lance-level feature request, not a Geneva change.

**Separate checkpoint bucket/account**: Same idea applied to checkpoints — the checkpoint store URI already supports pointing to a different bucket (`JOB__CHECKPOINT__OBJECT_STORE__PATH`), so this is partially available today. Full decoupling would mean different cloud projects or accounts with independent IOPS quotas.

**Local NVMe for checkpoints** (non-solution without architectural changes): Using local SSD on worker nodes instead of object store would eliminate checkpoint object store IOPS (~50% of total ops), but defeats the purpose of checkpointing. Checkpoints exist for crash recovery — if checkpoint data lives on the same node that crashed, it's gone exactly when you need it. For cheap CPU UDFs where recompute is cheap, simpler solutions exist (increase `checkpoint_size`, skip checkpoints entirely). For expensive GPU UDFs where checkpoint loss is most painful, these are also the long-running workloads most likely to experience a node failure. Local NVMe optimizes the wrong tradeoff without architectural changes like replicated local storage or a shared filesystem (NFS/Lustre).

**Coalesce checkpoint reads in the writer**: Currently the `FragmentWriter` reads each checkpoint individually (`store[checkpoint_key]` in `writer.py`). A batch-read API (multi-get / pipelined read) could fetch all checkpoints for a fragment in a single pipelined request, reducing object store round-trips from N to 1. This is a writer.py change + a `CheckpointStore.get_many()` API.

**Skip checkpoints for cheap CPU UDFs** (`@udf(checkpoint=False)`): For deterministic, fast UDFs where recomputation is cheap (e.g., simple transforms, string operations), skip checkpoint writes entirely. On failure, re-run the partition from scratch. This eliminates all checkpoint IOPS for qualifying UDFs. Not suitable for expensive GPU UDFs where recomputation cost is high.

**Streaming writes (bypass checkpoint round-trip)**: Current flow: applier → checkpoint (object store) → writer reads checkpoint (object store) → final Lance file (object store). If the applier could stream results directly to the writer actor via the existing Ray Queue, the object store round-trip for checkpoints is eliminated on the hot path. Checkpoints become periodic snapshots for crash recovery rather than per-batch writes. Significantly reduces steady-state IOPS at the cost of more complex failure recovery.

**Lazy fragment enumeration** (Lance level): Replace `dataset.get_fragments()` (which loads all metadata) with a paginated or streaming iterator. Planning and commit operations should process fragments in batches rather than loading 150K fragment metadata entries into memory at once.

**Incremental manifest updates** (Lance level): Instead of reading/writing the full manifest on every commit, support delta-based manifest updates where only modified fragment metadata is written. Reduces commit I/O from O(total_fragments) to O(modified_fragments).

### Actor Coordination & Fan-in

Single-actor serialization points that become latency bottlenecks as worker count grows.

#### JobTracker Mailbox Saturation

```mermaid
flowchart LR
    W0["Applier 0"] -- "batch_increment()" --> JT
    W1["Applier 1"] -- "batch_increment()" --> JT
    WN["Applier N"] -- "batch_increment()" --> JT

    JT["JobTracker\nmax_concurrency=1\nserial mailbox\n~80 msgs/sec at 800w"]

    style JT stroke:#c00,stroke-width:2px
```

**Architecture**: Every worker calls `JobTracker.batch_increment.remote()` after processing a partition. The JobTracker is a Ray actor with `max_concurrency=1` — all calls are serialized through a single-threaded mailbox. It throttles DB writes to one per 5 seconds (`min_time_between_updates_secs=5.0`).

**Pressure model**: At 800 workers each completing a task every ~10 seconds, the mailbox receives ~80 messages/sec. Ray does not expose `max_pending_calls` for actors, so the mailbox grows unboundedly. As queue depth increases, the round-trip latency for metric updates grows, which can trigger stall detection false positives.

**Key code paths**:
- `jobtracker.py` — `batch_increment()`, the hot path every worker calls
- `jobtracker.py` — `_save_with_throttle()`, DB write throttle
- `jobtracker.py` — `job_tracker_options()`, sets `max_concurrency=1`

**Risk**: Metric update latency exceeding the 600s stall timeout, causing false pipeline kills.

#### Commit Coordination (FragmentWriterManager)

```mermaid
flowchart LR
    FWM["FragmentWriterManager"] -- "commit()" --> Lance["Lance dataset"]
    Lance -- "version conflict" --> Retry["Exponential backoff\n0.5 × 2^n, cap 30s\nmax 12 retries"]
    Retry -- "retry" --> Lance

    style Retry stroke:#c00,stroke-width:2px
```

**Architecture**: A single `FragmentWriterManager` actor (`pipeline.py`) handles all fragment commits. `lance.LanceDataset.commit()` is synchronous and serialized. When multiple concurrent jobs backfill to the same dataset, Lance raises "Too many concurrent writers" errors.

**Retry behavior**: Exponential backoff with formula `min(30.0, 0.5 * 2^min(5, retry))`, up to 12 retries per commit and 10 retries for version conflicts. At high contention, total retry time can exceed minutes per commit.

**Key code paths**:
- `pipeline.py` — sequential commit logic with retry strategy
- `pipeline.py` — `GENEVA_COMMIT_MAX_RETRIES = 12`
- `pipeline.py` — `GENEVA_VERSION_CONFLICT_MAX_RETRIES = 10`

**Risk**: 12 retries becomes insufficient under heavy write contention, causing permanent commit failures.

#### Pipeline Stall Detection False Positives

```mermaid
flowchart LR
    Driver["Driver"] -- "poll every 30s" --> Pool["ActorPool\nget_next_unordered()"]
    Pool -- "no result" --> Timer["Stall timer\n600s default"]
    Timer -- "timeout exceeded" --> Kill["TimeoutError\njob killed"]

    style Kill stroke:#c00,stroke-width:2px
```

**Architecture**: The pipeline monitors `pool.get_next_unordered(timeout=30s)` for results. If no results arrive within `PIPELINE_STALL_TIMEOUT_S` (default 600s), the pipeline kills the job.

**Pressure model**: Under heavy load, legitimate work on large/slow partitions combined with queued actor calls can cause long gaps between result arrivals. The 600s timeout may be too aggressive for large-scale jobs.

**Key code paths**:
- `pipeline.py` — stall detection with 30s poll interval
- `pipeline.py` — writer drain detection, stalled after 6 idle rounds

#### Mitigations: Today (user-tunable)

| Mitigation | How | Impact |
|-----------|-----|--------|
| Increase stall timeout | `GENEVA_PIPELINE_STALL_TIMEOUT_S=1800` (default 600) | Prevents false kills at high concurrency |
| Increase writer stall rounds | `GENEVA_WRITER_STALL_IDLE_ROUNDS=12` (default 6) | Tolerates slower writers under load |
| Increase commit retries | `GENEVA_COMMIT_MAX_RETRIES=20`, `GENEVA_VERSION_CONFLICT_MAX_RETRIES=20` | More headroom for commit contention |

#### Mitigations: Future (code changes)

**Hierarchical JobTracker**: Tree of sub-trackers reduces fan-in from N:1 to √N:1 per level. Workers report to a local sub-tracker; sub-trackers aggregate and push to root. Reduces mailbox pressure by an order of magnitude at 2,000+ workers. See [Consideration 4](#4-hierarchical-jobtracker-tree-design) for full design.

**AIMD for commit coordination**: Replace stateless exponential backoff with Additive Increase / Multiplicative Decrease. The FragmentWriterManager maintains a shared contention signal; committers adjust delay based on recent success/failure rates. Converges faster than exponential backoff under heavy contention. See [Consideration 5](#5-aimd-for-commit-coordination).

**Adaptive stall detection**: Instead of a fixed timeout, scale the stall threshold with concurrency and partition size. A formula like `base_timeout + (concurrency * per_worker_overhead)` would auto-adjust.

### Resource Scaling

Hard limits on actors, memory, and ramp-up that determine whether a job can even be scheduled.

```mermaid
flowchart TB
    subgraph job["Single Job (concurrency = N)"]
        appliers["N Appliers\n1 CPU each"]
        writers["N FragWriters\n0.1 CPU, 1 GiB each"]
        queues["N Queues\n0 CPU"]
        jt["1 JobTracker\n0.1 CPU, 128 MiB"]
        driver["1 Driver"]
    end

    job -- "3N + 2 actors" --> cap["Ray State API\n10K actor listing cap"]
    writers -- "N × 1 GiB" --> mem["Writer memory\nN=2000 → 256 GiB"]
    appliers -- "pending tasks" --> unbounded["Unbounded queues\n_pending_submits\ncached_tasks\ncheckpoint_queue"]

    cap -- "N=3,333 → hits cap" --> limit["Monitoring degrades\n(reporting limit, not runtime)"]

    style cap stroke:#c00,stroke-width:2px
    style mem stroke:#c00,stroke-width:2px
    style unbounded stroke:#c00,stroke-width:2px
```

#### Ray GCS / State API Limits

**Architecture**: The Ray Global Control Store tracks all actors. The State API has a 10,000-actor listing cap — exceeding it raises `RayStateApiException`. This is a reporting limit, not a runtime limit (see [Consideration 2](#2-the-10000-actor-limit-is-a-reporting-limit-not-a-hard-limit)). Additionally, all workers simultaneously hit the GCS auth token metadata server during startup.

**Pressure model**: At `3N + 2` actors per job, a single job hits 10k at ~3,333 workers. Two concurrent 800-worker jobs create ~4,804 actors, well within the limit — but three concurrent jobs at 1,200 workers each would exceed it.

**Existing coverage**: `test_actor_overflow.py` verifies the 10k cap exists but doesn't test behavior near the boundary.

#### Actor Pool Ramp-up

**Architecture**: `ActorPool` (`actor_pool.py`) manages worker lifecycle. `_collect_ready_actors()` polls pending actor futures with `ray.wait(futs, num_returns=1, timeout=0.0)` in a tight loop. The poll timeout is 50ms.

**Pressure model**: At 2,000 pending readiness futures, the hot `ray.wait()` loop iterates over a large future set every 50ms. Only tested up to 120 actors in `test_actor_pool_ramp.py`.

**Key code paths**:
- `actor_pool.py` — `_collect_ready_actors()` with `ray.wait()` tight loop
- `actor_pool.py` — 50ms poll timeout

**Risk**: Ramp-up time scaling super-linearly, or the polling loop consuming excessive driver CPU.

#### FragmentWriter Queuing Memory

**Architecture**: Each fragment gets a `ray.util.queue.Queue` actor and a `FragmentWriterSession`. The `SequenceQueue` (`writer.py`) accumulates batches out-of-order and yields in-order, consuming memory proportional to the out-of-order window.

**Resource math at 2,000 fragments**: 2,000 queue actors (0 CPU) + 2,000 writer actors (0.1 CPU, 128 MiB) = 200 CPUs + 256 GiB just for writers.

#### Mitigations: Today (user-tunable)

| Mitigation | How | Impact |
|-----------|-----|--------|
| Use KubeRay cluster-per-job | Separate Ray clusters per job | Eliminates cross-job actor count and resource contention |
| Reduce concurrency for wide schemas | Lower `concurrency` | Fewer writers = less memory pressure |
| Use lower per-UDF CPU | `@udf(num_cpus=0.5)` if UDF is I/O-bound | Fits more workers per node |

#### Mitigations: Future (code changes)

**Writer actor pooling**: Instead of 1 writer per fragment (N writers for N fragments), use a pool of M writers (M << N) that process fragments sequentially. Reduces actor count from 3N+2 to N+M+2 and memory from N GiB to M GiB. Requires changes to `FragmentWriterManager` session management.

**Lazy writer creation**: Don't create writer actors until the first checkpoint for a fragment arrives. For datasets with many fragments but only a subset active at a time, this avoids pre-allocating resources for idle fragments.

**Bounded `_pending_submits`**: Add a high-water mark to `ActorPool._pending_submits` that pauses task submission when the queue exceeds a threshold (e.g., `2 × concurrency`). Prevents driver OOM from queuing millions of tasks.

### Data Volume & Error Handling

Pathological data conditions — mass UDF failures or extreme dataset shapes.

```mermaid
flowchart LR
    subgraph Appliers["N Appliers (mass UDF failure)"]
        A0["Applier 0\n❌ every row fails"]
        A1["Applier 1\n❌ every row fails"]
        AN["Applier N\n❌ every row fails"]
    end

    ET[("geneva_errors\nLance table\n(single dataset)")]

    A0 -- "log_errors()\nper batch" --> ET
    A1 -- "log_errors()\nper batch" --> ET
    AN -- "log_errors()\nper batch" --> ET

    ET --> OOM["In-memory:\n1M rows × 7 retries\n= 7M ErrorRecords"]
    ET --> CC["Lance commit\ncontention on\nerrors table"]

    style ET stroke:#c00,stroke-width:2px
    style OOM stroke:#c00,stroke-width:2px
    style CC stroke:#c00,stroke-width:2px
```

#### Geneva Errors Table Write Storm

**Architecture**: The `geneva_errors` system table (`error_store.py`) stores all UDF execution failures. `ErrorStore.log_errors()` performs a single Lance `.add()` call with all accumulated error records, decorated with `@retry_lance` (7 attempts, 0.5s–120s exponential backoff, configurable via `GENEVA_RETRY_LANCE_ATTEMPTS` and `GENEVA_RETRY_LANCE_MAX_SECS`).

**Write paths**:
- `SimpleApplier` (`simple.py`): calls `error_logger.log_errors()` per-batch — each batch with errors triggers an immediate Lance `.add()`.
- `MultiProcessApplier` (`multiprocess.py`): accumulates ALL errors across batches, single bulk `.add()` at applier completion.
- UDTF actors (`table.py`): return serialized error dicts, bulk write at job finalization.

**No limits exist**: no max error count, no throttling, no cap on error records per job. Configuration controls only `log_errors` (boolean, default `True`) and `log_retry_attempts` (boolean, default `False`). When `log_retry_attempts=True`, each retry attempt generates a separate error record, multiplying total records by `max_attempts`.

**Pressure model**: At 800 workers with a UDF that fails on every row (bad API key, schema mismatch, corrupted data):
- SimpleApplier path at `checkpoint_size=100`: each worker logs errors per-batch → 800 workers × 100 batches = 80,000 Lance `.add()` calls against the errors table.
- MultiProcessApplier path: 800 concurrent bulk writes at applier completion.
- With `max_attempts=7` and `log_retry_attempts=True`, 1M failing rows → 7M error records. Serializing 7M `ErrorRecord` objects via `attrs.asdict()` is itself a memory and CPU concern.
- The errors table uses the same Lance commit path — subject to "Too many concurrent writers" contention.

**Key code paths**:
- `error_store.py` — `log_errors()` bulk Lance `.add()`
- `error_store.py` — `log_error()` single-record `.add()`
- `simple.py` — per-batch error logging
- `multiprocess.py` — bulk error logging at applier completion
- `utils/__init__.py` — `GENEVA_RETRY_LANCE_ATTEMPTS=7`, `GENEVA_RETRY_LANCE_MAX_SECS=120`

**Risk**: Commit contention on errors table under mass UDF failures; OOM from accumulating millions of ErrorRecord objects in-memory before bulk write; error logging consuming the stall timeout window.

#### Mitigations: Today (user-tunable)

| Mitigation | How | Impact |
|-----------|-----|--------|
| Disable error logging | `ErrorHandlingConfig(log_errors=False)` | Eliminates error table IOPS entirely |
| Keep retry logging off | `log_retry_attempts=False` (default) | Avoids N× multiplication of error records |
| Increase Lance retry budget | `GENEVA_RETRY_LANCE_ATTEMPTS=12`, `GENEVA_RETRY_LANCE_MAX_SECS=300` | Error table writes survive contention |
| Use `on_error=fail_fast()` | Fail the job on first error | Avoids accumulating millions of error records |

#### Mitigations: Future (code changes)

**Error sampling / capping**: Instead of logging every error, cap at N errors per job (e.g., 10,000) or sample at a configurable rate. After the cap, stop writing to the error table. Most mass-failure scenarios produce the same error repeated millions of times — the first 100 are diagnostic, the rest are noise.

---

## Mitigations Summary

Quick-reference table of all user-tunable mitigations across bottlenecks. All environment variables are read at module import time — set before importing Geneva.

| Category | Bottleneck | Mitigation | How |
|---|---|---|---|
| **Actor Coordination** | JobTracker Mailbox | Increase stall timeout | `GENEVA_PIPELINE_STALL_TIMEOUT_S=1200` |
| | Commit Coordination | Increase retries | `GENEVA_COMMIT_MAX_RETRIES=20`, `GENEVA_VERSION_CONFLICT_MAX_RETRIES=20` |
| | Stall Detection | Increase timeouts | `GENEVA_PIPELINE_STALL_TIMEOUT_S=1800`, `GENEVA_WRITER_STALL_IDLE_ROUNDS=12` |
| **Resource Scaling** | Ray GCS / State API | Limit concurrent jobs | Total actors < 10k: `concurrency ≤ floor(10000 / (3 * num_jobs))` |
| | Actor Pool Ramp-up | No user knob | Extend stall timeout to cover ramp-up |
| | Writer Memory | Reduce concurrency | Lower `concurrency` for wide schemas |
| **Storage I/O** | Checkpoint Store | Use session-mode checkpointing | Default for Ray pipeline; avoid file-based |
| | High Fragment Count | Run compaction before backfill | `table.compact_files()` — target <10k fragments |
| | High Fragment Count | Increase commit retries | `GENEVA_COMMIT_MAX_RETRIES=20` for Merge fallback |
| | Object Store IOPS | Increase checkpoint size | `checkpoint_size=1000` reduces ops by 10x |
| | Object Store IOPS | Increase retries | `GENEVA_APPLIER_TRANSIENT_RETRIES=10`, `GENEVA_APPLIER_RETRY_MAX_BACKOFF_SECONDS=30` |
| | Object Store IOPS | Reduce concurrency | Stay within object store prefix/account limits |
| **Data Volume** | Error Table Storm | Disable error logging | `ErrorHandlingConfig(log_errors=False)` or keep `log_retry_attempts=False` |
| | Error Table Storm | Increase Lance retry budget | `GENEVA_RETRY_LANCE_ATTEMPTS=12`, `GENEVA_RETRY_LANCE_MAX_SECS=300` |

---

## Considerations

### 1. KubeRay Cluster Isolation

Geneva uses KubeRay and can provision a separate Ray cluster per job. While it is possible to run 10 jobs on the same Ray cluster, it is equally easy to run 10 jobs on 10 separate KubeRay clusters. This significantly changes the bottleneck profile:

**What cluster-per-job eliminates**:
- Ray GCS 10k actor limit — each cluster has its own GCS, so a single job's ~2,400 actors is well within limits.
- Cross-job commit contention — "Too many concurrent writers" from multiple jobs hitting the same Lance dataset is eliminated if jobs run against different datasets, or serialized via separate clusters.
- Cross-job checkpoint store contention — each cluster's workers only contend with each other.

**What cluster-per-job does NOT eliminate**:
- All single-job bottlenecks — the JobTracker mailbox, actor pool ramp-up, stall detection, writer memory, and error table storms are per-job problems regardless of cluster topology.
- Object store IOPS — prefix/account limits are per-bucket, not per-cluster. Ten clusters writing to the same bucket still share the same IOPS quota.
- High fragment count — manifest I/O and planning are per-dataset, not per-cluster.
- Lance commit contention within a single job — the FragmentWriterManager is still a single actor per job.

**Stress test implication**: Tests 1–3 and 6–9 should be designed as single-cluster tests. Tests 4–5 (checkpoint store and commit contention) should have variants for both shared-cluster and isolated-cluster topologies.

### 2. The 10,000 Actor Limit is a Reporting Limit, Not a Hard Limit

Investigation of `test_actor_overflow.py` confirms that the 10,000 actor cap is exclusively a **Ray State API listing limit**, not a runtime constraint:

- `test_actor_listing_explodes()` creates 11,000 actors successfully, then fails only when calling `list_actors()` with `raise_on_missing_output=True`.
- `test_actor_listing_truncates()` creates the same 11,000 actors and successfully queries them with `raise_on_missing_output=False` and a `limit` parameter.

**Geneva's State API usage is non-critical**: `_ray_status()` in `kuberay.py` calls `list_actors()` for monitoring only, wrapped in try-except. The pipeline continues normally if this call fails. It already sets `raise_on_missing_output=False` and `limit=10000` as failsafes.

**Implication**: A single job with >3,333 workers (~10k actors) would function correctly — only metrics reporting would be degraded. The stress tests should verify this by checking that pipeline completion is unaffected when actor count exceeds 10k, even if `_ray_status()` returns truncated results.

### 3. Checkpoint Throttling Knobs

Two existing parameters control checkpoint I/O frequency, and together they are the most impactful levers for reducing object store pressure:

**`batch_checkpoint_flush_interval_seconds`** (default: 10s, `apply/__init__.py`):
- Controls how often the background checkpoint flusher thread writes to storage.
- Multiple mapped batches are concatenated into a single checkpoint file per flush.
- Increasing to 30–120s significantly reduces object store PUTs, at the cost of coarser resume granularity.
- Setting to 0 disables async flushing — every batch is written immediately (maximum I/O).

**`checkpoint_size`** (default: 100 rows, `apply/task.py`):
- Controls how many rows per batch/checkpoint. Larger = fewer checkpoints.
- UDFs can override via `@udf(checkpoint_size=1000)`.
- Adaptive sizing (`AdaptiveCheckpointSizer`) targets ~10s of processing per batch, clamped to `[min_checkpoint_size, max_checkpoint_size]`.
- Increasing to 1,000–10,000 reduces checkpoint count by 10–100×.

**Combined effect**: At `checkpoint_size=100` and `flush_interval=10s`, a 10K-row task produces ~100 checkpoint files. At `checkpoint_size=1000` and `flush_interval=60s`, the same task might produce ~10 files batched into ~2 flushes — a 50× reduction in object store operations.

**Limitation**: Checkpointing cannot be fully disabled in the current design. Every job writes checkpoints for resume capability. The closest workaround is `CheckpointMode.IN_MEMORY` (for testing only, no resume) or very aggressive batching.

### 4. Hierarchical JobTracker (Tree Design)

The current JobTracker is a single actor with `max_concurrency=1`, creating a fan-in bottleneck at ~80 messages/sec (800 workers). A tree-structured design could distribute this load:

```mermaid
flowchart TB
    subgraph root["Root JobTracker"]
        RJT["RootJobTracker\nDB saves every 5s"]
    end

    subgraph sub0["SubTracker 0"]
        ST0["SubTracker\naggregates locally"]
    end
    subgraph sub1["SubTracker 1"]
        ST1["SubTracker\naggregates locally"]
    end
    subgraph sub2["SubTracker 2"]
        ST2["SubTracker\naggregates locally"]
    end

    W0["W0"] --> ST0
    W1["W1"] --> ST0
    W2["W2"] --> ST0
    W3["W3"] --> ST1
    W4["W4"] --> ST1
    W5["W5"] --> ST1
    W6["W6"] --> ST2
    W7["W7"] --> ST2
    W8["W8"] --> ST2

    ST0 -- "merged metrics\nevery ~5s" --> RJT
    ST1 -- "merged metrics\nevery ~5s" --> RJT
    ST2 -- "merged metrics\nevery ~5s" --> RJT
```

**How it would work**:
- N sub-trackers (e.g., `sqrt(concurrency)` ≈ 28 for 800 workers), each handling `concurrency/N` workers.
- Workers send `batch_increment` to their assigned sub-tracker (fire-and-forget, as today).
- Sub-trackers aggregate locally and periodically push merged metrics to the root tracker.
- Root tracker handles DB saves (already throttled to every 5s).

**Benefits**:
- Fan-in reduced from 800:1 to ~28:1 per sub-tracker, then 28:1 at the root.
- Sub-trackers can run with `max_concurrency > 1` since they only aggregate in-memory counters.
- Root tracker receives ~28 messages/5s instead of ~400/5s.

**Costs**:
- Additional actor lifecycle management (creation, fault tolerance, cleanup).
- Slight metric staleness — sub-tracker aggregation adds a delay before metrics reach the root.
- More complex stall detection — need to distinguish "sub-tracker slow" from "workers stalled".

**When to consider**: If stress tests show JobTracker p99 latency exceeding 10s at 800+ workers, this becomes a priority. If latency stays under 5s (likely, since `batch_increment` is just dict updates), the single-actor design may be sufficient to 2,000+ workers.

### 5. AIMD for Commit Coordination

The current commit retry strategy uses exponential backoff: `min(30.0, 0.5 * 2^min(5, retry))`. This is stateless — each committer backs off independently without knowledge of overall contention level. An AIMD (Additive Increase Multiplicative Decrease) approach could be more efficient:

**How it would work**:
- **Additive Increase**: After a successful commit, slightly decrease the delay before the next commit attempt (or proceed immediately).
- **Multiplicative Decrease**: On a "Too many concurrent writers" error, multiply the backoff by a factor (e.g., 2×) and add random jitter.
- **Shared signal**: The FragmentWriterManager could maintain a `contention_level` counter. On each conflict, increment it; on each success, decrement. Use this to set the initial backoff for new commit attempts.

**Advantage over exponential backoff**: AIMD converges to a fair throughput allocation across concurrent committers. Exponential backoff treats each retry independently, so two committers that happen to back off to the same slot will collide again. AIMD's multiplicative decrease separates them faster.

**Advantage over exponential backoff at scale**: With 200 concurrent committers, exponential backoff creates a thundering herd at each retry level. AIMD spreads attempts more evenly over time.

**Implementation sketch**:
```python
class AIMDCommitScheduler:
    def __init__(self, min_delay: float = 0.1, max_delay: float = 60.0):
        self.delay = min_delay
        self.min_delay = min_delay
        self.max_delay = max_delay

    def on_success(self) -> None:
        self.delay = max(self.min_delay, self.delay - 0.1)  # additive decrease

    def on_conflict(self) -> None:
        self.delay = min(self.max_delay, self.delay * 2)  # multiplicative increase

    async def wait(self) -> None:
        jitter = random.uniform(0, self.delay * 0.1)
        await asyncio.sleep(self.delay + jitter)
```

**When to consider**: If stress test 5 (`test_commit_contention.py`) shows that 12 retries with exponential backoff fails at moderate concurrency (e.g., <100 committers), AIMD should be explored. If exponential backoff holds to 200+, it may not be worth the complexity.

### 6. Unbounded Queues

Several internal queues and buffers grow without limit under load. These are potential OOM risks at scale:

| Component | Location | What Accumulates | Bound |
|-----------|----------|-----------------|-------|
| `ActorPool._pending_submits` | `actor_pool.py` | Tasks waiting for idle actors | None — grows when tasks arrive faster than actors complete |
| `ActorPool._future_to_actor` | `actor_pool.py` | In-flight task futures | None — one entry per submitted task |
| `FragmentWriterSession.cached_tasks` | `pipeline.py` | All (offset, result) pairs per fragment, for replay on writer restart | None — grows with every checkpoint |
| `ray.util.queue.Queue` (per fragment) | `pipeline.py` | Checkpoint batches queued for writer | No `maxsize` — created without bound |
| `FragmentWriterManager.to_commit` | `pipeline.py` | Completed fragments waiting for commit | None — clears only at `commit_granularity` threshold |
| `CheckpointingApplier.checkpoint_queue` | `apply/__init__.py` | Batches queued for checkpoint flusher thread | `queue.Queue()` without `maxsize` |
| `all_error_records` | `multiprocess.py` | Error records accumulated per task | None — grows with every failing row |
| JobTracker mailbox | `jobtracker.py` | Pending `batch_increment` calls | `max_concurrency=1`, no `max_pending_calls` |

**Most dangerous at scale**:
- `_pending_submits`: If a job has 100k tasks but only 800 actors, ~99,200 tasks queue up in memory on the driver.
- `cached_tasks`: For a fragment with 10k checkpoints, each session caches all 10k results. On writer restart, the entire cache is replayed into the unbounded Ray Queue.
- `checkpoint_queue`: If the flusher thread falls behind (slow object store), batches pile up in the producer thread's Python queue without bound.

**Mitigation considerations**:
- Add `maxsize` to `ray.util.queue.Queue` and `queue.Queue` instances, with backpressure (block on `put()` when full).
- Cap `cached_tasks` and switch from full replay to checkpoint-based recovery on writer restart.
- Add a high-water mark to `_pending_submits` that pauses task submission when the queue exceeds a threshold.
- These are code changes, not user-tunable knobs — they should be prioritized based on which queues stress tests reveal as actual OOM sources.
