# Scale Stress Tests

Companion to [Scale Stress Testing Design](scale_stress_testing.md), which covers the bottleneck analysis, mitigations, and design considerations. This document covers the concrete test specifications, implementation plan, and prioritization.

## Table of Contents

1. [Scaling Scenarios](#scaling-scenarios)
   - [Image Classification (CPU)](#image-classification-cpu) — 5B, 25B, 250B rows
   - [Image Embedding (CLIP/ViT)](#image-embedding-clipvit)
   - [Media Download (URL→Blob)](#media-download-urlblob)
   - [Record Linkage / Dedup (IOPS)](#record-linkage--dedup-iops)
2. [Proposed Stress Tests](#proposed-stress-tests)
   - [Regression vs Exploratory Tests](#regression-vs-exploratory-tests)
   - [Shared Utilities](#shared-utilities-stress_resultspy)
   - [Test 1: JobTracker Mailbox](#test-1-test_jobtracker_saturationpy--jobtracker-mailbox)
   - [Test 2: Pool Ramp-up](#test-2-test_actor_pool_ramppy--pool-ramp-up-at-scale)
   - [Test 3: Writer Actor Count](#test-3-test_fragment_writer_fanoutpy--writer-actor-count-stress)
   - [Test 4: Checkpoint Store](#test-4-test_checkpoint_scalepy--checkpoint-store-at-scale)
   - [Test 5: Lance Commit Races](#test-5-test_commit_contentionpy--lance-commit-races)
   - [Test 6: Full Pipeline Integration](#test-6-test_pipeline_e2e_scalepy--full-pipeline-integration)
   - [Test 7: Error Table Storm](#test-7-test_error_table_stormpy--error-table-write-contention)
   - [Test 8: Fragment Count Scaling](#test-8-test_high_fragment_countpy--fragment-count-scaling)
   - [Test 9: Object Store IOPS](#test-9-test_object_store_iopspy--object-store-throughput-limits)
3. [Implementation Plan](#implementation-plan)
   - [Runtime](#runtime)
   - [File Layout](#file-layout)
4. [Prioritization](#prioritization)
   - [Scenario Priority](#scenario-priority)
   - [Stress Test Priority](#stress-test-priority)
   - [Quick Wins Before Stress Testing](#quick-wins-before-stress-testing)

---

## Scaling Scenarios

Each scenario represents a real or planned Geneva workload with a distinct I/O profile (see [Workload-Dependent I/O Profile](scale_stress_testing.md#workload-dependent-io-profile)). The analysis identifies which bottlenecks hit first and at what scale, informing test prioritization and mitigation strategy.

### Image Classification (CPU)

#### Workload Description

A table with billions of rows where each row contains an image stored as a **Lance blob column**. A CPU-only UDF reads the full image and produces a small scalar or fixed-size output. Examples:

- **Perceptual hash (phash)**: Decode → resize to 32×32 → greyscale → DCT → 8-byte hash
- **Aesthetics score**: Decode → run a lightweight scoring model → single float
- **NSFW score**: Decode → run a classifier → single float or category label

The common pattern is extreme read amplification: each row reads 100KB–1MB of image data but writes only a few bytes of output (8 bytes for phash, 4 bytes for a float score). Compute is fast (milliseconds per image on CPU), so workers are almost entirely I/O-bound.

**Lance blob storage doubles read IOPS**: Blob columns don't store image bytes inline in the row data. Instead, the row contains a reference (pointer) to the blob, and reading the actual image requires a second I/O operation. The read path (`query.py`) is:

1. **First read**: Scan the fragment to get row data including `_rowid` and the blob reference
2. **Second read**: Call `dataset.take_blobs(col_name, ids=row_ids)` to fetch the actual image bytes from blob storage

This means every batch of 100 images requires **at least 2 object store reads** (row scan + blob fetch), not 1. In practice, `take_blobs()` may issue multiple reads depending on how blobs are laid out on disk. The effective per-batch read IOPS is 2× or more compared to inline storage.

#### I/O Profile

| Metric | 5B rows | 25B rows | 250B rows |
|--------|---------|----------|-----------|
| Input image size (typical) | 100 KB – 1 MB per row | same | same |
| Output phash size | 8 bytes per row | same | same |
| Read amplification | ~10,000–100,000× | same | same |
| Object store reads per batch (blob) | 2+ (row scan + `take_blobs()`) | same | same |
| Total source data (at 200 KB avg) | ~1 PB | ~5 PB | ~50 PB |
| Total output data | ~40 GB | ~200 GB | ~2 TB |
| Checkpoint data per batch (`checkpoint_size=100`) | ~800 bytes | same | same |

The key insight is that **reads dominate** this workload. Checkpoint writes and output writes are tiny compared to the source image reads. The bottleneck shifts from checkpoint write IOPS (the general case) to **source data read throughput**. Lance blob storage further compounds this by requiring 2× the read IOPS compared to inline column storage.

#### Read Throughput Math

Object store throughput limits vary by provider (see [Object Store Throttle Limits](scale_stress_testing.md#object-store-throttle-limits)). With Lance blob storage, each batch requires 2+ object store reads (row scan + blob fetch), so effective IOPS per batch is doubled.

| Workers | Images/sec (at 2s/batch of 100) | Read ops/sec (blob: 2× per batch) | Read bandwidth (at 200 KB/image) | vs. object store limits |
|---------|-------------------------------|---------------------------------------|--------------------------------|----------------|
| 580 | ~29,000/sec | ~580 (batches) × 2 = ~1,160 | ~5.8 GB/s | Within egress limits; blob IOPS within prefix limits |
| 1,160 | ~58,000/sec | ~1,160 × 2 = ~2,320 | ~11.6 GB/s | Approaching egress limits; blob IOPS nearing prefix limits |
| 2,325 | ~116,000/sec | ~2,325 × 2 = ~4,650 | ~23.2 GB/s | At egress ceiling (GCS/Azure); blob IOPS at prefix limit |

At 80+ nodes, **source data read throughput becomes the primary bottleneck** — not checkpoint writes. Workers spend most of their time blocked on object store reads of image bytes. This is distinct from the general-case IOPS analysis which focuses on checkpoint and commit operations.

#### Implications for This Workload

- **Checkpoint IOPS is negligible**: Output is 8 bytes per row, so checkpoint files are tiny. Even with `checkpoint_size=100`, each checkpoint is ~800 bytes — object store write IOPS from checkpoints is not the concern.
- **Source read bandwidth is the wall**: 580 workers each reading 200 KB images saturate a single object store prefix long before checkpoint writes matter. The source table's data files all live under one prefix.
- **CPU is underutilized**: Image decode + hash is fast (~ms per image). Workers are I/O-bound waiting for object store reads, so increasing `num_cpus` doesn't help. A lower `@udf(num_cpus=0.5)` would fit more workers per node, but they'd all be blocked on the same object store bandwidth.
- **Fragment layout matters for read parallelism**: If the source table's 5B rows are spread across many fragments in different object store prefixes (e.g., after partitioned ingestion), read bandwidth scales better. If all data is under a single prefix, adding workers beyond the prefix throughput limit produces diminishing returns.
- **Recomputation is cheap**: CPU classification (phash, aesthetics, NSFW) takes ~ms per image and is deterministic. The expensive part is the object store read, not the compute. This makes `@udf(checkpoint=False)` (skip checkpoint writes) a viable option for these workloads — on failure, re-reading the image and recomputing is cheap relative to the I/O cost already paid.
- **Amortize reads across multiple columns**: Since read I/O dominates and compute is cheap, running phash, aesthetics, and NSFW as three separate jobs reads the same images three times — 3× the I/O for ~1× the compute. A single UDF that produces all three outputs in one pass amortizes the read cost. Today this is possible using struct output columns with manual unnesting, which is cumbersome. See [multi-column UDF design](../designs/multi_column_udf_design.md) for first-class support that would allow a single UDF to write multiple top-level columns directly.

#### Cluster Resource Math

Resource math per worker (default `@udf(num_cpus=1.0)`):
- 1.0 CPU (applier) + 0.1 CPU (writer) + 0 CPU (queue) = **1.1 CPU per worker**
- Job overhead: 0.2 CPU (driver + jobtracker)

| | 5B rows | 25B rows | 250B rows |
|---|---------|----------|-----------|
| **Cluster (working today)** | 40 nodes × 16 CPUs = 640 CPUs | 200 nodes × 16 CPUs = 3,200 CPUs | 2,000 nodes × 16 CPUs = 32,000 CPUs |
| **Max workers** | ~580 | ~2,900 | ~29,000 |
| **Actor count** | ~1,742 | ~8,702 | ~87,002 |
| **Writer memory** | 580 GiB | 2.9 TiB | 29 TiB |
| **Fragment count (large appends)** | ~5,000 | ~25,000 | ~250,000 |
| **Fragment count (small appends)** | 50K–150K+ | 250K–750K+ | 2.5M–7.5M+ |
| **Wall time estimate (580 workers)** | ~weeks | ~months | not feasible at this scale |

At 25B rows, the actor count (~8,700) approaches the Ray State API 10K listing cap for a single cluster. At 250B rows, a single-cluster approach is not viable — would require sharding across multiple jobs/clusters, and fragment counts in the millions would make planning and commit operations intractable without lazy fragment enumeration (a Lance-level change).

The practical scaling path: 5B at 40–80 nodes, 25B at 80–200 nodes (requires storage sharding), 250B requires fundamental architectural changes (multi-cluster orchestration, sharded datasets, incremental manifests).

#### 5B Rows — What Breaks at 80 Nodes (1,280 CPUs, ~1,160 workers)

| Category | Bottleneck | Status | Why |
|----------|-----------|--------|-----|
| **Storage I/O** | **Source Read Throughput** | **FIRST TO HIT** | 1,160 workers reading ~200 KB images = ~11.6 GB/s aggregate read bandwidth. Exceeds object store per-prefix read IOPS (GCS: ~5K/sec, S3: ~5.5K/prefix, Azure: 20–40K/account) and approaches project egress limits. Workers block on object store reads, not on compute. |
| **Storage I/O** | **Object Store IOPS** | **Reduced for this workload** | Checkpoint writes are tiny (8 bytes × 100 = 800 bytes per checkpoint). Write IOPS is not the bottleneck — read IOPS from source images is. However, checkpoint _read_ IOPS (writer collecting checkpoints) still applies. |
| **Resource** | **Writer Memory** | **Hard constraint** | 1,160 writers × 1 GiB = 1.16 TiB reserved just for writers. On 80 nodes with 56 GiB each = 4.48 TiB total; writers consume 26% of cluster memory. Leaves less room for applier data. |
| **Storage I/O** | **High Fragment Count** | **Latent risk** | If dataset has 50K+ fragments, planning and commit phases already add minutes. More workers doesn't make this worse, but it doesn't help either — planning is single-threaded. |
| **Coordination** | **JobTracker Mailbox** | **Warm** | ~116 msgs/sec into a single-threaded actor. `batch_increment` is just dict updates (~μs each), so throughput is likely fine. But mailbox depth grows, adding latency to metric visibility. |
| **Coordination** | **Stall Detection** | **Increased risk** | Workers blocked on object store reads create variable batch completion times. Slow partitions (large images, cold storage cache) are more likely to trigger the 600s stall timeout. |
| **Resource** | **Actor Pool Ramp-up** | **Untested** | 1,160 actors is 10× beyond tested range (120). Likely works but ramp-up time unknown. |
| Resource | Ray Actor Limit | Fine | 3,482 actors — well under 10k. |
| Coordination | Commit Coordination | Same | Single FragmentWriterManager per job; contention doesn't change with worker count. |
| Data/Errors | Error Table | Same | Only matters if UDFs fail at high rates. |
| Storage I/O | Checkpoint Store | Fine | Session-mode handles concurrent access; checkpoint payloads are tiny for this workload. |

#### 5B Rows — What Breaks at 160 Nodes (2,560 CPUs, ~2,325 workers)

| Category | Bottleneck | Status | Why |
|----------|-----------|--------|-----|
| **Storage I/O** | **Source Read Throughput** | **CRITICAL** | 2,325 workers × 200 KB/image × 50 images/sec = ~23.2 GB/s. At the object store egress ceiling (GCS: 25 GB/s, Azure: 6–25 GB/s, S3: per-instance). Per-prefix read IOPS is 10× over limit. Workers are almost entirely I/O-bound — adding more nodes yields no speedup without storage sharding. |
| **Storage I/O** | **Object Store IOPS** | **Compounded** | Even though checkpoint writes are tiny, 2,325 workers still generate checkpoint _read_ IOPS in the writer. Combined with source reads, aggregate IOPS far exceeds prefix limits. |
| **Resource** | **Writer Memory** | **Critical constraint** | 2,325 writers × 1 GiB = 2.3 TiB. On 160 nodes with 56 GiB each = 8.96 TiB total; writers consume 26%. This is feasible but leaves 6.6 TiB for everything else. |
| **Resource** | **Unbounded Queues** | **OOM risk** | With 5B rows / 100 rows per checkpoint = 50M checkpoints. `_pending_submits` could hold millions of tasks. `cached_tasks` per fragment could hold 10K+ entries. Driver memory becomes the bottleneck. |
| **Coordination** | **JobTracker Mailbox** | **Hot** | ~232 msgs/sec. Still just dict updates, but mailbox depth at 2,325 concurrent senders means metric latency could reach seconds. Fire-and-forget pattern means this doesn't block workers, but stall detection relies on timely metrics. |
| **Storage I/O** | **High Fragment Count** | **Same latent risk** | More workers doesn't worsen this, but if the 5B-row dataset has 100K+ fragments, the planning phase and Merge commits are already multi-minute operations. |
| **Resource** | **Actor Pool Ramp-up** | **Unknown** | 2,325 actors — 20× beyond tested range. The `ray.wait()` polling loop processes one ready actor per iteration; ramp-up could take minutes. |
| Resource | Ray Actor Limit | Fine | 6,977 actors — under 10k for a single job. |
| Coordination | Commit Coordination | Same | Per-job, unchanged by worker count. |
| Data/Errors | Error Table | Same | Latent risk only. |
| Storage I/O | Checkpoint Store | Fine | Tiny checkpoint payloads; 2,325 concurrent session writers untested but likely manageable. |

#### 25B Rows — Additional Bottlenecks

Everything from the 5B analysis applies, plus:

| Category | Bottleneck | Status | Why |
|----------|-----------|--------|-----|
| **Resource** | **Ray Actor Limit** | **At limit** | ~8,700 actors for a single job approaches the 10K State API listing cap. Monitoring degrades; multiple concurrent jobs on the same cluster would exceed it. Cluster-per-job isolation becomes mandatory. |
| **Storage I/O** | **High Fragment Count** | **Critical** | 25K–750K+ fragments depending on ingestion history. Planning and commit phases at 100K+ fragments take minutes. Merge fallback enumerates all fragments. Compaction before backfill is essential. |
| **Resource** | **Unbounded Queues** | **Critical** | 250M checkpoints (25B rows / 100). Driver-side queues (`_pending_submits`, `cached_tasks`) would OOM without bounded queues or streaming task submission. |
| **Storage I/O** | **Source Read Throughput** | **Requires sharding** | 5× the data of 5B. A single bucket/prefix is completely saturated. Must shard source data across multiple buckets and potentially multiple cloud projects for independent IOPS quotas. |

#### 250B Rows — Architectural Limits

At 250B rows, Geneva hits fundamental limits that require architectural changes beyond tuning:

| Category | Bottleneck | Status | Why |
|----------|-----------|--------|-----|
| **Resource** | **Ray Actor Limit** | **Far exceeded** | ~87,000 actors — 9× the State API cap. Single-cluster is not viable. Requires multi-cluster orchestration with job sharding across clusters. |
| **Storage I/O** | **High Fragment Count** | **Intractable** | 2.5M–7.5M+ fragments. `get_fragments()` loads all metadata into memory — OOM at this scale. Lazy fragment enumeration (Lance-level change) and incremental manifests are prerequisites. |
| **Resource** | **Unbounded Queues** | **Intractable** | 2.5B checkpoints. No in-memory queue can hold this. Requires streaming task submission with bounded driver-side state. |
| **Storage I/O** | **Source Data** | **Multi-region** | ~50 PB of source data. Exceeds single-region storage and egress budgets. Requires data distribution across regions and compute co-located with data. |
| **Coordination** | **Single-job model** | **Not viable** | A single Geneva job cannot orchestrate 29,000 workers. Requires partitioning into many independent sub-jobs (e.g., by fragment range) with external orchestration to track overall progress. |

### Image Embedding (CLIP/ViT)

#### Workload Description

A table with billions of rows where each row contains an image stored as a **Lance blob column**. A GPU UDF runs a vision model (CLIP ViT-L/14, BLIP, etc.) to produce a dense embedding vector per image. Examples:

- **CLIP embedding**: Decode → resize/normalize → ViT forward pass → 768-float vector (~3KB)
- **ViT-L classification logits**: Decode → resize/normalize → ViT forward pass → 1000-float vector (~4KB)
- **BLIP captioning embedding**: Decode → multimodal model → embedding vector

The key difference from CPU classification: **compute is expensive** (50–200ms per image on GPU vs ~1ms on CPU), **output is large** (3–4KB per row vs 8 bytes), and **GPU memory constrains workers per node** rather than CPU count.

**Lance blob storage doubles read IOPS**: Same as the classification scenario — each batch requires 2+ object store reads (row scan + blob fetch). See [Image Classification](#image-classification-cpu) for details.

#### I/O Profile

| Metric | 5B rows | 25B rows |
|--------|---------|----------|
| Input image size (typical) | 100 KB – 1 MB per row | same |
| Output embedding size | ~3 KB per row (768 × float32) | same |
| Read amplification | ~30–300× (smaller than classification due to larger output) | same |
| Object store reads per batch (blob) | 2+ (row scan + `take_blobs()`) | same |
| Total source data (at 200 KB avg) | ~1 PB | ~5 PB |
| Total output data | ~15 TB | ~75 TB |
| Checkpoint data per batch (`checkpoint_size=100`) | ~300 KB (100 × 3 KB) | same |

Unlike CPU classification where checkpoint writes are negligible, embedding checkpoints are **300KB per batch** — 375× larger. At 100 workers each flushing checkpoints, this generates meaningful write IOPS. Both read and write bandwidth matter.

#### Compute Profile

| Metric | Value |
|--------|-------|
| Inference time per image | 50–200ms (model-dependent) |
| Inference time per batch (100 images) | 5–20s (with GPU batching) |
| GPU memory per model | 1–4 GB (ViT-L ~1.2GB, BLIP ~3.5GB) |
| GPU memory per batch (100 images, 224×224) | ~200 MB |
| Total GPU memory per worker | 2–6 GB |
| GPU utilization | High during inference, idle during I/O |

**Read/compute/write balance**: Unlike CPU classification where workers are almost entirely I/O-bound, GPU embedding workers split time between I/O and compute. At `checkpoint_size=100` and 200KB/image, each batch reads ~20MB of images (taking ~1–2s on object store) then runs inference for ~5–20s. Workers are **compute-bound** at small scale and transition to **I/O-bound** as worker count increases and object store bandwidth saturates.

**GPU pipelining**: The sequential read→compute→write flow means the GPU idles during I/O. The [GPU Pipelining Overview](gpu_pipelining_overview.md) proposes decoupling data loading from GPU execution across three phases:
- [Phase 1](gpu_pipelining_phase1.md): Separate DataLoader actors prefetch batches so the GPU always has work
- [Phase 1.1](gpu_pipelining_phase1_1.md): Auto-sizing loader actors based on cluster resources
- [Phase 2](gpu_pipelining_phase2.md): CPU preprocessing pipeline for heavy image transforms before GPU
- [Phase 3](gpu_pipelining_phase3.md): CUDA stream optimization for maximum GPU utilization

Pipelining directly addresses the compute profile above — the 1–2s of I/O idle time per batch can be hidden behind prefetch, potentially improving GPU utilization by 10–30% and reducing effective per-batch time.

#### Implications for This Workload

- **Checkpoint loss is costly**: Inference takes 5–20s per batch of 100. Losing checkpoints on node failure means re-reading images AND re-running GPU inference — potentially minutes of wasted GPU compute. `@udf(checkpoint=False)` is not viable for this workload.
- **Checkpoint sizing tradeoff shifts**: Larger checkpoints (1000 rows) reduce IOPS by 10× but increase the blast radius to ~50–200s of GPU compute per lost checkpoint. `AdaptiveCheckpointSizer` targeting ~10s per batch naturally lands at `checkpoint_size` ~50–200 for GPU workloads.
- **Write I/O is significant**: 15 TB of output for 5B rows means output writes are no longer negligible. Checkpoint data (300KB per batch) adds meaningful IOPS on top of this.
- **GPU memory constrains workers per node**: A node with 4 GPUs can run at most 4 embedding workers regardless of CPU count. Typical GPU nodes (g6e.xlarge on AWS) have 1 GPU + 4 CPUs, so the constraint is 1 worker per node.
- **Read amortization across columns applies**: Same as CPU classification — computing multiple embeddings (CLIP + BLIP) in a single UDF pass avoids re-reading source images. Even more valuable here since each read also triggers expensive GPU compute. See [multi-column UDF design](../designs/multi_column_udf_design.md).

#### Cluster Resource Math

Resource math per GPU worker (typical `@udf(num_gpus=1.0, num_cpus=1.0)`):
- 1.0 GPU (applier) + 1.0 CPU (applier) + 0.1 CPU (writer) + 0 CPU (queue) = **1 GPU + 1.1 CPU per worker**
- Job overhead: 0.2 CPU (driver + jobtracker)
- Writer memory: 1 GiB per worker (same as CPU workloads)

| | 5B rows | 25B rows |
|---|---------|----------|
| **Cluster** | 80 GPU nodes × 1 GPU = 80 GPUs | 400 GPU nodes × 1 GPU = 400 GPUs |
| **Max workers** | ~80 | ~400 |
| **Actor count** | ~242 | ~1,202 |
| **Writer memory** | 80 GiB | 400 GiB |
| **Throughput (at 10s/batch of 100)** | ~800 images/sec | ~4,000 images/sec |
| **Time to process 5B rows** | ~72 days | ~14 days |
| **Read bandwidth** | ~160 MB/s (well within limits) | ~800 MB/s (within limits) |
| **Checkpoint write IOPS** | ~80 PUTs/sec (80 workers × 1 flush/10s) | ~400 PUTs/sec |

**Key insight**: GPU workloads have far fewer concurrent workers than CPU workloads (80 vs 580 for the same node count) because each worker requires a full GPU. This means actor coordination bottlenecks (JobTracker, actor pool ramp) are much less stressed. The bottleneck shifts from I/O contention to **total compute time** — the job simply takes a long time because GPU inference is slow relative to the data volume.

#### 5B Rows — What Breaks at 80 GPU Nodes (~80 workers)

| Category | Bottleneck | Status | Why |
|----------|-----------|--------|-----|
| **Compute** | **GPU throughput** | **Dominant constraint** | 80 workers × 10 images/sec = 800 images/sec. Processing 5B rows takes ~72 days. The bottleneck is raw GPU compute time, not I/O or coordination. |
| Storage I/O | Source Read Throughput | Fine | 80 workers × 200 KB/image × 10 images/sec = ~160 MB/s. Well within single-prefix object store limits. GPU inference is slow enough that I/O is not the constraint. |
| Storage I/O | Checkpoint IOPS | Warm | 300 KB checkpoints × 80 workers = meaningful write load, but ~80 PUTs/sec is within object store limits. Becomes a concern if checkpoint flushing falls behind inference. |
| **Storage I/O** | **High Fragment Count** | **Latent risk** | Same as classification — if dataset has 50K+ fragments, planning and commit phases add minutes. Less impactful for embedding since the job runs for days anyway. |
| Coordination | JobTracker Mailbox | Fine | ~8 msgs/sec (80 workers completing a batch every ~10s). Far below saturation. |
| Coordination | Stall Detection | Low risk | Inference time is predictable (~10s per batch). Less variable than object store reads. False stall triggers unlikely. |
| Resource | Actor Pool Ramp-up | Fine | 242 actors — well within tested range. |
| Resource | Ray Actor Limit | Fine | 242 actors — far under 10k. |
| Resource | Writer Memory | Fine | 80 GiB — modest. |
| **Resource** | **GPU node cost** | **Practical constraint** | 80 GPU nodes for 72 days is expensive. Cost optimization (spot instances, preemption handling, checkpoint resume) matters more than performance tuning. |

#### 5B Rows — What Breaks at 160 GPU Nodes (~160 workers)

| Category | Bottleneck | Status | Why |
|----------|-----------|--------|-----|
| **Compute** | **GPU throughput** | **Still dominant** | 160 workers × 10 images/sec = 1,600 images/sec. Processing 5B rows takes ~36 days. Doubles throughput but job is still very long. |
| Storage I/O | Source Read Throughput | Fine | ~320 MB/s aggregate read. Still within object store limits. |
| Storage I/O | Checkpoint IOPS | Warm | ~160 PUTs/sec. Approaching levels where separate checkpoint storage is worthwhile. |
| Storage I/O | High Fragment Count | Same latent risk | Planning/commit overhead unchanged. |
| Coordination | JobTracker Mailbox | Fine | ~16 msgs/sec. No concern. |
| Resource | Actor Pool Ramp-up | Fine | 482 actors — within tested range. |
| Resource | Ray Actor Limit | Fine | 482 actors — far under 10k. |
| Resource | Writer Memory | Fine | 160 GiB — modest. |
| **Resource** | **GPU node cost** | **Critical** | 160 GPU nodes for 36 days. Cost per job may exceed budget. Spot instance preemption becomes likely over multi-week runs, making checkpoint durability essential. |

#### 25B Rows — Additional Bottlenecks

Everything from the 5B analysis applies, plus:

| Category | Bottleneck | Status | Why |
|----------|-----------|--------|-----|
| **Compute** | **Job duration** | **Impractical at small scale** | At 80 workers: ~360 days. At 400 workers: ~72 days. Even at 400 GPU nodes, processing 25B rows is a multi-month commitment. Requires either faster models, reduced precision (FP16/INT8), or architectural changes (batch inference services). |
| **Storage I/O** | **Checkpoint IOPS** | **Significant** | 400 workers × 300 KB checkpoints = ~400 PUTs/sec sustained over months. Separate checkpoint storage is mandatory. Total checkpoint data: ~750 TB over the life of the job. |
| **Storage I/O** | **High Fragment Count** | **Critical** | 25K–750K+ fragments. Planning and commit at this scale take minutes. Compaction before backfill is essential. |
| **Resource** | **Spot preemption** | **Inevitable** | Multi-month runs on 400 GPU nodes will experience node failures. Checkpoint resume must be robust — every lost checkpoint costs minutes of GPU compute. |
| Storage I/O | Output data volume | Significant | ~75 TB of embeddings. Write throughput matters at this scale. |

### Media Download (URL→Blob)

**I/O profile**: Write bandwidth bottlenecked. Source data is a small URL string (~100 bytes per row). The UDF fetches media from an external service and writes the downloaded content (100KB–10MB) into a Lance blob column. Reads from the source table are trivial; writes to object store dominate.

**Key differences from image classification (CPU)**:
- Read I/O is negligible — the source table is tiny URLs, not large blobs
- Write I/O dominates — each row writes 100KB–10MB of downloaded media to object store
- External service rate limits may gate throughput before object store limits are reached
- Network egress from external service → worker → object store creates a double-hop bandwidth constraint
- Checkpoint data is large (downloaded media bytes), making checkpoint IOPS a real concern

**TODO**: Full bottleneck analysis including external service rate limiting, write-heavy object store pressure, and checkpoint sizing for large payloads.

### Record Linkage / Dedup (IOPS)

**I/O profile**: IOPS bottlenecked. Each row requires multiple small random point lookups against a large reference index or table (e.g., checking near-duplicate candidates via a precomputed index). Individual I/Os are small (<1KB) but there are millions of them. Neither read nor write bandwidth is the constraint — the sheer number of small random operations exhausts the object store's per-prefix IOPS budget.

**Key differences from image classification (CPU)**:
- Per-row I/O is small but numerous — many sub-1KB random reads vs few large sequential reads
- Object store per-prefix IOPS limits (not bandwidth) are the first wall
- Prefix sharding and request batching are the primary mitigations, not bandwidth scaling
- CPU may be the bottleneck if the linkage/comparison logic is compute-intensive (e.g., vector similarity)
- Output is typically small (match IDs, scores), so write I/O is minimal

**TODO**: Full bottleneck analysis with IOPS math at scale, prefix distribution strategies, and request batching impact.


---

## Proposed Stress Tests

All tests use a standard scale sweep: `[100, 200, 400, 800, 1200, 1600, 2000]`, capped by `STRESS_MAX_SCALE` env var for resource-constrained CI environments.

### Regression vs Exploratory Tests

Every scale point in the sweep is classified as either **regression** or **exploratory** based on a per-test threshold (`explore_threshold`). This determines CI behavior:

| Category | Purpose | CI behavior | Assertion mode |
|----------|---------|-------------|----------------|
| **Regression** | Guard against regressions at known-good scales | Runs in `test-stress-gcp` (120 min timeout), failures block the PR | Hard assertions — test must pass |
| **Exploratory** | Probe for new breaking points at higher scales | Runs in `test-stress-gcp-explore` (240 min timeout), failures are informational | `xfail(strict=False)` — pass or fail is fine, results are logged |

The `scale_params()` helper in `stress_results.py` builds the `pytest.param` list. Scale points below `explore_threshold` are plain params; points at or above get the `stress_explore` marker plus `xfail(strict=False)`. CI separates them via pytest `-m` flag:

- `make test-stress-gcp` runs `-m "not xfail and not stress_explore"` (regression only)
- `make test-stress-gcp-explore` runs `-m "stress_explore"` (exploratory only)

**Per-test thresholds**: Each test sets its own `explore_threshold` based on where it has been validated:

| Test | `explore_threshold` | Regression scales | Exploratory scales |
|------|-------------------|-------------------|-------------------|
| `test_actor_pool_ramp` | 800 | 100, 200, 400 | 800, 1200, 1600, 2000 |
| `test_jobtracker_saturation` | 200 | 100 | 200, 400, 800, 1200, 1600, 2000 |
| `test_fragment_writer_fanout` | 200 | 100 | 200, 400, 800, 1200, 1600, 2000 |
| `test_object_store_iops` | 100 | 50 | 100, 200, 400, 800 |
| `test_metadata_stampede` | — | 10×500 | 40×10000 |

The default threshold is controlled by `STRESS_EXPLORE_THRESHOLD` env var (default: 800). As tests prove stable at higher scales, the `explore_threshold` is raised to promote those scales to regression.

### Shared Utilities: `stress_results.py`

A common module for collecting and reporting stress test metrics:

```python
@dataclass
class StressResult:
    scale: int
    throughput: float          # ops/sec
    p50_latency_s: float
    p90_latency_s: float
    p99_latency_s: float
    max_latency_s: float
    error_count: int
    error_rate: float
    elapsed_s: float
    metadata: dict[str, Any]   # test-specific details
```

### Test 1: `test_jobtracker_saturation.py` — JobTracker Mailbox

**Goal**: Find the worker count where JobTracker round-trip latency exceeds acceptable bounds.

**Setup**: Create a JobTracker with `enable_saves=False` (isolate mailbox from DB). Spawn N `StressCallerActor` instances, each calling `batch_increment.remote()` 50 times with 5 metrics per call. Use `ray.get()` per call to measure round-trip latency.

```python
@ray.remote(num_cpus=0.1, memory=128 * 1024**2)
class StressCallerActor:
    def flood_tracker(self, tracker_handle, num_calls: int) -> list[float]:
        latencies = []
        for _ in range(num_calls):
            t0 = time.monotonic()
            ray.get(tracker_handle.batch_increment.remote({
                "udf_processing_time": 100,
                "batch_checkpointing_time": 50,
                "read_io_time_ms": 200,
                "checkpoint_load_time_ms": 30,
                "read_task_total_time_ms": 400,
            }))
            latencies.append(time.monotonic() - t0)
        return latencies
```

| Parameter | Value |
|-----------|-------|
| `scale` | `[100, 200, 400, 800, 1200, 1600, 2000]` |
| `calls_per_actor` | 50 |
| `metrics_per_call` | 5 |

**Metrics**: throughput (calls/sec), p50/p90/p99/max latency, error rate.

**Pass criteria**: All metric totals correct (no lost updates), p99 < 30s, zero errors.

### Test 2: `test_actor_pool_ramp.py` — Pool Ramp-up at Scale

**Goal**: Verify that actor pool ramp-up time scales roughly linearly and doesn't regress.

**Setup**: Reuse `run_bench()` from existing `test_actor_pool_ramp.py`. Scale from 100 to 2,000 actors, each running 4 tasks with 5s busy time.

| Parameter | Value |
|-----------|-------|
| `scale` | `[100, 200, 400, 800, 1200, 1600, 2000]` |
| `tasks_per_actor` | 4 |
| `busy_ms` | 5000.0 |
| `timeout_s` | 1200.0 |

**Metrics**: `elapsed_to_full_saturation`, p50/p90/p99 first-task latency, `actors_per_second`.

**Pass criteria**: `actors_observed == num_actors`, ramp-up scales roughly linearly (no super-linear blowup).

### Test 3: `test_fragment_writer_fanout.py` — Writer Actor Count Stress

**Goal**: Find the actor count where writer creation fails, OOMs, or hits the 10k actor cap.

**Setup**: Create N `Queue` actors (num_cpus=0) + N `FakeWriter` actors (num_cpus=0.02, 128 MiB). Each writer drains its queue until a seal signal arrives.

```python
@ray.remote(num_cpus=0.02, memory=128 * 1024**2)
class FakeWriter:
    def write(self, queue) -> dict:
        t0 = time.monotonic()
        while True:
            msg = ray.get(queue.get.remote())
            if msg[0] < 0:  # seal signal
                break
        return {"elapsed_s": time.monotonic() - t0}
```

| Parameter | Value |
|-----------|-------|
| `num_fragments` | `[100, 200, 400, 800, 1200, 1600, 2000]` |
| `checkpoints_per_fragment` | 5 |

**Resource math at 2,000 fragments**: 2,000 queues (0 CPU) + 2,000 writers (0.1 CPU, 128 MiB) = 200 CPUs + 256 GiB.

**Metrics**: actor creation time, total actor count, completion time, OOM/death events.

**Pass criteria**: All fragments complete, total actor count stays under 10,000.

### Test 4: `test_checkpoint_scale.py` — Checkpoint Store at Scale

**Goal**: Verify checkpoint read/write performance under concurrent access at production scale.

**Setup**: Pre-seed 5,000 checkpoint keys. Spawn N actors, each performing 1,000 operations (80% reads, 20% writes) against random keys.

| Parameter | Value |
|-----------|-------|
| `scale` | `[100, 200, 400, 800]` |
| `ops_per_actor` | 1,000 |
| `keyset_size` | 5,000 |
| `checkpointer` | `"session"` |

**Metrics**: total ops/sec, error count by type, p50/p90/p99 latency.

**Pass criteria**: Error rate < 1%, latency remains bounded (no unbounded growth with scale).

**Note**: Requires `@pytest.mark.gcp_only` — needs an object store bucket for realistic checkpoint backend.

### Test 5: `test_commit_contention.py` — Lance Commit Races

**Goal**: Find the concurrency level where 12 commit retries becomes insufficient.

**Setup**: Create a Lance dataset. Spawn N actors that each write a tiny file and call `commit()` simultaneously (coordinated via a starting-gun queue).

| Parameter | Value |
|-----------|-------|
| `concurrent_committers` | `[10, 20, 50, 100, 200]` |
| `commit_retries` | 12 |

**Metrics**: success/failure counts by error type, max retries needed per commit, total commit wall time.

**Pass criteria**: All commits eventually succeed. Report the scale where 12 retries becomes insufficient.

### Test 6: `test_pipeline_e2e_scale.py` — Full Pipeline Integration

**Goal**: End-to-end validation that the full pipeline completes correctly at production scale.

**Setup**: Create a dataset with `concurrency * 2` fragments, 1,000 rows each. Run a trivial identity UDF through the full pipeline with elevated timeouts.

```python
@geneva.udf(num_cpus=0.1)
def identity_udf(batch: pa.RecordBatch) -> pa.RecordBatch:
    return pa.record_batch({"_stress_col": pa.array([1] * batch.num_rows)})
```

| Parameter | Value |
|-----------|-------|
| `concurrency` | `[100, 200, 400, 800]` |
| `num_fragments` | `concurrency * 2` |
| `rows_per_fragment` | 1,000 |
| `commit_granularity` | 64 |

**Env overrides**: `GENEVA_PIPELINE_STALL_TIMEOUT_S=1200`, `GENEVA_COMMIT_MAX_RETRIES=20`.

**Metrics**: total elapsed, rows/sec, actor count breakdown, errors/timeouts, JobTracker metrics dump.

**Pass criteria**: Job completes successfully, committed row count matches expected count.

### Test 7: `test_error_table_storm.py` — Error Table Write Contention

**Goal**: Find the failure count and concurrency where error table writes cause commit contention or OOM.

**Setup**: Spawn N actors, each bulk-writing error records via `ErrorStore.log_errors()` against a shared `geneva_errors` Lance table. Simulates the worst case of mass UDF failures across all workers.

```python
@ray.remote(num_cpus=0.1, memory=256 * 1024**2)
class ErrorStormActor:
    def flood_errors(self, error_store, num_errors: int, job_id: str) -> dict:
        from geneva.debug.error_store import ErrorRecord
        errors = [
            ErrorRecord(
                error_type="ValueError",
                error_message=f"synthetic failure {i}",
                error_trace="...",
                job_id=job_id,
                table_uri="gs://test/table.lance",
                table_name="test",
                column_name="test_col",
                udf_name="test_udf",
                udf_version="v1",
                batch_index=i,
            )
            for i in range(num_errors)
        ]
        t0 = time.monotonic()
        error_store.log_errors(errors)
        return {"elapsed_s": time.monotonic() - t0, "num_errors": num_errors}
```

| Parameter | Value |
|-----------|-------|
| `concurrent_writers` | `[10, 50, 100, 200, 400, 800]` |
| `errors_per_writer` | `[100, 1000, 10000]` |

**Metrics**: total write time, error count correctness, Lance commit retry count, OOM events, error rate.

**Pass criteria**: All errors logged correctly (count matches expected). Report the scale where concurrent writer retries are exhausted or OOM occurs.

### Test 8: `test_high_fragment_count.py` — Fragment Count Scaling

**Goal**: Find the fragment count where planning, commit, and metadata operations become unacceptable (>60s per operation).

**Setup**: Create a Lance dataset with increasing fragment counts via repeated appends. Measure time for key operations independently: `get_fragments()`, planning via `_get_fragments_missing_column_data()`, Merge commit, and manifest read I/O.

```python
def create_dataset_with_n_fragments(uri: str, n: int, rows_per_frag: int) -> None:
    """Create a Lance dataset by appending n times to produce n fragments."""
    for i in range(n):
        data = pa.table({
            "id": pa.array(range(rows_per_frag)),
            "val": pa.array([f"row_{i}_{j}" for j in range(rows_per_frag)]),
        })
        if i == 0:
            lance.write_dataset(data, uri)
        else:
            lance.write_dataset(data, uri, mode="append")
```

| Parameter | Value |
|-----------|-------|
| `fragment_counts` | `[1000, 5000, 10000, 50000, 100000, 150000]` |
| `rows_per_fragment` | 10 |

**Metrics**: time for `get_fragments()`, time for planning, time for Merge commit, manifest file size, peak memory during fragment enumeration.

**Pass criteria**: All operations complete. Report the fragment count where any single operation exceeds 60s. Report memory scaling curve.

**Note**: Can run locally with filesystem-backed Lance for planning benchmarks. Use `@pytest.mark.gcp_only` for realistic object store manifest I/O timing.

### Test 9: `test_object_store_iops.py` — Object Store Throughput Limits

**Goal**: Find the concurrency level where object store throttling causes task failures or stall detection false positives.

**Setup**: Spawn N actors that each perform a burst of small object store operations (1KB writes + reads) against the same object store prefix, measuring throughput and error rates.

```python
@ray.remote(num_cpus=0.02, memory=128 * 1024**2)
class IOPSStressActor:
    def burst_io(self, store_prefix: str, num_ops: int) -> dict:
        import pyarrow.fs as pafs
        fs, path = pafs.FileSystem.from_uri(store_prefix)
        write_times, read_times, errors = [], [], 0
        for i in range(num_ops):
            key = f"{path}/stress_{i}.bin"
            try:
                t0 = time.monotonic()
                with fs.open_output_stream(key) as f:
                    f.write(b"x" * 1024)  # 1KB writes
                write_times.append(time.monotonic() - t0)
                t0 = time.monotonic()
                with fs.open_input_stream(key) as f:
                    f.read()
                read_times.append(time.monotonic() - t0)
            except Exception:
                errors += 1
        return {
            "write_p50": statistics.median(write_times),
            "write_p99": _percentile(write_times, 99),
            "read_p50": statistics.median(read_times),
            "read_p99": _percentile(read_times, 99),
            "errors": errors,
            "total_ops": num_ops * 2,
        }
```

| Parameter | Value |
|-----------|-------|
| `concurrent_actors` | `[50, 100, 200, 400, 800]` |
| `ops_per_actor` | 200 |

**Metrics**: aggregate ops/sec, p50/p90/p99 per-op latency, error count and type (429/503), retry exhaustion events.

**Pass criteria**: Report the concurrency level where error rate exceeds 1% or p99 latency exceeds 10s. Verify retry logic handles throttling correctly.

**Note**: Requires `@pytest.mark.gcp_only` — needs a real object store bucket to capture actual throttling behavior.


---

## Implementation Plan

Tests are ordered from cheapest/fastest to implement to most expensive:

| Phase | Test | New Code | Dependencies | Runtime |
|-------|------|----------|-------------|---------|
| 1 | `stress_results.py` | Shared utilities | None | N/A |
| 2 | `test_jobtracker_saturation.py` | Standalone | `stress_results.py` | Local Ray |
| 3 | `test_actor_pool_ramp.py` | Minimal | `stress_results.py` | Local Ray |
| 4 | `test_fragment_writer_fanout.py` | Standalone simulation | `stress_results.py` | Local Ray |
| 5 | `test_checkpoint_scale.py` | Needs object store bucket | `stress_results.py`, GCP | Local Ray |
| 6 | `test_commit_contention.py` | Needs Lance dataset + object store | `stress_results.py`, GCP | Local Ray |
| 7 | `test_pipeline_e2e_scale.py` | Full integration | Everything | Local Ray |
| 8 | `test_error_table_storm.py` | Needs Lance dataset + error store | `stress_results.py`, GCP | Local Ray |
| 9 | `test_high_fragment_count.py` | Standalone (local or object store) | `stress_results.py` | Local Ray |
| 10 | `test_object_store_iops.py` | Needs object store bucket | `stress_results.py`, GCP | Local Ray |

### Runtime

All stress tests run on **local Ray** (no k8s cluster required). Tests use reduced per-actor resources (`num_cpus=0.02`) since they stress actor count and coordination, not CPU/memory throughput. The `STRESS_MAX_SCALE` env var caps the maximum concurrency for resource-constrained CI environments.

### File Layout

All new files go in `src/stress_tests/`:

```
src/stress_tests/
├── stress_results.py                  # shared utilities (implemented)
├── conftest.py                        # test fixtures (implemented)
├── test_jobtracker_saturation.py      # test 1 (implemented)
├── test_actor_pool_ramp.py            # test 2 (implemented)
├── test_fragment_writer_fanout.py     # test 3 (implemented)
├── test_checkpoint_scale.py           # test 4 (planned)
├── test_commit_contention.py          # test 5 (planned)
├── test_pipeline_e2e_scale.py         # test 6 (planned)
├── test_error_table_storm.py          # test 7 (planned)
├── test_high_fragment_count.py        # test 8 (implemented)
├── test_object_store_iops.py          # test 9 (implemented)
├── bench_worker.py                    # existing
├── test_metadata_stampede.py          # existing
└── test_metadata_stampede_write.py    # existing
```


---

## Prioritization

### Scenario Priority

| Priority | Scenario | Scale | Rationale |
|----------|----------|-------|-----------|
| **P1** | Image Classification (CPU) | 5B, 25B rows | Read-bandwidth bottlenecked. Represents the current largest real workload (phash, aesthetics score, NSFW score). 5B is near-term, 25B is the next scale target. |
| **P2** | Image Embedding (CLIP/ViT) | 5B, 25B rows | Read+write bottlenecked with GPU constraints. Second most common workload pattern; checkpoint sizing tradeoffs for expensive UDFs need analysis. |
| **P3** | Media Download (URL→Blob) | TBD | Write-bandwidth bottlenecked. External service rate limits add a dimension not covered by the other scenarios. |
| **P3** | Record Linkage / Dedup | TBD | IOPS bottlenecked. Different enough from bandwidth-limited workloads to warrant separate analysis when the use case matures. |

### Stress Test Priority

**Important distinction**: The first hard wall at 80+ nodes for read-amplified workloads is object store egress bandwidth — an external provider limit, not a Geneva code issue. Stress tests can measure where throttling kicks in but can't fix it. The mitigations for that wall are data layout decisions (source sharding across buckets/prefixes) and tuning (Quick Wins below), not code changes. The stress tests below validate *Geneva's internal* breaking points — actor coordination, resource scaling, commit contention — so that they don't compound the external I/O bottleneck when it arrives.

Ranked by "what kills the image classification (CPU) 80→160 node scale-up first":

| Priority | Test | Bottleneck | Rationale |
|----------|------|-----------|-----------|
| **P0** | `test_object_store_iops.py` (Test 9) | Source read throughput | First hard wall at 80 nodes. For read-heavy workloads, **read** IOPS and bandwidth from source images dominate — not checkpoint writes. Need to measure actual object store throttling behavior under read-heavy load with large objects. |
| **P0** | `test_high_fragment_count.py` (Test 8) | Fragment count | At 5B+ rows, fragment count is likely already in the tens of thousands. Need to know the planning/commit time curve to set expectations. Critical at 25B where fragment counts reach 100K+. |
| **P1** | `test_jobtracker_saturation.py` (Test 1) | JobTracker | Need to measure actual mailbox latency at 1K–2K senders. If p99 stays under 5s, this is fine and the hierarchical design is unnecessary. If it exceeds 30s, it becomes P0. |
| **P1** | `test_actor_pool_ramp.py` (Test 2) | Ramp-up | Never tested beyond 120. Need data to know if ramp-up at 2K actors is seconds or minutes. |
| **P1** | `test_fragment_writer_fanout.py` (Test 3) | Writer count | Validates that 2K+ writer actors can coexist without OOM or scheduling starvation. |
| **P2** | `test_pipeline_e2e_scale.py` (Test 6) | Integration | Full validation after individual bottlenecks are understood. |
| **P2** | `test_commit_contention.py` (Test 5) | Commits | Per-job, not affected by node count. Important but not on the critical path for this scale-up. |
| **P2** | `test_checkpoint_scale.py` (Test 4) | Checkpoints | Session-mode likely holds. Checkpoint payloads are tiny for phash (8 bytes × batch); becomes more important for embedding workloads (3KB × batch). |
| **P3** | `test_error_table_storm.py` (Test 7) | Errors | Only matters during mass failures. Defensive test, not on the scaling critical path. |

### Quick Wins Before Stress Testing

These mitigations can be applied immediately to 5B-row image classification workloads (phash, aesthetics, NSFW) and may unblock 80+ nodes without code changes:

1. **Separate checkpoint storage** (`JOB__CHECKPOINT__OBJECT_STORE__PATH=gs://other-bucket/ckpts`) — decouples checkpoint I/O from source data reads so they don't share the same prefix budget. Critical for this workload since source reads dominate.
2. **Increase `checkpoint_size` to 1,000–10,000** — reduces checkpoint file count. Less impactful for this workload (checkpoint payloads are tiny) but still reduces IOPS.
3. **Set `GENEVA_PIPELINE_STALL_TIMEOUT_S=1800`** — prevents false stall kills when workers are blocked on slow object store reads.
4. **Set `GENEVA_APPLIER_TRANSIENT_RETRIES=10` and `GENEVA_APPLIER_RETRY_MAX_BACKOFF_SECONDS=30`** — handles sustained object store throttling on source reads.
5. **Run `table.compact_files()` before backfill** — if the dataset has >10K fragments, compaction is the cheapest way to reduce planning/commit overhead.
6. **Set `GENEVA_COMMIT_MAX_RETRIES=20`** — extra headroom for commit contention.
7. **Consider `@udf(num_cpus=0.5)`** — phash computation is I/O-bound, so workers don't need a full CPU. Fits more workers per node, though all will be blocked on the same object store bandwidth.
8. **Shard source data across buckets** — if the source table can be stored across multiple object store buckets/prefixes, read bandwidth scales linearly. This is the only way to meaningfully scale past 80 nodes for this workload.
