# GPU Pipelining - Overview

## Problem Statement

GPU-based UDFs are underutilized because:

1. **Sequential I/O**: ApplierActors read batches sequentially - GPU idles during Lance I/O
2. **No overlap**: No overlap between data loading (I/O) and GPU computation
3. **No preprocessing pipeline**: No CPU preprocessing pipeline to prepare data for GPU actors
4. **Batch size mismatch**: Batch sizes may not be optimized for GPU memory/throughput

## Current Data Flow

```
ReadTask.to_batches() → [BLOCKING I/O] → ApplierActor → GPU UDF → Result → FragmentWriter
                              ↑
                        GPU IDLE HERE
```

The GPU sits idle while each ApplierActor reads the next batch from Lance. There's no pipelining between I/O (reading) and compute (GPU UDF execution).

## Target Data Flow (Pipelined)

```
DataLoaderActor (prefetch) ─┐
                            ├─→ Queue ─→ GPUApplierActor ─→ Results ─→ FragmentWriter
CPU Preprocessing Actor ────┘         (always has work)
```

## Implementation Strategy: Phased Approach

| Phase | Focus | Status | Complexity |
|-------|-------|--------|------------|
| **Phase 1** | Separate DataLoader Actors | 📋 Proposed / Planned | Quick win / Prototype |
| **Phase 1.1** | Auto-Sizing Loader Actors | 🔮 Future | Usability improvement |
| **Phase 2** | CPU Preprocessing Pipeline | 🔮 Future | For heavy preprocessing UDFs |
| **Phase 3** | CUDA Stream Optimization | 🔮 Future | Maximum GPU utilization |

## Phase Summaries

### Phase 1: DataLoader Actors (Proposed)

Decouples data loading from GPU execution using separate loader actors that prefetch batches into Ray's object store.

**Key Components:**
- `DataLoaderActor`: Reads from Lance, puts batches in object store
- `PreloadedReadTask`: Wrapper that returns pre-loaded batches
- `LoaderPool`: Manages loader actors with bounded prefetch queue

**Usage:**
```python
table.backfill(
    "caption",
    enable_gpu_pipelining=True,
    num_loader_actors=4,
    prefetch_depth=8,
)
```

See: [gpu_pipelining_phase1.md](gpu_pipelining_phase1.md)

### Phase 1.1: Auto-Sizing Loader Actors (Future)

Automatically determines optimal `num_loader_actors` and `prefetch_depth` so users don't need to manually tune these parameters.

**Sub-phases:**
- **1.1a**: Ratio-based - `num_loaders = max(2, applier_concurrency)`
- **1.1b**: Resource-based - Cap by cluster CPU availability (25%)
- **1.1c**: Dynamic scaling - Monitor queue depth and scale at runtime

See: [gpu_pipelining_phase1_1.md](gpu_pipelining_phase1_1.md)

### Phase 2: CPU Preprocessing Pipeline (Future)

For UDFs with heavy CPU preprocessing (image decode, tokenization), separates CPU work from GPU work with dedicated preprocessing actors.

**Key Concept:**
- UDFs define optional `preprocess()` method for CPU work
- CPUPreprocessorActors run preprocessing in parallel
- GPU actors only do inference on pre-processed tensors

See: [gpu_pipelining_phase2.md](gpu_pipelining_phase2.md)

### Phase 3: CUDA Stream Optimization (Future)

Maximum GPU utilization by overlapping data transfer with computation using CUDA streams.

**Key Concept:**
- Double-buffered GPU execution
- Overlap CPU→GPU transfer with GPU computation
- Requires UDF cooperation for stream-aware execution

See: [gpu_pipelining_phase3.md](gpu_pipelining_phase3.md)

## Configuration Options

```python
# In JobConfig (src/geneva/jobs/config.py)
enable_gpu_pipelining: bool = False  # Enable/disable pipelining
num_loader_actors: int = 4           # Number of data loader actors
prefetch_depth: int = 8              # Batches to prefetch ahead
```

## Key Files

| File | Purpose |
|------|---------|
| `src/geneva/runners/ray/loader.py` | DataLoaderActor, LoaderPool, PreloadedReadTask |
| `src/geneva/runners/ray/pipeline.py` | Integration with ColumnAddPipelineJob |
| `src/geneva/jobs/config.py` | Configuration options |
| `src/geneva/table.py` | API exposure in backfill() |

## Design Principles

1. **Backward compatible**: GPU pipelining is opt-in via config flag
2. **Transparent to UDFs**: Existing UDFs work without modification
3. **Fault tolerant**: Uses existing checkpoint mechanism for recovery
4. **Memory bounded**: Prefetch queue has configurable depth limit
