# GPU Pipelining - Phase 1.1: Auto-Sizing Loader Actors

**Status:** 🔮 Future / Not Implemented

## Problem

Users shouldn't need to manually tune `num_loader_actors` and `prefetch_depth`. The system should automatically determine optimal values based on cluster resources and workload.

## Current Behavior

Users must explicitly configure loader parameters:

```python
table.backfill(
    "caption",
    enable_gpu_pipelining=True,
    num_loader_actors=4,    # Manual tuning required
    prefetch_depth=8,       # Manual tuning required
)
```

## Target Behavior

Auto-size when parameters not specified:

```python
table.backfill(
    "caption",
    enable_gpu_pipelining=True,
    # num_loader_actors and prefetch_depth auto-calculated
)
```

---

## Implementation Strategy: Phased Approach

| Phase | Approach | Complexity |
|-------|----------|------------|
| **1.1a** | Ratio-based | Simple / Immediate |
| **1.1b** | Resource-based | Moderate |
| **1.1c** | Dynamic scaling | Complex / Future |

---

## Phase 1.1a: Ratio-Based Auto-Sizing

### Approach

Scale loaders based on GPU actor count (`applier_concurrency`):

- **Formula**: `num_loaders = max(2, applier_concurrency)`
- **Prefetch depth**: `prefetch_depth = num_loaders * 2`

### Rationale

- Each GPU actor needs ~1 loader to keep it fed
- I/O (Lance read) is typically slower than GPU inference
- 1:1 ratio is a good starting point
- Minimum of 2 loaders ensures some parallelism

### Proposed Implementation

```python
def auto_size_loaders(
    applier_concurrency: int,
    num_loader_actors: int | None = None,
    prefetch_depth: int | None = None,
) -> tuple[int, int]:
    """Auto-size loader parameters if not explicitly set.

    Args:
        applier_concurrency: Number of GPU applier actors
        num_loader_actors: User-specified loader count (None for auto)
        prefetch_depth: User-specified prefetch depth (None for auto)

    Returns:
        Tuple of (num_loader_actors, prefetch_depth)
    """
    # Auto-size num_loaders
    if num_loader_actors is None or num_loader_actors <= 0:
        # 1:1 ratio with GPU actors, minimum 2
        num_loader_actors = max(2, applier_concurrency)

    # Auto-size prefetch_depth
    if prefetch_depth is None or prefetch_depth <= 0:
        # 2 batches per loader actor
        prefetch_depth = num_loader_actors * 2

    return num_loader_actors, prefetch_depth
```

### Configuration Behavior

| User Specifies | Behavior |
|----------------|----------|
| Nothing | Auto-size both `num_loaders` and `prefetch_depth` |
| `num_loader_actors=4` | Use 4 loaders, auto-size prefetch_depth |
| `prefetch_depth=16` | Auto-size loaders, use prefetch_depth=16 |
| Both specified | Use user values (no auto-sizing) |

---

## Phase 1.1b: Resource-Aware Auto-Sizing

### Approach

Query Ray cluster resources and cap loaders based on available CPUs:

```python
def auto_size_loaders_v2(
    applier_concurrency: int,
    num_loader_actors: int | None = None,
    prefetch_depth: int | None = None,
) -> tuple[int, int]:
    """Auto-size with cluster resource awareness."""

    if num_loader_actors is None or num_loader_actors <= 0:
        # Start with 1:1 ratio
        desired = applier_concurrency

        # Cap by available cluster CPUs (use max 25%)
        try:
            resources = ray.cluster_resources()
            max_loaders = int(resources.get("CPU", 8) * 0.25)
            num_loader_actors = max(2, min(desired, max_loaders))
        except Exception:
            # Fallback if Ray not initialized
            num_loader_actors = max(2, desired)

    if prefetch_depth is None or prefetch_depth <= 0:
        prefetch_depth = num_loader_actors * 2

    return num_loader_actors, prefetch_depth
```

### Rationale

- Adapts to cluster size automatically
- Doesn't over-allocate on small clusters
- Scales up on large clusters
- Reserves CPUs for GPU actors, fragment writers, driver overhead

### Resource Allocation Strategy

| Component | CPU Allocation |
|-----------|---------------|
| GPU actors | `num_cpus` per actor (typically 0.5-1) |
| Fragment writers | ~1 CPU each |
| Driver overhead | ~1-2 CPUs |
| **Loaders** | **25% of remaining cluster CPUs** |

---

## Phase 1.1c: Dynamic Scaling (Future)

### Approach

Monitor queue depth at runtime and scale loaders up/down:

```python
class AdaptiveLoaderPool:
    """LoaderPool with dynamic scaling based on queue depth."""

    min_loaders: int = 2
    max_loaders: int = 16
    scale_up_threshold: float = 0.2   # Queue < 20% full
    scale_down_threshold: float = 0.8  # Queue > 80% full

    def _check_scaling(self) -> None:
        """Check if we should scale loaders up or down."""
        queue_depth = len(self._prefetched)
        queue_ratio = queue_depth / self.prefetch_depth

        if queue_ratio < self.scale_up_threshold and self._pending_loads:
            # GPU starved - loaders too slow
            self._scale_up()
        elif queue_ratio > self.scale_down_threshold:
            # Queue full - loaders faster than needed
            self._scale_down()

    def _scale_up(self) -> None:
        """Add more loader actors."""
        if len(self._actors) < self.max_loaders:
            new_actor = DataLoaderActor.options(num_cpus=1).remote()
            self._actors.append(new_actor)
            _LOG.info("Scaled up to %d loaders", len(self._actors))

    def _scale_down(self) -> None:
        """Remove idle loader actors."""
        if len(self._actors) > self.min_loaders and self._idle_actors:
            actor = self._idle_actors.pop()
            self._actors.remove(actor)
            ray.kill(actor)
            _LOG.info("Scaled down to %d loaders", len(self._actors))
```

### Rationale

- Adapts to actual I/O vs compute ratio at runtime
- Handles varying workloads (different batch sizes, I/O patterns)
- More complex to implement and test
- May cause resource contention during scaling

### Scaling Triggers

| Condition | Action | Reason |
|-----------|--------|--------|
| Queue empty, loads pending | Scale up | GPUs starving for data |
| Queue > 80% full | Scale down | Loaders outpacing GPUs |
| Actor failure | Replace | Maintain capacity |

---

## Files to Modify

| File | Changes |
|------|---------|
| `src/geneva/runners/ray/loader.py` | Add `auto_size_loaders()` function |
| `src/geneva/runners/ray/pipeline.py` | Call auto-sizing before creating LoaderPool |
| `src/geneva/jobs/config.py` | Change defaults to `None` for auto-sizing |

## Config Changes

```python
# Before (Phase 1)
enable_gpu_pipelining: bool = False
num_loader_actors: int = 4           # Fixed default
prefetch_depth: int = 8              # Fixed default

# After (Phase 1.1)
enable_gpu_pipelining: bool = False
num_loader_actors: int | None = None  # None = auto-size
prefetch_depth: int | None = None     # None = auto-size
```

## Logging

Add info-level logging when auto-sizing:

```
INFO: Auto-sized GPU pipelining: num_loaders=4, prefetch_depth=8 (based on 4 GPU actors)
```

---

## Open Questions

1. **Scaling frequency**: How often should dynamic scaling check queue depth?
2. **Hysteresis**: Should there be a cooldown period between scaling events?
3. **Memory awareness**: Should auto-sizing consider Ray object store memory limits?
4. **Heterogeneous clusters**: How to handle clusters with varying node capabilities?
