# Modal + LanceDB Enterprise + Geneva: Serverless GPU Inference for the Multimodal Lakehouse

Modal provides serverless GPU compute — auto-scaled, pay-per-second, zero
infrastructure. LanceDB Enterprise + Geneva provides data lifecycle management —
checkpointing, incremental processing, versioning, orchestration. Together they
fill each other's gaps: neither team manages GPU infrastructure, and neither team
builds custom data pipelines.

| Concern | Modal provides | Geneva provides |
|---------|---------------|-----------------|
| GPU provisioning | Automatic (serverless) | — |
| Pay-per-second billing | Yes | — |
| Auto-scaling inference | Yes | — |
| Container/model caching | Yes | — |
| Checkpointing | — | Automatic (fragment + batch) |
| Incremental processing | — | Automatic (only new rows) |
| Data versioning | — | Automatic (Lance format) |
| Retry / error handling | — | Configurable (tenacity-based) |
| Distributed orchestration | — | Ray Data |
| Vector search / serving | — | LanceDB Enterprise |

---

## Why Combine Them?

Each tool has a gap the other fills.

| Problem | Modal alone | Geneva alone | Modal + Geneva |
|---------|------------|--------------|----------------|
| GPU provisioning | Automatic | Manual cluster | Automatic |
| Pay-per-use GPU | Yes | No (idle costs) | Yes |
| Any model (not just APIs) | Yes | Yes (needs GPU cluster) | Yes (no cluster) |
| Checkpointing | Build yourself | Automatic | Automatic |
| Incremental processing | Build yourself | Automatic | Automatic |
| Data versioning | Build yourself | Automatic | Automatic |
| Auto-scaling inference | Yes | Manual | Yes |

**The key insight from `modal_vs_geneva.md`:**

> **Modal asks**: "Where should this function run?"
> **Geneva asks**: "How do I maintain this data over time?"

When combined, you get both answers: Modal runs GPU functions, Geneva maintains
the data. You write ~20 lines of pipeline code and get production-grade data
management with serverless GPU inference.

---

## Architecture

```mermaid
flowchart TB
    A["<b>Your Code</b><br/>UDF definitions · table.backfill()"]
    A --> LH

    subgraph LH["Multimodal Lakehouse (CPU-only cluster)"]
        direction TB

        subgraph G["Geneva (compute layer)"]
            direction TB
            G1["Checkpointing"] ~~~ G2["Incremental Processing"] ~~~ G3["Retry / Error Handling"]
            G4["UDF Versions"] ~~~ G5["Adaptive Batching"] ~~~ G6["Orchestration"]
        end

        subgraph R["Ray Workers (CPU)"]
            W1["Worker 1<br/>(UDF)"]
            W2["Worker 2<br/>(UDF)"]
            W3["Worker 3<br/>(UDF)"]
        end

        subgraph L["LanceDB Enterprise (storage + serving)"]
            direction TB
            L1["Lance Tables"] ~~~ L2["Vector Search"] ~~~ L3["Data Versions"]
        end

        G --> R
        R --> L
    end

    subgraph M["Modal (serverless GPU platform)"]
        direction TB
        subgraph MW["Auto-scaled GPU Workers"]
            MW1["GPU Worker<br/>(Model)<br/>A10G"]
            MW2["GPU Worker<br/>(Model)<br/>A10G"]
            MW3["GPU Worker<br/>(Model)<br/>A10G"]
        end
        MC["Cached container images<br/>Warm pool · Pay-per-second"]
    end

    W1 -->|"gRPC / HTTP"| M
    W2 -->|"gRPC / HTTP"| M
    W3 -->|"gRPC / HTTP"| M
```

### What Runs Where

| Concern | Owner | Notes |
|---------|-------|-------|
| Data storage + versioning | LanceDB Enterprise | Lance format, vector indices |
| Checkpointing + resume | Geneva | Fragment + batch level |
| Incremental processing | Geneva | Only new/changed rows |
| Retry / error handling | Geneva | Configurable tenacity-based |
| Distributed orchestration | Geneva + Ray | CPU-only cluster |
| GPU inference | Modal | Serverless, auto-scaled |
| Model container images | Modal | Cached, warm pool |
| Scaling GPU workers | Modal | Automatic |
| Billing | Modal (GPU) + cluster (CPU) | GPU is pay-per-second |

**Key point:** The Geneva cluster is **CPU-only** — cheap to run. All GPU cost is
pure pay-per-use via Modal. No idle GPU costs.

---

## Two Integration Modes

From the inference UDF design doc (Approach 3), Geneva's Modal UDFs support two modes:

### Mode 1: Connect to Existing Endpoint

For teams that already have a Modal deployment running:

```python
# Using the ModalEmbeddingUDF sketched below in the E2E section
udf = ModalEmbeddingUDF(
    app_name="geneva-inference",  # Existing Modal app
    function_name="Embedder",
)

table.add_columns({"embedding": udf})
table.backfill("embedding")
```

### Mode 2: Self-Provisioning

The UDF auto-provisions the Modal endpoint in `setup()`:

```python
# A provisioning variant would deploy the Modal app in setup()
udf = ModalProvisionedEmbeddingUDF(
    model="sentence-transformers/all-MiniLM-L6-v2",
    gpu="A10G",           # or "A100", "H100"
    container_idle_timeout=300,
)

table.add_columns({"embedding": udf})
table.backfill("embedding")
```

**What happens under the hood:**

1. **First `setup()` call** — UDF provisions/deploys the inference microservice on Modal
2. **Subsequent calls** — Reuses the deployed endpoint
3. **Warm pool** — Modal maintains warm containers for low latency
4. **Teardown** — Optional cleanup after job completes (or keep warm for future jobs)

### Mode Comparison

| Aspect | Mode 1: Existing Endpoint | Mode 2: Self-Provisioning |
|--------|--------------------------|--------------------------|
| Pre-deployment needed | Yes (user deploys Modal app) | No (Geneva handles it) |
| Who manages Modal app | User | Geneva UDF |
| Container config | User controls image/GPU | Specified in UDF params |
| Best for | Shared inference services | Per-pipeline ephemeral GPU |
| Cold start | Warm (always running) | First call provisions (~10s) |

---

## End-to-End Example

Image embedding + captioning via Modal, with Geneva handling checkpointing,
incremental processing, and versioning:

### The Modal UDF (sketch)

Modal's `Function.lookup()` lets you call any deployed Modal function by name.
The Geneva UDF wraps this into a standard `setup()` / `__call__` pattern — no
GPU needed on the Geneva side.

```python
import geneva
from typing import Any


@geneva.udf(version="0.1")
class ModalEndpointUDF:
    """
    Generic Geneva UDF that calls a deployed Modal function.

    No GPU needed on the Geneva side — the GPU runs on Modal's
    serverless infrastructure. This UDF runs on CPU Ray workers
    and makes remote calls via Modal's Python client.
    """

    def __init__(
        self,
        app_name: str,
        function_name: str,
    ):
        self.app_name = app_name
        self.function_name = function_name

    def setup(self):
        import modal

        self.fn = modal.Function.lookup(
            self.app_name, self.function_name
        )

    def __call__(self, row: dict[str, Any]) -> dict[str, Any]:
        """
        Send a row to the Modal function, return the response.

        Users subclass this for their specific model's payload format.
        """
        return self.fn.remote(row)


# Concrete UDFs built on top of the generic one:

@geneva.udf(version="0.1")
class ModalEmbeddingUDF(ModalEndpointUDF):
    """CLIP embedding via a Modal function."""

    def __call__(self, image: bytes) -> list[float]:
        return self.fn.remote(image)


@geneva.udf(version="0.1")
class ModalCaptionUDF(ModalEndpointUDF):
    """BLIP captioning via a Modal function."""

    def __call__(self, image: bytes) -> str:
        return self.fn.remote(image)
```

**Key points:**
- **No `num_gpus`** — the UDF runs on CPU. The GPU runs on Modal's platform.
- **`modal` is the only dependency** — no ML frameworks needed on the Geneva cluster.
- **`Function.lookup()`** connects to an already-deployed Modal app in `setup()`.
- **`.remote()`** calls the function on Modal's GPU — serialization handled by Modal.
- **Batch support** — could use `.map()` instead of `.remote()` for parallel
  execution across a batch, complementing Geneva's `batch_size`.

### Pipeline Code

With the UDFs above, the pipeline is straightforward:

```python
import geneva

conn = geneva.connect("s3://my-bucket/workspace")
table = conn.open_table("images")

# Embedding via Modal — CLIP on A10G
table.add_columns({
    "embedding": ModalEmbeddingUDF(
        app_name="geneva-inference",
        function_name="clip-embed",
    ),
})

# Captioning via Modal — BLIP on A10G
table.add_columns({
    "caption": ModalCaptionUDF(
        app_name="geneva-inference",
        function_name="blip-caption",
    ),
})

# Run — Geneva handles everything else
table.backfill("embedding", batch_size=100)
table.backfill("caption", batch_size=50)

# Later: add new images → only new rows processed
table.add(new_images)
table.backfill("embedding")
table.backfill("caption")
```

**What Geneva handles:**
- Checkpointing (fragment + batch level, resume from exact failure point)
- Incremental processing (only rows where column IS NULL)
- Data versioning (every backfill creates a new version)
- Retry with exponential backoff on transient failures
- Adaptive batch sizing based on execution time
- Distributed orchestration across Ray workers

**What Modal handles:**
- GPU provisioning (A10G containers spun up automatically)
- Container caching (model weights baked into image, fast cold starts)
- Model loading (`@modal.enter()` loads once per container)
- Auto-scaling (more containers as Geneva sends more requests)
- Pay-per-second billing (no cost when idle)

---

## Comparison with Alternatives

Geneva supports three approaches to GPU inference. Each has different trade-offs:

| Factor | Third-Party API (Gemini UDFs) | Local GPU (`num_gpus=1`) | Modal + Geneva |
|--------|------------------------------|--------------------------|----------------|
| **Setup complexity** | Low | High | Medium |
| **GPU management** | None | Full responsibility | None |
| **Model flexibility** | Provider models only | Any model | Any model |
| **Cost at 1M rows** | $10–65 | $4–10 | $2.50–5 |
| **Cost at 1B rows** | $10K–65K | $1–2K | $2–5K |
| **Data privacy** | Low (data leaves infra) | High (data stays local) | Medium (crosses network) |
| **Latency** | High (50–500ms) | Low (<10ms) | Medium (10–100ms) |
| **Scaling** | Automatic | Manual / complex | Automatic |
| **Cold start** | None | 10–60s | 3–30s (warm pools) |
| **Idle cost** | None | ~$1–3/hr per GPU | None |

### Cost Deep Dive (1M embeddings, 384-dim)

Assumptions: 500 tokens/row avg, 50ms inference time per batch of 32.

| Approach | Compute Cost | Notes |
|----------|-------------|-------|
| OpenAI API (text-embedding-3-small) | $10 | Pay-per-token |
| Local A100 (1hr job) | $4 | Assumes full utilization |
| Modal A10G | $2.50 | ~15,000 sec total inference |
| Modal A100 | $4.20 | Faster but more expensive per second |

**Modal + Geneva is the sweet spot** for teams that want custom models without
GPU ops. You get the model flexibility of local GPU with the operational
simplicity of third-party APIs — and often at the lowest cost.

---

## When to Use This Pattern

### Decision Criteria

**Use Modal + Geneva when:**
- You want custom or open-source models (not just API providers)
- You don't want to manage GPU infrastructure
- Workloads are variable or bursty (not sustained 24/7)
- You need Geneva's data lifecycle (checkpointing, incremental, versioning)
- Cost optimization matters (pay-per-second, no idle GPUs)

**Use Third-Party APIs (Gemini UDFs) when:**
- Provider models are sufficient for your use case
- You want the simplest possible setup
- Dataset is < 10M rows
- No custom model requirements

**Use Local GPU when:**
- Sustained 24/7 workloads (GPU amortization makes sense)
- Latency-critical applications (< 10ms required)
- Strict data privacy (data cannot cross network boundary)
- You already have GPU infrastructure
- Large-scale production (100M+ rows, predictable volume)

### Decision Flowchart

```mermaid
flowchart TB
    Start["Need GPU inference<br/>in Geneva pipeline?"]
    Start --> Q1{"Provider model<br/>sufficient?<br/>(GPT, Gemini, Claude)"}

    Q1 -->|Yes| Q2{"Dataset < 10M rows?"}
    Q1 -->|"No — need custom<br/>or open-source model"| Q3{"Have GPU<br/>infrastructure?"}

    Q2 -->|Yes| API["<b>Third-Party API</b><br/>(Gemini UDFs)<br/>Simplest setup"]
    Q2 -->|"No — cost matters<br/>at scale"| Q3

    Q3 -->|"Yes — sustained<br/>24/7 workload"| LOCAL["<b>Local GPU</b><br/>(num_gpus=1)<br/>Lowest cost at scale"]
    Q3 -->|"No — or workload<br/>is variable"| MODAL["<b>Modal + Geneva</b><br/>Serverless GPU<br/>Pay-per-second"]
```

---

## Modal Capabilities Reference

How Modal features map to Geneva integration points:

| Modal Feature | What It Does | Geneva Integration |
|---------------|-------------|-------------------|
| `Function.lookup()` | Connect to deployed function | Mode 1: existing endpoint |
| `.remote()` | Call function on Modal GPU | Per-row or per-batch inference |
| `.map()` | Parallel execution across inputs | Batch processing within UDF |
| `@modal.enter()` | One-time setup (model loading) | Mirrors Geneva `setup()` |
| `min_containers` | Keep N containers warm | Reduces cold start for backfill |
| `container_idle_timeout` | Auto-shutdown after idle | Cost control between batches |
| `allow_concurrent_inputs` | Batching on Modal side | Complements Geneva batch_size |
| Pay-per-second billing | Only pay during inference | No idle GPU cost |
| GPU selection (T4/A10G/A100/H100) | Right-size GPU for model | Specified in UDF params |
| Container image caching | Fast cold starts | Model weights baked in |

---

## Implementation Status

| Phase | Description | Status |
|-------|-------------|--------|
| Phase 1 | API provider UDFs (Gemini embedding + generative) | **Implemented** |
| Phase 2 | Local GPU support (`num_gpus=1` on Ray workers) | **Implemented** |
| Phase 3 | Serverless GPU integration (Modal, Nebius) | **Planned** |
| Phase 4 | Unified `InferenceBackend` abstraction | **Planned** |

### Phase 3 Scope (Modal Integration)

- `ModalInferenceUDF` base class
- `NebiusInferenceUDF` base class
- Connection pooling and request batching
- Automatic failover between providers
- Cost optimization (spot instances, region selection)

### Open Questions

1. **Should Geneva manage inference endpoints directly?** Or leave deployment to users?
2. **How to handle model versioning?** Same UDF code should produce consistent results across runs.
3. **Connection pooling strategy?** Per-actor vs shared across actors?
4. **Cost attribution?** How to track inference costs per job/table?

---

## Related Documents

- [Modal vs Geneva](modal_vs_geneva.md) — Competitive comparison (Modal as standalone alternative)
- Inference UDF Design Doc (`geneva/internal_docs/inference-udf-design.md`) — Full design for all three inference approaches
- [Pixeltable vs Geneva](pixeltable_vs_geneva.md) — Closest competitor comparison (architecture patterns)
- [Comparison Overview](comparison_overview.md) — Category map of all competitors
