# Vertex AI + LanceDB Enterprise + Geneva: Managed GPU Inference for the Multimodal Lakehouse

Vertex AI provides managed GPU compute — GCP-integrated, auto-scaling endpoints,
Model Garden for pre-built models. LanceDB Enterprise + Geneva provides data
lifecycle management — checkpointing, incremental processing, versioning,
orchestration. Together they fill each other's gaps: Vertex AI handles GPU
infrastructure and model serving within GCP, Geneva handles the data pipeline
that feeds it.

| Concern | Vertex AI provides | Geneva provides |
|---------|-------------------|-----------------|
| GPU provisioning | Managed (endpoint or batch prediction) | — |
| Auto-scaling inference | Yes (endpoint auto-scaling) | — |
| Preemptible VM support | Yes (for custom training/batch jobs) | — |
| Model registry | Yes (Vertex AI Model Registry) | — |
| Model Garden | Yes (pre-built models, one-click deploy) | — |
| GCP ecosystem integration | Native (GCS, IAM, Cloud Monitoring, Artifact Registry) | — |
| Model training checkpoints | Yes — epoch-level for tuning; manual GCS saves for custom training ([docs](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/tuning-checkpoints)) | — (Geneva doesn't train models) |
| Data processing checkpoints | Via Dataflow/Apache Beam — but bundle-level, not row-level; batch jobs fail after 4 bundle retries with no resume ([docs](https://docs.cloud.google.com/dataflow/docs/pipeline-lifecycle)) | Automatic (fragment + batch, row-level resume) |
| Incremental processing | Via Dataflow — parameterized file inputs or streaming; no row-level change tracking. Vertex Pipelines step caching ([docs](https://docs.cloud.google.com/vertex-ai/docs/pipelines/rerun-pipeline-task)); Feature Store streaming ingestion | Automatic (only new rows, per-column) |
| Data versioning | Partial — dataset snapshots (image only, REST API); ML Metadata lineage ([docs](https://docs.cloud.google.com/vertex-ai/docs/datasets/manage-dataset-versions)) | Automatic (every backfill, Lance format) |
| Retry / error handling | Yes — pipeline task retry policies; training node restart ([docs](https://docs.cloud.google.com/vertex-ai/docs/pipelines/rerun-pipeline-task)) | Automatic per-row (tenacity-based, exponential backoff) |
| Distributed orchestration | Yes — distributed training clusters; Ray on Vertex AI ([docs](https://docs.cloud.google.com/vertex-ai/docs/open-source/ray-on-vertex-ai/overview)) | Ray Core (data-aware distribution) |
| Vector search / serving | Vertex AI Vector Search (ScaNN) ([docs](https://cloud.google.com/vertex-ai/docs/vector-search/overview)) | LanceDB Enterprise (Lance ANN, co-located with data) |
| Feature store / management | Feature Store (BigQuery-backed) | Computed columns (Lance-backed) |
| Pipeline orchestration | Vertex AI Pipelines (Kubeflow) | Ray Core orchestration |
| Built-in generative AI | Gemini API on Vertex | Built-in Gemini & OpenAI UDFs |

**Different concerns:** Vertex AI's checkpointing saves model weights during training
([docs](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/tuning-checkpoints))
— a fundamentally different concern from Geneva's data processing checkpoints (resume
feature computation from exact row). The GCP-native answer for data processing is
Dataflow/Apache Beam ([docs](https://docs.cloud.google.com/dataflow/docs/overview)),
which checkpoints at **bundle boundaries** — if a batch job's bundle fails 4 times,
the entire job fails with no resume
([docs](https://docs.cloud.google.com/dataflow/docs/pipeline-lifecycle)). Dataflow
does integrate with Vertex AI via `RunInference`
([docs](https://docs.cloud.google.com/dataflow/docs/machine-learning)), but even
that combination lacks row-level checkpointing for batch pipelines. Geneva provides
row-level resume, per-column incremental processing, and automatic versioning —
capabilities that don't exist in the Vertex AI + Dataflow stack today.

---

## Why Combine Them?

Each tool has a gap the other fills.

| Problem | Vertex AI alone | Geneva alone | Vertex AI + Geneva |
|---------|----------------|--------------|---------------------|
| GPU provisioning | Managed | Manual cluster | Managed |
| GCP-native integration | Yes | Limited | Yes |
| Any model (not just APIs) | Yes | Yes (needs GPU cluster) | Yes (no cluster) |
| Container management | You build + push to Artifact Registry | Not needed (pyproject.toml) | Vertex AI handles serving |
| Pre-built models | Model Garden (one-click deploy) | — | Model Garden + data lifecycle |
| Model training checkpoints | Epoch/step level (tuning jobs) | — (doesn't train models) | Vertex AI for model training |
| Data processing checkpoints | Dataflow: bundle-level, no batch resume | Row-level (fragment + batch) | Geneva for data pipelines |
| Incremental processing | Dataflow: file-based or streaming; Pipelines: step-level caching | Row-level (only NULL rows processed) | Row-level (Geneva), step-level (Vertex Pipelines) |
| Data versioning | Dataset snapshots (image only); ML Metadata lineage | Automatic per-backfill (Lance format) | Automatic per-backfill (Geneva), plus ML Metadata for models |
| Preemptible savings | Yes (up to 80% off) | Manual | Yes |
| Vector search | Vertex AI Vector Search (separate index) | LanceDB Enterprise (co-located) | LanceDB preferred (data + index co-located) |
| Feature management | Feature Store (BigQuery-backed) | Computed columns | Geneva computes; optionally export to Feature Store for Bigtable online serving |
| Pipeline orchestration | Vertex AI Pipelines (Kubeflow) | Ray Core | Geneva for data pipelines; Vertex Pipelines for ML workflows |

**The key insight from `cloud_managed_vs_geneva.md`:**

> **Vertex AI asks**: "Give me managed infrastructure for my ML code"
> **Geneva asks**: "Give me a managed feature column"

When combined, Vertex AI manages the GPU inference service and Geneva manages
the data lifecycle around it. The ~300 lines of glue code from the DIY approach
drops to ~20 lines of pipeline code.

**Bonus:** Vertex AI's Model Garden lets teams deploy open-source models
(CLIP, BLIP, Llama, etc.) with minimal container work.

---

## Full Data Prep Pipeline: Google-Native vs Geneva

A concrete comparison: compute CLIP embeddings + captions over 10M images,
serve them for vector search, and keep features fresh as new images arrive.

### Google-Native Stack

```mermaid
flowchart TB
    subgraph Sources["Data Sources"]
        GCS["GCS Bucket<br/>(raw images)"]
    end

    subgraph Orchestration["Vertex AI Pipelines (Kubeflow)"]
        direction TB
        P1["Step 1: Ingest"]
        P2["Step 2: Preprocess"]
        P3["Step 3: Inference"]
        P4["Step 4: Post-process"]
        P5["Step 5: Index"]
        P6["Step 6: Feature Ingest"]
        P1 --> P2 --> P3 --> P4 --> P5 & P6
    end

    subgraph Processing["Dataflow / Apache Beam"]
        direction TB
        DF1["Beam Pipeline<br/>(RunInference transform)"]
        DF2["Bundle-level checkpointing<br/>Batch: 4 retries then fail<br/>No row-level resume"]
    end

    subgraph Inference["Vertex AI Endpoints"]
        EP1["CLIP Endpoint<br/>(GPU)"]
        EP2["BLIP Endpoint<br/>(GPU)"]
    end

    subgraph Storage["Storage & Serving (separate systems)"]
        BQ["BigQuery<br/>(embeddings + captions)"]
        VS["Vertex AI Vector Search<br/>(ScaNN index, separate deploy)"]
        FS["Feature Store<br/>(BigQuery offline +<br/>Bigtable online)"]
    end

    subgraph YouBuild["You Build & Maintain"]
        direction TB
        C1["Container images<br/>(Artifact Registry)"]
        C2["Incremental logic<br/>(track processed files)"]
        C3["Schema management<br/>(BigQuery DDL)"]
        C4["Index sync pipeline<br/>(BQ → Vector Search)"]
        C5["Feature ingestion<br/>(BQ → Feature Store)"]
    end

    GCS --> Orchestration
    P3 --> Processing
    DF1 --> Inference
    Processing --> Storage
    YouBuild -.->|"you wire these together"| Orchestration & Processing & Storage
```

**Services involved:** GCS, Vertex AI Pipelines, Dataflow, Vertex AI Endpoints,
BigQuery, Vertex AI Vector Search, Vertex AI Feature Store, Artifact Registry,
Cloud Monitoring, IAM — **10+ services** to configure, connect, and maintain.

**What you build yourself:**
- Container images for each pipeline step → push to Artifact Registry
- Incremental processing logic (which files are new?)
- Row-level checkpointing (Dataflow only provides bundle-level)
- BigQuery schema definitions and migrations
- Sync pipeline: BigQuery → Vector Search index (separate deploy + rebuild)
- Sync pipeline: BigQuery → Feature Store (ingest job)
- Retry and error handling beyond Dataflow's 4-bundle-retry limit

### LanceDB Enterprise + Geneva Stack

```mermaid
flowchart TB
    subgraph Sources["Data Sources"]
        GCS["GCS Bucket<br/>(raw images)"]
    end

    subgraph Geneva["Geneva + LanceDB Enterprise"]
        direction TB

        subgraph UserCode["Your Code (~15 lines)"]
            U1["table.add_columns(embedding=ClipUDF)"]
            U2["table.add_columns(caption=BlipUDF)"]
            U3["table.backfill('embedding')"]
            U4["table.backfill('caption')"]
        end

        subgraph Auto["Automatic (Geneva)"]
            A1["Row-level checkpointing"]
            A2["Incremental processing<br/>(only NULL rows)"]
            A3["Schema from type hints"]
            A4["Retry per-row<br/>(exponential backoff)"]
            A5["Data versioning<br/>(every backfill)"]
        end

        subgraph LDB["LanceDB Enterprise"]
            L1["Lance Table<br/>(images + embeddings +<br/>captions in one table)"]
            L2["Vector Index<br/>(auto-updated,<br/>co-located)"]
            L3["SQL-like Queries<br/>(filtered search)"]
        end

        UserCode --> Auto --> LDB
    end

    subgraph Inference["Vertex AI Endpoints"]
        EP1["CLIP Endpoint<br/>(GPU)"]
        EP2["BLIP Endpoint<br/>(GPU)"]
    end

    GCS --> Geneva
    Auto -->|"HTTPS calls"| Inference
```

**Services involved:** GCS, LanceDB Enterprise, Geneva, Vertex AI Endpoints
— **4 services**. Vector search is co-located. Features are co-located.
No separate index deploy, no separate feature store, no sync pipelines.

**What you build yourself:**
- UDF functions (type-hinted Python, ~10 lines each)
- That's it

### Side-by-Side Summary

| Concern | Google-Native Stack | Geneva Stack |
|---------|-------------------|--------------|
| Services to configure | 10+ (GCS, Pipelines, Dataflow, Endpoints, BQ, Vector Search, Feature Store, Artifact Registry, IAM, Monitoring) | 4 (GCS, Geneva, LanceDB Enterprise, Endpoints) |
| Data processing checkpoints | Bundle-level (Dataflow); batch fails after 4 retries | Row-level; resume from exact failure point |
| Incremental processing | Custom logic (track processed files) | Automatic (only NULL rows) |
| Vector index sync | Separate pipeline (BQ → Vector Search rebuild) | Co-located (auto-updated with data) |
| Feature serving | Separate pipeline (BQ → Feature Store → Bigtable) | Co-located (query Lance table directly) |
| Schema management | BigQuery DDL, manual migrations | Inferred from UDF type hints |
| Container management | Build + push to Artifact Registry per step | None (pyproject.toml) |
| Data versioning | Dataset snapshots (image only, manual) | Automatic (every backfill) |
| Lines of pipeline code | ~200-300 + Terraform | ~15 |
| Ongoing maintenance | Sync pipelines, index rebuilds, schema migrations | `table.backfill()` on new data |

---

## Architecture

```mermaid
flowchart TB
    A["<b>Your Code</b><br/>UDF definitions · table.backfill()"]
    A --> LH

    subgraph LH["Multimodal Lakehouse (CPU-only cluster)"]
        direction TB

        subgraph G["Geneva (compute layer)"]
            direction TB
            G1["Checkpointing"] ~~~ G2["Incremental Processing"] ~~~ G3["Retry / Error Handling"]
            G4["UDF Versions"] ~~~ G5["Adaptive Batching"] ~~~ G6["Orchestration"]
        end

        subgraph R["Ray Workers (CPU)"]
            W1["Worker 1<br/>(UDF)"]
            W2["Worker 2<br/>(UDF)"]
            W3["Worker 3<br/>(UDF)"]
        end

        subgraph L["LanceDB Enterprise (storage + serving)"]
            direction TB
            L1["Lance Tables"] ~~~ L2["Vector Search"] ~~~ L3["Data Versions"]
        end

        G --> R
        R --> L
    end

    subgraph VAI["Google Vertex AI (managed GPU platform)"]
        direction TB
        subgraph EP["Vertex AI Endpoint"]
            EP1["GPU Node<br/>(Model)<br/>n1-standard + T4"]
            EP2["GPU Node<br/>(Model)<br/>n1-standard + T4"]
            EP3["GPU Node<br/>(Model)<br/>n1-standard + T4"]
        end
        VAIC["Artifact Registry images<br/>Auto-scaling · Cloud Monitoring"]
        VAIS["Other Vertex AI Services<br/>Vector Search · Feature Store<br/>Pipelines · Gemini API<br/>Model Garden · RAG Engine"]
    end

    W1 -->|"HTTPS (aiplatform SDK)"| VAI
    W2 -->|"HTTPS (aiplatform SDK)"| VAI
    W3 -->|"HTTPS (aiplatform SDK)"| VAI
```

### What Runs Where

| Concern | Owner | Notes |
|---------|-------|-------|
| Data storage + versioning | LanceDB Enterprise | Lance format on GCS, vector indices |
| Checkpointing + resume | Geneva | Fragment + batch level |
| Incremental processing | Geneva | Only new/changed rows |
| Retry / error handling | Geneva | Configurable tenacity-based |
| Distributed orchestration | Geneva + Ray | CPU-only cluster |
| GPU inference | Vertex AI | Managed endpoint, auto-scaled |
| Model container images | Vertex AI + Artifact Registry | Built once, or Model Garden pre-built |
| Scaling GPU workers | Vertex AI | Auto-scaling on request metrics |
| Model registry | Vertex AI | Model versioning, evaluation |
| Billing | Vertex AI (GPU) + cluster (CPU) | GPU is per-node-hour (preemptible available) |

**Key point:** The Geneva cluster is **CPU-only** — cheap to run. All GPU cost is
via Vertex AI endpoints, which auto-scale and support preemptible VMs. No GPU
management in the Geneva cluster.

---

## Two Integration Modes

### Mode 1: Call Existing Vertex AI Endpoint

For teams that already have a Vertex AI endpoint deployed:

```python
# Using the VertexAIEmbeddingUDF sketched below in the E2E section
udf = VertexAIEmbeddingUDF(
    endpoint_id="1234567890",             # Existing endpoint
    project="my-gcp-project",
    location="us-central1",
)

table.add_columns({"embedding": udf})
table.backfill("embedding")
```

This is the most common pattern — the ML platform team deploys and manages
Vertex AI endpoints (or deploys from Model Garden), and data teams consume
them via Geneva.

### Mode 2: Self-Provisioning

The UDF deploys from Model Garden or a custom container in `setup()`:

```python
# A provisioning variant would deploy from Model Garden in setup()
udf = VertexAIProvisionedEmbeddingUDF(
    model_garden_id="google/clip-vit-base-patch32",  # Model Garden model
    machine_type="n1-standard-4",
    accelerator_type="NVIDIA_TESLA_T4",
    teardown_on_complete=True,
)

table.add_columns({"embedding": udf})
table.backfill("embedding")
```

**What happens under the hood:**

1. **First `setup()` call** — UDF uploads model + creates endpoint + deploys model to endpoint
2. **Endpoint becomes ready** — ~5-15 minutes for cold start (container pull + model load)
3. **Subsequent calls** — Reuses the live endpoint
4. **Teardown** — Optional cleanup after job completes (undeploy model, delete endpoint)

### Mode Comparison

| Aspect | Mode 1: Existing Endpoint | Mode 2: Self-Provisioning |
|--------|--------------------------|--------------------------|
| Pre-deployment needed | Yes (ML platform team or Model Garden) | No (Geneva handles it) |
| Who manages endpoint | ML platform / MLOps team | Geneva UDF |
| Container + model config | Team controls via console/Terraform | Specified in UDF params |
| Best for | Shared inference services, enterprise | Per-pipeline ephemeral GPU |
| Cold start | Warm (always running) | ~5-15 min (endpoint creation) |
| Idle cost | ~$0.35-1.20/hr per node | None if teardown enabled |
| IAM required | Endpoint predict permission | Full Vertex AI create/delete permissions |

---

## End-to-End Example

Image embedding + captioning via Vertex AI, with Geneva handling checkpointing,
incremental processing, and versioning:

### The Vertex AI UDF (sketch)

The Geneva UDF wraps the `google-cloud-aiplatform` SDK's `Endpoint.predict()`
call. This would be built by the Geneva team or the user — it's a generic
wrapper, not per-model code.

```python
import geneva
import json
from typing import Any


@geneva.udf(version="0.1")
class VertexAIEndpointUDF:
    """
    Generic Geneva UDF that calls a Vertex AI online prediction endpoint.

    No GPU needed on the Geneva side — the GPU runs behind the
    Vertex AI endpoint. This UDF runs on CPU Ray workers and makes
    HTTP calls via the aiplatform SDK.
    """

    def __init__(
        self,
        endpoint_id: str,
        project: str,
        location: str = "us-central1",
    ):
        self.endpoint_id = endpoint_id
        self.project = project
        self.location = location

    def setup(self):
        from google.cloud import aiplatform

        aiplatform.init(project=self.project, location=self.location)
        self.endpoint = aiplatform.Endpoint(self.endpoint_id)

    def __call__(self, row: dict[str, Any]) -> dict[str, Any]:
        """
        Send a row to the Vertex AI endpoint, return the response.

        The payload format depends on the model behind the endpoint.
        Users subclass or configure this for their specific model.
        """
        response = self.endpoint.predict(instances=[row])
        return response.predictions[0]


# Concrete UDFs built on top of the generic one:

@geneva.udf(version="0.1")
class VertexAIEmbeddingUDF(VertexAIEndpointUDF):
    """CLIP embedding via a Vertex AI endpoint."""

    def __call__(self, image: bytes) -> list[float]:
        import base64

        instance = {"image": base64.b64encode(image).decode()}
        response = self.endpoint.predict(instances=[instance])
        return response.predictions[0]["embedding"]


@geneva.udf(version="0.1")
class VertexAICaptionUDF(VertexAIEndpointUDF):
    """BLIP captioning via a Vertex AI endpoint."""

    def __call__(self, image: bytes) -> str:
        import base64

        instance = {"image": base64.b64encode(image).decode()}
        response = self.endpoint.predict(instances=[instance])
        return response.predictions[0]["caption"]
```

**Key points:**
- **No `num_gpus`** — the UDF runs on CPU. The GPU is behind the Vertex AI endpoint.
- **`google-cloud-aiplatform` is the only dependency** — no ML frameworks needed on the Geneva cluster.
- **Generic base class** — `VertexAIEndpointUDF` handles the SDK plumbing;
  subclasses define the payload format and return type for schema inference.
- **`Endpoint.predict()`** accepts a list of instances — could batch multiple
  rows per call for throughput.
- **Batch Prediction alternative** — for very large offline jobs, Vertex AI Batch
  Prediction can process GCS→GCS directly, which Geneva could orchestrate instead
  of per-row endpoint calls.

### Pipeline Code

With the UDFs above, the pipeline is straightforward:

```python
import geneva

conn = geneva.connect("gs://my-bucket/workspace")
table = conn.open_table("images")

# Embedding via Vertex AI — CLIP on T4
table.add_columns({
    "embedding": VertexAIEmbeddingUDF(
        endpoint_id="1234567890",         # Pre-deployed by ML platform
        project="my-gcp-project",
        location="us-central1",
    ),
})

# Captioning via Vertex AI — BLIP on T4
table.add_columns({
    "caption": VertexAICaptionUDF(
        endpoint_id="9876543210",         # Pre-deployed by ML platform
        project="my-gcp-project",
        location="us-central1",
    ),
})

# Run — Geneva handles everything else
table.backfill("embedding", batch_size=100)
table.backfill("caption", batch_size=50)

# Later: add new images → only new rows processed
table.add(new_images)
table.backfill("embedding")
table.backfill("caption")
```

**What Geneva handles:**
- Checkpointing (fragment + batch level, resume from exact failure point)
- Incremental processing (only rows where column IS NULL)
- Data versioning (every backfill creates a new version)
- Retry with exponential backoff on transient failures
- Adaptive batch sizing based on execution time
- Distributed orchestration across Ray workers

**What Vertex AI handles:**
- GPU provisioning (managed nodes behind endpoint)
- Model serving (container with model weights, health checks)
- Auto-scaling (request-based scaling)
- Model Garden (deploy open-source models without building containers)
- Model Registry (versioned models, evaluation metrics)
- Cloud Monitoring and logging
- Preemptible VM support (for batch/training jobs)

---

## Comparison: Vertex AI + Geneva vs DIY Vertex AI

The `cloud_managed_vs_geneva.md` doc shows ~280 lines + config for a DIY
cloud ML pipeline. With the Vertex AI + Geneva pattern:

| Component | DIY Vertex AI | Vertex AI + Geneva |
|-----------|--------------|---------------------|
| Inference code | ~60 lines (prediction container) | Endpoint already deployed |
| Pipeline/orchestration | ~40 lines (batch_predict call) | ~15 lines (table.backfill) |
| Container + Artifact Registry | ~20 lines + build steps | 0 (endpoint or Model Garden) |
| Data processing checkpoints | ~40 lines (Dataflow provides bundle-level only; Vertex training checkpoints are a different concern) | 0 (Geneva automatic, row-level) |
| IAM/permissions | ~30 lines (Terraform) | Predict-only IAM |
| Compute setup | ~20 lines (Terraform) | 0 (endpoint pre-exists) |
| **TOTAL** | **~210 lines + config** | **~15 lines** |

The container, model serving, and GPU management still exist — they're just
owned by the ML platform team via Vertex AI, not reimplemented in every pipeline.

---

## Comparison with Other Geneva Inference Approaches

| Factor | Third-Party API (Gemini/OpenAI UDFs) | Local GPU (`num_gpus=1`) | Modal + Geneva | Vertex AI + Geneva |
|--------|--------------------------------------|--------------------------|----------------|--------------------|
| **Setup complexity** | Low | High | Medium | Medium-High |
| **GPU management** | None | Full responsibility | None | Vertex AI manages |
| **Model flexibility** | Provider models only | Any model | Any model | Any model + Model Garden |
| **Cloud integration** | None | Manual | None | GCP-native |
| **Data privacy** | Low (data leaves infra) | High (stays local) | Medium (crosses network) | High (GCP VPC-SC) |
| **Latency** | High (50-500ms) | Low (<10ms) | Medium (10-100ms) | Medium (10-50ms) |
| **Scaling** | Automatic | Manual / complex | Automatic | Automatic |
| **Cold start** | None | 10-60s | 3-30s | 5-15 min |
| **Idle cost** | None | ~$1-3/hr per GPU | None | ~$0.35-1.20/hr |
| **Preemptible/Spot** | N/A | Manual | N/A | Preemptible built-in |
| **Governance** | Limited | Manual | Limited | Full (IAM, VPC-SC, Audit) |
| **Pre-built models** | Provider only | Any | Any | Model Garden |

### Cost Deep Dive (1M embeddings, 384-dim)

Assumptions: 500 tokens/row avg, 50ms inference time per batch of 32.

| Approach | Compute Cost | Notes |
|----------|-------------|-------|
| OpenAI API (text-embedding-3-small) | $10 | Pay-per-token |
| Local A100 (1hr job) | $4 | Assumes full utilization |
| Modal A10G | $2.50 | ~15,000 sec total inference, pay-per-second |
| Vertex AI n1+T4 (on-demand) | $4.10 | ~1hr runtime + endpoint overhead |
| Vertex AI n1+T4 (preemptible) | $1.50 | ~60% discount, but can be preempted |

**Vertex AI + Geneva is the GCP enterprise choice** — Model Garden reduces setup
friction, and GCP's T4 pricing is competitive. Geneva eliminates the data
pipeline glue code.

---

## When to Use This Pattern

### Decision Criteria

**Use Vertex AI + Geneva when:**
- Your organization is standardized on GCP
- You need VPC Service Controls / data residency within GCP
- ML platform team already manages Vertex AI endpoints
- You want to leverage Model Garden for quick model deployment
- Compliance requires GCP IAM, audit logging, and governance controls
- You want managed GPU infrastructure without building pipelines

**Use Modal + Geneva when:**
- You want simplest possible GPU setup (no containers, no IAM)
- Workloads are bursty (pay-per-second matters)
- No cloud provider lock-in requirement
- Startup / small team without ML platform engineers

**Use Third-Party APIs (Gemini/OpenAI UDFs) when:**
- Provider models are sufficient
- Simplest possible setup
- Dataset is < 10M rows

**Use Local GPU when:**
- Sustained 24/7 workloads
- Latency-critical (< 10ms)
- Strict data isolation (no network boundary crossing)
- Already have GPU infrastructure

### Decision Flowchart

```mermaid
flowchart TB
    Start["Need GPU inference<br/>in Geneva pipeline?"]
    Start --> Q1{"Provider model<br/>sufficient?<br/>(GPT, Gemini, Claude)"}

    Q1 -->|Yes| Q2{"Dataset < 10M rows?"}
    Q1 -->|"No — need custom<br/>or open-source model"| Q3{"Which cloud?"}

    Q2 -->|Yes| API["<b>Third-Party API</b><br/>(Gemini/OpenAI UDFs)<br/>Simplest setup"]
    Q2 -->|"No — cost matters<br/>at scale"| Q3

    Q3 -->|"GCP"| Q4G{"Have GPU<br/>infrastructure<br/>already?"}
    Q3 -->|"None / multi-cloud"| MODAL["<b>Modal + Geneva</b><br/>Serverless GPU<br/>Pay-per-second"]

    Q4G -->|"Yes — sustained<br/>24/7 workload"| LOCAL["<b>Local GPU</b><br/>(num_gpus=1)<br/>Lowest cost at scale"]
    Q4G -->|"No — let Vertex AI<br/>manage it"| VAI["<b>Vertex AI + Geneva</b><br/>Managed GPU<br/>GCP-native governance"]
```

---

## Vertex AI Capabilities Reference

How Vertex AI features map to Geneva integration points:

| Vertex AI Feature | What It Does | Geneva Integration |
|-------------------|-------------|-------------------|
| Online Prediction Endpoints | Low-latency model serving | Mode 1: call via `Endpoint.predict()` |
| Batch Prediction | Batch inference on GCS data | Alternative to endpoint for large jobs |
| Custom Training Jobs | Run arbitrary containers on managed GPU | Could wrap Geneva UDF as custom job |
| Model Garden | One-click deploy of open-source models | Fastest path to a working endpoint |
| Model Registry | Version and evaluate models | Track model version in UDF `version` param |
| Auto-scaling | Scale nodes on request metrics | Complements Geneva's distributed workers |
| Preemptible VMs | Up to 80% cost savings | Use for batch/training, not endpoints |
| VPC Service Controls | Network + data isolation | Data stays within GCP perimeter |
| Cloud Monitoring | Monitoring and alerting | Monitor inference latency/errors |
| IAM | Fine-grained access control | Predict-only permissions for Geneva |
| Vertex AI Pipelines | ML workflow orchestration | Geneva replaces this for data pipelines |
| **Overlaps** Vector Search | Managed ANN index (ScaNN) | LanceDB Enterprise provides this natively |
| **Overlaps** Feature Store | BigQuery-backed feature management | Geneva computed columns provide this natively |
| **Overlaps** Gemini API | Gemini models on Vertex | Geneva has built-in Gemini UDFs (same API) |
| Vertex AI Search | Enterprise search over documents | Complementary — different use case |
| Agent Builder | Build conversational AI agents | Complementary — not a Geneva concern |
| RAG Engine | Managed retrieval-augmented generation | Complementary — could use LanceDB as retriever |
| Model Monitoring | Detect drift, skew, feature attribution | Complementary — monitors deployed models |
| Experiments / ML Metadata | Track experiments and lineage | Complementary — Geneva tracks data versions, not experiments |
| AutoML | No-code model training | Complementary — Geneva consumes models, doesn't train them |

---

## Vertex AI Services: Overlap vs Complement

Vertex AI is a broad platform. Some services complement Geneva, some overlap.

### Service Classification

| Vertex AI Service | Relationship | Notes |
|-------------------|-------------|-------|
| Online Prediction Endpoints | Complementary | Geneva calls these for GPU inference |
| Batch Prediction | Complementary | Alternative to endpoint for large offline jobs |
| Custom Training Jobs | Complementary | Geneva doesn't train models |
| Model Garden | Complementary | Quick path to a deployable endpoint |
| Model Registry | Complementary | Geneva tracks data versions, not model versions |
| Auto-scaling | Complementary | Vertex scales GPUs, Geneva scales CPU workers |
| VPC Service Controls | Complementary | Security perimeter for both |
| Cloud Monitoring | Complementary | Monitor inference latency from Geneva UDFs |
| AutoML | Complementary | Geneva consumes models, doesn't train them |
| Model Monitoring | Complementary | Drift detection on deployed models |
| Experiments / ML Metadata | Complementary | Experiment tracking is orthogonal to feature pipelines |
| Vertex AI Search | Complementary | Enterprise search, different use case |
| Agent Builder | Complementary | Conversational AI, not feature engineering |
| RAG Engine | Complementary | Could use LanceDB as the retrieval backend |
| Ray on Vertex AI | Complementary | Managed Ray clusters; Geneva could run on Vertex AI Ray ([docs](https://docs.cloud.google.com/vertex-ai/docs/open-source/ray-on-vertex-ai/overview)) |
| Distributed Training | Complementary | Multi-node training clusters ([docs](https://docs.cloud.google.com/vertex-ai/docs/training/distributed-training)); Geneva doesn't train models |
| **Vector Search** | **Overlapping** | Both provide ANN search over embeddings |
| **Feature Store** | **Overlapping** | Both manage computed features |
| **Vertex AI Pipelines** | **Overlapping** | Both orchestrate data/ML workflows |
| **Gemini API on Vertex** | **Overlapping** | Same Gemini models, different lifecycle management |

### Vertex AI Vector Search vs LanceDB Enterprise

| Dimension | Vertex AI Vector Search | LanceDB Enterprise |
|-----------|------------------------|-------------------|
| Algorithm | ScaNN (Google Research) | Lance ANN (IVF-PQ, DiskANN) |
| Deployment model | Separate managed index, deployed to endpoint | Co-located with source data in Lance tables |
| Co-location with features | No — index is separate from features in BigQuery/GCS | Yes — embeddings, features, and source data in one table |
| Filtered search | Metadata filtering via restricts/numeric restricts | SQL-like filtering on any column |
| Versioning | Manual index rebuilds | Automatic (Lance format versions) |
| Update model | Full rebuild or streaming update (extra cost) | Incremental (new fragments auto-indexed) |
| Scaling | Managed replicas + auto-scaling | LanceDB Enterprise cluster |
| Pricing | Per-node-hour + per-query | Included with LanceDB Enterprise |

**Positioning:** LanceDB Enterprise is preferred when using Geneva — embeddings are
co-located with source data and other computed features, so there's no ETL step to
sync vectors to a separate index. Vertex AI Vector Search makes sense for teams with
existing Vertex AI investments who need a standalone vector search service independent
of their feature pipeline.

### Vertex AI Feature Store vs Geneva Computed Columns

| Dimension | Vertex AI Feature Store | Geneva Computed Columns |
|-----------|------------------------|------------------------|
| Storage backend | BigQuery (offline) + Bigtable (online) | Lance format (columnar, versioned) |
| Feature definition | SQL transforms or custom pipelines → ingest | UDF functions with type hints |
| Computation engine | External (Dataflow, Spark, custom) — Feature Store only stores results | Built-in (Ray workers execute UDFs) |
| Online serving | Yes (Bigtable-backed, low-latency) | Via LanceDB Enterprise |
| Incremental updates | Streaming ingestion + periodic sync to online store | Automatic (only new rows processed, per-column) |
| Multimodal support | Limited (tabular-focused) | Native (images, video, PDFs, embeddings) |
| Checkpointing | Not included (your pipeline's responsibility) | Automatic (fragment + batch level) |
| Versioning | Feature Store snapshots | Automatic (Lance format) |

**Positioning:** Feature Store excels in tabular ML within BigQuery-centric stacks —
especially when you need Bigtable online serving for real-time ML. Geneva excels at
multimodal feature engineering where computation (not just storage) is the hard part.
The two can work together: Geneva computes features → export to Feature Store for
Bigtable online serving.

### Vertex AI Pipelines vs Geneva Orchestration

| Dimension | Vertex AI Pipelines | Geneva Orchestration |
|-----------|--------------------|--------------------|
| Engine | Kubeflow Pipelines (managed) | Ray Core |
| Abstraction level | Multi-step DAG (train → eval → deploy) | Per-column backfill (table.backfill) |
| Checkpointing granularity | Per-step (step succeeds or fails) | Per-fragment + per-batch within a step |
| Incremental support | Step-level caching (reuse unchanged step outputs; [Preview](https://docs.cloud.google.com/vertex-ai/docs/pipelines/rerun-pipeline-task)) | Row-level (only NULL rows processed) |
| Container requirements | Yes (each step is a container) | No (UDFs packaged from pyproject.toml) |
| Scheduling | Managed (cron, event-based) | External (cron, Airflow, etc.) |
| Artifact tracking | Built-in (ML Metadata) | Lance versions |

**Positioning:** Different granularity. Vertex AI Pipelines orchestrates multi-step ML
workflows (data prep → training → evaluation → deployment). Geneva orchestrates feature
computation within a single step. Realistic pattern: a Vertex AI Pipeline DAG triggers
`table.backfill()` as one step in a larger ML workflow.

### Gemini API on Vertex vs Geneva Built-in Gemini UDFs

| Dimension | Gemini API on Vertex | Geneva Gemini UDFs |
|-----------|---------------------|--------------------|
| Underlying model | Gemini (same) | Gemini (same) |
| API | `generativeai` SDK or Vertex SDK | Geneva UDF wrapping the same API |
| Batching | Manual | Automatic (adaptive batch sizing) |
| Checkpointing | None (your code) | Automatic (fragment + batch) |
| Incremental processing | None (your code) | Automatic (only new rows) |
| Rate limiting | Manual (your code) | Built-in (configurable) |
| Retry | Manual (your code) | Automatic (tenacity-based) |
| Billing | Vertex AI (per-token) | Same Vertex AI billing (per-token) |

**Positioning:** Same API, same model, same billing. Geneva UDFs add lifecycle
management (checkpointing, incremental processing, retry, rate limiting) on top of
the raw Gemini API. Use Geneva Gemini UDFs for batch feature computation over tables.
Use Gemini on Vertex directly for real-time inference, agent workflows, or interactive
use cases.

---

## Implementation Status

| Phase | Description | Status |
|-------|-------------|--------|
| Phase 1 | API provider UDFs (Gemini/OpenAI embedding + generative) | **Implemented** |
| Phase 2 | Local GPU support (`num_gpus=1` on Ray workers) | **Implemented** |
| Phase 3 | Serverless GPU integration (Modal) | **Planned** |
| Phase 3b | Managed GPU integration (Vertex AI) | **Planned** |
| Phase 4 | Unified `InferenceBackend` abstraction | **Planned** |

### Phase 3b Scope (Vertex AI Integration)

- `VertexAIInferenceUDF` base class
- `VertexAIEndpointUDF` for calling existing endpoints (Mode 1)
- `VertexAIProvisionedUDF` for auto-provisioning from Model Garden (Mode 2)
- Batch Prediction mode for large offline jobs
- Request batching via `instances` list in `predict()` call
- GCP credential handling (service account or application default)

### Open Questions

1. **Endpoint lifecycle management?** Should Geneva create/delete endpoints, or always expect pre-existing ones?
2. **Batch Prediction vs Online Prediction?** When should Geneva use Batch Prediction (GCS→GCS) vs endpoint calls?
3. **Model Garden integration depth?** Should Geneva have a catalog of Model Garden model IDs for common tasks (CLIP, BLIP, etc.)?
4. **Cost attribution?** Vertex AI endpoint costs are per-node-hour — how to attribute across Geneva jobs?
5. **Gemini UDF overlap?** Geneva already has built-in Gemini UDFs. Vertex AI endpoints can also serve Gemini. When to use which?
6. **Vector Search coexistence?** Should Geneva write embeddings to both Lance and Vertex AI Vector Search for teams that need both?
7. **Feature Store export?** Should Geneva support exporting computed columns to Vertex AI Feature Store for Bigtable online serving?
8. **Vertex Pipelines component?** Should we provide a Kubeflow component for triggering `table.backfill()` from within a Vertex AI Pipeline DAG?

---

## Related Documents

- [Cloud Managed vs Geneva](cloud_managed_vs_geneva.md) — Competitive comparison (cloud ML platforms as standalone alternatives)
- [Modal + Geneva](modal_plus_geneva.md) — Serverless GPU partner integration (similar pattern, different trade-offs)
- [Comparison Overview](comparison_overview.md) — Category map of all competitors
