# Geneva Architecture Overview

High-level and detailed architecture diagrams for Geneva deployments.
See also: [deployment.md](deployment.md), [resource_guide.md](resource_guide.md)

## 1. High-Level Architecture

Two primary usage modes: **managed deployment** (via `db://` URI through the data
plane) and **direct access** (client connects to storage and K8s directly).

### 1A. Managed Deployment (`db://` URI)

The client connects through the data plane (Router → Phalanx). Backfills can be
**client-initiated** (explicit `table.backfill()`) or **server-initiated**
(auto-backfill triggered by the Lance Agent when new data arrives).

```mermaid
flowchart TB
    subgraph customer["Customer Environment"]
        code["User Code<br/>(Python)"]
        client["Geneva Client<br/>(SDK)"]
        code --> client
    end

    subgraph k8s["Kubernetes Cluster (EKS / GKE / AKS)"]
        subgraph lancedb_ns["lancedb namespace (Data Plane)"]
            router["Router<br/>(Pingora LB)"]
            phalanx["Query Node<br/>(Phalanx)"]
            kafka["Kafka"]
            agent["Lance Agent"]
            indexer["Indexer Jobs"]

            router --> phalanx
            phalanx -->|"table events"| kafka
            kafka --> agent
            agent -->|"index/compact"| indexer
        end

        subgraph geneva_ns["geneva namespace"]
            console["Geneva Console<br/>(UI + read-only API)"]
            driver["Geneva Driver<br/>(K8s Job, spawned<br/>by Lance Agent)"]
            kuberay["KubeRay<br/>Operator"]
            subgraph ray["Ray Cluster"]
                head["Ray Head"]
                w1["Workers: CPU"]
                w2["Workers: GPU"]
                head --- w1
                head --- w2
            end
            kuberay -->|provisions| ray
        end

        agent -->|"auto-backfill:<br/>spawns K8s Job"| driver
        driver -->|"IN_CLUSTER auth"| kuberay
        driver -->|"table.backfill()"| head
        phalanx -.->|"user-triggered backfill<br/>(roadmap / TODO)"| driver
        console -->|"reads job/cluster<br/>metadata"| phalanx
    end

    subgraph storage["Cloud Storage (S3 / GCS / Azure Blob)"]
        data["Lance Tables<br/>(fragments + indexes)"]
        systables["System Tables<br/>(jobs, manifests,<br/>cluster defs)"]
        manifests["Manifest ZIPs"]
    end

    client -->|"db:// table catalog +<br/>data writes<br/>(HTTP / Flight SQL)"| router
    client -.->|"client-initiated backfill:<br/>K8s API (to deprecate)"| kuberay
    client -.->|"client-initiated backfill:<br/>submit job (to deprecate)"| head
    client -.->|"manifest upload<br/>(to deprecate)"| manifests

    phalanx -->|"read/write"| data
    indexer -->|"build indexes"| data
    w1 & w2 -->|"read/write<br/>(direct storage)"| data
    driver -->|"read metadata"| systables
    w1 & w2 -->|"download"| manifests

    linkStyle 14 stroke:#f44336,stroke-dasharray:5 5
    linkStyle 15 stroke:#f44336,stroke-dasharray:5 5
    linkStyle 16 stroke:#f44336,stroke-dasharray:5 5
    linkStyle 11 stroke:#4caf50,stroke-dasharray:5 5
```

**Three backfill paths:**

| Path | Trigger | Who provisions Ray? | Client needs K8s access? | Status |
|------|---------|-------------------|------------------------|--------|
| **Auto-backfill** | Lance Agent detects when to backfill `auto_backfill=True` UDF columns | Geneva Driver pod (IN_CLUSTER) | No | Active |
| **User-triggered via QN** | User calls backfill through Phalanx → Geneva Driver | Geneva Driver pod (IN_CLUSTER) | No | Roadmap |
| **Client-initiated** | User calls `table.backfill()` directly from SDK | Client via K8s API | Yes | To deprecate (roadmap) |

**Auto-backfill flow:**
1. Client writes data via `db://` → Router → Phalanx
2. Phalanx publishes Kafka event
3. Lance Agent's `BackfillColumnCriterion` scans for UDF columns with `virtual_column.auto_backfill=true` metadata and NULL values exceeding threshold
4. Agent spawns a **Geneva Driver** K8s Job in the `geneva` namespace
5. Geneva Driver calls `geneva.connect()` → `table.backfill(column_name)` with IN_CLUSTER K8s auth
6. KubeRay provisions a Ray cluster, workers execute the UDF
7. Console API exposes job status for monitoring

### 1B. Direct Access (`s3://` / `gs://` / `az://` URI)

The client connects directly to storage and manages Ray clusters via the K8s API.
No data plane services are involved. No auto-backfill — all backfills are
client-initiated.

```mermaid
flowchart TB
    subgraph customer["Customer Environment"]
        code["User Code<br/>(Python)"]
        client["Geneva Client<br/>(SDK)"]
        code --> client
    end

    subgraph k8s["Kubernetes Cluster (EKS / GKE / AKS)"]
        subgraph geneva_ns["geneva namespace"]
            kuberay["KubeRay<br/>Operator"]
            subgraph ray["Ray Cluster"]
                head["Ray Head"]
                w1["Workers: CPU"]
                w2["Workers: GPU"]
                head --- w1
                head --- w2
            end
            kuberay -->|provisions| ray
        end
    end

    subgraph storage["Cloud Storage (S3 / GCS / Azure Blob)"]
        data["Lance Tables<br/>(fragments + indexes)"]
        manifests["Manifest ZIPs"]
        checkpoints["Checkpoints"]
        systables["System Tables<br/>(jobs, errors,<br/>cluster defs)"]
    end

    subgraph external["External Services"]
        external_api["ML APIs<br/>(OpenAI, Gemini,<br/>HuggingFace)"]
    end

    client -->|"K8s API"| kuberay
    client -->|"submit jobs"| head
    client -->|"direct storage<br/>(upload manifests,<br/>read/write metadata)"| storage

    w1 & w2 -->|"read/write<br/>(direct storage)"| data
    w1 & w2 -->|"emit"| checkpoints
    w1 & w2 -.->|"optional"| external_api
    w1 & w2 -->|"download"| manifests
```

**Key differences from managed deployment:**

| Aspect | Managed (`db://`) | Direct Access (`s3://`) |
|--------|------------------|----------------------|
| **Data plane** | Router, Phalanx, Kafka, Agent | Not used |
| **Table catalog** | Via Phalanx REST/Flight SQL | Direct to storage |
| **Auto-backfill** | Supported (Lance Agent + Geneva Driver) | Not available |
| **Client-initiated backfill** | Optional (client needs K8s access) | Required (only path) |
| **Indexing / compaction** | Automatic (Lance Agent) | Manual |
| **Console monitoring** | Available | Not available |
| **Client K8s access required** | Only for client-initiated backfill | Always |

## 1a. Server-Side Services (from Sophon)

The server-side components are deployed via two Helm charts from the
[Sophon repo](https://github.com/lancedb/sophon):
- `helm/lancedb` — data plane (query node, router, agent, indexer, Kafka)
- `helm/geneva` — management console (API + UI)

```mermaid
flowchart LR
    subgraph data_plane["Data Plane (helm/lancedb)"]
        direction TB
        router["**Router**<br/>Rust / Pingora<br/>:6188 HTTP<br/>:6189 Admin<br/>:6190 Metrics"]
        phalanx["**Query Node (Phalanx)**<br/>Rust / LanceDB<br/>:10024 REST + Flight SQL<br/>:9090 Metrics"]
        plan_exec["**Plan Executor**<br/>Rust<br/>:12345 gRPC<br/>SSD query cache"]
        agent["**Lance Agent**<br/>Python<br/>Kafka consumer<br/>K8s job scheduler"]
        indexer["**Indexer**<br/>Rust / Lance<br/>K8s Jobs (CPU or GPU)<br/>vector/scalar/FTS/compact/GC"]
        kafka["**Kafka**<br/>Event bus<br/>topic: phalanx-events"]

        router --> phalanx
        phalanx --> plan_exec
        phalanx -->|"publishes events"| kafka
        kafka -->|"consumes events"| agent
        agent -->|"creates K8s jobs"| indexer
    end

    subgraph geneva_services["Geneva Services (helm/geneva)"]
        direction TB
        ui["**Geneva Console UI**<br/>NextJS / React<br/>:3000"]
        api["**Geneva Console API**<br/>Python / FastAPI<br/>:8000"]
        driver["**Geneva Driver**<br/>Python / Geneva SDK<br/>K8s Job (ephemeral)<br/>Spawned by Lance Agent"]
        ui --> api
    end

    agent -->|"auto-backfill:<br/>creates K8s job"| driver
    api -->|"REST :10024"| phalanx

    subgraph storage["Object Storage"]
        tables["Lance Tables"]
    end

    phalanx --> tables
    indexer --> tables
    driver -->|"table.backfill()<br/>via KubeRay"| tables
```

### Service Reference

| Service | Technology | Port(s) | Helm Chart | Purpose |
|---------|-----------|---------|------------|---------|
| **Router** | Rust (Pingora) | 6188, 6189, 6190 | `helm/lancedb` | HTTP load balancer, consistent-hash routing to query nodes |
| **Query Node (Phalanx)** | Rust (LanceDB + DataFusion) | 10024, 9090 | `helm/lancedb` | REST API + Flight SQL for table operations, query execution |
| **Plan Executor** | Rust | 12345, 9090 | `helm/lancedb` | Distributed query sub-plan execution with SSD cache |
| **Lance Agent** | Python | 8000 (health) | `helm/lancedb` | Consumes Kafka events, schedules indexing/compaction/backfill jobs |
| **Indexer** | Rust (Lance) | K8s Job (ephemeral) | Spawned by Agent | Vector/scalar/FTS indexing, compaction, garbage collection |
| **Kafka** | Kafka | 9092 | `helm/lancedb` | Event bus for table modification events (`phalanx-events` topic) |
| **Geneva Driver** | Python (Geneva SDK) | K8s Job (ephemeral) | Spawned by Agent | Executes auto-backfills: provisions KubeRay cluster, runs `table.backfill()` |
| **Geneva Console API** | Python (FastAPI) | 8000 | `helm/geneva` | Read-only monitoring API — jobs, tables, clusters, manifests |
| **Geneva Console UI** | NextJS / React | 3000 | `helm/geneva` | Monitoring dashboard for Geneva operations |

### Lance Agent: Event-Driven Triggers

The Lance Agent monitors Kafka for table write events and schedules
indexing, compaction, and GC jobs when configurable thresholds are exceeded
(see Helm chart values for full configuration). For Geneva, the key trigger is:

| Trigger | Default Threshold | Action |
|---------|------------------|--------|
| **UDF column NULLs** | Configurable | **Spawn Geneva Driver for auto-backfill** |

### Geneva Driver: Auto-Backfill Execution

The Geneva Driver is a K8s Job spawned by the Lance Agent when auto-backfill
is triggered. It runs in the `geneva` namespace with `IN_CLUSTER` K8s auth.

**Execution flow:**
1. Lance Agent creates K8s Job with `geneva-driver` image
2. Driver calls `geneva.connect(db_uri)` → `conn.open_table(table_name)`
3. Driver selects a Ray cluster based on the configured mode (see below)
4. Calls `table.backfill(column_name)` within a `conn.context(cluster=..., manifest=...)`
5. Workers execute the UDF
6. Job status tracked in `geneva_jobs` system table

**Ray cluster modes** (configurable at deployment time):

| Mode | Description |
|------|-------------|
| **Provided cluster definition** | Uses a `geneva_cluster_definitions` entry specified by name |
| **Default cluster definition** | Auto-creates a default KubeRay cluster when no cluster is specified |
| **Static RayCluster** | Uses a pre-deployed RayCluster from the `helm/lancedb` chart (same as distributed indexing) |

**Configurable via environment variables:**
- `GENEVA_DRIVER_HEAD_CPUS` / `GENEVA_DRIVER_HEAD_MEMORY`
- `GENEVA_DRIVER_WORKER_CPUS` / `GENEVA_DRIVER_WORKER_MEMORY`
- `GENEVA_DRIVER_NUM_WORKERS`

## 2. Deployment Models

Three deployment models are supported, differing in who manages infrastructure.

```mermaid
flowchart LR
    subgraph managed["Managed (Dedicated)"]
        direction TB
        m_tf["Terraform<br/>(LanceDB-managed)"]
        m_k8s["K8s Cluster<br/>(LanceDB-provisioned)"]
        m_dp["Data Plane<br/>(Router, Phalanx,<br/>Agent, Kafka)"]
        m_console["Geneva Console<br/>(API + UI)"]
        m_kuberay["KubeRay Operator"]
        m_ray["Ray Cluster"]
        m_iam["IAM / SA<br/>(auto-configured)"]
        m_storage["Object Storage<br/>(LanceDB-managed buckets<br/>or customer BYOB)"]

        m_tf --> m_k8s
        m_tf --> m_iam
        m_k8s --> m_dp
        m_k8s --> m_console
        m_k8s --> m_kuberay --> m_ray
        m_dp --> m_storage
        m_ray --> m_storage
    end

    subgraph byoc["Self-Managed (BYOC / Helm)"]
        direction TB
        b_helm["Helm Charts<br/>(helm/lancedb +<br/>helm/geneva)"]
        b_k8s["K8s Cluster<br/>(customer-owned)"]
        b_dp["Data Plane<br/>(Router, Phalanx,<br/>Agent, Kafka)"]
        b_console["Geneva Console<br/>(API + UI)"]
        b_kuberay["KubeRay Operator"]
        b_ray["Ray Cluster"]
        b_iam["IAM / SA<br/>(customer configures)"]
        b_storage["Object Storage<br/>(customer buckets)"]

        b_helm --> b_dp
        b_helm --> b_console
        b_helm --> b_kuberay
        b_k8s --> b_kuberay --> b_ray
        b_iam --> b_ray
        b_dp --> b_storage
        b_ray --> b_storage
    end

    subgraph direct["Direct Connect"]
        direction TB
        d_ray["Existing Ray Cluster<br/>(customer-managed)"]
        d_iam["IAM / SA<br/>(customer configures)"]
        d_storage["Object Storage<br/>(customer buckets)"]
        d_note["No KubeRay, Helm,<br/>data plane, or<br/>console needed"]

        d_iam --> d_ray
        d_ray --> d_storage
    end

    client_m["Geneva Client"] --> managed
    client_b["Geneva Client"] --> byoc
    client_d["Geneva Client"] --> direct
```

### Deployment Model Comparison

| Aspect | Managed (Dedicated) | Self-Managed (BYOC) | Direct Connect |
|--------|-------------------|-------------------|----------------|
| **K8s cluster** | Provisioned by LanceDB (Terraform) | Customer-owned | Customer-owned (or no K8s) |
| **Data plane** (Router, Phalanx, Agent, Kafka) | Auto-deployed | Helm chart (`helm/lancedb`) | Not included |
| **Geneva Console** (API + UI) | Auto-deployed | Helm chart (`helm/geneva`) | Not included |
| **KubeRay operator** | Auto-deployed | Helm chart install | Not needed |
| **Ray cluster lifecycle** | Auto-provisioned per job | Auto-provisioned per job | Pre-existing, externally managed |
| **Indexing / compaction** | Lance Agent + Indexer (auto) | Lance Agent + Indexer (auto) | Manual or external |
| **IAM / service accounts** | Terraform-managed | Customer configures (can reuse LanceDB SA) | Customer configures |
| **Object storage** | LanceDB-managed buckets or customer BYOB | Customer buckets | Customer buckets |
| **Cloud support** | AWS, GCP, Azure | Any K8s (incl. neoclouds — requires SA/SE engagement) | Any Ray cluster |
| **Operational burden** | Low (LanceDB manages infra) | Medium (customer manages K8s + IAM) | High (customer manages everything) |
| **Network control** | LanceDB VPC/peering | Full customer control | Full customer control |
| **Best for** | Fastest time to value | Enterprise with existing K8s | Orgs with existing Ray infrastructure |

## 3. Cloud Provider Mapping

```mermaid
flowchart TB
    subgraph aws["AWS"]
        direction TB
        eks["EKS<br/>(Elastic Kubernetes Service)"]
        s3["S3<br/>(s3://)"]
        iam["IAM Roles + IRSA<br/>(Roles for Service Accounts)"]
        sts["STS Tokens<br/>(30min expiry)"]
        ec2["EC2 Instances<br/>(m5, g6, g6e, r6i)"]

        eks --> ec2
        iam --> sts
    end

    subgraph gcp["GCP"]
        direction TB
        gke["GKE<br/>(Google Kubernetes Engine)"]
        gcs["GCS<br/>(gs://)"]
        gcp_sa["Service Accounts<br/>+ Workload Identity"]
        adc["Application Default<br/>Credentials"]
        gce["GCE Instances<br/>(n2, a2, g2)"]

        gke --> gce
        gcp_sa --> adc
    end

    subgraph azure["Azure"]
        direction TB
        aks["AKS<br/>(Azure Kubernetes Service)"]
        blob["Blob Storage<br/>(az://)"]
        az_id["Managed / Workload<br/>Identity"]
        az_cred["DefaultAzure<br/>Credential"]
        vm["VM Scale Sets<br/>(Standard_NC, Standard_D)"]

        aks --> vm
        az_id --> az_cred
    end
```

### Cloud Provider Feature Matrix

| Component | AWS | GCP | Azure |
|-----------|-----|-----|-------|
| **Kubernetes** | EKS | GKE | AKS |
| **Object storage** | S3 (`s3://`) | GCS (`gs://`) | Blob Storage (`az://`) |
| **Client → K8s auth** | EKS token (STS, 30min expiry) | gcloud kubeconfig | az aks get-credentials |
| **Pod → storage auth** | IRSA (IAM Roles for SA) | Workload Identity | Workload Identity |
| **Terraform support** | Full | Full | In development |
| **Managed deployment** | Yes | Yes | Yes |
| **GPU instances** | g6.xlarge, g6e.xlarge | a2-highgpu, g2-standard | Standard_NC series |
| **CPU instances** | m5.xlarge, r6i.8xlarge | n2-standard, n2-highmem | Standard_D series |
| **Test region** | us-east-1 | us-central1 | eastus |
| **Geneva SDK extra** | `pip install geneva[aws]` | `pip install geneva[gcp]` | `pip install geneva[azure]` |

## 4. Security and Trust Boundaries

### 4a. Managed (Dedicated) Trust Boundaries

In managed deployments, LanceDB provisions and operates the K8s cluster and IAM.
The customer controls the client environment and optionally brings their own storage bucket.
A VPC peering or private link connects the customer network to the LanceDB-managed cluster.

```mermaid
flowchart TB
    subgraph lancedb_managed["LanceDB-Managed Infrastructure"]
        subgraph lancedb_ops["LanceDB Operations"]
            terraform["Terraform<br/>(provisions all infra)"]
            lancedb_iam["IAM / SA<br/>(auto-configured)"]
            terraform --> lancedb_iam
        end

        subgraph k8s["K8s Cluster (LanceDB-provisioned)"]
            subgraph lancedb_ns["lancedb namespace"]
                router["Router (Pingora)"]
                phalanx["Query Node (Phalanx)"]
                agent["Lance Agent"]
                indexer["Indexer Jobs"]
                kafka["Kafka"]
                router --> phalanx
                phalanx --> kafka --> agent --> indexer
            end

            subgraph geneva_ns["geneva namespace (RBAC)"]
                console["Geneva Console<br/>(API + UI)"]
                kuberay_op["KubeRay Operator"]
                subgraph ray["Ray Cluster"]
                    head["Ray Head"]
                    workers["Ray Workers<br/>(run customer UDFs)"]
                    head --- workers
                end
                kuberay_op --> ray
            end

            console -->|"REST"| phalanx
        end
        terraform --> k8s
    end

    subgraph customer_env["Customer Environment"]
        client["Geneva Client<br/>(Python SDK)"]
        client_creds["Customer Cloud Creds<br/>(K8s access + storage)"]
        client --- client_creds
    end

    subgraph storage["Cloud Storage"]
        lancedb_bucket["LanceDB-Managed Bucket<br/>(default)"]
        customer_bucket["Customer BYOB<br/>(optional)"]
    end

    client -->|"HTTP / Flight SQL"| router
    client -->|"K8s API (TLS)<br/>via peering / private link"| geneva_ns
    client -->|"direct storage<br/>(cloud SDK)"| lancedb_bucket
    client -->|"direct storage<br/>(cloud SDK)"| customer_bucket
    lancedb_iam -->|"pod identity"| workers
    phalanx -->|"read/write"| lancedb_bucket
    phalanx -->|"read/write"| customer_bucket
    indexer -->|"build indexes"| lancedb_bucket
    indexer -->|"build indexes"| customer_bucket
    workers -->|"read/write"| lancedb_bucket
    workers -->|"read/write"| customer_bucket

    style lancedb_managed stroke:#ffa726,stroke-width:2px,fill:none
    style lancedb_ops stroke:#ffa726,stroke-width:1px,stroke-dasharray:5 5,fill:none
    style k8s stroke:#ab47bc,stroke-width:2px,fill:none
    style customer_env stroke:#42a5f5,stroke-width:2px,fill:none
    style storage stroke:#66bb6a,stroke-width:2px,fill:none
```

**Key trust properties:**
- LanceDB operates the K8s cluster and IAM — customer does not need K8s expertise
- Customer UDFs run inside LanceDB-managed infrastructure (code leaves customer env)
- Storage can be LanceDB-managed or customer-owned (BYOB)
- Network boundary crossed via VPC peering or private endpoint

### 4b. Self-Managed (BYOC / Helm) Trust Boundaries

In BYOC, everything runs inside the customer's cloud account. LanceDB provides
the Helm chart and Geneva client — no data or credentials flow to LanceDB.

```mermaid
flowchart TB
    subgraph customer_cloud["Customer Cloud Account"]
        subgraph customer_env["Customer Environment"]
            client["Geneva Client<br/>(Python SDK)"]
            client_creds["Cloud Credentials<br/>(IAM role / SA key)"]
            kubeconfig["Kubeconfig"]
            client --- client_creds
            client --- kubeconfig
        end

        subgraph k8s["Customer K8s Cluster (EKS / GKE / AKS / other)"]
            subgraph lancedb_ns["lancedb namespace"]
                router["Router (Pingora)"]
                phalanx["Query Node (Phalanx)"]
                agent["Lance Agent"]
                indexer["Indexer Jobs"]
                kafka["Kafka"]
                router --> phalanx
                phalanx --> kafka --> agent --> indexer
            end

            subgraph geneva_ns["geneva namespace (RBAC)"]
                console["Geneva Console<br/>(API + UI)"]
                kuberay_op["KubeRay Operator<br/>(Helm-installed)"]
                subgraph ray["Ray Cluster"]
                    head["Ray Head<br/>+ K8s Service Account"]
                    workers["Ray Workers<br/>(run user UDFs)"]
                    head --- workers
                end
                kuberay_op --> ray
            end

            console -->|"REST"| phalanx
        end

        subgraph cloud_iam["Customer IAM"]
            role["IAM Role / SA"]
            policy["Storage Policy<br/>(scoped to buckets)"]
            role --> policy
        end

        subgraph storage["Customer Cloud Storage"]
            encryption["Encryption at Rest<br/>(KMS / SSE)"]
            buckets["Data Buckets<br/>(Lance tables, manifests,<br/>checkpoints)"]
            encryption --- buckets
        end

        client -->|"HTTP / Flight SQL"| router
        client -->|"K8s API (TLS)"| geneva_ns
        phalanx -->|"assumes"| role
        head -->|"assumes"| role
        workers -->|"assumes"| role
        indexer -->|"assumes"| role
        policy -->|"scoped access"| buckets
        client -->|"cloud SDK (TLS)"| buckets
    end

    subgraph lancedb_provided["LanceDB-Provided (artifacts only)"]
        helm_lancedb["Helm Chart<br/>(helm/lancedb)"]
        helm_geneva["Helm Chart<br/>(helm/geneva)"]
        sdk["Geneva SDK<br/>(PyPI package)"]
    end

    helm_lancedb -.->|"install-time only"| lancedb_ns
    helm_geneva -.->|"install-time only"| geneva_ns
    sdk -.->|"install-time only"| client

    style customer_cloud stroke:#42a5f5,stroke-width:2px,fill:none
    style customer_env stroke:#42a5f5,stroke-width:1px,stroke-dasharray:5 5,fill:none
    style k8s stroke:#ab47bc,stroke-width:2px,fill:none
    style cloud_iam stroke:#ffa726,stroke-width:2px,fill:none
    style storage stroke:#66bb6a,stroke-width:2px,fill:none
    style lancedb_provided stroke:#888,stroke-width:1px,stroke-dasharray:5 5,fill:none
```

**Key trust properties:**
- All infrastructure runs in customer's cloud account — nothing leaves their boundary
- LanceDB provides only build artifacts (Helm chart + PyPI SDK), no runtime access
- Customer controls IAM, network, encryption, and storage
- Ideal for strict data residency and compliance requirements
- Neoclouds supported with SA/SE engagement

### 4c. Direct Connect Trust Boundaries

In direct connect, the customer manages an existing Ray cluster. Geneva connects
to it as a client — no KubeRay, no Helm chart, no K8s dependency required.

```mermaid
flowchart TB
    subgraph customer_infra["Customer Infrastructure"]
        subgraph customer_env["Customer Environment"]
            client["Geneva Client<br/>(Python SDK)"]
            client_creds["Cloud Credentials"]
            ray_addr["Ray Cluster Address"]
            client --- client_creds
            client --- ray_addr
        end

        subgraph compute["Customer Compute (K8s, VMs, bare metal)"]
            subgraph ray["Existing Ray Cluster (customer-managed)"]
                head["Ray Head"]
                workers["Ray Workers<br/>(run user UDFs)"]
                head --- workers
            end
        end

        subgraph cloud_iam["Customer IAM"]
            role["IAM Role / SA"]
            policy["Storage Policy"]
            role --> policy
        end

        subgraph storage["Customer Cloud Storage"]
            buckets["Data Buckets<br/>(Lance tables, manifests,<br/>checkpoints)"]
        end

        client -->|"Ray client protocol"| head
        workers -->|"assumes"| role
        policy -->|"scoped access"| buckets
        client -->|"cloud SDK (TLS)"| buckets
    end

    subgraph lancedb_provided["LanceDB-Provided (artifact only)"]
        sdk["Geneva SDK<br/>(PyPI package)"]
    end

    sdk -.->|"install-time only"| client

    style customer_infra stroke:#42a5f5,stroke-width:2px,fill:none
    style customer_env stroke:#42a5f5,stroke-width:1px,stroke-dasharray:5 5,fill:none
    style compute stroke:#ab47bc,stroke-width:2px,fill:none
    style cloud_iam stroke:#ffa726,stroke-width:2px,fill:none
    style storage stroke:#66bb6a,stroke-width:2px,fill:none
    style lancedb_provided stroke:#888,stroke-width:1px,stroke-dasharray:5 5,fill:none
```

**Key trust properties:**
- Smallest LanceDB footprint — only the PyPI SDK
- Customer fully owns compute, network, storage, and identity
- Ray cluster can run on K8s, VMs, or bare metal
- No KubeRay operator or Helm chart needed
- Customer responsible for Ray cluster lifecycle and scaling

### Trust Boundary Comparison

| Property | Managed | Self-Managed (BYOC) | Direct Connect |
|----------|---------|-------------------|----------------|
| **Data residency** | LanceDB-managed or customer BYOB | Fully in customer account | Fully in customer account |
| **UDF code execution** | In LanceDB-managed cluster | In customer cluster | In customer cluster |
| **Data plane services** | In LanceDB-managed cluster | In customer cluster | Not included |
| **Geneva Console** | In LanceDB-managed cluster | In customer cluster | Not included |
| **Indexing / compaction** | Lance Agent (auto, in LanceDB cluster) | Lance Agent (auto, in customer cluster) | Manual or external |
| **K8s cluster operator** | LanceDB | Customer | Customer (or none) |
| **IAM management** | LanceDB (Terraform) | Customer | Customer |
| **Network boundary** | VPC peering / private link | No cross-boundary traffic | No cross-boundary traffic |
| **LanceDB runtime access** | Yes (operates infra) | No (artifacts only) | No (artifact only) |
| **Encryption control** | LanceDB-managed keys or customer CMK | Customer-managed | Customer-managed |
| **Compliance posture** | Shared responsibility | Customer-controlled | Customer-controlled |
| **Operational burden** | Low | Medium | High |

## 5. Data Flow: Backfill Pipeline

```mermaid
sequenceDiagram
    participant User as User Code
    participant Client as Geneva Client
    participant Packager as UDF Packager
    participant Storage as Object Storage
    participant KubeRay as KubeRay Operator
    participant Head as Ray Head
    participant Workers as Ray Workers
    participant Lance as Lance Tables
    participant SysTables as System Tables

    User->>Client: Define UDF + call backfill()
    Client->>Packager: Package UDF + dependencies
    Packager->>Storage: Upload manifest ZIP

    Client->>SysTables: Create JobRecord (PENDING)
    Client->>KubeRay: Provision Ray cluster (if needed)
    KubeRay-->>Head: Cluster ready

    Client->>Head: Submit backfill job
    Head->>SysTables: Update JobRecord (RUNNING)
    Head->>Storage: Download manifest

    loop For each fragment batch
        Head->>Workers: Schedule applier actors
        Workers->>Lance: Read fragment data
        Workers->>Workers: Apply UDF to batch
        Workers->>Lance: Write result fragments
        Workers->>Storage: Emit checkpoint
        Workers->>Head: Report progress
    end

    Head->>SysTables: Update JobRecord (DONE)
    Head->>SysTables: Store metrics + errors

    alt Job failed
        Head->>SysTables: Update JobRecord (FAILED)
        Head->>SysTables: Write to error store
        Note over Client: Can resume from last checkpoint
    end
```

### System Tables

Geneva stores operational metadata in Lance tables alongside user data. These are
created automatically when features are first used.

#### `geneva_jobs` — Job Lifecycle Tracking

Defined in `src/geneva/jobs/jobs.py` (`JobRecord`).

| Field | Type | Description |
|-------|------|-------------|
| `job_id` | `str` (UUID) | Unique job identifier |
| `table_name` | `str` | Target table name |
| `column_name` | `str` | Target computed column |
| `job_type` | `str` | `BACKFILL` (default) |
| `status` | `enum` | `PENDING` → `RUNNING` → `DONE` / `FAILED` |
| `launched_at` | `timestamp` | Job start time (UTC) |
| `completed_at` | `timestamp` | Job end time (UTC, nullable) |
| `updated_at` | `timestamp` | Last status update (UTC, nullable) |
| `launched_by` | `str` | Username that launched the job |
| `config` | `str` (JSON) | Serialized backfill parameters |
| `manifest_id` | `str` | Associated manifest name (nullable) |
| `manifest_checksum` | `str` | Manifest content hash (nullable) |
| `metrics` | `list[str]` | Progress metrics (name, n, total, done) |
| `events` | `list[str]` | Lifecycle event log |
| `object_ref` | `str` | Ray object reference for running jobs (nullable) |

#### `geneva_errors` — UDF Error Store

Defined in `src/geneva/debug/error_store.py` (`ErrorRecord`).

| Field | Type | Description |
|-------|------|-------------|
| `error_id` | `str` (UUID) | Unique error identifier |
| `error_type` | `str` | Exception class name |
| `error_message` | `str` | Exception message |
| `error_trace` | `str` | Full traceback |
| `job_id` | `str` | Parent job ID |
| `table_uri` | `str` | Full URI to the table |
| `table_name` | `str` | Table name |
| `table_version` | `int` | Read version (nullable) |
| `column_name` | `str` | Target column |
| `udf_name` | `str` | UDF function name |
| `udf_version` | `str` | UDF version |
| `actor_id` | `str` | Ray actor ID (nullable) |
| `fragment_id` | `int` | Lance fragment being processed (nullable) |
| `batch_index` | `int` | Batch sequence within fragment |
| `row_address` | `int` | Failing row address for scalar UDFs (nullable) |
| `attempt` | `int` | Retry attempt number |
| `max_attempts` | `int` | Max configured attempts |
| `timestamp` | `timestamp` | Error time (UTC) |

#### `geneva_cluster_definitions` — Cluster Configurations

Defined in `src/geneva/cluster/mgr.py` (`GenevaCluster`).

| Field | Type | Description |
|-------|------|-------------|
| `name` | `str` | Cluster name |
| `cluster_type` | `enum` | `KUBERAY`, `EXTERNAL_RAY`, `LOCAL` |
| `kuberay` | `str` (JSON) | Full KubeRay config: namespace, head/worker group specs (CPU, memory, image, node selectors, tolerations, replicas, autoscaling) |
| `ray_address` | `str` | Address for external Ray clusters (nullable) |
| `ray_init_kwargs` | `str` (JSON) | Extra `ray.init()` kwargs (nullable) |
| `created_at` | `timestamp` | Creation time (UTC) |
| `created_by` | `str` | Username that created the cluster |

#### `geneva_manifests` — UDF Dependency Manifests

Defined in `src/geneva/manifest/mgr.py` (`GenevaManifest`).

| Field | Type | Description |
|-------|------|-------------|
| `name` | `str` | Manifest name |
| `version` | `str` | Version label (nullable) |
| `pip` | `list[str]` | pip dependencies |
| `py_modules` | `list[str]` | Python module paths to package |
| `head_image` | `str` | Custom Ray head container image (nullable) |
| `worker_image` | `str` | Custom Ray worker container image (nullable) |
| `conda` | `str` (JSON) | Conda environment spec (nullable) |
| `zips` | `list[list[str]]` | Uploaded zip file references |
| `checksum` | `str` | Content hash for deduplication |
| `created_at` | `timestamp` | Creation time (UTC) |
| `created_by` | `str` | Username that created the manifest |

## 6. Scaling Architecture

```mermaid
flowchart TB
    subgraph scaling["Scaling Dimensions"]
        direction TB

        subgraph data_scale["Data Scale"]
            fragments["Lance Fragments<br/>(unit of parallelism)"]
            partitions["Table Partitions"]
            partitions --> fragments
        end

        subgraph compute_scale["Compute Scale"]
            subgraph ray_head["Ray Head Node"]
                driver["Driver Task<br/>(0.1 CPU)"]
                tracker["JobTracker<br/>(0.1 CPU, 128MB)"]
                admission["Admission<br/>Control"]
            end
            subgraph cpu_pool["CPU Worker Pool"]
                cpu1["Applier + Writer<br/>(N CPUs each)"]
                cpu2["Applier + Writer"]
                cpu3["Applier + Writer"]
                cpuN["... × concurrency"]
            end
            subgraph gpu_pool["GPU Worker Pool"]
                gpu1["Applier + Writer<br/>(1 GPU + N CPUs each)"]
                gpu2["Applier + Writer"]
                gpuN["... × concurrency"]
            end
        end

        subgraph controls["Scaling Controls"]
            concurrency["concurrency<br/>(# parallel appliers)"]
            checkpoint_size["checkpoint_size<br/>(rows per checkpoint)"]
            intra["intra_applier_concurrency<br/>(parallelism within applier)"]
            workers["worker replicas<br/>(K8s pod count)"]
            autoscale["KubeRay autoscaling<br/>(min/max replicas)"]
        end
    end

    fragments -->|"distributed across"| cpu_pool
    fragments -->|"distributed across"| gpu_pool
    concurrency --> cpu_pool
    concurrency --> gpu_pool
    workers --> cpu_pool
    workers --> gpu_pool
    autoscale --> workers
```

### Sizing Reference

| Workload | Cluster | Concurrency | Checkpoint Size | Notes |
|----------|---------|-------------|------------|-------|
| **Dev / CI** | 1 head (1 CPU) + 4 workers (2 CPU each) | 2 | 32 | ~9 CPUs total |
| **Staging** | 1 head (2 CPU) + 4 workers (4 CPU each) | 4 | 64 | ~18 CPUs total |
| **Production (CPU)** | 1 head (4 CPU) + 8 workers (8 CPU each) | 8 | 128 | ~68 CPUs total |
| **Production (GPU)** | 1 head (m5.24xl) + 40 GPU workers (g6e.xl) + 64 CPU workers (r6i.8xl) | 8-16 | 64 | Large-scale embeddings |

### Resource Formula

```
Total CPU needed ≈ concurrency × UDF_cpus + concurrency × 0.1 (writers) + 0.2 (driver + tracker)
Total Memory ≈ concurrency × 1GB (writers) + concurrency × UDF_memory + 128MB (tracker)
Total GPU = concurrency × UDF_gpus (if GPU workload)
```

## 7. Operational View

```mermaid
flowchart TB
    subgraph lifecycle["Cluster Lifecycle"]
        provision["Provision<br/>(KubeRay creates<br/>RayCluster CRD)"]
        running["Running<br/>(jobs execute)"]
        teardown["Teardown<br/>(cluster deleted<br/>after job completes)"]
        orphan["Orphan Cleanup<br/>(scheduled cron:<br/>delete clusters > N hours)"]

        provision --> running --> teardown
        running -->|"failure/timeout"| orphan
    end

    subgraph monitoring["Monitoring & Observability"]
        direction TB
        job_metrics["Job Metrics<br/>(UDF time, write time,<br/>checkpoint time, conflicts)"]
        error_store["Error Store<br/>(exceptions, stack traces,<br/>per-fragment errors)"]
        progress["Progress Tracking<br/>(tqdm progress bar)"]
        slack["Slack Notifications<br/>(optional, via TqdmSlackConfig)"]

        job_metrics --> progress
        progress --> slack
    end

    subgraph versioning["Version Management"]
        client_ver["Geneva Client<br/>version"]
        ray_img["Ray Docker<br/>Image version"]
        manifest_ver["UDF Manifest<br/>version"]
        lance_ver["LanceDB/Lance<br/>version"]
    end

    subgraph recovery["Fault Tolerance"]
        checkpoint_store["Checkpoint Store<br/>(periodic snapshots)"]
        retry_policy["Retry Policy<br/>(transient vs permanent<br/>errors)"]
        conflict_res["Conflict Resolution<br/>(overlapping backfills)"]
        skip_errors["Skip-on-Error<br/>(configurable)"]
    end

    running --> monitoring
    running --> recovery
```

### Operational Checklist

| Concern | Mechanism | Details |
|---------|-----------|---------|
| **Cluster provisioning** | KubeRay Operator | Creates RayCluster CRD, manages pod lifecycle |
| **Cluster cleanup** | Cron job | `cleanup_old_rayclusters.sh` removes clusters older than N hours |
| **Job monitoring** | System tables | `geneva_jobs` stores metrics, events, status per job |
| **Error debugging** | Error store | `geneva_errors` captures exceptions with full context |
| **Cluster tracking** | System tables | `geneva_cluster_definitions` stores Ray cluster configs |
| **Dependency tracking** | System tables | `geneva_manifests` stores UDF packaging and image info |
| **Progress visibility** | tqdm + Slack | Real-time progress bar, optional Slack webhook |
| **Failure recovery** | Checkpoints | Resume from last checkpoint on job restart |
| **Retry policy** | Configurable | Per-UDF: max retries, transient vs permanent classification |
| **Write conflicts** | Auto-retry | Conflict resolution for concurrent fragment writes |
| **Version compat** | Manifest pinning | UDF manifest captures exact dependency versions |
