# Architecture

This document provides a high-level view of Geneva's components and how they fit
into a typical execution flow.

## Components
- Client library for user APIs and orchestration
- UDF packaging and manifest generation
- Ray-based execution for distributed backfills
- LanceDB tables for storage and metadata
- Optional Kubernetes and cloud integrations for cluster provisioning

## Data Flow Overview
```mermaid
flowchart LR
  User[Client Code] --> API[Geneva Client]
  API --> Packager[UDF Packager]
  API --> Query[Query Builder]
  API --> Runner[Job Runner]
  Runner --> Ray[Ray Cluster]
  Ray --> Storage[Object Storage]
  Ray --> Tables[Lance Tables]
  API --> System[System Tables]
```

## Key Modules
- `src/geneva/transformer.py` defines UDFs and execution behavior.
- `src/geneva/query.py` extends query building and materialized views.
- `src/geneva/packager/` packages UDF dependencies into manifests.
- `src/geneva/runners/` executes jobs on Ray.
- `src/geneva/manifest/` manages manifest definitions and uploads.
- `src/geneva/config/` loads configuration from files and env vars.

## Backfill Lifecycle
1. A user defines a UDF and configures computed columns.
2. The client packages UDF dependencies into a manifest.
3. A job is created and scheduled on a Ray cluster.
4. Workers scan data from Lance tables, apply the UDF, and write results.
5. Checkpoints are emitted for recovery and progress tracking.

Refer to `internal_docs/backfills.md` for more information.
