# Materialized Views Architecture

**Audience:** Sales engineers, support engineers, architects, and new contributors

**Purpose:** Understand how Geneva materialized views work at a conceptual level to support customers, debug issues, and contribute to the codebase.

---

## What are Materialized Views?

Materialized views are **precomputed query results** that are stored as physical tables. Instead of running expensive queries repeatedly, you compute the results once and store them for fast access.

**Business Value:**
- **Performance:** Query results cached as tables (milliseconds vs. minutes)
- **Cost Efficiency:** Compute once, query many times
- **Efficient Data Loading:** Precomputed transformations for exploratory analysis and ML training

**Geneva's Approach:**
Geneva materialized views are built on Lance tables with support for:
- Filtering (SQL WHERE clauses)
- Shuffling (randomized ordering for ML training)
- User-defined functions (computed columns like embeddings, predictions)
- Incremental refresh (process only new data, not everything)

---

## High-Level Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                     MATERIALIZED VIEW SYSTEM                    │
└─────────────────────────────────────────────────────────────────┘

┌──────────────┐                                  ┌──────────────┐
│ SOURCE TABLE │                                  │ MATERIALIZED │
│  (Lance DB)  │                                  │     VIEW     │
│              │                                  │  (Lance DB)  │
│ ┌──────────┐ │                                  │ ┌──────────┐ │
│ │Fragment 0│ │                                  │ │Fragment 0│ │
│ │Fragment 1│ │                                  │ │Fragment 1│ │
│ │Fragment 2│ │                                  │ │Fragment 2│ │
│ └──────────┘ │                                  │ └──────────┘ │
└──────┬───────┘                                  └──────▲───────┘
       │                                                 │
       │ 1. Read source data                 4. Write results
       ▼                                                  │
┌─────────────────────────────────────────────────────────────────┐
│                      RAY PROCESSING CLUSTER                     │
│                                                                 │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐           │
│  │   Worker 1   │  │   Worker 2   │  │   Worker 3   │           │
│  │              │  │              │  │              │           │
│  │  Apply UDFs  │  │  Apply UDFs  │  │  Apply UDFs  │           │
│  │   Process    │  │   Process    │  │   Process    │           │
│  │   Batches    │  │   Batches    │  │   Batches    │           │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘           │
│         │                 │                 │                   │
│         └─────────────────┼─────────────────┘                   │
│                           │ 2. Store checkpoints                │
│                           ▼                                     │
│                   ┌─────────────────┐                           │
│                   │ CHECKPOINT STORE│                           │
│                   │                 │                           │
│                   │  Deduplication  │                           │
│                   │  Progress Track │                           │
│                   │  Fault Tolerant │                           │
│                   └────────┬────────┘                           │
│                            │ 3. Fragment Writers read ckpts     │
└────────────────────────────┼────────────────────────────────────┘
                             ▼
                     Fragment Writers
                     (see fragment_writers.md)
```

### Key Components

1. **Source Table**: Original Lance table with raw data
2. **Ray Workers**: Distributed processors that apply transformations (filters, UDFs)
3. **Checkpoint Store**: Intermediate storage for fault tolerance and deduplication
4. **Fragment Writers**: Specialized writers that commit results to the materialized view
5. **Materialized View**: Destination Lance table with computed results

---

## How It Works: Creation & Refresh

### 1. Creating a Materialized View

```
User Code:
  query.where("age > 21").add_column(sentiment_udf).create_materialized_view()

Flow:
┌─────────┐     ┌─────────────┐     ┌─────────────┐     ┌──────────┐
│ Extract │────>│   Create    │────>│ Initialize  │────>│ Ready to │
│  Query  │     │ Placeholder │     │  Metadata   │     │ Refresh  │
│ Config  │     │    Table    │     │             │     │          │
└─────────┘     └─────────────┘     └─────────────┘     └──────────┘

Result: Empty table with:
- Schema matching query output
- Metadata storing query definition
- Placeholder rows with NULL values
```

**What happens:**
1. Query is serialized and stored in table metadata
2. Empty destination table created with correct schema
3. Placeholder rows added (one per matching source row)
4. Rows contain `__source_row_id` (pointer to source data)
5. Table ready for refresh operation

### 2. Refreshing the View (Full Refresh)

```
Step 1: Plan
┌─────────────────────────────────────────────────────────────┐
│  Read destination table → Find all rows with __is_set=False │
│  Create tasks: one per batch of rows                        │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
Step 2: Execute (Parallel)
┌───────────────────────────────────────────────────────────────────┐
│  Ray Worker 1            Ray Worker 2            Ray Worker 3     │
│  ┌──────────────┐        ┌──────────────┐        ┌──────────────┐ │
│  │Read batch via│        │Read batch via│        │Read batch via│ │
│  │__source_row_id        │__source_row_id        │__source_row_id │
│  ├──────────────┤        ├──────────────┤        ├──────────────┤ │
│  │Apply filters │        │Apply filters │        │Apply filters │ │
│  ├──────────────┤        ├──────────────┤        ├──────────────┤ │
│  │Run UDFs      │        │Run UDFs      │        │Run UDFs      │ │
│  ├──────────────┤        ├──────────────┤        ├──────────────┤ │
│  │Write to      │        │Write to      │        │Write to      │ │
│  │checkpoint    │        │checkpoint    │        │checkpoint    │ │
│  └──────────────┘        └──────────────┘        └──────────────┘ │
└───────────────────────────────────────────────────────────────────┘
                            │
                            ▼
Step 3: Commit
┌─────────────────────────────────────────────────────────────┐
│  Fragment Writers read checkpoints → Write to destination   │
│  Update __is_set=True for processed rows                    │
│  Commit new version of materialized view table              │
└─────────────────────────────────────────────────────────────┘
```

**Key Points:**
- **Destination-driven**: Tasks created based on destination table structure
- **Parallel**: Multiple workers process different batches simultaneously
- **Checkpointed**: Progress saved to handle failures
- **Idempotent**: Safe to re-run, skips already-processed work

### 3. Incremental Refresh (Processing New Data)

When source table gets new data, only process the additions:

```
Before:
  Source: [Frag 0] [Frag 1] [Frag 2]
  View:   [Frag 0] ← contains results from source frags 0,1,2

After source update:
  Source: [Frag 0] [Frag 1] [Frag 2] [Frag 3] [Frag 4] ← NEW

Incremental Refresh:
  ┌─────────────────────────────────────────┐
  │ 1. Identify new source fragments (3,4)  │
  │ 2. Apply original query to new data     │
  │ 3. Append placeholder rows to view      │
  │ 4. Map destination fragments to source  │
  │ 5. Process only new rows (parallel)     │
  │ 6. Mark new data as processed           │
  └─────────────────────────────────────────┘

Result:
  View:   [Frag 0] [Frag 1] ← Frag 0 untouched, Frag 1 is new data
                    (old)     (new computations)
```

**Benefits:**
- Only compute results for new source data
- Existing results remain unchanged (no recomputation)
- Proportional cost: process 1MB of new data, not 1TB total

---

## Key Concepts Explained

### Fragment Mapping

Materialized views maintain a mapping from destination rows back to source rows:

```
SOURCE TABLE                      MATERIALIZED VIEW
┌──────────────┐                 ┌────────────────────────┐
│ Fragment 0   │                 │ Fragment 0             │
│  Row 0       │                 │  Row 0 (__source_row_id│
│  Row 1       │                 │         = Frag2:Row15)─┼─┐
│  Row 2       │                 │  Row 1 (__source_row_id│ │
├──────────────┤                 │         = Frag0:Row7) <┼─┤
│ Fragment 1   │                 │  Row 2 (__source_row_id│ │
│  Row 0       │                 │         = Frag1:Row3) <┼─┤
│  Row 1       │                 └────────────────────────┘ │
├──────────────┤                                            │
│ Fragment 2   │               After filtering & shuffle,   │
│  Row 0       │               destination rows can come    │
│  Row 1       │               from any source fragment     │
│  Row 15 ─────┼───────────────in any order ────────────────┘
└──────────────┘
```

**Why this matters:**
- Filters reduce rows (1000 source rows → 100 view rows)
- Shuffle reorders rows (random order for ML training)
- Source fragment boundaries don't match destination fragments
- `__source_row_id` tracks the relationship

### Checkpointing for Fault Tolerance

```
Without checkpointing:
  Process 1000 batches → Crash at batch 999 → Start over (1000 batches)

With checkpointing:
  Process 1000 batches → Crash at batch 999 → Resume from 999 (1 batch)

Checkpoint Store:
  ┌─────────────────────────────────────┐
  │ Key: fragment_3_batch_0_100         │
  │ Value: /path/to/computed/data.lance │
  ├─────────────────────────────────────┤
  │ Key: fragment_3_batch_100_200       │
  │ Value: /path/to/computed/data.lance │
  └─────────────────────────────────────┘
```

**Benefits:**
- Failed jobs can resume from last checkpoint
- Multiple workers coordinate via checkpoint store
- Deduplication prevents double-processing

### Source Version Management

Track which version of the source table was used:

```
Timeline:
  T1: Source v1 (1000 rows) → Create view → View refs source v1
  T2: Source v2 (1100 rows) → Source grows
  T3: Refresh view → Automatically uses latest (v2)
  T4: Source v3 (1200 rows) → Source grows
  T5: Refresh to specific version: view.refresh(src_version=2)
```

**Use cases:**
- **Reproducibility**: Pin to specific source version for testing
- **Rollback**: Revert to earlier source version
- **Cross-database**: Source can be in different database than view

---

## Common Scenarios

### Scenario 1: Filtered Dataset for Team Sharing

```
Problem: Team B only needs customers in EMEA region

Solution:
  1. Create view: .where("region == 'EMEA'")
  2. Team B queries view (only EMEA data)
  3. Automatic access control via separate table
  4. Incremental refresh keeps view up-to-date

Result: Simple data sharing with filtering built-in
```

### Scenario 2: ML Training Data with Embeddings

```
Problem: Computing embeddings is expensive (GPU required)

Solution:
  1. Create view with embedding UDF: .add_column(embedding_udf)
  2. First refresh computes all embeddings (one-time cost)
  3. Incremental refresh only computes embeddings for new data
  4. Training reads precomputed view (no GPU needed)

Result: Train on 1M rows without recomputing embeddings each time
```

---

## Quick Reference

**When to use materialized views:**
- Expensive queries run repeatedly
- Need precomputed aggregations or transformations
- Want to cache intermediate results
- ML training requires consistent dataset snapshots

**Key operations:**
- `create_materialized_view()`: Initialize empty view
- `refresh()`: Compute/update results
- `refresh(src_version=N)`: Refresh to specific source version

**Performance characteristics:**
- **Creation**: Fast (just metadata, no computation)
- **First refresh**: Slow (compute all results)
- **Incremental refresh**: Fast (only new data)
- **Query**: Fast (materialized results)

**For more details:**
- Incremental refresh mechanics: See "Incremental Refresh with New Source Data" below
- Fragment writer details: See `fragment_writers.md`
- Persistent formats: See "Persistent Formats Reference" below

---

## Implementation Internals

**Note:** The following sections provide detailed implementation information for debugging and supporting materialized views. These are intended for on-call engineers, support engineers, and contributors working on the codebase.

1. [Persistent Formats Reference](#persistent-formats-reference)
2. [Fragment Mapping: Source to Checkpoint to Destination](#fragment-mapping-source-to-checkpoint-to-destination)
3. [Checkpoint Mechanism and Progress Tracking](#checkpoint-mechanism-and-progress-tracking)
4. [Materialized View Refresh Process](#materialized-view-refresh-process)
5. [Incremental Refresh with New Source Data](#incremental-refresh-with-new-source-data)
6. [Source Version Management](#source-version-management)
7. [Query Operations Impact on Fragments and Layouts](#query-operations-impact-on-fragments-and-layouts)
8. [Checkpoint Store Implementations](#checkpoint-store-implementations)
9. [Error Handling and Recovery](#error-handling-and-recovery)
10. [Performance Tuning](#performance-tuning)
11. [Testing Examples](#testing-examples)
12. [Key Architectural Components](#key-architectural-components)

---

## Persistent Formats Reference

This section documents all persistent data formats and encoding schemes used by materialized views. These formats are critical for understanding how the system tracks data across source, checkpoints, and destination tables.

### Row Address Encoding (`__source_row_id`)


Row addresses are 64-bit integers encoding both fragment ID and row offset within that fragment:

```python
# Encoding (source code reference)
row_id = (fragment_id << 32) | row_offset

# Decoding
fragment_id = row_id >> 32
row_offset = row_id & 0xFFFFFFFF

# Example
row_id = 0x0000000300000005  # Fragment 3, row 5
fragment_id = 0x0000000300000005 >> 32  # = 3
row_offset = 0x0000000300000005 & 0xFFFFFFFF  # = 5
```

**Properties:**
- **Max fragment ID**: 2^32 - 1 (4,294,967,295)
- **Max rows per fragment**: 2^32 - 1 (4,294,967,295)
- **Storage type**: PyArrow `int64`
- **Column name**: `__source_row_id` in destination tables

This encoding allows Geneva to:
1. Quickly extract source fragment IDs from destination tables
2. Determine which destination fragments need reprocessing during incremental refresh
3. Efficiently track the mapping from destination rows back to source rows

### Checkpoint Key Formats


Checkpoint keys identify unique units of work that can be deduplicated and resumed.

#### Fragment Deduplication Key

```python
# Format
key = f"{uri}:{frag_id}:{map_task.checkpoint_key()}"
dedupe_key = hashlib.sha256(key.encode()).hexdigest()

# Example - DESTINATION fragment ID is used!
uri = "/path/to/dest/table.lance"
frag_id = 3  # DESTINATION fragment ID (from task.dest_frag_id())
checkpoint_key = "video_udf:abc123def456"
key = "/path/to/dest/table.lance:3:video_udf:abc123def456"
dedupe_key = "7a8b9c..." # SHA-256 hash
```

**Purpose:** Prevents duplicate processing of the same fragment across workers.

**Critical Note:** The `frag_id` is the **destination fragment ID**, not the source fragment ID. This is crucial for incremental refresh to correctly identify checkpointed work. For materialized views with filters/shuffle, source and destination fragment IDs differ.

**Stored as:** Key in checkpoint store with RecordBatch value containing file path reference.

#### Batch Checkpoint Key

```python
# Format (from apply/__init__.py:148)
checkpoint_key = f"{data_key}:{self.map_task.checkpoint_key()}"

# Example
data_key = "fragment_3_batch_0_50"
map_task_key = "video_udf:abc123def456"
checkpoint_key = "fragment_3_batch_0_50:video_udf:abc123def456"
```

**Purpose:** Tracks completion of individual batches within a fragment for fine-grained resumption.

### Checkpoint File Storage


Checkpoint files are stored as Lance tables containing references to staged data files.

```python
# Storage format
checkpoint_store[dedupe_key] = pa.RecordBatch.from_pydict({
    "file": [new_file.path]  # Path to staged .lance file
})

# File locations
checkpoint_dir/
  ├── {dedupe_key_1}.lance  # Checkpoint entry
  ├── {dedupe_key_2}.lance
  └── ...

staged_data_dir/
  ├── {fragment_id}_{batch_start}_{batch_end}.lance  # Actual data
  └── ...
```

**Properties:**
- **Format**: PyArrow RecordBatch with single column "file"
- **File type**: `.lance` files
- **Naming**: SHA-256 hash of checkpoint key
- **Content**: File path reference to staged data

### Schema Metadata Keys


Materialized view tables store query metadata in their schema:

```python
# Metadata keys (as UTF-8 encoded bytes)
MATVIEW_META_QUERY = b"geneva.query"           # Serialized query JSON
MATVIEW_META_BASE_TABLE = b"geneva.base_table"  # Source table name
MATVIEW_META_BASE_DBURI = b"geneva.base_dburi"  # Source DB URI
MATVIEW_META_BASE_VERSION = b"geneva.source_version"  # Source version

# Example metadata
schema.metadata[b"geneva.query"] = b'{"filter": "age > 21", ...}'
schema.metadata[b"geneva.base_table"] = b"users"
schema.metadata[b"geneva.base_dburi"] = b"/path/to/source.lance"
schema.metadata[b"geneva.source_version"] = b"42"
```

**Properties:**
- **Encoding**: UTF-8 bytes (required for PyArrow schema metadata)
- **Query format**: JSON serialization of GenevaQuery object
- **Version format**: Integer as string bytes
- **Persistence**: Stored in Lance table schema, survives table operations

### Internal Column Formats

All materialized view tables include these system columns:

```python
# Column schemas
__source_row_id: int64         # Row address encoding (see above)
__is_set: bool                 # Whether row has been computed

# Initial state after create_materialized_view()
__source_row_id: [1234, 5678, ...]  # Populated with target row IDs
__is_set: [False, False, ...]       # All False (not yet computed)

# After refresh()
__is_set: [True, True, ...]         # All True (computed)
```

### Checkpoint Store Data Model


The checkpoint store maps checkpoint keys to file references:

```python
# Interface
checkpoint_store: Mapping[str, pa.RecordBatch]

# Key: Checkpoint deduplication key (SHA-256 hash)
key: str = hashlib.sha256(...).hexdigest()

# Value: RecordBatch with file path
value: pa.RecordBatch = pa.RecordBatch.from_pydict({
    "file": ["/path/to/staged/data.lance"]
})

# Operations
if key in checkpoint_store:
    # Fragment already processed, skip
    pass
else:
    # Process fragment and store result
    checkpoint_store[key] = result_batch
```

---

## Fragment Mapping: Source to Checkpoint to Destination

### Mapping Example

**Scenario:** Materialized view with filter and shuffle

```
SOURCE TABLE (3 fragments)
Fragment 0: rows 0-99     → _rowaddr: [0, 1, ..., 99]
Fragment 1: rows 100-199  → _rowaddr: [4294967396, ..., 4294967495]
Fragment 2: rows 200-299  → _rowaddr: [8589934792, ..., 8589934891]

QUERY: where("x > 150").shuffle(seed=42)
Matching rows: [151, 152, ..., 199, 200, 201, ..., 299]
After shuffle: [287, 163, 241, ..., 189]

DESTINATION TABLE (created with 2 fragments)
Fragment 0: __source_row_id = [8589935079, 4294967563, 8589935033, ...]
                              = [287, 163, 241, ...]
Fragment 1: __source_row_id = [..., 8589934981]
                              = [..., 189]

REFRESH PROCESS:
CopyTask for dest fragment 0:
  1. Read __source_row_id from dest fragment 0
  2. Take rows [287, 163, 241, ...] from source
     - Row 287 from source fragment 2, offset 87
     - Row 163 from source fragment 1, offset 63
     - Row 241 from source fragment 2, offset 41
  3. Apply UDFs to create computed columns
  4. Write to dest fragment 0 with _rowaddr = [0, 1, 2, ...]
```

**Key observation:** Source fragment boundaries are irrelevant to destination fragments. The mapping is purely based on the logical query result.

---

## Checkpoint Mechanism and Progress Tracking

### Fragment-Level Checkpointing


Each fragment + map task combination gets a unique deduplication key:

```python
def _get_fragment_dedupe_key(uri: str, frag_id: int, map_task: MapTask) -> str:
    key = f"{uri}:{frag_id}:{map_task.checkpoint_key()}"
    return hashlib.sha256(key.encode()).hexdigest()
```

This enables:
- **Deduplication:** Skip processing if fragment already computed
- **Idempotency:** Safe to retry failed fragments
- **Partial completion:** Only process remaining fragments after failure

### Batch-Level Checkpointing


The `CheckpointingApplier` stores each batch with MD5-based key:

```python
class CheckpointingApplier:
    def apply(self, task: ReadTask) -> Iterator[pa.RecordBatch]:
        checkpoint_key = task.checkpoint_key()  # MD5 of task params

        if checkpoint_key in self._checkpoint_store:
            yield self._checkpoint_store[checkpoint_key]
        else:
            for batch in task.to_batches():
                batch_with_schema = self._apply_map(batch)
                self._checkpoint_store[checkpoint_key] = batch_with_schema
                yield batch_with_schema
```

**Checkpoint key includes:**
- URI and version
- Column list
- Fragment ID, offset, limit
- Where clause (if any)

This granularity allows resumption at batch level if pipeline fails.

### Checkpoint Lifecycle

```
┌─────────────────────────────────────────────────────────────┐
│ CHECKPOINT STORE                                            │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Fragment 0, Batch 0 → RecordBatch (key: sha256(...))       │
│  Fragment 0, Batch 1 → RecordBatch                          │
│  Fragment 1, Batch 0 → RecordBatch                          │
│  ...                                                        │
│                                                             │
│  Data File for Fragment 0 → "/path/to/data-file-0.lance"    │
│  Data File for Fragment 1 → "/path/to/data-file-1.lance"    │
│                                                             │
└────────────┬────────────────────────────┬───────────────────┘
             ↑                            ↓
        Write batches              Read batches
             │                            │
   ┌─────────┴─────────┐        ┌─────────┴──────────┐
   │ CheckpointingAp   │        │ FragmentWriter     │
   │ plier (Ray)       │        │ (Ray)              │
   └───────────────────┘        └────────────────────┘
```

**Cleanup:** Checkpoint store can be cleared after commit succeeds, but is often retained for debugging and potential re-use.

---

## Materialized View Refresh Process

**Why this matters:** The refresh process recomputes materialized view results based on the latest (or specified) source data. Understanding how refresh works helps debug issues with stale data, version mismatches, or performance problems.

### Key Concepts

The refresh process reconstructs the original query from metadata stored in the materialized view's schema:
- **Same source table** and version for consistent reads
- **Same filter/shuffle/UDF operations** applied
- **Destination-driven**: Iterates through existing view fragments, not source fragments
- **Checkpointed**: Can resume from failures without reprocessing completed work

### Version Management

**Why it matters:** Controlling source versions enables reproducibility, testing with historical data, and proper incremental refresh.

**Key capabilities:**
- **Automatic**: `view.refresh()` uses latest source version by default
- **Explicit**: `view.refresh(src_version=42)` pins to specific version
- **Cross-database**: Source can be in different database than view
- **Tracked**: Version metadata stored in view schema for audit trail

**Use cases:**
- Testing with historical snapshots
- Rollback scenarios when source data has issues
- Compliance requirements for version tracking

---

## Incremental Refresh with New Source Data


When new data is added to the source table, the materialized view can incrementally process only the new rows without recomputing existing results.

### Append-Only Strategy

The implementation uses a simple **append-only strategy** that avoids race conditions and complex coordination:

**Key Insight:** Use `table.add()` to append new placeholder rows as a NEW fragment instead of using DataReplacement operations that can conflict with concurrent writers.

### Implementation Overview

The incremental refresh process follows these steps:

1. **Build destination-to-source mapping** by reading `__source_row_id` from destination fragments
2. **Check destination fragment checkpoints** (not source fragment checkpoints)
3. **Identify source fragments needing processing** by checking if their data exists in checkpointed destination fragments
4. **Query new fragments** with filters and shuffle applied
5. **Append placeholder rows** using `table.add()` (creates new destination fragment)
6. **Process only affected fragments** using destination-driven `CopyTask`s

**Critical Design Detail:** Checkpoint lookups use **destination fragment IDs**, not source fragment IDs. This is because:
- Checkpoints are keyed by `_get_fragment_dedupe_key(dst_uri, dest_frag_id, map_task)`
- For materialized views with filters/shuffle, source and destination fragment IDs differ
- A source fragment check would always fail, causing full reprocessing on every refresh
- The mapping from source to destination is determined by inspecting `__source_row_id` values


### Complete Example

**Initial state:**
```
Source table: 3 fragments (0, 1, 2) with 300 rows
Query: where("value > 150") → 150 matches

After initial refresh:
  Destination fragment 0: 150 rows
  Checkpoints: Source fragments [0, 1, 2] marked as processed
```

**Add new data:**
```
Source table: Add fragments [3, 4] with 100 rows
Query filter "value > 150" matches: 50 new rows
```

**Incremental refresh execution:**

1. **Build destination-to-source mapping:**
   - Scan existing destination fragment 0 for `__source_row_id`
   - Extract source fragment IDs: `dst_to_src_map[0] = {0, 1, 2}`
   - Check destination fragment 0 checkpoint: **exists** (from initial refresh)

2. **Identify unprocessed sources:**
   - Check which source fragments have checkpointed data in destination
   - Source fragments [0, 1, 2] are in destination fragment 0 which has a checkpoint
   - Source fragments [3, 4] are **not** in any checkpointed destination fragment
   - `source_fragments_without_checkpoint = [3, 4]`

3. **Query with filter:**
   - Apply "value > 150" to source fragments [3, 4]
   - Extract 50 matching row addresses

4. **Append placeholders:**
   - `table.add(placeholder_data)`
   - Creates destination fragment 1 with 50 rows
   - All `__is_set = False`, all data columns NULL

5. **Update mapping for new fragment:**
   - Destination fragment 1 contains rows from source fragments [3, 4]
   - `fragments_to_process = {1}` (fragment 0 has checkpoint, fragment 1 is new)

6. **Execute CopyTasks:**
   - Skip destination fragment 0 (has checkpoint)
   - Process destination fragment 1 (no checkpoint, contains source [3, 4])
   - `CopyTask` reads `__source_row_id`, fetches from source, applies UDFs
   - Writes results to destination fragment 1

7. **Create checkpoints:**
   - Mark destination fragment 1 as processed
   - Checkpoint key: `_get_fragment_dedupe_key(dst_uri, frag_id=1, map_task)`
   - Now source fragments [3, 4] are checkpointed (via destination fragment 1)

**Final state:**
```
Destination table: 2 fragments
  Fragment 0: 150 rows (checkpointed, not recomputed)
  Fragment 1: 50 rows (newly computed)
Total: 200 rows
```

### Checkpoint Lookup Algorithm

The incremental refresh checkpoint lookup follows this precise algorithm:

```python
# 1. Build mapping from destination fragments to source fragments
dst_to_src_map = {}
dest_fragments_with_checkpoint = set()

for dst_frag in dst_dataset.get_fragments():
    # Check if this DESTINATION fragment has a checkpoint
    checkpoint_exists = _check_fragment_data_file_exists(
        dst_uri, dst_frag.fragment_id, map_task, checkpoint_store
    )

    if checkpoint_exists:
        dest_fragments_with_checkpoint.add(dst_frag.fragment_id)

    # Read __source_row_id to extract source fragment IDs
    scanner = dst_dataset.scanner(
        columns=["__source_row_id"],
        fragments=[dst_frag]
    )
    dst_frag_data = scanner.to_table()
    source_frag_ids = {
        row_id >> 32 for row_id in dst_frag_data["__source_row_id"]
    }
    dst_to_src_map[dst_frag.fragment_id] = source_frag_ids

# 2. Determine which source fragments need processing
for src_frag in src_dataset.get_fragments():
    # Is this source fragment's data in ANY checkpointed destination fragment?
    is_checkpointed = any(
        src_frag.fragment_id in src_frags and
        dst_frag_id in dest_fragments_with_checkpoint
        for dst_frag_id, src_frags in dst_to_src_map.items()
    )

    if not is_checkpointed:
        source_fragments_without_checkpoint.append(src_frag.fragment_id)
```

**Why this works:**
- Correctly handles many-to-one source-to-destination mapping (filters consolidate fragments)
- Uses destination fragment IDs for checkpoint keys (matches how checkpoints are written)
- Efficiently scans destination fragments once using scanner API (not to_table())
- Reuses the mapping to identify which destination fragments need processing

### Why This Design Works

1. **Append-only safety:** `table.add()` avoids race conditions with parallel writes
2. **Destination-driven:** Correct handling of filters/shuffle that consolidate fragments
3. **Fragment immutability:** Source fragments never change once created
4. **Idempotent:** Safe to re-run, skips already-processed fragments
5. **Incremental cost:** Processing proportional to new data only
6. **Correct consolidation:** Multiple source fragments → single destination fragment handled correctly
7. **Correct checkpoint lookups:** Uses destination fragment IDs, matching how checkpoints are written

### Shuffle Behavior During Incremental Refresh

**Important:** Shuffle only applies to NEW data during refresh, preserving original shuffle order for existing data.

**Initial creation:**
```python
# All row IDs are shuffled together
row_ids = query.to_arrow()["_rowid"].to_pylist()
if query.shuffle:
    rng = np.random.default_rng(query.shuffle_seed)
    rng.shuffle(row_ids)
```

**Incremental refresh:**
```python
# Only new row IDs are extracted (with same shuffle seed applied to new data)
new_row_ids = query_new_fragments.to_arrow()["_rowid"].to_pylist()
# Shuffle applied if original query had shuffle
# New data appended, not interleaved with existing data
```

**Result:** Original data maintains its shuffle order, new data added with consistent shuffle semantics.

### Test Reference

See `test_materialized_view_refresh_with_updated_source` in `src/tests/test_runners_ray_matview.py` for a complete working example demonstrating incremental refresh.

---

## Query Operations Impact on Fragments and Layouts

**Why this matters:** Understanding how query operations affect the materialized view structure helps debug performance issues, understand storage costs, and design efficient queries.

### Filter (`where()`)

**Impact:**
- **Row count:** Reduced based on filter selectivity
- **Fragment structure:** Unchanged (same number of fragments as source)
- **Physical layout:** Sparse (filtered rows represented as NULLs)

**Example:**
```
Source fragment 0: 1000 rows
Filter "age > 21": 300 matches

Destination fragment 0:
  Physical rows: 1000
  Logical rows: 300 (rows 0-699 are NULL)
```

**Key point:** Physical layout preserves fragment boundaries for efficient mapping.

### Shuffle

**Impact:**
- **Row order:** Randomized based on seed
- **Fragment structure:** Unchanged
- **Physical layout:** Same density, different order

**Reproducibility:** Same seed always produces same shuffle order. Critical for ML training data consistency.

**Use cases:**
- Randomize data for ML training
- Break locality for better load balancing
- Create different views of same filtered data

### Limit

**Impact:**
- **Row count:** Exactly `limit` rows (or fewer if source has fewer)
- **Fragment structure:** May have fewer fragments than source
- **Physical layout:** Dense (all rows present)

**Example:** `table.limit(100)` with batch_size=1024 creates 1 destination fragment.

### Offset

**Impact:**
- **Row selection:** Skip first N rows
- **Fragment structure:** Unchanged from base query
- **Physical layout:** Same as base query (offset not stored)

**Use case:** Pagination combined with limit: `table.offset(50).limit(100)` returns rows 50-149.

### UDF Columns

**Impact:**
- **Schema:** Additional computed columns
- **Fragment structure:** Unchanged
- **Physical layout:** Same as base query

**Performance note:** UDFs are **not** applied during view creation, only during refresh. This enables fast initialization with lazy computation.

**Use cases:**
- Add embeddings (expensive GPU operations)
- Add predictions from ML models
- Add derived features for analysis

### Column Selection

**Impact:**
- **Schema:** Subset of source columns
- **Fragment structure:** Unchanged
- **Physical layout:** Smaller data files (fewer columns written)

**Use case:** Reduce storage costs by excluding unused columns from materialized view.

---


**Note:** Fragment writer and commit/versioning details are in `fragment_writers.md` (shared with backfill operations).

---

## Key Architectural Components

### Task Hierarchy


```
ReadTask (abstract base)
├── ScanTask
│   - Read from source table with fragment ID
│   - Uses Lance scanner with offset/limit
│   - Supports where clause filtering
│   - Returns batches with _rowaddr column
│
└── CopyTask
    - Read from source using __source_row_id
    - Maps to specific destination fragment
    - Uses Lance take() for random access
    - Encodes _rowaddr for destination

MapTask (abstract base)
├── BackfillUDFTask
│   - Applies single UDF to batches
│   - Used for adding columns to existing tables
│   - Handles BACKFILL_SELECTED filtering
│
└── CopyTableTask
    - Applies multiple UDFs from query
    - Used for materialized view refresh
    - Supports column exclusion
    - Handles null values for filtered rows
```

### Pipeline Components


```
ColumnAddPipelineJob
├── ActorPool[CheckpointingApplier]
│   - Ray remote actors
│   - Apply map tasks to read tasks
│   - Store results in checkpoint store
│   - Return checkpoint keys
│
├── FragmentWriterManager
│   ├── FragmentWriterSession (per fragment)
│   │   ├── Ray.remote(FragmentWriter)
│   │   │   - Buffer and sort batches
│   │   │   - Align to physical layout
│   │   │   - Filter columns
│   │   │   - Write Lance file
│   │   │
│   │   ├── Checkpoint queue
│   │   └── Inflight futures (max 3)
│   │
│   └── Commit coordinator
│       - Track completed fragments
│       - Batch commits every N fragments
│       - Handle version conflicts
│
└── JobTracker
    - Progress tracking
    - Fragment completion metrics
    - Task counts
```

### Data Flow Diagram

```
┌──────────────────────┐
│   SOURCE TABLE       │
│   (Lance)            │
└──────────┬───────────┘
           │
           │ CopyTask reads __source_row_id
           │ from destination, takes rows
           │ from source
           ↓
┌──────────────────────┐
│ CheckpointingApplier │  Apply UDFs
│ (Ray)                │ ───────────→ CopyTableTask
└──────────┬───────────┘
           │
           │ Store batches with
           │ checkpoint key
           ↓
┌──────────────────────┐
│ CHECKPOINT STORE     │
│ (Lance)              │
└──────────┬───────────┘
           │
           │ Retrieve batches
           │ for fragment
           ↓
┌──────────────────────┐
│ FragmentWriter       │  1. Buffer & sort
│ (Ray)                │  2. Align to physical layout
│                      │  3. Filter columns
│                      │  4. Write Lance file
└──────────┬───────────┘
           │
           │ Return (frag_id, DataFile, rows)
           ↓
┌──────────────────────┐
│ FragmentWriter       │  Record completion
│ Manager              │  Batch commits
└──────────┬───────────┘
           │
           │ lance.LanceOperation.DataReplacement
           ↓
┌──────────────────────┐
│ DESTINATION TABLE    │
│ (Lance)              │
│ - Updated fragments  │
│ - New version        │
└──────────────────────┘
```

### Query Metadata Storage


Metadata keys stored in Lance schema.metadata:

```python
MATVIEW_META = "geneva::view::"
MATVIEW_META_QUERY = f"{MATVIEW_META}query"           # Serialized GenevaQuery JSON
MATVIEW_META_BASE_TABLE = f"{MATVIEW_META}base_table" # Source table name
MATVIEW_META_BASE_DBURI = f"{MATVIEW_META}base_table_db_uri"  # Source DB URI
MATVIEW_META_BASE_VERSION = f"{MATVIEW_META}base_table_version"  # Source version
```

**Example:**
```python
schema = lance.dataset(view_uri).schema
metadata = schema.metadata

query_json = metadata[b"geneva::view::query"]
# {
#   "table": "users",
#   "columns": ["name", "age"],
#   "where": "age > 21",
#   "shuffle": true,
#   "shuffle_seed": 42,
#   "column_udfs": [...]
# }
```

This metadata enables:
- **Refresh without query object:** Reconstruct from metadata
- **Schema evolution:** Detect source table changes
- **Audit trail:** Know exact query that created view

---

## Summary

Geneva materialized views provide a robust, distributed system for precomputing query results:

### Key Innovations

1. **Explicit row mapping:** `__source_row_id` enables precise source-to-destination mapping
2. **Checkpoint-based recovery:** Granular deduplication at fragment and batch level
3. **Physical layout alignment:** Sparse logical views represented as dense physical fragments
4. **Incremental commits:** Atomic multi-fragment updates with conflict detection
5. **Lazy computation:** Fast initialization with deferred UDF execution

### Performance Characteristics

- **Creation:** O(n) scan to extract row IDs, minimal compute
- **Refresh:** O(n) for full refresh, parallelized across Ray actors
- **Partial refresh:** O(k) where k = changed rows (if using where-as-bool-column)
- **Storage:** O(n) physical rows (includes NULLs for filtered data)

### Future Enhancements

- **Rank/window functions:** Not yet implemented - Would require sorted/partitioned fragments

---
