# Scalar UDTFs (1:N Row Expansion)

## The Problem

Today, MVs maintain **1:1 row correspondence** with source tables. Each MV row maps to exactly one source row via `__source_row_id`.

But many AI/ML workflows need **1:N expansion**:

| Source Table (videos) | Derived Table (clips) |
|-----------------------|-----------------------|
| 1 video row | → N clip rows |
| 1 document row | → N chunk rows |
| 1 image row | → N tile/region rows |

### Example Use Case: Video → Clips

```
videos table:
┌────────────┬──────────┬────────────┐
│ video_path │ duration │ metadata   │
├────────────┼──────────┼────────────┤
│ /v/a.mp4   │ 120.0    │ {fps: 30}  │
│ /v/b.mp4   │ 60.0     │ {fps: 24}  │
└────────────┴──────────┴────────────┘

clips table (derived):
┌────────────┬──────────┬────────────┬────────────┬──────────┐
│ video_path │ duration │ metadata   │ clip_start │ clip_end │
├────────────┼──────────┼────────────┼────────────┼──────────┤
│ /v/a.mp4   │ 120.0    │ {fps: 30}  │ 0.0        │ 10.0     │
│ /v/a.mp4   │ 120.0    │ {fps: 30}  │ 10.0       │ 20.0     │
│ /v/a.mp4   │ 120.0    │ {fps: 30}  │ ...        │ ...      │
│ /v/b.mp4   │ 60.0     │ {fps: 24}  │ 0.0        │ 10.0     │
│ /v/b.mp4   │ 60.0     │ {fps: 24}  │ 10.0       │ 20.0     │
│ ...        │ ...      │ ...        │ ...        │ ...      │
└────────────┴──────────┴────────────┴────────────┴──────────┘
```

**Key requirements:**
1. Parent columns (`video_path`, `duration`, `metadata`) inherited "for free"
2. UDTF adds new columns (`clip_start`, `clip_end`)
3. Linkage back to parent row for updates/deletes
4. MV-like incremental refresh semantics
5. **Add new UDF computed columns and backfill** — after creating derived table, add columns (e.g., embeddings) that backfill on existing rows

---

## Current MV Architecture (for reference)

```
Source Table                    Materialized View
┌─────────────────┐            ┌──────────────────────────────┐
│ row_id: 1       │ ────────── │ __source_row_id: 1           │
│ col_a: "foo"    │   1:1      │ col_a: "foo"                 │
│ col_b: 123      │            │ udf_col: <computed>          │
└─────────────────┘            └──────────────────────────────┘
```

- `__source_row_id` points to exactly one source row
- Refresh: find new source rows → create new MV rows
- Schema: source columns + UDF-computed columns

---

## Proposed UDTF Architecture

```
Source Table (videos)           Materialized View (clips) - 1:N
┌─────────────────┐            ┌──────────────────────────────┐
│ row_id: 1       │ ────┬───── │ __source_row_id: 1           │
│ video_path: ... │     │      │ __child_index: 0             │
│ duration: 120   │     │ 1:N  │ video_path: ... (inherited)  │
│ metadata: {...} │     │      │ duration: 120 (inherited)    │
└─────────────────┘     │      │ clip_start: 0.0 (from UDTF)  │
                        │      │ clip_end: 10.0 (from UDTF)   │
                        │      ├──────────────────────────────┤
                        ├───── │ __source_row_id: 1           │
                        │      │ __child_index: 1             │
                        │      │ video_path: ... (inherited)  │
                        │      │ clip_start: 10.0             │
                        │      │ clip_end: 20.0               │
                        │      ├──────────────────────────────┤
                        └───── │ ...                          │
                               └──────────────────────────────┘
```

### Internal Columns

| Column | Type | Purpose |
|--------|------|---------|
| `__source_row_id` | int64 | Links to source row (reuses existing MV column) |
| `__child_index` | int32 | Disambiguates children from same parent |
| `__is_set` | bool | Marks placeholder vs computed (same as MV) |

**Note**: We reuse `__source_row_id` (not a new `__parent_row_id`) for consistency with existing MVs. The difference is cardinality: 1:1 for regular MVs, 1:N for UDTFs.

---

## UDTF Interface Options

### Option A: Generator/Yield Pattern

```python
from geneva import scalar_udtf
from typing import Iterator, NamedTuple

class Clip(NamedTuple):
    clip_start: float
    clip_end: float
    clip_bytes: bytes

@scalar_udtf
def extract_clips(video_path: str, duration: float) -> Iterator[Clip]:
    """Yields multiple clips per video"""
    clip_length = 10.0
    for start in range(0, int(duration), int(clip_length)):
        end = min(start + clip_length, duration)
        clip_data = extract_video_segment(video_path, start, end)
        yield Clip(clip_start=start, clip_end=end, clip_bytes=clip_data)

# Usage
clips_table = db.create_materialized_view(
    "clips",
    query=videos.search(None),
    udtf=extract_clips
)
```

**Pros**: Pythonic, familiar generator pattern, memory efficient
**Cons**: Can't know output count upfront

### Option B: List Return Pattern

```python
@scalar_udtf
def extract_clips(video_path: str, duration: float) -> list[Clip]:
    """Returns list of clips per video"""
    clips = []
    for start in range(0, int(duration), 10):
        clips.append(Clip(...))
    return clips
```

**Pros**: Simple, output count known
**Cons**: Must materialize all in memory

### Option C: Batched Scalar UDTF Pattern

```python
@scalar_udtf(batch=True)
def extract_clips(batch: pa.RecordBatch) -> pa.RecordBatch:
    """Process rows in batches for efficiency, but semantic is still 1:N per row.
    Must include __source_row_id in output to map expanded rows back to inputs."""
    ...
```

**Pros**: Vectorized, efficient, same 1:N semantic as scalar_udtf
**Cons**: User must include row mapping in output

### Option D: Arbitrary Batch UDTF (N:M)

```python
@batch_udtf  # alias for existing @udtf
def transform_batch(batch: pa.RecordBatch) -> pa.RecordBatch:
    """Arbitrary N:M batch transformation (already implemented in PR #573)"""
    ...
```

**Pros**: Full flexibility
**Cons**: No per-row expansion semantic, user manages everything

### Recommendation

**Option A (Generator)** for `@scalar_udtf` — most Pythonic, natural for 1:N

**Option A + batch=True** for `@scalar_udtf(batch=True)` — vectorized variant, processes batches but same 1:N semantic per row

**Option C (Batch)** for `@batch_udtf` — already implemented (PR #573) for N:M

### UDTF Naming Taxonomy

Geneva has five decorator forms for user-defined functions:

| Decorator | Cardinality | Description |
|-----------|-------------|-------------|
| `@udf` | 1:1 | Per-row function, returns one value per row (currently infers scalar vs batch) |
| `@udf(batch=True)` | N:N | Vectorized per-batch, returns one value per input row **(TODO: rename from `@batch_udf`)** |
| `@scalar_udtf` | 1:N | Per-row expansion, yields multiple rows per input row **(new)** |
| `@scalar_udtf(batch=True)` | 1:N (batched) | Vectorized per-row expansion, processes batches but expands each row to multiple rows **(new)** |
| `@batch_udtf` | N:M | Per-batch transformation, returns a different-sized batch (alias for existing `@udtf`) |

---

## Inherited Columns: How It Works

When creating a 1:N MV, parent columns are available "for free" via the query:

```python
# Query defines which columns to inherit
clips_table = db.create_materialized_view(
    "clips",
    query=videos.search(None).select(["video_path", "duration", "metadata"]),
    udtf=extract_clips,
)

# Or inherit all source columns (default when no select)
clips_table = db.create_materialized_view(
    "clips",
    query=videos.search(None),  # All columns inherited
    udtf=extract_clips,
)
```

### Implementation Options

**Option 1: Physical Copy (Denormalized)**
- Parent column values copied into each child row
- Pros: Fast reads, self-contained rows
- Cons: Storage bloat, especially for large columns (video bytes!)

**Option 2: Lazy Join (Normalized)**
- Only store `__source_row_id`, join at query time
- Pros: No duplication, storage efficient
- Cons: Query overhead, requires parent table access

**Option 3: Hybrid (Recommended)**
- Small/scalar columns: physically copied
- Large columns (bytes, embeddings): lazy join via `__source_row_id`
- Configurable per-column

```python
clips_table = db.create_materialized_view(
    "clips",
    query=videos.search(None).select(["video_path", "duration", "metadata"]),
    udtf=extract_clips,
    column_storage={
        "video_path": "copy",      # Small, copy it
        "duration": "copy",        # Small, copy it
        "metadata": "copy",        # Small, copy it
        "video_bytes": "reference" # Large, join at query time
    }
)
```

---

## Refresh Semantics

### When Parent Row Added
1. Run UDTF on new parent row
2. Insert N child rows with `__source_row_id` pointing to parent

### When Parent Row Deleted
1. Find all child rows with matching `__source_row_id`
2. Delete them (cascade delete)

### When Parent Row Updated
Options:
- **Full recompute**: Delete old children, rerun UDTF, insert new children
- **Smart diff**: Only if UDTF is deterministic and we can detect column changes

### Incremental Refresh Flow

```
1. Find new parent rows (same as MV)
   new_parents = source.rows_since(last_refresh_version)

2. For each new parent:
   children = udtf(parent)
   for i, child in enumerate(children):
       child.__source_row_id = parent.row_id
       child.__child_index = i
       mv_table.append(child)

3. Update checkpoint
```

---

## API Design

### Unified MV API with `udtf=` Parameter

The UDTF API extends the existing `create_materialized_view` pattern rather than introducing a new concept. The key difference is cardinality: regular MVs are 1:1, UDTFs enable 1:N.

```python
# Current 1:1 MV pattern (unchanged)
mv = (
    videos.search(None)
    .select({
        "video_path": "video_path",
        "embedding": embedding_udf
    })
    .create_materialized_view(db, "video_embeddings")
)

# New 1:N MV pattern with udtf=
clips = db.create_materialized_view(
    name="clips",
    query=videos.search(None),
    udtf=extract_clips,  # <-- New parameter for 1:N expansion
)
```

### Fitting into the Query Pattern

Geneva's current MV creation uses a **query-based** pattern where a `GenevaQueryBuilder` is passed to `create_materialized_view`. The query captures:
- Source table
- Column selection (via `.select()`)
- Filters (via `.where()`)
- UDF-computed columns

For UDTFs, we extend this pattern:

```python
# Current: Query defines what columns to select + UDF columns
query = videos.search(None).select({
    "video_path": "video_path",
    "video": load_video,  # UDF column
})
mv = query.create_materialized_view(db, "processed_videos")

# Extended: Query + udtf= for 1:N expansion
query = videos.search(None).select(["video_path", "duration"])
clips = db.create_materialized_view(
    "clips",
    query=query,
    udtf=extract_clips,
)
```

### `select("*")` + Additional Columns (Sugar)

For the common case of "all source columns plus computed columns", we can add syntactic sugar:

```python
# Option A: Explicit wildcard in select dict
clips = (
    videos.search(None)
    .select({
        "*": "*",                      # All source columns
        "embedding": clip_embedding,   # Plus computed column
    })
    .create_materialized_view(db, "clips", udtf=extract_clips)
)

# Option B: select_all() method
clips = (
    videos.search(None)
    .select_all()                      # All source columns
    .add_udf_columns({                 # Plus computed columns
        "embedding": clip_embedding
    })
    .create_materialized_view(db, "clips", udtf=extract_clips)
)

# Option C: columns= parameter on create_materialized_view
clips = db.create_materialized_view(
    "clips",
    query=videos.search(None),  # Inherits all source columns
    udtf=extract_clips,
    columns={"embedding": clip_embedding},  # Additional computed columns
)
```

**Recommendation**: Option C is most explicit and aligns with the existing pattern where `create_materialized_view` takes a query and additional configuration.

### Full API Example

```python
from geneva import udtf
from typing import Iterator, NamedTuple

class Clip(NamedTuple):
    clip_start: float
    clip_end: float
    clip_bytes: bytes

@scalar_udtf
def extract_clips(video_path: str, duration: float) -> Iterator[Clip]:
    clip_length = 10.0
    for start in range(0, int(duration), int(clip_length)):
        end = min(start + clip_length, duration)
        clip_data = extract_video_segment(video_path, start, end)
        yield Clip(clip_start=start, clip_end=end, clip_bytes=clip_data)

# Create 1:N MV with UDTF
clips = db.create_materialized_view(
    "clips",
    query=videos.search(None).select(["video_path", "duration", "metadata"]),
    udtf=extract_clips,
)

# Refresh populates clip rows
clips.refresh()

# Add computed columns after creation, backfill on refresh
clips.add_columns({"embedding": clip_embedding_udf})
clips.refresh()  # Backfills embedding on all existing clips

# Query - parent columns available automatically
clips.search(None).select(["video_path", "clip_start", "embedding"]).to_pandas()
```

### GenevaQuery Extension

The `GenevaQuery` model would be extended to include UDTF metadata:

```python
class GenevaQuery(BaseModel):
    base: Query
    shuffle: bool | None = None
    shuffle_seed: int | None = None
    fragment_ids: list[int] | None = None
    with_row_address: bool | None = None
    column_udfs: list[ColumnUDF] | None = None
    # New for 1:N MVs
    scalar_udtf: PydanticUDFSpec | None = None  # The 1:N expansion function
```

### Refreshing

```python
# Same as MV - refresh is unified
clips.refresh()

# Incremental: only processes new source rows
clips.refresh(src_version=latest_version)
```

### Querying with Parent Columns

```python
# Parent columns available transparently (inherited into MV schema)
clips.to_pandas()[["video_path", "clip_start", "clip_end"]]

# For large columns stored by reference
clips.with_parent_columns(["video_bytes"])  # Lazy join
```

---

## Comparison with Existing Systems

| System | Feature | Cardinality | Inherited Columns | Custom Logic | Incremental | Add Cols + Backfill |
|--------|---------|-------------|-------------------|--------------|-------------|---------------------|
| **Pixeltable** | `create_view` + iterator | 1:N | Automatic via parent ref | ✅ Subclass `ComponentIterator` | ✅ | ✅ |
| **Databricks** | `@udtf` class + `eval()` | 1:N | Via `lateralJoin` | ✅ Python | ❌ | ❌ |
| **Flink** | `TableFunction` + `collect()` | 1:N | Via `LATERAL TABLE` | ✅ Java/Python | ✅ Streaming | ❌ |
| **Snowflake** | UDTF class + `process()` | 1:N | Via LATERAL JOIN | ✅ Python/SQL | ❌ | ❌ |
| **Trino** | `UNNEST` | 1:N | Via `CROSS JOIN` | ❌ Arrays only | ❌ | ❌ |
| **Polars** | `explode()` / `unnest()` | 1:N | Preserved | ❌ Arrays only | ❌ | ❌ |
| **Pandas** | `explode()` | 1:N | Index preserved | ❌ Arrays only | ❌ | ❌ |
| **Geneva (proposed)** | `@scalar_udtf` + `create_materialized_view(udtf=)` | 1:N | Via query `.select()` | ✅ Python | ✅ MV-style | ✅ |

---

## Deep Dive: Pixeltable

[Pixeltable](https://github.com/pixeltable/pixeltable) is the closest prior art — a declarative, incremental multimodal data infrastructure with first-class support for 1:N expansion via "iterators".

### Pixeltable's Iterator Pattern

```python
import pixeltable as pxt
from pixeltable.iterators import FrameIterator, DocumentSplitter

# Create base table
videos = pxt.create_table('videos', {'video': pxt.Video})

# Create view with iterator - 1 video → N frames
frames = pxt.create_view('frames', videos,
    iterator=FrameIterator.create(video=videos.video, fps=0.5))

# Parent columns accessible via reference
frames.add_computed_column(
    detections=yolox(frames.frame, model_id='yolox_s')
)

# Document chunking example
docs = pxt.create_table('docs', {'doc': pxt.Document})
chunks = pxt.create_view('chunks', docs,
    iterator=DocumentSplitter.create(
        document=docs.doc,
        separators='sentence,token_limit',
        overlap=50,
        limit=500
    ))
```

### Built-in Iterators

| Iterator | Input | Output | Parameters |
|----------|-------|--------|------------|
| `FrameIterator` | Video | Frames | `fps` (0 = original rate) |
| `DocumentSplitter` | Document | Chunks | `separators`, `overlap`, `limit` |
| `AudioSplitter` | Audio/Video | Audio chunks | `chunk_duration_sec` |
| `TileIterator` | Image | Tiles | tile size, overlap |
| `VideoSplitter` | Video | Clips | clip parameters |

### Custom Iterators via ComponentIterator

Pixeltable **does** support custom iterators by subclassing `ComponentIterator`:

```python
from pixeltable.iterators.base import ComponentIterator
from pixeltable.type_system import VideoType, FloatType

class MyChunker(ComponentIterator):
    def __init__(self, video, chunk_duration: float):
        self.video = video
        self.chunk_duration = chunk_duration
        self.pos = 0
        self.chunks = self._extract_chunks()

    @classmethod
    def input_schema(cls) -> dict[str, ColumnType]:
        """Define input parameter types (must match __init__ params)"""
        return {'video': VideoType(), 'chunk_duration': FloatType()}

    @classmethod
    def output_schema(cls, *args, **kwargs) -> tuple[dict, list]:
        """Define output columns and which are unstored (lazy)"""
        return {
            'chunk_start': FloatType(),
            'chunk_end': FloatType(),
            'chunk_bytes': VideoType()
        }, ['chunk_bytes']  # chunk_bytes is lazy/unstored

    def __next__(self) -> dict[str, Any]:
        if self.pos >= len(self.chunks):
            raise StopIteration
        chunk = self.chunks[self.pos]
        self.pos += 1
        return chunk

    def close(self) -> None:
        """Release resources"""
        pass

    def set_pos(self, pos: int) -> None:
        """Seek to position (for random access)"""
        self.pos = pos

# Usage
chunks = pxt.create_view('chunks', videos,
    iterator=MyChunker.create(video=videos.video, chunk_duration=10.0))
```

### Key Pixeltable Concepts

| Concept | Description |
|---------|-------------|
| **Views** | Virtual tables via `create_view()` with iterator |
| **ComponentIterator** | Base class for custom 1:N expansion |
| **input_schema()** | Declare iterator input types |
| **output_schema()** | Declare output columns + unstored (lazy) columns |
| **Lazy columns** | Not stored; re-extracted on access (e.g., frame pixels) |
| **Parent access** | Child views reference parent via `parent_table.column` |
| **Incremental** | Changes to parent cascade to views automatically |

### Pixeltable Strengths

1. **Custom iterators**: Full control via `ComponentIterator` subclass
2. **Lazy storage**: Output columns can be unstored (re-computed on access)
3. **Declarative**: `create_view(..., iterator=X)` is clean
4. **Parent column access**: `videos.video` from child view works
5. **Incremental**: Only recomputes affected rows on parent change

### Pixeltable Limitations

1. **Class-based**: Must subclass `ComponentIterator` (vs simple `@scalar_udtf` decorator)
2. **Schema methods**: Requires `input_schema()` + `output_schema()` classmethods
3. **Virtual only**: Views are always lazy, no option to materialize for performance
4. **Seek required**: Must implement `set_pos()` for random access

### Geneva vs Pixeltable

| Aspect | Pixeltable | Geneva (Proposed) |
|--------|------------|-------------------|
| **Custom 1:N** | Subclass `ComponentIterator` | `@scalar_udtf` decorator (simpler) |
| **Schema declaration** | `input_schema()` + `output_schema()` methods | Type hints on function |
| **Storage** | Virtual/lazy only | Choice: materialized or lazy |
| **Parent columns** | `parent.column` syntax | Via query `.select()` |
| **Incremental refresh** | ✅ Automatic | ✅ MV-style refresh |
| **Seek/random access** | Required (`set_pos`) | Optional |
| **Built-in iterators** | FrameIterator, DocumentSplitter, etc. | TBD |
| **Add cols + backfill** | ✅ `add_computed_column()` | ✅ `add_columns()` + `refresh()` |

### Design Insight from Pixeltable

Geneva can offer a **simpler interface** than Pixeltable's class-based approach:

```python
# Pixeltable: Subclass with 5 methods + create_view()
class MyChunker(ComponentIterator):
    def __init__(...): ...
    def input_schema(cls): ...
    def output_schema(cls): ...
    def __next__(self): ...
    def close(self): ...
    def set_pos(self, pos): ...

chunks = pxt.create_view('chunks', videos,
    iterator=MyChunker.create(video=videos.video, chunk_duration=10.0))

# Geneva: Simple decorator + generator + create_materialized_view()
@scalar_udtf
def my_chunker(video_path: str, duration: float) -> Iterator[Clip]:
    for start in range(0, int(duration), 10):
        yield Clip(start=start, end=start+10)

clips = db.create_materialized_view(
    "clips",
    query=videos.search(None),
    udtf=my_chunker
)
```

---

## Deep Dive: Databricks/PySpark UDTF

[Databricks UDTF Docs](https://docs.databricks.com/aws/en/udf/python-udtf)

Databricks uses a **class-based pattern** with an `eval()` method that yields rows:

```python
from pyspark.sql.functions import lit, udtf

@udtf(returnType="clip_start: float, clip_end: float")
class ExtractClips:
    def eval(self, video_path: str, duration: float):
        for start in range(0, int(duration), 10):
            yield start, min(start + 10, duration)

# Direct usage
ExtractClips(lit("/path/to/video"), lit(120.0)).show()

# Register for SQL
spark.udtf.register("extract_clips", ExtractClips)
spark.sql("SELECT * FROM extract_clips('/path/to/video', 120.0)")

# DataFrame API with lateralJoin (Spark 4.0+)
videos_df.lateralJoin(ExtractClips(videos_df.video_path, videos_df.duration))
```

**Key characteristics:**
- Class with `eval()` method that `yield`s tuples
- Schema declared in `returnType` string
- Parent columns via `lateralJoin` (Spark 4.0) or manual join
- No incremental refresh

---

## Deep Dive: Apache Flink

[Flink TableFunction Docs](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/python/table/udfs/python_udfs/)

Flink uses `TableFunction` with a `collect()` method for streaming scenarios:

```python
from pyflink.table.udf import udtf
from pyflink.table import DataTypes

@udtf(result_types=[DataTypes.FLOAT(), DataTypes.FLOAT()])
def extract_clips(video_path: str, duration: float):
    for start in range(0, int(duration), 10):
        yield start, min(start + 10, duration)

# SQL usage with LATERAL TABLE
table_env.sql_query("""
    SELECT v.video_path, c.clip_start, c.clip_end
    FROM videos v, LATERAL TABLE(extract_clips(v.video_path, v.duration)) AS c(clip_start, clip_end)
""")

# Table API
videos.join_lateral(extract_clips(videos.video_path, videos.duration))
      .select(videos.video_path, col("clip_start"), col("clip_end"))
```

**Key characteristics:**
- `@udtf` decorator with `result_types`
- `LATERAL TABLE()` syntax in SQL
- Streaming-native with incremental processing
- Parent columns via explicit join

---

## Deep Dive: Trino/Presto

[Trino UNNEST Docs](https://trino.io/docs/current/sql/select.html)

Trino uses `UNNEST` for array expansion (no custom logic):

```sql
-- Explode array column to rows
SELECT video_path, tag
FROM videos
CROSS JOIN UNNEST(tags) AS t(tag)

-- With ordinality (row index)
SELECT video_path, tag, idx
FROM videos
CROSS JOIN UNNEST(tags) WITH ORDINALITY AS t(tag, idx)

-- LEFT JOIN preserves rows with empty arrays
SELECT video_path, tag
FROM videos
LEFT JOIN UNNEST(tags) AS t(tag) ON TRUE

-- Multiple arrays (zipped)
SELECT video_path, start_time, end_time
FROM videos
CROSS JOIN UNNEST(clip_starts, clip_ends) AS t(start_time, end_time)
```

**Key characteristics:**
- Arrays only — no custom expansion logic
- `CROSS JOIN UNNEST` pattern
- `WITH ORDINALITY` for row index
- Parent columns via join

---

## Deep Dive: Polars

[Polars explode](https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.explode.html) | [Polars unnest](https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.unnest.html)

Polars has `explode()` for lists and `unnest()` for structs:

```python
import polars as pl

# Explode list column to rows (1:N)
df = pl.DataFrame({
    "video_path": ["/a.mp4", "/b.mp4"],
    "tags": [["action", "comedy"], ["drama"]]
})
df.explode("tags")
# video_path | tags
# /a.mp4     | action
# /a.mp4     | comedy
# /b.mp4     | drama

# Unnest struct column to separate columns
df.unnest("metadata")

# Common pattern: list[struct] → flat table
df.explode("clips").unnest("clips")
```

**Key characteristics:**
- `explode()` for lists → rows
- `unnest()` for structs → columns
- No custom logic — operates on existing array/struct columns
- Parent columns preserved automatically

---

## Design Patterns Summary

| Pattern | Systems | Pros | Cons |
|---------|---------|------|------|
| **Class + yield** | Databricks, Flink, Snowflake, Pixeltable | Explicit schema, stateful | Boilerplate |
| **Decorator + generator** | Geneva (proposed) | Simple, Pythonic | Schema from type hints |
| **LATERAL JOIN** | All SQL systems | Standard SQL | Manual parent column handling |
| **Built-in explode** | Polars, Pandas, Trino | Simple | No custom logic |

---

### Geneva's Advantage

Other systems require manual joins to get parent columns:

```sql
-- Snowflake: Manual LATERAL JOIN
SELECT v.video_path, v.duration, c.*
FROM videos v, LATERAL extract_clips(v.video_path, v.duration) c;
```

Geneva makes parent columns automatic via the query:

```python
# Parent columns inherited from query
clips = db.create_materialized_view(
    "clips",
    query=videos.search(None).select(["video_path", "duration"]),
    udtf=extract_clips
)
clips.search(None).select(["video_path", "clip_start"]).to_pandas()  # video_path available
```

---

## Open Questions

### Resolved

1. ~~**Naming**: `create_derived_table` vs `create_child_table` vs `explode` vs `chunk`?~~
   **Resolved**: Use `create_materialized_view(..., udtf=)` — unified under existing MV concept.

2. ~~**Nested derivation**: Can a 1:N MV have its own MVs? (clips → frames)~~
   **Resolved**: Yes. Since we use the unified MV concept, chaining works naturally. Geneva already supports MV-on-MV via stable row IDs. A 1:N MV is just an MV, so `clips.search(None).create_materialized_view(db, "frames", udtf=extract_frames)` should work.

3. ~~**Lazy vs eager inheritance**: Default to copy or reference for inherited columns?~~
   **Resolved**: Inherited columns come from the query's `.select()`. Default is physical copy (same as current MVs). Use `column_storage={"large_col": "reference"}` for lazy join on large columns.

4. ~~**`__child_index` ordering**: Is order deterministic? Should we store the original order from the UDTF generator?~~
   **Resolved**: Yes, order is preserved. The refresh flow uses `enumerate()` on the generator output, storing the original yield order in `__child_index`.

5. ~~**Query-on-Query**: Can the query builder chain with `udtf=`?~~
   **Resolved**: No chaining on query builder. Use `db.create_materialized_view(query=..., udtf=...)` pattern. The `udtf=` is a parameter on the MV creation, not a query method.

6. ~~**Determinism**: If UDTF is non-deterministic (e.g., random sampling), how to handle refresh?~~
   **Resolved**: User responsibility. Document that non-deterministic scalar UDTFs may produce different results on re-refresh, but don't enforce or detect it.

7. ~~**Delete propagation**: Auto-cascade vs manual cleanup?~~
   **Resolved**: Punt for now. Same limitations as existing MVs with deletes.

8. ~~**Schema evolution**: What if UDTF output schema changes between refreshes?~~
   **Resolved**: Punt for now. Same issue exists for regular MV UDFs.

9. ~~**Input binding**: How does the framework map source columns to scalar_udtf parameters?~~
   **Resolved**: Implicit binding by parameter name, same as UDFs. Parameter `video_path` binds to column `video_path`.

10. ~~**Error handling**: If the scalar_udtf fails on one source row, what happens?~~
    **Resolved**: Use the same parameterized error handling mechanism as UDFs (e.g., skip/fail/default).

11. ~~**Empty yields**: Is it valid for a scalar_udtf to yield zero rows for a source row?~~
    **Resolved**: Yes. A source row can produce zero child rows. The source row is still marked as processed (not retried on next refresh).

12. ~~**`batch=True` contract**: What's the exact signature for `@scalar_udtf(batch=True)`?~~
    **Resolved**: Takes implicit bindings as Arrow Arrays (same column name matching as scalar variant), returns a RecordBatch of expanded rows.

13. ~~**Row ID stability for child rows**: Do child rows get stable IDs for MV chaining?~~
    **Resolved**: Yes. Child rows get their own stable `row_id` in the MV, and `__source_row_id` tracks lineage. This enables chaining (video → clips → frames) where each level references the parent via `__source_row_id`.

14. ~~**Re-expansion on parent update**: What happens when a parent column used by the scalar_udtf changes?~~
    **Resolved**: Full recompute — delete old children, rerun scalar_udtf, insert new children. Correctness over performance for now.

15. ~~**`column_storage="reference"` for lazy join**: Planned for v1?~~
    **Resolved**: No. Physical copy only for v1. Lazy join via reference is a future optimization.
