# UDTF Future Work

Unimplemented features and optimizations for the batch UDTF system. These are
tracked here for planning purposes — they are explicitly out of scope for the
initial production hardening (Phases 1+2).

For the implemented UDTF system, see [../udtf.md](../udtf.md).

---

## Phase 3: Hardening and Validation

**Goal:** Battle-test the pipeline under realistic failure conditions.

| Task | Status | Description |
|---|---|---|
| Failure injection tests | Not started | Actor OOM, worker node death, storage write timeout, checkpoint corruption. Verify recovery in each case. |
| Stress tests | Not started | Large partition counts (1000+), large output batches (100k+ rows), concurrent UDTF refreshes. |
| Observability | Not started | Structured logging with partition/batch context. Error summaries in JobTracker. |

---

## Incremental UDTF Refresh

### Per-Partition Skip Optimization

**Goal:** Avoid re-executing unchanged partitions when only a subset of the
source data has changed.

| Task | Description |
|---|---|
| Partition change detection | Implement approach B (transaction log walk) or D (partition-aligned ingestion), depending on feasibility analysis. See "Partition Change Detection" section below. |
| Per-partition fingerprint store | Extend checkpoint key scheme with per-partition fingerprints. On refresh, compare fingerprints to skip unchanged partitions. |
| Partial overwrite semantics | When some partitions are skipped, the writer must merge new partition output with existing unchanged partition data rather than full overwrite. |

The checkpoint key scheme (`udtf/{udtf_name}:{udtf_version}/src-{source_version}/...`)
is designed to accommodate this.

### Partition Change Detection — Design Space

#### The core problem

Partitions (e.g., `date='2026-01-15'`) are a logical concept. Lance stores data
in fragments (physical). A partition's data may span multiple fragments, and a
fragment may contain data for multiple partitions. We need a signal that says
"the source data for this partition value has changed since the last refresh."

#### Candidate approaches

| Approach | How it works | Cost | Pros | Cons |
|---|---|---|---|---|
| **A. Source table version only** | If `src_version == last_refresh_version`, skip all. Otherwise re-run all. | O(1) | Trivial. No per-partition tracking. | All-or-nothing — one changed row re-runs every partition. |
| **B. Transaction log walk** | Walk `dataset.get_transactions()` from `last_refresh_version` to current. Collect touched `fragment_id`s. Map fragments→partitions via `frag.count_rows(filter=...)`. | O(txns × frags) | No row data read. Uses existing Lance API. | Fragment→partition mapping requires scanning each changed fragment. Expensive if many fragments changed. |
| **C. Per-partition data file fingerprint** | For each partition, compute `hash_source_files(files)` where `files` = union of `data_files()` from all fragments containing that partition's rows. Store fingerprint per-partition in checkpoint store. On next refresh, recompute and compare. | O(partitions × frags) | Reuses existing `hash_source_files` pattern from MV pipeline. Fine-grained. | Requires mapping partitions→fragments (same scan problem as B). Fingerprint must be recomputed each refresh. |
| **D. Partition-aligned ingestion convention** | Require/recommend that source data is ingested with one fragment per partition value (e.g., daily append creates one fragment per day). Then fragment change = partition change. | O(changed_frags) | Exact, cheap — just check if fragment file paths changed. | Requires upstream discipline. Breaks if compaction merges fragments across partitions. |
| **E. Source-side partition manifest** | Maintain a sidecar manifest mapping `(partition_value → [fragment_ids] → file_hash)`. Update on ingest. UDTF refresh reads manifest diff. | O(changed_partitions) | Cheapest at refresh time. | Requires maintaining the manifest — new write-path concern. |
| **F. Content hash via Arrow compute** | For each partition, read the actual data, compute a hash (e.g., IPC serialize + MD5). Compare against stored hash. | O(partition_data_size) | Exact — detects any data change regardless of physical layout. | Reads all partition data just to decide whether to skip. Defeats the purpose for large partitions. |

#### Recommended path

- **Phase 1 (done):** Approach A only — whole-table version check.
- **Follow-on:** Approach B or C, gated by whether partition-aligned ingestion
  (D) is feasible for our use cases. If source data is already partitioned by
  the `partition_by` column at the fragment level, approach D is ideal and
  essentially free. If not, approach B (transaction walk + fragment scan) is
  the most practical.

---

## Reduce Phase (Map-Reduce Style)

**Goal:** Add an optional reduce step that runs after all partitions complete,
enabling cross-partition aggregation within a single UDTF refresh.

### Motivation

Today's partition-parallel UDTFs are pure **map** — each partition runs
independently and outputs are concatenated. Many workflows need a final
**reduce** that combines partition-level results into a global output:

| Workflow | Map (per-partition) | Reduce (cross-partition) |
|---|---|---|
| Image dedup | Compute per-partition duplicate clusters | Merge clusters that span partition boundaries |
| K-means | Compute per-partition centroids | Run global k-means on partition centroids |
| Feature statistics | Compute per-partition stats (min, max, mean) | Combine into global stats |
| Graph analysis | Build per-partition edge lists | Run connected-components across partitions |

Without a reduce phase, users must chain two separate UDTF views (partition →
intermediate table → global reduce), which doubles the storage and operational
overhead.

### Strawman API

```python
@geneva.udtf(
    output_schema=FINAL_SCHEMA,
    partition_by="partition_id",
)
class DedupWithReduce:
    def __call__(self, source: GenevaQueryBuilder) -> Iterator[pa.RecordBatch]:
        # Map: per-partition duplicate detection
        data = source.to_arrow()
        yield compute_partition_clusters(data)

    def reduce(self, partitions: GenevaQueryBuilder) -> Iterator[pa.RecordBatch]:
        # Reduce: merge cross-partition clusters
        all_clusters = partitions.to_arrow()
        yield merge_clusters(all_clusters)
```

- `__call__` is the **map** — runs once per partition (unchanged from today)
- `reduce` is optional — runs once after all partitions complete, receiving the
  concatenated map output as its source
- If `reduce` is absent, behavior is identical to today (map-only)
- The reduce output replaces the map output as the final view contents

### Design Considerations

| Question | Notes |
|---|---|
| **Reduce input** | Should `reduce` receive the map output as a `GenevaQueryBuilder` (reading from staged fragments) or as in-memory `pa.Table`? QBR is more scalable but requires staged intermediate writes. |
| **Checkpointing** | Map phase checkpoints work as today. Reduce checkpoint is a single key. On resume, if all map partitions are done, skip straight to reduce. |
| **Multi-level reduce** | A single reduce step covers most use cases. Tree-reduce (hierarchical) is future-future work. |
| **Output schema** | Map and reduce may have different schemas. The reduce `output_schema` becomes the view's final schema. Need to decide if map schema is declared separately or inferred. |
| **Error handling** | If reduce fails, do we re-run reduce only (map output is checkpointed) or the entire refresh? Reduce-only retry is strongly preferred. |
| **Resource requests** | Reduce may need different resources than map (e.g., more memory for global aggregation). May need separate resource annotations. |

### Execution Model

```
_refresh_udtf_matview (with reduce)
├── Map phase (unchanged)
│   ├── ActorPool dispatches partitions
│   ├── Each partition yields batches → staged fragments
│   └── Checkpoint per-batch and per-partition
├── Barrier: wait for all map partitions to complete
├── Reduce phase
│   ├── Open staged map output as GenevaQueryBuilder
│   ├── Call udtf.reduce(map_output)
│   ├── Yields final batches → output fragments
│   └── Checkpoint reduce completion
└── Commit final output
```

---

## Future Optimizations from Batch Design

### Incremental support for batch UDTFs

Batch UDTFs always do a full refresh today. Framework-managed state persistence
(e.g., persisting centroids or dedup indices between refreshes to enable
incremental processing) is a potential future extension. In the meantime, users
who need cross-refresh state can manage it externally (e.g., write reduced state
to a separate table, read it back in `__init__`).

### Schema evolution

What if UDTF output schema changes between refreshes? Options include:
- Error on schema mismatch
- Schema migration
- Versioned UDTFs

This is the same problem as regular MV UDF schema evolution — no solution
implemented yet for either path.

---

## Related Design Docs

- [udtf_chunking_design.md](udtf_chunking_design.md) — per-row 1:N UDTFs (unimplemented)
- [../udtf.md](../udtf.md) — implemented batch UDTF reference
