# Daft vs Geneva: Image Embedding & Captioning Pipeline

This file compares Daft (distributed DataFrame for multimodal data) with Geneva
for building production image embedding pipelines.

## Key Insight

Daft is an excellent distributed DataFrame engine for multimodal data.
Geneva is a feature engineering platform that handles the full lifecycle.

The difference shows up in production requirements:
- **Daft**: Great for one-shot batch jobs, requires custom code for production ops
- **Geneva**: Built-in checkpointing, incremental refresh, versioning, resume

## Summary

| Approach | Lines of Code |
|----------|---------------|
| Daft | ~150 lines for computation + ~100 lines for production ops (custom) |
| Geneva | ~80 lines total, production ops included automatically |

---

## Daft Approach

### Daft Excels At

- Multimodal data processing (images, audio, video)
- Distributed DataFrame API familiar to pandas/polars users
- Rust-powered performance
- GPU-accelerated UDFs

### You'll Need to Add

- Checkpointing for fault tolerance
- Incremental processing logic
- Schema versioning
- Dependency packaging for clusters

---

### UDF Definitions

```python
import daft
from daft import udf, Series, DataType
from daft.io import IOConfig, S3Config
import pyarrow as pa


@udf(return_dtype=DataType.embedding(DataType.float32(), 512), num_gpus=1)
class ComputeEmbeddingDaft:
    """
    CLIP embedding UDF for Daft.

    Similar to Geneva's UDF pattern:
    - Class-based for model caching
    - GPU allocation via decorator
    - Batch processing via Series
    """

    def __init__(self):
        import open_clip
        import torch

        self.model, _, self.preprocess = open_clip.create_model_and_transforms(
            'ViT-B-32', pretrained='laion2b_s34b_b79k'
        )
        self.model = self.model.cuda().eval()
        self.device = "cuda"

    def __call__(self, images: Series) -> list:
        from PIL import Image
        import torch
        import io

        results = []
        image_list = images.to_pylist()

        # Batch process for GPU efficiency
        tensors = []
        for img_bytes in image_list:
            if img_bytes is None:
                tensors.append(None)
                continue
            img = Image.open(io.BytesIO(img_bytes)).convert('RGB')
            tensors.append(self.preprocess(img))

        # Stack valid tensors
        valid_indices = [i for i, t in enumerate(tensors) if t is not None]
        if valid_indices:
            batch = torch.stack([tensors[i] for i in valid_indices]).cuda()
            with torch.no_grad():
                embeddings = self.model.encode_image(batch).cpu().numpy()

            # Map back to results
            embed_idx = 0
            for i in range(len(tensors)):
                if i in valid_indices:
                    results.append(embeddings[embed_idx].tolist())
                    embed_idx += 1
                else:
                    results.append(None)
        else:
            results = [None] * len(image_list)

        return results


@udf(return_dtype=DataType.string(), num_gpus=1)
class GenerateCaptionDaft:
    """
    BLIP captioning UDF for Daft.
    """

    def __init__(self):
        from transformers import BlipProcessor, BlipForConditionalGeneration
        import torch

        self.processor = BlipProcessor.from_pretrained(
            "Salesforce/blip-image-captioning-base"
        )
        self.model = BlipForConditionalGeneration.from_pretrained(
            "Salesforce/blip-image-captioning-base"
        ).cuda().eval()

    def __call__(self, images: Series) -> list:
        from PIL import Image
        import torch
        import io

        results = []
        for img_bytes in images.to_pylist():
            if img_bytes is None:
                results.append(None)
                continue

            try:
                img = Image.open(io.BytesIO(img_bytes)).convert('RGB')
                inputs = self.processor(img, return_tensors="pt")
                inputs = {k: v.cuda() for k, v in inputs.items()}

                with torch.no_grad():
                    output = self.model.generate(**inputs, max_new_tokens=50)

                caption = self.processor.decode(output[0], skip_special_tokens=True)
                results.append(caption)
            except Exception as e:
                results.append(None)

        return results
```

### Basic Pipeline (no production ops)

```python
def run_daft_pipeline_basic():
    """
    Basic Daft pipeline - clean and simple for one-shot batch jobs.

    This is where Daft shines: simple, expressive DataFrame API.
    """
    # Configure Ray cluster (assumes already running)
    daft.context.set_runner_ray(address="auto")

    # Read data
    df = daft.read_parquet("s3://my-bucket/images/")

    # Add computed columns
    df = df.with_column(
        "embedding",
        ComputeEmbeddingDaft(df["image"])
    )

    df = df.with_column(
        "caption",
        GenerateCaptionDaft(df["image"])
    )

    # Write results
    df.write_parquet("s3://my-bucket/embeddings/")

    print("Pipeline complete!")
```

### Production Pipeline (with custom ops)

For production, you need to add:
1. **Checkpointing** - Resume from failures
2. **Incremental processing** - Only process new data
3. **Progress tracking** - Know where you are
4. **Error handling** - Don't lose work on failures

Daft doesn't provide these out of the box.

```python
import json
from pathlib import Path
from datetime import datetime
import hashlib


class DaftCheckpointManager:
    """
    Custom checkpoint manager for Daft pipelines.

    You build this yourself - Daft doesn't include it.
    """

    def __init__(self, checkpoint_dir: str):
        self.checkpoint_dir = Path(checkpoint_dir)
        self.checkpoint_dir.mkdir(parents=True, exist_ok=True)
        self.state_file = self.checkpoint_dir / "state.json"
        self.state = self._load_state()

    def _load_state(self) -> dict:
        if self.state_file.exists():
            return json.loads(self.state_file.read_text())
        return {
            "processed_files": [],
            "last_checkpoint": None,
            "total_rows": 0,
        }

    def _save_state(self):
        self.state["last_checkpoint"] = datetime.now().isoformat()
        self.state_file.write_text(json.dumps(self.state, indent=2))

    def mark_file_processed(self, file_path: str, row_count: int):
        self.state["processed_files"].append(file_path)
        self.state["total_rows"] += row_count
        self._save_state()

    def get_unprocessed_files(self, all_files: list[str]) -> list[str]:
        processed = set(self.state["processed_files"])
        return [f for f in all_files if f not in processed]

    def get_stats(self) -> dict:
        return self.state.copy()


class DaftIncrementalProcessor:
    """
    Incremental processing wrapper for Daft.

    Handles:
    - Tracking which files have been processed
    - Processing only new files
    - Checkpointing progress

    You build this yourself - Daft doesn't include it.
    """

    def __init__(
        self,
        input_path: str,
        output_path: str,
        checkpoint_dir: str,
    ):
        self.input_path = input_path
        self.output_path = output_path
        self.checkpoint = DaftCheckpointManager(checkpoint_dir)

    def _list_input_files(self) -> list[str]:
        """List all parquet files in input path."""
        # In practice, use fsspec or boto3 for S3
        import subprocess
        result = subprocess.run(
            ["aws", "s3", "ls", self.input_path, "--recursive"],
            capture_output=True, text=True
        )
        files = []
        for line in result.stdout.strip().split("\n"):
            if line.endswith(".parquet"):
                parts = line.split()
                files.append(f"{self.input_path}{parts[-1]}")
        return files

    def run(self):
        """Run incremental processing."""
        # Get unprocessed files
        all_files = self._list_input_files()
        pending_files = self.checkpoint.get_unprocessed_files(all_files)

        print(f"Found {len(pending_files)} new files to process")
        print(f"Already processed: {len(all_files) - len(pending_files)} files")

        if not pending_files:
            print("Nothing to process!")
            return

        # Process each file with checkpointing
        for i, file_path in enumerate(pending_files):
            print(f"Processing file {i+1}/{len(pending_files)}: {file_path}")

            try:
                # Read single file
                df = daft.read_parquet(file_path)

                # Compute features
                df = df.with_column(
                    "embedding",
                    ComputeEmbeddingDaft(df["image"])
                )
                df = df.with_column(
                    "caption",
                    GenerateCaptionDaft(df["image"])
                )

                # Collect to get row count
                result = df.collect()
                row_count = len(result)

                # Write to output (append mode via unique filename)
                output_file = f"{self.output_path}/batch_{i:06d}.parquet"
                result.write_parquet(output_file)

                # Checkpoint
                self.checkpoint.mark_file_processed(file_path, row_count)
                print(f"  Processed {row_count} rows, checkpointed")

            except Exception as e:
                print(f"  ERROR: {e}")
                print(f"  Stopping. Resume from checkpoint to continue.")
                raise

        stats = self.checkpoint.get_stats()
        print(f"Complete! Total rows processed: {stats['total_rows']}")


def run_daft_pipeline_production():
    """
    Production Daft pipeline with checkpointing and incremental processing.

    Note: This requires ~100 lines of custom infrastructure code
    that you write and maintain yourself.
    """
    daft.context.set_runner_ray(address="auto")

    processor = DaftIncrementalProcessor(
        input_path="s3://my-bucket/images/",
        output_path="s3://my-bucket/embeddings/",
        checkpoint_dir="/tmp/daft_checkpoints/",
    )

    processor.run()
```

### Dependency Management

Daft on Ray clusters requires manual dependency setup:
- Option 1: Docker image with all dependencies
- Option 2: Ray runtime_env
- Option 3: Conda environment

You configure this yourself - Daft doesn't package dependencies automatically.

```python
def configure_daft_with_dependencies():
    """Configure Ray with required dependencies."""
    import ray

    runtime_env = {
        "pip": [
            "torch>=2.1.0",
            "open-clip-torch>=2.24.0",
            "transformers>=4.36.0",
            "Pillow>=10.0.0",
            "getdaft",  # Daft itself
        ],
        # Or use a pre-built conda env
        # "conda": "path/to/environment.yml",
    }

    ray.init(address="auto", runtime_env=runtime_env)
    daft.context.set_runner_ray()
```

---

## Geneva Approach

Geneva provides:
- Same UDF simplicity as Daft
- Built-in checkpointing at fragment AND batch level
- Automatic incremental processing
- Dependency isolation per UDF
- Table versioning
- No custom infrastructure code needed

### UDF Definitions

```python
import geneva


@geneva.udf(num_gpus=1, version="0.1")
class ComputeEmbeddingGeneva:
    """
    CLIP embedding UDF for Geneva.

    Nearly identical to Daft UDF, but:
    - Type annotation -> schema (no return_dtype needed)
    - setup() instead of __init__() (called once per worker)
    - Returns single value, Geneva handles batching
    """

    def setup(self):
        import open_clip
        self.model, _, self.preprocess = open_clip.create_model_and_transforms(
            'ViT-B-32', pretrained='laion2b_s34b_b79k'
        )
        self.model = self.model.cuda().eval()

    def __call__(self, image: bytes) -> list[float]:
        from PIL import Image
        import torch
        import io

        img = Image.open(io.BytesIO(image)).convert('RGB')
        tensor = self.preprocess(img).unsqueeze(0).cuda()
        with torch.no_grad():
            embedding = self.model.encode_image(tensor)
        return embedding.cpu().numpy().flatten().tolist()


@geneva.udf(num_gpus=1, version="0.1")
class GenerateCaptionGeneva:
    """BLIP captioning UDF for Geneva."""

    def setup(self):
        from transformers import BlipProcessor, BlipForConditionalGeneration
        self.processor = BlipProcessor.from_pretrained(
            "Salesforce/blip-image-captioning-base"
        )
        self.model = BlipForConditionalGeneration.from_pretrained(
            "Salesforce/blip-image-captioning-base"
        ).cuda()

    def __call__(self, image: bytes) -> str:
        from PIL import Image
        import torch
        import io

        img = Image.open(io.BytesIO(image)).convert('RGB')
        inputs = self.processor(img, return_tensors="pt")
        inputs = {k: v.cuda() for k, v in inputs.items()}
        with torch.no_grad():
            output = self.model.generate(**inputs, max_new_tokens=50)
        return self.processor.decode(output[0], skip_special_tokens=True)
```

### Pipeline

```python
def run_geneva_pipeline():
    """
    Geneva pipeline - same simplicity as Daft basic, but production-ready.

    No custom checkpoint code needed.
    No incremental processing code needed.
    No dependency management code needed.
    """
    conn = geneva.connect("s3://my-bucket/geneva-workspace")
    table = conn.open_table("images")

    # Add computed columns
    table.add_columns({
        "embedding": ComputeEmbeddingGeneva,
        "caption": GenerateCaptionGeneva,
    })

    # Run with automatic checkpointing, incremental processing, versioning
    table.backfill("embedding", batch_size=100, commit_granularity=10)
    table.backfill("caption", batch_size=50, commit_granularity=10)

    print(f"Processed {table.count_rows()} images")
    print(f"Table version: {table.version}")


def run_geneva_incremental():
    """
    Incremental processing in Geneva.

    Add new data -> backfill -> only new rows are processed.
    No custom code needed.
    """
    conn = geneva.connect("s3://my-bucket/geneva-workspace")
    table = conn.open_table("images")

    # Add new images
    new_images = load_new_images()  # Your data loading
    table.add(new_images)

    # Backfill only processes rows where embedding IS NULL
    # (i.e., the new rows). Existing embeddings are untouched.
    table.backfill("embedding", batch_size=100)
    table.backfill("caption", batch_size=50)

    print("Incremental update complete!")


def run_geneva_with_resume():
    """
    Resume from failure in Geneva.

    If a job fails at row 999,000 of 1,000,000:
    - Re-run the same backfill command
    - Geneva detects checkpoint, resumes from row 999,000
    - No custom checkpoint code needed
    """
    conn = geneva.connect("s3://my-bucket/geneva-workspace")
    table = conn.open_table("images")

    table.add_columns({
        "embedding": ComputeEmbeddingGeneva,
    })

    # If this fails partway through...
    try:
        table.backfill("embedding", batch_size=100, commit_granularity=10)
    except Exception as e:
        print(f"Job failed: {e}")
        print("Run the same command again to resume from checkpoint")

    # ...just run again. Geneva resumes automatically.
    table.backfill("embedding", batch_size=100, commit_granularity=10)
    # ^ Only processes remaining rows, not from scratch
```

---

## Side-by-Side Comparison

### Feature Comparison

| Feature | Daft | Geneva |
|---------|------|--------|
| DataFrame API | Excellent | Table-centric |
| Multimodal native | Yes | Yes |
| GPU UDFs | num_gpus=1 | num_gpus=1 |
| Class-based UDFs | __init__/__call__ | setup()/__call__ |
| Distributed execution | Ray backend | Ray backend |
| **Row expansion (1:N)** | `explode()` (arrays only, no custom logic) | `@scalar_udtf` (custom generator, per-row compute) |
| **Batch transforms (N:M)** | Not included | `@udtf` (dedup, clustering, aggregation) |
| **Checkpointing** | Build yourself | Automatic |
| **Incremental processing** | Build yourself | Automatic |
| **Resume from failure** | Build yourself | Automatic |
| **Dependency isolation** | Docker/runtime_env | Per-UDF automatic |
| **Table versioning** | Not included | Automatic |
| **Schema from type hints** | return_dtype req | Automatic |

### Code Comparison

| Component | Daft | Geneva |
|-----------|------|--------|
| UDF definitions | ~60 lines | ~50 lines |
| Basic pipeline | ~15 lines | ~15 lines |
| Checkpoint system | ~60 lines (custom) | 0 (included) |
| Incremental processor | ~80 lines (custom) | 0 (included) |
| Dependency config | ~20 lines | pyproject.toml |
| **TOTAL (production)** | **~235 lines** | **~65 lines** |

---

## Row Expansion & Table Transforms: Daft `explode()` vs Geneva UDTFs

Daft has `explode()` for expanding list columns into rows — the same pattern as
Polars and Pandas. Geneva has two UDTF types that go significantly further.

### Daft `explode()`

Daft's `explode()` operates on **pre-computed array columns**. It flattens a
list column into individual rows, preserving other columns:

```python
import daft

df = daft.from_pydict({
    "video_path": ["/a.mp4", "/b.mp4"],
    "tags": [["action", "comedy"], ["drama"]],
})

# Explode list column to rows — no custom logic, arrays only
df.explode("tags")
# video_path | tags
# /a.mp4     | action
# /a.mp4     | comedy
# /b.mp4     | drama
```

**What `explode()` can do:**
- Flatten existing list/array columns to rows
- Preserve parent columns automatically
- Chain with other DataFrame operations

**What `explode()` cannot do:**
- Custom computation (the array must already exist)
- Generate rows from non-array data (e.g., split a video file into clips)
- N:M transformations (dedup, clustering, aggregation)

### Geneva `@scalar_udtf` — Custom 1:N Expansion

Geneva's scalar UDTFs **compute** the expansion — the function runs per-row and
yields output rows. This is fundamentally different from exploding a pre-computed
array.

```python
from typing import Iterator, NamedTuple
from geneva import scalar_udtf

class Clip(NamedTuple):
    clip_start: float
    clip_end: float
    clip_bytes: bytes

@scalar_udtf
def extract_clips(video_path: str, duration: float) -> Iterator[Clip]:
    """Per-row compute: reads video file, extracts and yields clip segments."""
    clip_length = 10.0
    for start in range(0, int(duration), int(clip_length)):
        end = min(start + clip_length, duration)
        clip_data = extract_video_segment(video_path, start, end)
        yield Clip(clip_start=start, clip_end=end, clip_bytes=clip_data)

clips = db.create_materialized_view(
    "clips",
    query=videos.search(None).select(["video_path", "duration"]),
    scalar_udtf=extract_clips,
)
clips.refresh()  # Distributed, checkpointed, incremental
```

### Geneva `@udtf` — N:M Batch Transforms

Batch UDTFs handle cross-row operations that neither Daft's `explode()` nor
any DataFrame API can express as a column operation:

```python
@geneva.udtf(
    output_schema=pa.schema([
        pa.field("row_id", pa.int64()),
        pa.field("cluster_id", pa.int64()),
        pa.field("is_duplicate", pa.bool_()),
    ]),
    partition_by_indexed_column="phash",
    num_cpus=4,
)
class PHashDedupe:
    def __init__(self, threshold: int = 4):
        self.threshold = threshold

    def __call__(self, source: geneva.GenevaQueryBuilder) -> Iterator[pa.RecordBatch]:
        data = source.to_arrow()
        clusters = self._cluster(data)
        yield pa.RecordBatch.from_pydict({...})
```

### Comparison

| Capability | Daft `explode()` | Geneva `@scalar_udtf` | Geneva `@udtf` |
|-----------|-----------------|----------------------|----------------|
| **Operation** | Flatten existing array | Per-row compute → yield N rows | Whole-table/partition → yield M rows |
| **Custom logic** | ❌ Arrays only | ✅ Any generator | ✅ Any computation |
| **Cardinality** | 1:N (array length) | 1:N (yield count) | N:M (arbitrary) |
| **Use cases** | Pre-computed tags, labels | Video→clips, doc→chunks, image→tiles | Dedup, clustering, aggregation |
| **Incremental** | N/A (one-shot) | ✅ MV-style refresh | ✅ Version-aware refresh |
| **Checkpointing** | N/A | ✅ Per-batch | ✅ Per-partition + per-batch |
| **Distributed** | Via Daft/Ray | Via Ray ActorPool | Via Ray ActorPool |

**The key difference:** Daft's `explode()` is a **data reshaping** operation —
the array already exists. Geneva's UDTFs are **compute** operations — the
function produces the expanded rows. This means Geneva can express chunking,
splitting, deduplication, and clustering as first-class table operations with
full lifecycle management (checkpointing, incremental refresh, versioning).

In Daft, to achieve the same thing, you'd first compute the array column via
a UDF, then `explode()` it — two separate steps with no lifecycle management.
In Geneva, it's one operation with built-in production ops.

---

## When to Use Each

### Daft is great for:
- Ad-hoc data exploration and analysis
- One-shot batch processing jobs
- Teams that want DataFrame semantics
- Workloads where you control the full pipeline
- When you already have checkpoint/incremental infrastructure

### Geneva is great for:
- Production feature engineering pipelines
- Ongoing incremental updates (new data daily)
- Teams that want to focus on ML, not infrastructure
- When fault tolerance and resume are critical
- Multi-team environments (admin + researchers)

---

## The Key Insight

**Daft asks**: "How do I compute this transformation?"
**Geneva asks**: "How do I maintain this feature over time?"

Both compute embeddings efficiently. Geneva also handles:
- What if the job fails at 3am? → Resume from exact batch
- What if new data arrives tomorrow? → Incremental processing
- What if I need to recompute with a new model? → UDF version tracking
- What if another team needs these features? → Table versioning

These production concerns are where Geneva shines.

---

## Addressing the "Library vs Platform" Critique

**Critics say**: "You're comparing Daft (library) + custom ops vs Geneva (platform).
That's not fair—the line counts aren't comparable."

**Response**: Yes, exactly. That's the real choice teams face.

| Approach | Trade-off |
|----------|-----------|
| **Option A:** Daft (library) + build your own production ops | More flexibility, more code to write and maintain. You own the checkpoint logic, incremental logic, versioning. |
| **Option B:** Geneva (platform) with production ops included | More opinionated, less code to write and maintain. Trade flexibility for velocity. |

**The analogy:** Django vs Flask + SQLAlchemy + Alembic + Flask-Login + ...

- Django "hides complexity"—yes, that's the value proposition
- Flask gives more control—yes, that's its value proposition

**Geneva is Django for feature engineering:**
- Opinionated defaults that work for 90% of use cases
- Integrated components that work together
- If you need maximum flexibility, use Daft directly
- If you want to ship features faster, use Geneva

---

## Why "Checkpointing Isn't Hard" Misses the Point

**Critics say**: "Your 60-line checkpoint manager is easy to write once and reuse."

**Reality**: Checkpoints aren't just for failure recovery—they enable INCREMENTAL
PROCESSING, which matters for every production pipeline.

```
Day 1: 1M images → compute 1M embeddings
Day 2: 10K new images added

Without Geneva:
  Option A: Recompute all 1.01M (wasteful, expensive)
  Option B: Build custom "what's new?" detection + checkpoint logic

With Geneva:
  table.backfill("embedding")  # Only computes 10K new rows
```

The checkpoint system tracks:
1. What's been computed (failure recovery)
2. What's new (incremental processing)
3. What changed upstream (source mutation detection)
4. Which UDF version was used (model versioning)

It's not "just checkpointing"—it's the foundation for maintaining features
as data evolves. That's harder to "write once and reuse" than it looks.

---

## Architecture

### Daft Architecture (you build the boxes in gray)

**Your Code** (you build these):

| Checkpoint Store | Incremental Processor | Progress Tracking | Error Handling |
|------------------|----------------------|-------------------|----------------|
| (custom) | (custom) | (custom) | (custom) |

↓

**Daft**

| DataFrame API | UDF System | GPU Scheduling | Distributed Execution |
|---------------|------------|----------------|----------------------|

↓

**Ray Cluster**

### Geneva Architecture (everything included)

**Your Code**

| UDFs (ML logic) | Pipeline (backfill) |
|-----------------|---------------------|

↓

**Geneva**

| Checkpoint Store | Incremental Processor | Progress Tracking | Error Handling | Version Control |
|------------------|----------------------|-------------------|----------------|-----------------|

| UDF System | GPU Scheduling | Distributed Execution | Dependency Isolation |
|------------|----------------|----------------------|---------------------|

↓

**Ray Cluster + Lance Storage**
