# Kubeflow Pipelines vs Geneva: Image Embedding & Captioning Pipeline

This file compares Kubeflow Pipelines (KFP) with Geneva
for building production image embedding pipelines.

## Key Insight

Kubeflow Pipelines is a workflow orchestration platform for ML on Kubernetes.
Geneva is a feature engineering platform with built-in data management.

**The fundamental difference:**
- **KFP**: "Define and orchestrate ML workflow steps as containers"
- **Geneva**: "Define feature transformations, we handle the workflow"

## Summary

| Approach | Lines of Code |
|----------|---------------|
| KFP | ~350 lines for pipeline + component definitions + Kubernetes config |
| Geneva | ~60 lines total, no Kubernetes expertise needed |

---

## Kubeflow Pipelines Approach

### Kubeflow Pipelines Excels At

- Complex multi-step ML workflows (train→evaluate→deploy)
- Kubernetes-native execution
- Pipeline versioning and experiment tracking
- Component reusability across pipelines
- Visual pipeline DAG editor

### You'll Need to Handle

- Container builds for each component
- Kubernetes cluster management
- Manual checkpointing between steps
- Custom data passing between components
- GPU scheduling via Kubernetes resources

---

### components/compute_embeddings.py

Kubeflow component for computing CLIP embeddings.
Each component is a containerized step in the pipeline.
You build, push, and version each container separately.

```python
from kfp import dsl
from kfp.dsl import Input, Output, Dataset, Model, Artifact
from typing import NamedTuple


@dsl.component(
    base_image="python:3.11",
    packages_to_install=[
        "torch>=2.1.0",
        "open-clip-torch>=2.24.0",
        "pyarrow>=14.0.0",
        "Pillow>=10.0.0",
    ],
)
def compute_embeddings(
    input_data: Input[Dataset],
    output_data: Output[Dataset],
    checkpoint: Input[Artifact],
    checkpoint_out: Output[Artifact],
    batch_size: int = 100,
):
    """
    Compute CLIP embeddings for images.

    This component runs in a container with GPU access.
    """
    import json
    import torch
    import open_clip
    from PIL import Image
    import pyarrow.parquet as pq
    import pyarrow as pa
    import io
    from pathlib import Path

    # Load checkpoint state
    processed_ids = set()
    checkpoint_path = Path(checkpoint.path) if checkpoint else None
    if checkpoint_path and checkpoint_path.exists():
        with open(checkpoint_path) as f:
            processed_ids = set(json.load(f).get("processed_ids", []))

    # Load model
    model, _, preprocess = open_clip.create_model_and_transforms(
        'ViT-B-32', pretrained='laion2b_s34b_b79k'
    )
    device = "cuda" if torch.cuda.is_available() else "cpu"
    model = model.to(device).eval()

    # Read input data
    table = pq.read_table(input_data.path)

    # Filter to unprocessed rows
    ids = []
    embeddings = []

    for i in range(len(table)):
        row_id = str(table['id'][i].as_py())
        if row_id in processed_ids:
            continue

        image_bytes = table['image'][i].as_py()
        try:
            img = Image.open(io.BytesIO(image_bytes)).convert('RGB')
            tensor = preprocess(img).unsqueeze(0).to(device)
            with torch.no_grad():
                embedding = model.encode_image(tensor)
            ids.append(row_id)
            embeddings.append(embedding.cpu().numpy().flatten().tolist())
            processed_ids.add(row_id)
        except Exception as e:
            print(f"Error processing {row_id}: {e}")
            ids.append(row_id)
            embeddings.append(None)

        if len(ids) % batch_size == 0:
            print(f"Progress: {len(ids)} rows processed")

    # Write output
    output_table = pa.table({
        'id': pa.array(ids),
        'embedding': pa.array(embeddings),
    })
    pq.write_table(output_table, output_data.path)

    # Save checkpoint
    with open(checkpoint_out.path, 'w') as f:
        json.dump({"processed_ids": list(processed_ids)}, f)

    print(f"Complete! Processed {len(ids)} embeddings")


@dsl.component(
    base_image="python:3.11",
    packages_to_install=[
        "torch>=2.1.0",
        "transformers>=4.36.0",
        "pyarrow>=14.0.0",
        "Pillow>=10.0.0",
    ],
)
def compute_captions(
    input_data: Input[Dataset],
    output_data: Output[Dataset],
    checkpoint: Input[Artifact],
    checkpoint_out: Output[Artifact],
    batch_size: int = 50,
):
    """
    Generate BLIP captions for images.

    Separate component with different dependencies.
    """
    import json
    import torch
    from transformers import BlipProcessor, BlipForConditionalGeneration
    from PIL import Image
    import pyarrow.parquet as pq
    import pyarrow as pa
    import io
    from pathlib import Path

    # Load checkpoint state
    processed_ids = set()
    checkpoint_path = Path(checkpoint.path) if checkpoint else None
    if checkpoint_path and checkpoint_path.exists():
        with open(checkpoint_path) as f:
            processed_ids = set(json.load(f).get("processed_ids", []))

    # Load model
    processor = BlipProcessor.from_pretrained(
        "Salesforce/blip-image-captioning-base"
    )
    model = BlipForConditionalGeneration.from_pretrained(
        "Salesforce/blip-image-captioning-base"
    )
    device = "cuda" if torch.cuda.is_available() else "cpu"
    model = model.to(device).eval()

    # Read input data
    table = pq.read_table(input_data.path)

    # Process
    ids = []
    captions = []

    for i in range(len(table)):
        row_id = str(table['id'][i].as_py())
        if row_id in processed_ids:
            continue

        image_bytes = table['image'][i].as_py()
        try:
            img = Image.open(io.BytesIO(image_bytes)).convert('RGB')
            inputs = processor(img, return_tensors="pt").to(device)
            with torch.no_grad():
                output = model.generate(**inputs, max_new_tokens=50)
            caption = processor.decode(output[0], skip_special_tokens=True)
            ids.append(row_id)
            captions.append(caption)
            processed_ids.add(row_id)
        except Exception as e:
            print(f"Error processing {row_id}: {e}")
            ids.append(row_id)
            captions.append(None)

        if len(ids) % batch_size == 0:
            print(f"Progress: {len(ids)} rows processed")

    # Write output
    output_table = pa.table({
        'id': pa.array(ids),
        'caption': pa.array(captions),
    })
    pq.write_table(output_table, output_data.path)

    # Save checkpoint
    with open(checkpoint_out.path, 'w') as f:
        json.dump({"processed_ids": list(processed_ids)}, f)

    print(f"Complete! Generated {len(ids)} captions")


@dsl.component(
    base_image="python:3.11",
    packages_to_install=["pyarrow>=14.0.0"],
)
def merge_results(
    original_data: Input[Dataset],
    embeddings: Input[Dataset],
    captions: Input[Dataset],
    output_data: Output[Dataset],
):
    """
    Merge embeddings and captions with original data.

    KFP requires explicit merge steps - data doesn't flow automatically.
    """
    import pyarrow.parquet as pq
    import pyarrow as pa

    # Read all data
    original = pq.read_table(original_data.path)
    embed_table = pq.read_table(embeddings.path)
    caption_table = pq.read_table(captions.path)

    # Convert to pandas for easy join (or use pyarrow join)
    import pandas as pd
    orig_df = original.to_pandas()
    embed_df = embed_table.to_pandas()
    caption_df = caption_table.to_pandas()

    # Merge on id
    result = orig_df.merge(embed_df, on='id', how='left')
    result = result.merge(caption_df, on='id', how='left')

    # Write output
    output_table = pa.Table.from_pandas(result)
    pq.write_table(output_table, output_data.path)

    print(f"Merged {len(result)} rows")
```

### pipeline_definition.py

Pipeline definition that orchestrates the components.

```python
@dsl.pipeline(
    name="image-embedding-pipeline",
    description="Compute embeddings and captions for images",
)
def image_embedding_pipeline(
    input_path: str,
    output_path: str,
    embedding_batch_size: int = 100,
    caption_batch_size: int = 50,
):
    """
    Main pipeline definition.

    NOTE: Checkpointing between runs requires:
    1. External checkpoint storage
    2. Manual checkpoint passing
    3. Resume logic in each component
    """
    # Import data
    import_data = dsl.importer(
        artifact_uri=input_path,
        artifact_class=dsl.Dataset,
    )

    # Initialize empty checkpoints (would be loaded from storage for resume)
    init_embed_checkpoint = dsl.importer(
        artifact_uri="gs://my-bucket/checkpoints/embed_checkpoint.json",
        artifact_class=dsl.Artifact,
    )
    init_caption_checkpoint = dsl.importer(
        artifact_uri="gs://my-bucket/checkpoints/caption_checkpoint.json",
        artifact_class=dsl.Artifact,
    )

    # Compute embeddings (runs on GPU node)
    embed_task = compute_embeddings(
        input_data=import_data.output,
        checkpoint=init_embed_checkpoint.output,
        batch_size=embedding_batch_size,
    )
    embed_task.set_gpu_limit(1)
    embed_task.set_cpu_limit("4")
    embed_task.set_memory_limit("16G")

    # Compute captions (runs on GPU node, depends on embeddings for ordering)
    caption_task = compute_captions(
        input_data=import_data.output,
        checkpoint=init_caption_checkpoint.output,
        batch_size=caption_batch_size,
    )
    caption_task.set_gpu_limit(1)
    caption_task.set_cpu_limit("4")
    caption_task.set_memory_limit("16G")
    caption_task.after(embed_task)  # Ordering dependency

    # Merge results
    merge_task = merge_results(
        original_data=import_data.output,
        embeddings=embed_task.outputs["output_data"],
        captions=caption_task.outputs["output_data"],
    )

    # Export results
    # (would need additional component to write to final destination)
```

### compile_and_run.py

Compile and submit the pipeline.

```python
from kfp import compiler
from kfp.client import Client


def compile_pipeline():
    """Compile pipeline to YAML."""
    compiler.Compiler().compile(
        pipeline_func=image_embedding_pipeline,
        package_path="image_embedding_pipeline.yaml",
    )
    print("Pipeline compiled to: image_embedding_pipeline.yaml")


def run_pipeline(
    host: str,
    input_path: str,
    output_path: str,
):
    """Submit pipeline to KFP cluster."""
    client = Client(host=host)

    run = client.create_run_from_pipeline_func(
        image_embedding_pipeline,
        arguments={
            "input_path": input_path,
            "output_path": output_path,
            "embedding_batch_size": 100,
            "caption_batch_size": 50,
        },
        experiment_name="image-embeddings",
        run_name="embedding-run-001",
    )

    print(f"Pipeline run submitted: {run.run_id}")
    return run
```

### kubernetes/gpu-node-pool.yaml

Kubernetes configuration for GPU nodes (separate from KFP).
You manage this infrastructure.

```yaml
apiVersion: v1
kind: NodePool
metadata:
  name: gpu-pool
spec:
  machineType: n1-standard-8
  acceleratorType: nvidia-tesla-t4
  acceleratorCount: 1
  minNodes: 0
  maxNodes: 10
  autoscaling: true
```

---

## What Kubeflow Pipelines Requires (that Geneva handles automatically)

### Kubeflow Pipelines Operational Requirements

1. **Kubernetes Cluster Management:**
   - Set up and maintain K8s cluster
   - Configure GPU node pools
   - Manage cluster autoscaling
   - Handle node failures and rescheduling

2. **KFP Installation & Maintenance:**
   - Install KFP on cluster
   - Manage KFP version upgrades
   - Configure persistent storage
   - Set up authentication

3. **Container Management:**
   - Build containers for each component
   - Push to container registry
   - Manage image versions
   - Handle dependency conflicts between components

4. **Pipeline Configuration:**
   - Define resource limits per component
   - Configure GPU scheduling
   - Set up artifact storage
   - Manage secrets and credentials

5. **Checkpointing:**
   - Design checkpoint strategy
   - Implement checkpoint loading/saving per component
   - Handle checkpoint storage
   - Resume logic for each step

6. **Data Passing:**
   - Configure artifact storage
   - Handle large data serialization
   - Merge results from parallel steps
   - Schema management between components

**Lines of code:** 300-400 lines + Kubernetes YAML + cluster management

---

## Geneva Approach

Geneva provides all of the above automatically:
- No Kubernetes cluster management (uses Ray)
- No container builds (dependencies from pyproject.toml)
- Automatic checkpointing at fragment + batch level
- Automatic data management
- No pipeline DAG definition needed

### UDF Definitions

```python
import geneva


@geneva.udf(num_gpus=1, version="0.1")
class ComputeEmbeddingGeneva:
    """
    CLIP embedding UDF for Geneva.

    Compare to KFP:
    - No container build
    - No Kubernetes resource spec
    - No checkpoint code
    - No data passing code
    """

    def setup(self):
        import open_clip
        self.model, _, self.preprocess = open_clip.create_model_and_transforms(
            'ViT-B-32', pretrained='laion2b_s34b_b79k'
        )
        self.model = self.model.cuda().eval()

    def __call__(self, image: bytes) -> list[float]:
        from PIL import Image
        import torch
        import io

        img = Image.open(io.BytesIO(image)).convert('RGB')
        tensor = self.preprocess(img).unsqueeze(0).cuda()
        with torch.no_grad():
            embedding = self.model.encode_image(tensor)
        return embedding.cpu().numpy().flatten().tolist()


@geneva.udf(num_gpus=1, version="0.1")
class GenerateCaptionGeneva:
    """BLIP captioning UDF for Geneva."""

    def setup(self):
        from transformers import BlipProcessor, BlipForConditionalGeneration
        self.processor = BlipProcessor.from_pretrained(
            "Salesforce/blip-image-captioning-base"
        )
        self.model = BlipForConditionalGeneration.from_pretrained(
            "Salesforce/blip-image-captioning-base"
        ).cuda()

    def __call__(self, image: bytes) -> str:
        from PIL import Image
        import torch
        import io

        img = Image.open(io.BytesIO(image)).convert('RGB')
        inputs = self.processor(img, return_tensors="pt")
        inputs = {k: v.cuda() for k, v in inputs.items()}
        with torch.no_grad():
            output = self.model.generate(**inputs, max_new_tokens=50)
        return self.processor.decode(output[0], skip_special_tokens=True)
```

### Pipeline

```python
def run_geneva_pipeline():
    """
    Geneva pipeline - no DAG definition needed.

    No container builds.
    No Kubernetes YAML.
    No checkpoint code.
    No merge steps.
    """
    conn = geneva.connect("s3://my-bucket/geneva-workspace")
    table = conn.open_table("images")

    # Add computed columns
    table.add_columns({
        "embedding": ComputeEmbeddingGeneva,
        "caption": GenerateCaptionGeneva,
    })

    # Run with automatic checkpointing and data management
    table.backfill("embedding", batch_size=100, commit_granularity=10)
    table.backfill("caption", batch_size=50, commit_granularity=10)

    print(f"Processed {table.count_rows()} images")
    print(f"Table version: {table.version}")
```

---

## Side-by-Side Comparison

### Feature Comparison

| Feature | Kubeflow Pipelines | Geneva |
|---------|-------------------|--------|
| Visual pipeline DAG | Built-in UI | Code-first |
| Experiment tracking | Built-in | Via table versions |
| Multi-step workflows | Core strength | Table-centric |
| Component reusability | Across pipelines | UDF reusability |
| Kubernetes native | Yes | Ray-based |
| **Container management** | Per component | Automatic |
| **Cluster management** | Your responsibility | Via Ray |
| **Checkpointing** | Per component | Automatic |
| **Incremental processing** | Build yourself | Automatic |
| **Data passing** | Artifact config | Table columns |
| **Resume from failure** | Step-level only | Batch-level |
| **GPU scheduling** | Via K8s resources | num_gpus=1 |

### Code + Configuration Comparison

| Component | Kubeflow Pipelines | Geneva |
|-----------|-------------------|--------|
| Processing components | ~150 lines | ~45 lines (UDFs) |
| Pipeline definition | ~60 lines | ~15 lines |
| Merge/data passing | ~40 lines | 0 (automatic) |
| Compile/run script | ~30 lines | 0 (not needed) |
| Kubernetes config | ~50 lines YAML | 0 (not needed) |
| Checkpoint logic | ~40 lines/component | 0 (automatic) |
| **TOTAL** | **~370 lines + K8s** | **~60 lines** |

### Failure Recovery Comparison

| Scenario | Kubeflow Pipelines | Geneva |
|----------|-------------------|--------|
| Step fails after processing 500K of 1M rows | Restart from step beginning or custom checkpoint logic | Resume from exact row where it failed |
| GPU node preempted | Step restarts from beginning | Resume from last committed batch |
| Pod OOM killed | Step fails, manual restart | Retry with smaller batch automatically |

---

## When to Use Each

### Kubeflow Pipelines is great for:
- Complex ML workflows (train → evaluate → deploy)
- Teams already running Kubernetes
- When you need visual pipeline editor
- Multi-step workflows with different frameworks
- When component reusability across pipelines is important
- CI/CD integration for ML models

### Geneva is great for:
- Feature engineering specifically
- Teams without Kubernetes expertise
- When fault tolerance down to batch level matters
- Ongoing incremental feature updates
- When you want to avoid container management
- Simpler "compute feature columns" use case

---

## The Key Insight

**Kubeflow Pipelines thinks in workflow steps**: "Run Step A, then Step B, then Step C"
**Geneva thinks in feature columns**: "This table has these computed features"

KFP is powerful for orchestrating complex ML workflows.
Geneva is focused on one thing: maintaining feature columns over time.

For a train → evaluate → deploy workflow, KFP is the right tool.
For "compute embeddings on my image table and keep them updated," Geneva is simpler.

**Think of it this way:**
- KFP: Workflow orchestration engine
- Geneva: Feature column manager

**The abstraction level is different:**
- KFP: You define the workflow graph
- Geneva: You define the features, workflow is implicit

**One concrete example:**
- KFP requires explicit merge steps to combine results
- Geneva columns are automatically part of the table

---

## Architecture

### Kubeflow Pipelines Architecture (you manage all boxes)

**Your Code** (you manage all these):

| Component 1 (Container) | Component 2 (Container) | Component 3 (Container) | Pipeline Definition |
|-------------------------|-------------------------|-------------------------|---------------------|

| Checkpoint Logic | Merge Logic | K8s Config |
|------------------|-------------|------------|

↓

**Kubeflow Pipelines**

| Pipeline Executor | Artifact Store | Experiment Tracking | Visual Editor |
|-------------------|----------------|---------------------|---------------|

↓

**Kubernetes Cluster** *(You manage cluster scaling, node pools, GPU quotas)*

| Pods | Nodes | GPU Scheduling | Storage (PV/PVC) |
|------|-------|----------------|------------------|

### Geneva Architecture (everything included)

**Your Code**

| UDFs (ML logic) | Pipeline (backfill) |
|-----------------|---------------------|

↓

**Geneva**

| Dep Packaging | Checkpoint Store | Incremental Processor | Data Merging | Version Control |
|---------------|------------------|----------------------|--------------|-----------------|

| UDF System | GPU Scheduling | Distributed Execution | Error Handling |
|------------|----------------|----------------------|----------------|

↓

**Ray Cluster + Lance Storage** *(Managed via KubeRay or cloud Ray services)*

### Workflow Comparison

**KFP Workflow (you define):**

| Step | Load Data → | Compute Embedding → | Compute Captions → | Merge Results |
|------|-------------|---------------------|--------------------|---------------|
| Storage | Input Storage | Checkpoint File | Checkpoint File | Output Storage |

**Geneva Workflow (implicit):**

**Table: images**

| id (key) | image (input) | metadata (input) | embedding (computed) | caption (computed) |
|----------|---------------|------------------|---------------------|-------------------|
| ... | ... | ... | `backfill()` | `backfill()` |

*No separate load/merge steps - columns are part of the table.*
