# Cloud Managed ML Services vs Geneva: Image Embedding & Captioning Pipeline

This file compares cloud managed ML services (AWS SageMaker, Google Vertex AI,
Azure ML) with Geneva for building production image embedding pipelines.

## Key Insight

Cloud ML services are general-purpose ML platforms - training, inference, deployment.
Geneva is purpose-built for feature engineering on multimodal data.

**The fundamental difference:**
- **Cloud ML**: "Run any ML workload in a managed environment"
- **Geneva**: "Compute and maintain feature columns over time"

## Summary

| Approach | Lines of Code |
|----------|---------------|
| Cloud ML | ~300-500 lines depending on service, plus console configuration |
| Geneva | ~60 lines total, no console configuration needed |

---

## AWS SageMaker Approach

### SageMaker Processing Jobs Excel At

- Managed infrastructure (no cluster management)
- Integration with AWS ecosystem (S3, IAM, CloudWatch)
- Batch Transform for inference at scale
- Spot instance support for cost savings

### You'll Need to Handle

- Custom container builds
- Manual checkpointing
- No built-in incremental processing
- Complex IAM and networking setup
- Separate code for data management

---

### sagemaker_pipeline/inference.py (runs inside container)

This script runs inside the SageMaker Processing container.
You package this into a Docker image and push to ECR.

```python
import os
import json
import torch
import open_clip
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration
import pyarrow.parquet as pq
import pyarrow as pa
from pathlib import Path


def load_clip_model():
    """Load CLIP model for embeddings."""
    model, _, preprocess = open_clip.create_model_and_transforms(
        'ViT-B-32', pretrained='laion2b_s34b_b79k'
    )
    model = model.cuda().eval()
    return model, preprocess


def load_blip_model():
    """Load BLIP model for captions."""
    processor = BlipProcessor.from_pretrained(
        "Salesforce/blip-image-captioning-base"
    )
    model = BlipForConditionalGeneration.from_pretrained(
        "Salesforce/blip-image-captioning-base"
    ).cuda().eval()
    return model, processor


def process_images(input_dir: str, output_dir: str, checkpoint_dir: str):
    """Main processing function for SageMaker Processing Job."""
    import io

    # Load checkpoint state
    checkpoint_file = Path(checkpoint_dir) / "progress.json"
    processed_files = set()
    if checkpoint_file.exists():
        with open(checkpoint_file) as f:
            processed_files = set(json.load(f).get("processed_files", []))

    # Load models
    print("Loading CLIP model...")
    clip_model, clip_preprocess = load_clip_model()
    print("Loading BLIP model...")
    blip_model, blip_processor = load_blip_model()

    # Process each parquet file
    input_path = Path(input_dir)
    output_path = Path(output_dir)
    output_path.mkdir(parents=True, exist_ok=True)

    for parquet_file in input_path.glob("*.parquet"):
        if parquet_file.name in processed_files:
            print(f"Skipping already processed: {parquet_file.name}")
            continue

        print(f"Processing: {parquet_file.name}")
        table = pq.read_table(parquet_file)

        embeddings = []
        captions = []

        for i in range(len(table)):
            image_bytes = table['image'][i].as_py()

            # Compute embedding
            try:
                img = Image.open(io.BytesIO(image_bytes)).convert('RGB')
                tensor = clip_preprocess(img).unsqueeze(0).cuda()
                with torch.no_grad():
                    embedding = clip_model.encode_image(tensor)
                embeddings.append(embedding.cpu().numpy().flatten().tolist())
            except Exception as e:
                print(f"Embedding error row {i}: {e}")
                embeddings.append(None)

            # Compute caption
            try:
                img = Image.open(io.BytesIO(image_bytes)).convert('RGB')
                inputs = blip_processor(img, return_tensors="pt")
                inputs = {k: v.cuda() for k, v in inputs.items()}
                with torch.no_grad():
                    output = blip_model.generate(**inputs, max_new_tokens=50)
                caption = blip_processor.decode(output[0], skip_special_tokens=True)
                captions.append(caption)
            except Exception as e:
                print(f"Caption error row {i}: {e}")
                captions.append(None)

            if (i + 1) % 100 == 0:
                print(f"  Progress: {i + 1}/{len(table)}")

        # Write output
        output_table = pa.table({
            'id': table['id'],
            'embedding': pa.array(embeddings),
            'caption': pa.array(captions),
        })
        pq.write_table(output_table, output_path / parquet_file.name)

        # Update checkpoint
        processed_files.add(parquet_file.name)
        with open(checkpoint_file, 'w') as f:
            json.dump({"processed_files": list(processed_files)}, f)

    print("Processing complete!")


if __name__ == "__main__":
    # SageMaker paths (configured by the job)
    input_dir = "/opt/ml/processing/input"
    output_dir = "/opt/ml/processing/output"
    checkpoint_dir = "/opt/ml/processing/checkpoint"

    process_images(input_dir, output_dir, checkpoint_dir)
```

### sagemaker_pipeline/run_job.py (local orchestration)

This script launches the SageMaker Processing Job from your local machine
or CI/CD pipeline.

```python
import boto3
import sagemaker
from sagemaker.processing import (
    ProcessingInput,
    ProcessingOutput,
    ScriptProcessor,
)


def run_sagemaker_processing_job(
    input_s3_path: str,
    output_s3_path: str,
    checkpoint_s3_path: str,
    image_uri: str,
    role: str,
    instance_type: str = "ml.g4dn.xlarge",
    instance_count: int = 1,
):
    """
    Launch a SageMaker Processing Job for image embeddings.

    Prerequisites:
    1. Build and push Docker image to ECR
    2. Configure IAM role with S3 and SageMaker permissions
    3. Set up VPC/security groups if needed
    """
    session = sagemaker.Session()

    processor = ScriptProcessor(
        role=role,
        image_uri=image_uri,
        instance_type=instance_type,
        instance_count=instance_count,
        command=["python3"],
        sagemaker_session=session,
    )

    processor.run(
        code="inference.py",
        inputs=[
            ProcessingInput(
                source=input_s3_path,
                destination="/opt/ml/processing/input",
                s3_data_type="S3Prefix",
            ),
            ProcessingInput(
                source=checkpoint_s3_path,
                destination="/opt/ml/processing/checkpoint",
                s3_data_type="S3Prefix",
            ),
        ],
        outputs=[
            ProcessingOutput(
                source="/opt/ml/processing/output",
                destination=output_s3_path,
            ),
            ProcessingOutput(
                source="/opt/ml/processing/checkpoint",
                destination=checkpoint_s3_path,
            ),
        ],
    )

    print(f"Job complete. Output at: {output_s3_path}")
```

### Dockerfile for SageMaker

```dockerfile
FROM nvidia/cuda:11.8-runtime-ubuntu22.04

RUN apt-get update && apt-get install -y python3 python3-pip

RUN pip3 install \
    torch>=2.1.0 \
    open-clip-torch>=2.24.0 \
    transformers>=4.36.0 \
    Pillow>=10.0.0 \
    pyarrow>=14.0.0 \
    boto3

COPY inference.py /opt/ml/code/

ENTRYPOINT ["python3", "/opt/ml/code/inference.py"]
```

---

## Google Vertex AI Approach

Similar complexity to SageMaker:
- Custom container required
- Manual checkpointing
- Custom Training jobs or Batch Prediction

```python
from google.cloud import aiplatform


def run_vertex_batch_prediction(
    project_id: str,
    location: str,
    model_id: str,
    input_gcs_path: str,
    output_gcs_path: str,
    machine_type: str = "n1-standard-4",
    accelerator_type: str = "NVIDIA_TESLA_T4",
    accelerator_count: int = 1,
):
    """
    Run Vertex AI Batch Prediction for image embeddings.

    NOTE: This requires:
    1. Pre-deployed model in Model Registry
    2. Custom container with prediction code
    3. Proper IAM setup
    4. No built-in checkpointing - job restarts from scratch on failure
    """
    aiplatform.init(project=project_id, location=location)

    model = aiplatform.Model(model_id)

    batch_prediction_job = model.batch_predict(
        job_display_name="image-embedding-batch",
        gcs_source=input_gcs_path,
        gcs_destination_prefix=output_gcs_path,
        machine_type=machine_type,
        accelerator_type=accelerator_type,
        accelerator_count=accelerator_count,
        starting_replica_count=1,
        max_replica_count=10,
    )

    batch_prediction_job.wait()
    print(f"Job complete: {batch_prediction_job.display_name}")


# For Custom Training (more control), similar setup to SageMaker:
# - Build container
# - Push to Artifact Registry
# - Submit CustomJob
# - Manage checkpoints manually
```

---

## Azure ML Approach

Azure ML Pipelines have similar patterns:
- Environment definition
- Compute cluster setup
- Pipeline component definition
- Manual checkpoint management

```python
from azure.ai.ml import MLClient
from azure.ai.ml.dsl import pipeline
from azure.ai.ml import command, Input, Output
from azure.identity import DefaultAzureCredential


def create_azure_ml_pipeline():
    """
    Create Azure ML Pipeline for image embeddings.

    NOTE: Requires:
    1. Azure ML Workspace
    2. Compute cluster with GPUs
    3. Environment with dependencies
    4. Custom checkpointing code
    """
    credential = DefaultAzureCredential()
    ml_client = MLClient(
        credential=credential,
        subscription_id="<subscription>",
        resource_group_name="<resource-group>",
        workspace_name="<workspace>",
    )

    # Define the command component
    embedding_component = command(
        name="compute_embeddings",
        display_name="Compute Image Embeddings",
        inputs={
            "input_data": Input(type="uri_folder"),
            "checkpoint": Input(type="uri_folder", optional=True),
        },
        outputs={
            "output_data": Output(type="uri_folder"),
            "checkpoint_out": Output(type="uri_folder"),
        },
        code="./src",
        command=(
            "python inference.py "
            "--input ${{inputs.input_data}} "
            "--output ${{outputs.output_data}} "
            "--checkpoint ${{inputs.checkpoint}} "
            "--checkpoint-out ${{outputs.checkpoint_out}}"
        ),
        environment="embeddings-env:1",  # Must be pre-created
        compute="gpu-cluster",  # Must be pre-created
    )

    @pipeline(
        compute="gpu-cluster",
        description="Image embedding pipeline",
    )
    def embedding_pipeline(input_data):
        embedding_job = embedding_component(input_data=input_data)
        return {"embeddings": embedding_job.outputs.output_data}

    # Submit the pipeline
    pipeline_job = ml_client.jobs.create_or_update(
        embedding_pipeline(
            input_data=Input(
                type="uri_folder",
                path="azureml://datastores/images/paths/input/",
            )
        ),
        experiment_name="image-embeddings",
    )

    return pipeline_job
```

---

## What All Cloud ML Services Require (that Geneva handles automatically)

### Common Requirements Across AWS/GCP/Azure

1. **Container Management:**
   - Build Dockerfile with all dependencies
   - Push to container registry (ECR/GCR/ACR)
   - Manage image versions
   - Handle dependency conflicts

2. **Infrastructure Setup:**
   - IAM roles/service accounts with correct permissions
   - Networking (VPC, security groups, private endpoints)
   - Storage permissions (S3/GCS/Blob)
   - GPU quota requests

3. **Custom Checkpointing:**
   - Track processed files/rows
   - Persist checkpoint to durable storage
   - Resume logic in your code
   - Handle partial failures

4. **Incremental Processing:**
   - Compare input vs. output timestamps
   - Track which rows are new
   - Custom logic to process only new data

5. **Schema Management:**
   - Define output schema manually
   - Handle schema evolution
   - Validate input/output compatibility

6. **Orchestration:**
   - Trigger jobs on schedule or event
   - Monitor job status
   - Handle job failures
   - Manage job dependencies

**Lines of code:** 300-500 lines + console/terraform configuration

---

## Geneva Approach

Geneva provides all of the above automatically:
- No container builds (dependencies from pyproject.toml)
- No IAM setup beyond storage access
- Automatic checkpointing at fragment + batch level
- Automatic incremental processing
- Schema from type hints
- Built-in orchestration via backfill/backfill_async

### UDF Definitions

```python
import geneva


@geneva.udf(num_gpus=1, version="0.1")
class ComputeEmbeddingGeneva:
    """
    CLIP embedding UDF for Geneva.

    Compare to cloud ML:
    - No Dockerfile
    - No container registry
    - No IAM setup
    - Dependencies in pyproject.toml
    """

    def setup(self):
        import open_clip
        self.model, _, self.preprocess = open_clip.create_model_and_transforms(
            'ViT-B-32', pretrained='laion2b_s34b_b79k'
        )
        self.model = self.model.cuda().eval()

    def __call__(self, image: bytes) -> list[float]:
        from PIL import Image
        import torch
        import io

        img = Image.open(io.BytesIO(image)).convert('RGB')
        tensor = self.preprocess(img).unsqueeze(0).cuda()
        with torch.no_grad():
            embedding = self.model.encode_image(tensor)
        return embedding.cpu().numpy().flatten().tolist()


@geneva.udf(num_gpus=1, version="0.1")
class GenerateCaptionGeneva:
    """BLIP captioning UDF for Geneva."""

    def setup(self):
        from transformers import BlipProcessor, BlipForConditionalGeneration
        self.processor = BlipProcessor.from_pretrained(
            "Salesforce/blip-image-captioning-base"
        )
        self.model = BlipForConditionalGeneration.from_pretrained(
            "Salesforce/blip-image-captioning-base"
        ).cuda()

    def __call__(self, image: bytes) -> str:
        from PIL import Image
        import torch
        import io

        img = Image.open(io.BytesIO(image)).convert('RGB')
        inputs = self.processor(img, return_tensors="pt")
        inputs = {k: v.cuda() for k, v in inputs.items()}
        with torch.no_grad():
            output = self.model.generate(**inputs, max_new_tokens=50)
        return self.processor.decode(output[0], skip_special_tokens=True)
```

### Pipeline

```python
def run_geneva_pipeline():
    """
    Geneva pipeline - everything included.

    No container builds.
    No IAM configuration.
    No checkpoint code.
    No incremental processing code.
    No schema definitions.
    """
    conn = geneva.connect("s3://my-bucket/geneva-workspace")
    table = conn.open_table("images")

    # Add computed columns
    table.add_columns({
        "embedding": ComputeEmbeddingGeneva,
        "caption": GenerateCaptionGeneva,
    })

    # Run with automatic checkpointing and incremental processing
    table.backfill("embedding", batch_size=100, commit_granularity=10)
    table.backfill("caption", batch_size=50, commit_granularity=10)

    print(f"Processed {table.count_rows()} images")
    print(f"Table version: {table.version}")
```

---

## Side-by-Side Comparison

### Feature Comparison

| Feature | Cloud ML Services | Geneva |
|---------|-------------------|--------|
| Managed infrastructure | Yes | Via Ray cluster |
| Cloud provider integration | Native | S3/GCS compatible |
| GPU support | Various types | num_gpus=1 |
| Autoscaling | Built-in | Via Ray |
| Spot/preemptible support | Built-in | Via Ray |
| **Container management** | Build yourself | Automatic |
| **Checkpointing** | Build yourself | Automatic |
| **Incremental processing** | Build yourself | Automatic |
| **Resume from failure** | Limited | Exact batch |
| **Schema from type hints** | Manual | Automatic |
| **Data versioning** | Not included | Automatic |
| **Dependency isolation** | Per container | Per-UDF |

### Code + Configuration Comparison

| Component | Cloud ML Services | Geneva |
|-----------|-------------------|--------|
| UDF/inference code | ~80 lines | ~45 lines |
| Pipeline/orchestration | ~60 lines | ~15 lines |
| Dockerfile | ~20 lines | 0 (not needed) |
| Checkpoint logic | ~40 lines | 0 (included) |
| IAM/permissions | ~50 lines (Terraform) | 0 (storage only) |
| Compute setup | ~30 lines (Terraform) | 0 (cluster config) |
| Console configuration | Many clicks | 0 |
| **TOTAL** | **~280 lines + config** | **~60 lines** |

### Failure Recovery Comparison

| Scenario | Cloud ML Services | Geneva |
|----------|-------------------|--------|
| Job fails at row 999,000 of 1,000,000 | Custom checkpoint code determines restart point | Resume from 999,000 automatically |
| Spot instance preempted | Job restarts from beginning or custom checkpoint | Resume from last committed batch |
| OOM on large batch | Manual retry with smaller batch | Automatic retry with backoff |

---

## When to Use Each

### Cloud ML Services are great for:
- General ML training workflows (not just feature engineering)
- When you need specific cloud-native integrations
- Organizations standardized on a specific cloud provider
- Teams with dedicated MLOps/Platform engineers
- When you need managed experiment tracking, model registry, etc.
- Training jobs (Geneva is for feature computation, not training)

### Geneva is great for:
- Feature engineering specifically
- Teams without dedicated MLOps resources
- When fault tolerance is critical
- Ongoing incremental feature updates
- When you want to avoid container management
- Multimodal data (images, video, PDFs)

---

## The Key Insight

Cloud ML services are general-purpose ML platforms.
They handle training, inference, deployment, monitoring.
Feature engineering is one use case among many.

Geneva is purpose-built for feature engineering.
Every design decision optimizes for:
- Computing features on large datasets
- Maintaining features as data evolves
- Handling failures in long-running jobs

For a simple batch job, cloud ML services work fine.
For an ongoing feature pipeline, Geneva eliminates the ops burden.

**Think of it this way:**
- Cloud ML: "Give me managed infrastructure for my ML code"
- Geneva: "Give me a managed feature column"

**The abstraction level is different:**
- Cloud ML abstracts infrastructure (you still write pipeline code)
- Geneva abstracts the entire feature lifecycle

---

## Architecture

### Cloud ML Services Architecture (you build the boxes in gray)

**Your Code** (you build these):

| Dockerfile + Build + Push | Checkpoint Logic | Incremental Processing | Schema Definition |
|---------------------------|------------------|------------------------|-------------------|
| (custom) | (custom) | (custom) | (custom) |

| IAM Setup | Networking Config | Compute Config |
|-----------|-------------------|----------------|
| (Terraform) | (Terraform) | (Terraform) |

↓

**Cloud ML Service** (SageMaker / Vertex AI / Azure ML)

| Container Runtime | GPU Instances | Job Scheduler | Logging Monitoring |
|-------------------|---------------|---------------|--------------------|

↓

**Cloud Storage** (S3/GCS/Blob)

### Geneva Architecture (everything included)

**Your Code**

| UDFs (ML logic) | Pipeline (backfill) |
|-----------------|---------------------|

↓

**Geneva**

| Dep Packaging | Checkpoint Store | Incremental Processor | Schema Inference | Version Control |
|---------------|------------------|----------------------|------------------|-----------------|

| UDF System | GPU Scheduling | Distributed Execution | Error Handling |
|------------|----------------|----------------------|----------------|

↓

**Ray Cluster + Lance Storage**
