# Distributed Image Deduplication Pipeline

## Status: Draft (batch UDTF infrastructure production-hardened — 2026-03-01)

## Goal

Given a table of images with perceptual hashes (pHash), produce a table that
maps each image to its duplicate cluster — a representative image and its
duplicates. Two images are considered duplicates if their pHash hamming
distance is within a user-specified threshold.

The naive approach (all-pairs comparison) is O(n^2) and doesn't parallelize.
This design uses IVF-style partitioning to divide the work into independent
chunks that can run in parallel, then merges results and computes connected
components to produce the final output.

## Pipeline Overview

```mermaid
flowchart LR
    Source["Source Table<br>(image)"]
    S0["Step 0<br>UDF: pHash"]
    S0b["Step 0b<br>UDF: partition_id<br>(IVF centroid)"]

    subgraph "Batch UDTF 1: Edge Detection (partition_by)"
        S1["Step 1<br>Framework<br>partitions by<br>partition_id"]
        S2["Step 2<br>Pairwise<br>Comparison<br>(per-partition)"]
        S3["Step 3<br>Framework<br>merges outputs"]
        S1 --> S2 --> S3
    end

    Edges["Edge List Table<br>(row_id_a, row_id_b,<br>hamming_dist)"]

    subgraph "Batch UDTF 2: Clustering"
        S4["Step 4<br>Connected<br>Components"]
        S5["Step 5<br>Output<br>Formatting"]
        S4 --> S5
    end

    Output["Output Table<br>A1 → {A2}<br>D1 → {A3, D2}<br>B1 → {B2}"]

    Source --> S0 --> S0b --> S1
    S3 --> Edges --> S4
    S5 --> Output
```

| Step | Name | Input | Output | Geneva primitive |
|------|------|-------|--------|-----------------|
| 0 | pHash computation | image bytes | phash column | UDF (per-row) |
| 0b | Partition assignment | phash | partition_id column | UDF (per-row, IVF centroid) |
| 1-3 | Edge detection | phash + partition_id | edge list table | **Batch UDTF 1** (with `partition_by`) |
| 1 |   Partitioning | partition_id column | grouped partitions | Framework (`partition_by` key) |
| 2 |   Pairwise comparison | partition of phashes | per-partition edge list | Per-partition Ray task |
| 3 |   Merge edges | per-partition edge lists | global edge list | Framework (concat partition outputs) |
| 4-5 | Clustering | edge list table | dedup output table | **Batch UDTF 2** |
| 4 |   Connected components | global edge list | component → member list | Internal to UDTF |
| 5 |   Output formatting | components | representative + duplicates table | Internal to UDTF |

---

## Step 0: pHash Computation (UDF — unchanged)

Per-row UDF that computes a perceptual hash from image bytes. This is the
existing Geneva UDF model — fragment-parallel, incremental on new rows.

```python
@geneva.udf(data_type=pa.list_(pa.uint8(), 16), version="1.0")
def compute_phash(img_bytes: bytes | None) -> list[int] | None:
    if img_bytes is None:
        return None
    image = PILImage.open(BytesIO(img_bytes))
    return hash_to_u8_hex_vector(imagehash.phash(image))

images.add_columns({"phash": compute_phash})
images.backfill("phash")
```

After this step, every row in the source table has a `phash` column.

---

## Step 0b: Partition Assignment (UDF)

**Goal:** Assign each row to one of N partitions so that likely duplicates end
up in the same partition. This reduces the O(n^2) all-pairs problem to N
independent O((n/N)^2) sub-problems.

**Method:** Compute N centroids over the phash vectors using IVF-FLAT with
hamming distance, then assign each row to its nearest centroid. The resulting
`partition_id` column is what the framework uses as the `partition_by` key in
the next step.

```python
@geneva.udf(data_type=pa.int32(), version="1.0")
def assign_ivf_partition(phash: list[int] | None) -> int | None:
    if phash is None:
        return None
    # Assign to nearest centroid (centroids computed offline or lazily)
    return find_nearest_centroid(phash, centroids, metric="hamming")

images.add_columns({"partition_id": assign_ivf_partition})
images.backfill("partition_id")
```

```mermaid
flowchart TD
    All["All rows: A1, A2, A3, B1, B2, D1, D2"]
    C1["Centroid 1<br>{A1, A2, B1, B2}"]
    C2["Centroid 2<br>{A3, D1, D2}"]

    All -- "IVF-FLAT<br>(hamming distance)" --> C1
    All --> C2
```

**Key properties:**
- N (number of partitions) controls the parallelism vs accuracy tradeoff.
  More partitions = smaller sub-problems but higher chance of splitting true
  duplicates across partitions.
- IVF assignment is approximate — true duplicates may land in different
  partitions if they're equidistant to two centroids. Research literature
  covers techniques to mitigate this (multi-probe, multiple hash functions,
  overlapping partitions). For the initial version, single-probe IVF is
  sufficient.

**Implementation options for centroid computation:**
- Use Lance's built-in IVF-FLAT index (already exists for vector search)
- Build a lightweight k-means on phash vectors with hamming distance

**Output:** Each row gets a `partition_id` column. This is a plain per-row UDF
— incremental on new rows, fragment-parallel, same as `compute_phash`.

---

## Step 1: Partitioning (framework-managed)

The framework reads the `partition_id` column and groups rows by this key. Each
group is dispatched as an independent UDTF invocation (Ray task). The UDTF
author does not write any partitioning logic — the framework handles it via
`partition_by="partition_id"` on the UDTF decorator.

---

## Step 2: Pairwise Comparison (per-partition, parallelizable)

**Goal:** Within each partition, compute all pairs of images within the hamming
distance threshold. This is the expensive step, but it's embarrassingly
parallel across partitions.

**Input:** One partition of (row_id, phash) tuples.

**Output:** Edge list — pairs of row_ids whose hamming distance <= threshold.

```mermaid
flowchart LR
    subgraph "Centroid 1: {A1, A2, B1, B2}"
        P1["Pairwise comparisons"]
        P1 --> E1["(A1,A2) → h=1 ✓"]
        P1 --> E2["(A1,B1) → h=8 ✗"]
        P1 --> E3["(A1,B2) → h=9 ✗"]
        P1 --> E4["(A2,B1) → h=5 ✗"]
        P1 --> E5["(A2,B2) → h=6 ✗"]
        P1 --> E6["(B1,B2) → h=0 ✓"]
    end
    subgraph "Centroid 2: {A3, D1, D2}"
        P2["Pairwise comparisons"]
        P2 --> E7["(A3,D1) → h=4 ✓"]
        P2 --> E8["(A3,D2) → h=8 ✗"]
        P2 --> E9["(D1,D2) → h=2 ✓"]
    end

    E1 --> Edges1["Edges: {(A1,A2), (B1,B2)}"]
    E6 --> Edges1
    E7 --> Edges2["Edges: {(A3,D1), (D1,D2)}"]
    E9 --> Edges2

    style E1 stroke:#2E8B57,stroke-width:3px
    style E6 stroke:#2E8B57,stroke-width:3px
    style E7 stroke:#2E8B57,stroke-width:3px
    style E9 stroke:#2E8B57,stroke-width:3px
    style E2 stroke:#CD5C5C,stroke-width:3px,stroke-dasharray:5 5
    style E3 stroke:#CD5C5C,stroke-width:3px,stroke-dasharray:5 5
    style E4 stroke:#CD5C5C,stroke-width:3px,stroke-dasharray:5 5
    style E5 stroke:#CD5C5C,stroke-width:3px,stroke-dasharray:5 5
    style E8 stroke:#CD5C5C,stroke-width:3px,stroke-dasharray:5 5
```

**Parallelism:** Each partition runs as an independent Ray task. For N
partitions, we get up to N-way parallelism. The partition size determines
the per-task cost — O((n/N)^2) comparisons per partition.

**This is the core of Batch UDTF 1** — each partition runs as an independent
Ray task via `partition_by`. The UDTF author writes the per-partition comparison
logic; the framework handles partitioning, dispatch, and merging the results.

---

## Step 3: Merge Edges (framework-managed)

**Goal:** Collect edge lists from all partitions into a single global edge
list. This is a simple concatenation — no deduplication needed because
partitions are disjoint (each row belongs to exactly one partition).

```mermaid
flowchart LR
    P1["Partition 1 edges<br>{(A1,A2), (B1,B2)}"]
    P2["Partition 2 edges<br>{(A3,D1), (D1,D2)}"]
    Merged["Merged edges<br>{(A1,A2), (B1,B2),<br>(A3,D1), (D1,D2)}"]

    P1 --> Merged
    P2 --> Merged
```

**Implementation:** The framework concatenates per-partition outputs into
the edge list materialized view. This is cheap — just metadata manipulation,
no data copying.

> **Note on cross-partition duplicates:** If true duplicates are split across
> partitions (e.g., A3 ends up in partition 1, D1 in partition 2), their edge
> will be missed. Multi-probe IVF or overlapping partitions can mitigate this
> at the cost of more comparisons. For the initial version, we accept this
> approximation.

---

## Step 4: Connected Components

**Goal:** Given the global edge list, compute connected components. Each
component is a set of images that are transitively similar — if A~B and B~C,
then {A, B, C} form a component even if A and C are not directly similar.

```mermaid
flowchart TD
    subgraph "Component 0"
        A1 --- A2
    end
    subgraph "Component 1"
        B1 --- B2
    end
    subgraph "Component 2 (transitive)"
        A3 --- D1 --- D2
    end
```

Edges: {(A1,A2), (B1,B2), (A3,D1), (D1,D2)} → three connected components.

**Implementation:** Union-Find (disjoint set) data structure. Linear in the
number of edges — cheap compared to the pairwise comparison step.

---

## Step 5: Output Formatting

**Goal:** For each connected component, choose one image as the representative
and list the rest as duplicates. Produce the final output table.

```mermaid
flowchart LR
    subgraph "Component 0"
        A1_r["A1 (representative)"] --> A2_d["A2 (duplicate)"]
    end
    subgraph "Component 1"
        B1_r["B1 (representative)"] --> B2_d["B2 (duplicate)"]
    end
    subgraph "Component 2"
        D1_r["D1 (representative)"] --> A3_d["A3 (duplicate)"]
        D1_r --> D2_d["D2 (duplicate)"]
    end
```

**Representative selection:** Choose the image with the lowest row_id (or
earliest ingestion time, or by a user-specified criterion). The choice is
arbitrary — what matters is consistency.

**Output schema:**

| row_id | is_representative | cluster_id | duplicate_row_ids |
|--------|-------------------|------------|-------------------|
| A1     | true              | 0          | [A2]              |
| A2     | false             | 0          | []                |
| B1     | true              | 1          | [B2]              |
| B2     | false             | 1          | []                |
| D1     | true              | 2          | [A3, D2]          |
| A3     | false             | 2          | []                |
| D2     | false             | 2          | []                |

Images not in any duplicate cluster are their own representative with an
empty duplicate list (singleton components).

---

## How This Maps to Geneva Primitives

The pipeline uses three Geneva primitives chained together:

```mermaid
flowchart LR
    T1["images table"] -- "UDF: compute_phash<br>(per-row)" --> T2["images + phash"]
    T2 -- "UDF: assign_ivf_partition<br>(per-row)" --> T2b["images + phash<br>+ partition_id"]
    T2b -- "Batch UDTF 1<br>EdgeDetection<br>(partition_by)" --> T3["edge_list table<br>(materialized view)"]
    T3 -- "Batch UDTF 2<br>DedupClustering" --> T4["dedup_output table<br>(materialized view)"]
```

| Primitive | Steps | Input table | Output table | Notes |
|-----------|-------|-------------|--------------|-------|
| UDF (`compute_phash`) | 0 | images | images + phash | Per-row, incremental |
| UDF (`assign_ivf_partition`) | 0b | images + phash | images + phash + partition_id | Per-row, incremental |
| **Batch UDTF 1** (`EdgeDetection`) | 1-3 | images (phash + partition_id) | edge_list | Partition-parallel via `partition_by` |
| **Batch UDTF 2** (`DedupClustering`) | 4-5 | edge_list | dedup_output | Single-worker, global |

The intermediate **edge list table** is a materialized view that serves as the
contract between the two UDTFs. It's also independently useful — users can
query it directly for debugging, analysis, or alternate downstream uses.

### Batch UDTF 1: Edge Detection (Steps 1-3)

Per-partition pairwise comparison. The framework handles partitioning (Step 1)
and merging (Step 3) — the UDTF author only writes the comparison logic
(Step 2). Each partition runs as an independent Ray task.

```python
EDGE_SCHEMA = pa.schema([
    ("row_id_a", pa.string()),
    ("row_id_b", pa.string()),
    ("hamming_dist", pa.int32()),
])

@geneva.udtf(
    output_schema=EDGE_SCHEMA,
    input_columns=["row_id", "phash"],
    partition_by="partition_id",
    num_cpus=2,
)
class EdgeDetection:
    def __init__(self, threshold: int = 4):
        self.threshold = threshold

    def __call__(self, source) -> Iterator[pa.RecordBatch]:
        # source contains only this partition's rows
        data = source.to_arrow()
        row_ids = data["row_id"].to_pylist()
        phashes = data["phash"].to_pylist()

        # Step 2: Pairwise comparison within this partition
        edges = self._pairwise_compare(row_ids, phashes, self.threshold)
        if edges:
            yield pa.RecordBatch.from_pydict({
                "row_id_a": [e[0] for e in edges],
                "row_id_b": [e[1] for e in edges],
                "hamming_dist": [e[2] for e in edges],
            })
```

### Batch UDTF 2: Dedup Clustering (Steps 4-5)

Reads the edge list table, computes connected components via Union-Find,
and outputs the final representative + duplicates table.

```python
DEDUP_OUTPUT_SCHEMA = pa.schema([
    ("row_id", pa.string()),
    ("is_representative", pa.bool_()),
    ("cluster_id", pa.int64()),
    ("duplicate_row_ids", pa.list_(pa.string())),
])

@geneva.udtf(
    output_schema=DEDUP_OUTPUT_SCHEMA,
    input_columns=["row_id_a", "row_id_b"],
    num_cpus=1,
)
class DedupClustering:
    def __call__(self, source) -> Iterator[pa.RecordBatch]:
        data = source.to_arrow()

        # Step 4: Connected components via union-find
        components = self._connected_components(
            data["row_id_a"].to_pylist(),
            data["row_id_b"].to_pylist(),
        )

        # Step 5: Format output — pick representative per component
        yield from self._format_output(components)
```

### Wiring it together

```python
# Step 0: UDF — compute phash
images.add_columns({"phash": compute_phash})
images.backfill("phash")

# Step 0b: UDF — assign IVF partition
images.add_columns({"partition_id": assign_ivf_partition})
images.backfill("partition_id")

# Steps 1-3: Batch UDTF 1 — edge detection (partition-parallel)
edge_list = conn.create_udtf_view(
    "edge_list",
    source=images,
    udtf=EdgeDetection(threshold=4),
)
edge_list.refresh()

# Steps 4-5: Batch UDTF 2 — clustering (single-worker)
dedup_output = conn.create_udtf_view(
    "dedup_output",
    source=edge_list,
    udtf=DedupClustering(),
)
dedup_output.refresh()
```

---

## Scaling Analysis

### Cost model

| Step | Cost | Notes |
|------|------|-------|
| Centroid training (k-means) | O(k² × sample_rate × d × i) | Single-node, independent of n |
| Partition assignment (Step 0b) | O(n × k × d) | Per-row UDF, fragment-parallel |
| Pairwise comparison (Step 2) | O(n²/k × d) total | Dominant cost, parallelized across k partitions |

Where d = 16 bytes (pHash), i = 20–50 iterations, sample_rate = 256.

Pairwise comparison dominates. Each partition performs O((n/k)² × d) work
independently, so the total work is O(n²/k × d) but wall-clock time is
O(n²/k² × d) with k parallel workers.

### Reference machine

96 vCPU, 384 GB RAM (4 GB per vCPU).

**Memory constraint:** Each row is ~64–128 bytes (16B pHash + row ID +
overhead). At 50K rows per partition, that's 3–6 MB — well within the 4 GB
per-vCPU budget. Memory is not the bottleneck; compute is.

**Compute constraint:** pHash hamming distance (XOR + popcount on 128 bits) runs
at ~10 ns/pair. Per-partition task time at different sizes:

| Rows per partition | Pairs | Task time |
|--------------------|-------|-----------|
| 5,000 | ~12.5M | ~0.1s |
| 50,000 | ~1.25B | ~12s |
| 100,000 | ~5B | ~50s |

With these machines, each vCPU can comfortably handle 50K–100K rows per
partition (tasks complete in 12–50s). This is 10× larger than a minimal
deployment, which means 10× fewer centroids and much cheaper training.

### Choosing k (number of partitions)

The recommended formula is:

```
k = n / target_partition_size
```

Where `target_partition_size` ≈ 50,000 rows on the reference machine (yielding
~1.25B pairs and ~12s of compute per task).

Why this is better than `sqrt(n)`:

- **Bounded per-partition work.** Each partition always has `target_partition_size`
  rows regardless of dataset size, so per-task memory and compute stay constant.
- **Natural parallelism scaling.** k grows linearly with n, so adding more data
  automatically creates more parallel tasks.
- **Predictable resource planning.** Given n and a target partition size, you know
  exactly how many machines you need.

### Scaling examples

Assumes `target_partition_size` = 50,000 rows (~12s per task) on reference
machines (96 vCPU, 384 GB RAM each).

| | 10M rows | 1B rows | 10B rows |
|---|---|---|---|
| k (centroids) | 200 | 20,000 | 200,000 |
| Rows per partition | 50,000 | 50,000 | 50,000 |
| Pairs per partition | ~1.25B | ~1.25B | ~1.25B |
| Task time | ~12s | ~12s | ~12s |
| Machines | 1 | 5 | 42 |
| Total vCPUs | 96 | 480 | 4,032 |
| Partitions per vCPU | ~2 | ~42 | ~50 |
| Pairwise wall time | ~36s | ~8 min | ~10 min |
| Training samples (k × 256) | 51.2K | 5.12M | 51.2M |
| Training cost | seconds | seconds | minutes (single node) |

The machine counts above target ~10 min wall time for the pairwise step.
For faster completion, scale horizontally: halving wall time requires
doubling the machine count.

**Note on centroid training at large k:** At k = 200K, training is still
tractable on a single machine (d = 16 for pHash). For k beyond ~1M,
mini-batch k-means or hierarchical IVF (train coarse centroids, then refine)
can help.

### Accuracy vs parallelism tradeoff

The IVF partitioning introduces an approximation: true duplicates may be split
across partitions. The key parameters:

| Parameter | Effect on accuracy | Effect on performance |
|-----------|-------------------|---------------------|
| k (partitions) | Fewer = more accurate | Fewer = less parallel, larger sub-problems |
| Probe count | More probes = more accurate | More probes = more comparisons |
| Overlap ratio | More overlap = more accurate | More overlap = more comparisons |

### Indexing considerations

No indexes are needed on the pHash column or the edge list table. Every step
in this pipeline is a bulk scan or a partitioned bulk scan:

- **Partition assignment (Step 0b):** Full table scan — every row gets a centroid.
- **Pairwise comparison (Step 2):** Full scan within each partition — all pairs.
- **Connected components (Step 4):** Full scan of the edge list to build a graph.

The `partition_id` column already serves the role an index would — it narrows
the working set for the expensive pairwise step. Adding a secondary index on
pHash or the edge columns would cost write-time overhead with no read-time
benefit.

### Future optimizations

The following approaches can improve accuracy without changing the pipeline
structure:

- **Multi-probe IVF:** Query multiple nearby centroids per image
- **Multiple hash functions:** Use several hash algorithms and union the results
- **Locality-sensitive hashing (LSH):** Alternative to IVF for hamming space
- **Overlapping partitions:** Assign borderline images to multiple partitions

---

## Relationship to UDTF Batch Design

This pipeline is a concrete instance of the patterns described in
[`internal_docs/udtf.md`](../../internal_docs/udtf.md):

- **Step 0** uses the existing UDF model (per-row, 1:1)
- **Steps 1-3** are Batch UDTF 1 with `partition_by` — the framework partitions
  the source data, dispatches each partition as an independent Ray task, and
  merges the outputs. The UDTF author only writes the per-partition comparison.
- **Steps 4-5** are Batch UDTF 2 — reads the edge list (output of UDTF 1),
  produces the final dedup table. This is a lightweight, single-worker UDTF
  (connected components is inherently global).
- The two UDTFs are chained via an intermediate materialized view (`edge_list`),
  demonstrating the **UDTF composition** pattern: complex pipelines built from
  simple, independently refreshable stages.

This document focuses on the deduplication algorithm and pipeline structure.
For the UDTF execution model, API surface, and materialized view integration,
see [`internal_docs/udtf.md`](../../internal_docs/udtf.md).
