# Image Dedupe E2E Test Suite

End-to-end tests for the distributed image deduplication pipeline using Geneva
batch UDTFs with IVF-style partitioning.

## Overview

This pipeline finds near-duplicate images by perceptual hash (pHash) similarity.
It chains two batch UDTFs — partition-parallel edge detection followed by global
connected-components clustering — to produce a table mapping each image to its
duplicate cluster.

See [DESIGN.md](DESIGN.md) for the full pipeline design, scaling analysis, and
architecture diagrams.

## Structure

```
image_dedupe/
├── README.md              # This file
├── DESIGN.md              # Full pipeline design doc
├── conftest.py            # Fixtures: table creation, manifest upload
├── pyproject.toml         # Test driver dependencies
├── test_drivers/          # Test drivers
│   └── test_dedup.py      # End-to-end dedup pipeline test
└── udfs/                  # UDF packages
    └── phash/
        ├── pyproject.toml
        └── manifest.py
```

## How to Run

```bash
# From workspace root
cd e2e/image_dedupe
RAY_ENABLE_UV_RUN_RUNTIME_ENV=0 uv run -m pytest -c ../pytest.ini test_drivers -v
```

## Pipeline Steps

1. **UDF: compute_phash** — per-row perceptual hash computation
2. **UDF: assign_ivf_partition** — IVF centroid-based partition assignment
3. **Batch UDTF 1: EdgeDetection** — partition-parallel pairwise comparison
4. **Batch UDTF 2: DedupClustering** — connected components + output formatting
