{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "87bb60f4",
   "metadata": {},
   "source": [
    "# filoma dedup tutorial\n",
    "\n",
    "This notebook explains duplicate detection and near-duplicate matching in a friendly, beginner-oriented way.\n",
    "\n",
    "Why care about duplicates?\n",
    "\n",
    "- Duplicate detection helps you find identical files (byte-for-byte) and near-duplicates (same content with small changes).\n",
    "- Use exact hashing (SHA256) to find perfect duplicates quickly.\n",
    "- Use text similarity (shingles + Jaccard or MinHash) to find documents that are very similar but not identical.\n",
    "- Use perceptual image hashes (aHash/dHash/pHash) to find images that look the same even if they were resized, compressed, or slightly edited.\n",
    "\n",
    "What is covered in this tutorial:\n",
    "\n",
    "- How to run quick, practical checks on small datasets.\n",
    "- How the core algorithms work at a high level and what the parameters mean.\n",
    "- How to use `FileProfiler`, `ImageProfiler`, and `DataFrame.evaluate_duplicates()` to integrate dedup checks into your workflow.\n",
    "\n",
    "Each section has a short \"big idea\" explanation followed by a compact runnable example."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b415e03d",
   "metadata": {},
   "source": [
    "### Installing optional dependencies\n",
    "\n",
    "What's needed to use the dedup tools?\n",
    "\n",
    "- Some features (image perceptual hashing, MinHash acceleration) require third-party packages that are not part of the small core.\n",
    "- `Pillow` lets us open and process images in Python so we can compute perceptual hashes.\n",
    "- `datasketch` provides a MinHash implementation and LSH helpers that scale text similarity to large collections.\n",
    "\n",
    "When to install\n",
    "\n",
    "- If you only need exact duplicate checks (SHA256) and small-scale text comparisons, you can skip installation.\n",
    "- If you plan to deduplicate images or run similarity over many documents, install the optional packages.\n",
    "\n",
    "Run this cell to install the optional tools (or use your environment manager):\n",
    "\n",
    "```\n",
    "!uv pip install --upgrade pillow datasketch\n",
    "```\n",
    "or\n",
    "```\n",
    "!uv sync --extra dedup\n",
    "```\n",
    "\n",
    "If you run in an environment that already has these packages (for example the project's dev environment), you can skip installation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6f835e17",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dedup module loaded: <module 'filoma.dedup' from '/home/kalfasy/repos/filoma/src/filoma/dedup.py'>\n",
      "Pillow available: True\n"
     ]
    }
   ],
   "source": [
    "# This cell prepares the environment and imports the filoma dedup utilities.\n",
    "# We'll create temporary files and run short examples in-memory so nothing is written to your project.\n",
    "\n",
    "import os\n",
    "import tempfile\n",
    "\n",
    "from filoma import dedup\n",
    "from filoma.dataframe import DataFrame\n",
    "from filoma.files.file_profiler import FileProfiler\n",
    "from filoma.images.image_profiler import ImageProfiler\n",
    "\n",
    "# check Pillow availability for image examples\n",
    "try:\n",
    "    from PIL import Image\n",
    "\n",
    "    _HAS_PIL = True\n",
    "except Exception:\n",
    "    _HAS_PIL = False\n",
    "\n",
    "print(\"dedup module loaded:\", dedup)\n",
    "print(\"Pillow available:\", _HAS_PIL)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93ef5d93",
   "metadata": {},
   "source": [
    "## Standalone text dedup example\n",
    "\n",
    "What do we use for text deduplication?\n",
    "\n",
    "- Text near-duplicate detection often uses \"shingles\": short sequences of consecutive words (for example 3-word shingles).\n",
    "- The Jaccard similarity between the sets of shingles of two documents gives a simple measure of how similar they are (intersection / union).\n",
    "- MinHash is a probabilistic technique that approximates Jaccard similarity much faster for large document collections; `datasketch` provides an implementation.\n",
    "\n",
    "Create two small text files that are near-duplicates and run `find_duplicates` with a lower threshold for short texts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9fc5d07e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created two files:\n",
      " - /tmp/tmpjudpnmpf/a.txt\n",
      "   contents: the quick brown fox jumps over the lazy dog\n",
      " - /tmp/tmpjudpnmpf/b.txt\n",
      "   contents: the quick brown fox jumped over the lazy dog\n",
      "\n",
      "Computed 3-word shingles for each file:\n",
      " shingles a.txt: {'the lazy dog', 'the quick brown', 'brown fox jumps', 'quick brown fox', 'over the lazy', 'fox jumps over', 'jumps over the'}\n",
      " shingles b.txt: {'jumped over the', 'the quick brown', 'quick brown fox', 'over the lazy', 'fox jumped over', 'the lazy dog', 'brown fox jumped'}\n",
      " Jaccard similarity (intersection/union): 4/10 = 0.400\n",
      "\n",
      "Calling dedup.find_duplicates(text_k=3, text_threshold=0.4)\n",
      "Raw return value from find_duplicates():\n",
      "{'exact': [], 'text': [['/tmp/tmpjudpnmpf/a.txt', '/tmp/tmpjudpnmpf/b.txt']], 'image': []}\n",
      "\n",
      "No exact (byte-for-byte) duplicates found.\n",
      "\n",
      "Text-based near-duplicate groups (based on shingles/jaccard):\n",
      "  Group:\n",
      "   - /tmp/tmpjudpnmpf/a.txt\n",
      "   - /tmp/tmpjudpnmpf/b.txt\n",
      "   (Computed 3-word Jaccard between the two files = 0.400; threshold was 0.4)\n"
     ]
    }
   ],
   "source": [
    "# Create two small text files that are almost the same, then run dedup and explain the results.\n",
    "with tempfile.TemporaryDirectory() as td:\n",
    "    p1 = os.path.join(td, \"a.txt\")\n",
    "    p2 = os.path.join(td, \"b.txt\")\n",
    "\n",
    "    text1 = \"the quick brown fox jumps over the lazy dog\"\n",
    "    text2 = \"the quick brown fox jumped over the lazy dog\"  # only \"jumps\" -> \"jumped\"\n",
    "\n",
    "    # Write files\n",
    "    with open(p1, \"w\") as f:\n",
    "        f.write(text1)\n",
    "    with open(p2, \"w\") as f:\n",
    "        f.write(text2)\n",
    "\n",
    "    print(\"Created two files:\")\n",
    "    print(\" -\", p1)\n",
    "    print(\"   contents:\", text1)\n",
    "    print(\" -\", p2)\n",
    "    print(\"   contents:\", text2)\n",
    "    print()\n",
    "\n",
    "    # Helper to compute k-word shingles and a human-readable Jaccard similarity\n",
    "    def word_shingles(text, k=3):\n",
    "        toks = text.lower().split()\n",
    "        if len(toks) < k:\n",
    "            return set()\n",
    "        return {\" \".join(toks[i : i + k]) for i in range(len(toks) - k + 1)}\n",
    "\n",
    "    k = 3\n",
    "    s1 = word_shingles(text1, k=k)\n",
    "    s2 = word_shingles(text2, k=k)\n",
    "    inter = s1 & s2\n",
    "    union = s1 | s2\n",
    "    jaccard = len(inter) / len(union) if union else 0.0\n",
    "\n",
    "    print(f\"Computed {k}-word shingles for each file:\")\n",
    "    print(\" shingles a.txt:\", s1)\n",
    "    print(\" shingles b.txt:\", s2)\n",
    "    print(f\" Jaccard similarity (intersection/union): {len(inter)}/{len(union)} = {jaccard:.3f}\")\n",
    "    print()\n",
    "\n",
    "    # Parameters used by the dedup routine\n",
    "    text_threshold = 0.4\n",
    "    print(f\"Calling dedup.find_duplicates(text_k={k}, text_threshold={text_threshold})\")\n",
    "    res = dedup.find_duplicates([p1, p2], text_k=k, text_threshold=text_threshold)\n",
    "    print(\"Raw return value from find_duplicates():\")\n",
    "    print(res)\n",
    "    print()\n",
    "\n",
    "    # Interpret the result in plain language\n",
    "    if res.get(\"exact\"):\n",
    "        print(\"Exact duplicate groups (byte-for-byte identical files):\")\n",
    "        for group in res[\"exact\"]:\n",
    "            print(\"  -\", group)\n",
    "    else:\n",
    "        print(\"No exact (byte-for-byte) duplicates found.\")\n",
    "\n",
    "    if res.get(\"text\"):\n",
    "        print(\"\\nText-based near-duplicate groups (based on shingles/jaccard):\")\n",
    "        for group in res[\"text\"]:\n",
    "            # group is a list of file paths (or items); print paths and our computed jaccard for clarity\n",
    "            print(\"  Group:\")\n",
    "            for item in group:\n",
    "                # item may be a path string or a dict; handle both gracefully\n",
    "                path = item if isinstance(item, str) else item.get(\"path\", str(item))\n",
    "                print(\"   -\", path)\n",
    "            # since we only compared two files here, print the computed Jaccard as additional context\n",
    "            print(f\"   (Computed {k}-word Jaccard between the two files = {jaccard:.3f}; threshold was {text_threshold})\")\n",
    "    else:\n",
    "        print(\"\\nNo text near-duplicate groups found (threshold was not met).\")\n",
    "\n",
    "    if res.get(\"image\"):\n",
    "        print(\"\\nImage-based groups (not expected here):\")\n",
    "        print(res[\"image\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4564f4d0",
   "metadata": {},
   "source": [
    "## Standalone image dedup example\n",
    "\n",
    "What do we use for image deduplication?\n",
    "\n",
    "- Perceptual hashing converts an image into a compact fingerprint that captures visual structure rather than raw bytes.\n",
    "- Simple hashes like aHash and dHash are fast and robust to small changes like resizing or compression.\n",
    "- Compare hashes with Hamming distance: small distances mean visually similar images.\n",
    "\n",
    "If Pillow is available this creates two identical images and demonstrates perceptual hashing and grouping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fdb8f646",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created two images:\n",
      " - /tmp/tmpnn_uqkgz/img1.png (original)\n",
      " - /tmp/tmpnn_uqkgz/img2.png (one pixel changed)\n",
      "\n",
      "Perceptual hashes (aHash):\n",
      " aHash img1: ffffffffffffffff\n",
      " aHash img2: ffffffffffffffff\n",
      "\n",
      "Hamming distance between the two aHashes: 0\n",
      "\n",
      "Calling dedup.find_duplicates([...], image_max_distance=2)\n",
      "Raw return value from find_duplicates():\n",
      "{'exact': [], 'text': [], 'image': [['/tmp/tmpnn_uqkgz/img1.png', '/tmp/tmpnn_uqkgz/img2.png']]}\n",
      "\n",
      "Image-based near-duplicate groups found:\n",
      " Group:\n",
      "  - /tmp/tmpnn_uqkgz/img1.png\n",
      "  - /tmp/tmpnn_uqkgz/img2.png\n",
      "\n",
      "Interpretation: the two images were grouped because their aHash Hamming distance (0) is <= the threshold (2). Small visual changes (like one pixel) often produce small Hamming distances so perceptual hashing is useful to catch visually-similar images.\n"
     ]
    }
   ],
   "source": [
    "# Image dedup demo with clear, human-friendly explanations\n",
    "if not _HAS_PIL:\n",
    "    print(\"Pillow is not available; skipping image dedup example\")\n",
    "else:\n",
    "    with tempfile.TemporaryDirectory() as td:\n",
    "        # Create two images: start identical, then make a very small change to the second\n",
    "        p1 = os.path.join(td, \"img1.png\")\n",
    "        p2 = os.path.join(td, \"img2.png\")\n",
    "\n",
    "        # Base image (uniform color)\n",
    "        img = Image.new(\"RGB\", (64, 64), color=(123, 200, 100))\n",
    "        img.save(p1)\n",
    "\n",
    "        # Make a nearly imperceptible change: alter a single pixel in a copy\n",
    "        img2 = img.copy()\n",
    "        img2.putpixel((0, 0), (124, 200, 100))  # tiny change, one pixel different\n",
    "        img2.save(p2)\n",
    "\n",
    "        print(\"Created two images:\")\n",
    "        print(\" -\", p1, \"(original)\")\n",
    "        print(\" -\", p2, \"(one pixel changed)\\n\")\n",
    "\n",
    "        # Compute perceptual hashes (aHash) for both files\n",
    "        h1 = dedup.ahash_image(p1)\n",
    "        h2 = dedup.ahash_image(p2)\n",
    "        print(\"Perceptual hashes (aHash):\")\n",
    "        print(\" aHash img1:\", h1)\n",
    "        print(\" aHash img2:\", h2)\n",
    "\n",
    "        # Compute Hamming distance between the two hex hashes.\n",
    "        # Convert hex to int, xor, then count differing bits.\n",
    "        try:\n",
    "            xor = int(h1, 16) ^ int(h2, 16)\n",
    "            ham_dist = xor.bit_count()  # Python 3.8+: number of 1-bits in XOR\n",
    "        except Exception:\n",
    "            # Fallback if hash format is unexpected\n",
    "            ham_dist = sum(c1 != c2 for c1, c2 in zip(h1, h2))\n",
    "\n",
    "        print(f\"\\nHamming distance between the two aHashes: {ham_dist}\")\n",
    "\n",
    "        # Use the high-level dedup helper to find duplicate/near-duplicate groups\n",
    "        threshold = 2\n",
    "        print(f\"\\nCalling dedup.find_duplicates([...], image_max_distance={threshold})\")\n",
    "        res = dedup.find_duplicates([p1, p2], image_max_distance=threshold)\n",
    "        print(\"Raw return value from find_duplicates():\")\n",
    "        print(res)\n",
    "        print()\n",
    "\n",
    "        # Interpret the results in plain language\n",
    "        if res.get(\"image\"):\n",
    "            print(\"Image-based near-duplicate groups found:\")\n",
    "            for group in res[\"image\"]:\n",
    "                print(\" Group:\")\n",
    "                for item in group:\n",
    "                    path = item if isinstance(item, str) else item.get(\"path\", str(item))\n",
    "                    print(\"  -\", path)\n",
    "            print(\n",
    "                f\"\\nInterpretation: the two images were grouped because their aHash Hamming distance ({ham_dist}) \"\n",
    "                f\"is <= the threshold ({threshold}). Small visual changes (like one pixel) often produce small \"\n",
    "                \"Hamming distances so perceptual hashing is useful to catch visually-similar images.\"\n",
    "            )\n",
    "        else:\n",
    "            print(\"No image-based groups found.\")\n",
    "            print(\n",
    "                f\"Interpretation: the two images were considered different for the chosen threshold ({threshold}). \"\n",
    "                \"You can increase the threshold to treat more-distorted images as near-duplicates, or try a different \"\n",
    "                \"hash (dHash/pHash) if you need different sensitivity characteristics.\"\n",
    "            )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70bbdc41",
   "metadata": {},
   "source": [
    "## Using FileProfiler for dedup fingerprints\n",
    "\n",
    "What does filoma's `FileProfiler` provide for deduplication?\n",
    "\n",
    "- `FileProfiler` gathers filesystem metadata (size, timestamps) and can compute a SHA256 fingerprint of file contents.\n",
    "- `fingerprint_for_dedup()` is a compact representation useful when scanning many files: you can store fingerprints and later group files with identical fingerprints or similar derived features (text shingles or image hashes).\n",
    "\n",
    "`FileProfiler` exposes `fingerprint_for_dedup()` which produces a compact dict with `sha256` and optional `text_shingles` or `image_hash`. This is handy for pipeline-style scanning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9ade1978",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fingerprint for dedup (human-friendly explanation):\n",
      "\n",
      "1) Basic file info\n",
      "   path : /tmp/tmpiucdpm15/doc.txt\n",
      "   size : 48 bytes  -> size quickly rules out equality when different\n",
      "\n",
      "2) Exact fingerprint\n",
      "   sha256: 064d7354bd3bf25c401f0899a9cde918cedf90f80392ff43080028a551e15782\n",
      "   -> SHA256 is an exact content fingerprint: two files with the same SHA256 are byte-for-byte identical.\n",
      "\n",
      "3) Text shingles (used for near-duplicate text detection)\n",
      "   7 shingles (overlapping phrases):\n",
      "    - a sample document\n",
      "    - document used for\n",
      "    - for dedup test\n",
      "    - is a sample\n",
      "    - sample document used\n",
      "    - thi is a\n",
      "    - used for dedup\n",
      "\n",
      "   What this means:\n",
      "    - Each shingle is a short overlapping phrase (here 3-word phrases).\n",
      "    - To estimate text similarity between two files we compute the Jaccard index between their shingle sets\n",
      "      (intersection size / union size). More shared shingles -> more similar text.\n",
      "\n",
      "   Verification (recomputing 3-word shingles from the file contents):\n",
      "    {'this is a', 'for dedup testing', 'sample document used', 'is a sample', 'a sample document', 'document used for', 'used for dedup'}\n",
      "\n",
      "4) Image hash field (not relevant here): None\n",
      "\n",
      "Summary: the fingerprint contains an exact hash (SHA256) for byte-perfect deduping and\n",
      "a set of text shingles useful for near-duplicate detection using Jaccard/MinHash approaches.\n"
     ]
    }
   ],
   "source": [
    "prof = FileProfiler()  # create a FileProfiler instance for the example\n",
    "# FileProfiler example with clear, human-friendly explanations.\n",
    "with tempfile.TemporaryDirectory() as td:\n",
    "    p = os.path.join(td, \"doc.txt\")\n",
    "    text = \"this is a sample document used for dedup testing\"\n",
    "    with open(p, \"w\") as f:\n",
    "        f.write(text)\n",
    "\n",
    "    # ask the profiler for the compact fingerprint used by dedup routines\n",
    "    fp = prof.fingerprint_for_dedup(p, compute_text=True)\n",
    "\n",
    "    # Friendly, step-by-step explanation of the results\n",
    "    print(\"Fingerprint for dedup (human-friendly explanation):\\n\")\n",
    "\n",
    "    print(\"1) Basic file info\")\n",
    "    print(\"   path :\", fp.get(\"path\"))\n",
    "    print(\"   size :\", fp.get(\"size\"), \"bytes  -> size quickly rules out equality when different\")\n",
    "    print()\n",
    "\n",
    "    print(\"2) Exact fingerprint\")\n",
    "    print(\"   sha256:\", fp.get(\"sha256\"))\n",
    "    print(\"   -> SHA256 is an exact content fingerprint: two files with the same SHA256 are byte-for-byte identical.\")\n",
    "    print()\n",
    "\n",
    "    print(\"3) Text shingles (used for near-duplicate text detection)\")\n",
    "    shingles = fp.get(\"text_shingles\")\n",
    "    if shingles:\n",
    "        print(f\"   {len(shingles)} shingles (overlapping phrases):\")\n",
    "        for s in sorted(shingles):\n",
    "            print(\"    -\", s)\n",
    "        print(\"\\n   What this means:\")\n",
    "        print(\"    - Each shingle is a short overlapping phrase (here 3-word phrases).\")\n",
    "        print(\"    - To estimate text similarity between two files we compute the Jaccard index between their shingle sets\")\n",
    "        print(\"      (intersection size / union size). More shared shingles -> more similar text.\")\n",
    "        print()\n",
    "\n",
    "        # quick verification so the user sees where the shingles come from\n",
    "        def word_shingles(text, k=3):\n",
    "            toks = text.lower().split()\n",
    "            if len(toks) < k:\n",
    "                return set()\n",
    "            return {\" \".join(toks[i : i + k]) for i in range(len(toks) - k + 1)}\n",
    "\n",
    "        print(\"   Verification (recomputing 3-word shingles from the file contents):\")\n",
    "        print(\"   \", word_shingles(text, k=3))\n",
    "    else:\n",
    "        print(\"   No text shingles were computed for this file.\")\n",
    "        print(\"   (If compute_text=True was passed, check profiler configuration or file contents.)\")\n",
    "\n",
    "    print()\n",
    "    print(\"4) Image hash field (not relevant here):\", fp.get(\"image_hash\"))\n",
    "    print(\"\\nSummary: the fingerprint contains an exact hash (SHA256) for byte-perfect deduping and\")\n",
    "    print(\"a set of text shingles useful for near-duplicate detection using Jaccard/MinHash approaches.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2645f5b",
   "metadata": {},
   "source": [
    "## Using ImageProfiler to compute perceptual hashes\n",
    "\n",
    "What does filoma's `ImageProfiler` provide for deduplication?\n",
    "\n",
    "- Use `ImageProfiler` when you already treat images as data (numpy arrays) or when you want a consistent interface for computing image statistics and perceptual hashes.\n",
    "- Perceptual hashes let you cluster or filter visually-similar images before more expensive visual comparisons.\n",
    "\n",
    "The `ImageProfiler` exposes `compute_ahash` / `compute_dhash` which delegate to `filoma.dedup` for consistent hash computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3f4ca395",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using existing ImageProfiler instance: <filoma.images.image_profiler.ImageProfiler object at 0x79862ffc6950>\n",
      "\n",
      "Created a test image at: /tmp/tmpuks7fj71/img.png\n",
      " Image size: (32, 32) mode: RGB\n",
      "\n",
      "Computed perceptual hashes:\n",
      " - aHash (average hash): ffffffffffffffff\n",
      "   -> aHash summarizes average light/dark pattern; small visual edits usually produce small Hamming distances.\n",
      " - dHash (difference hash): 0000000000000000\n",
      "   -> dHash captures gradient differences; complementary sensitivity to aHash.\n",
      "\n",
      "Verification: recomputing aHash on the same file yields identical value: True\n",
      " Recomputed aHash: ffffffffffffffff\n",
      "\n",
      "Example Hamming distance (aHash vs same aHash): 0 (0 means identical)\n",
      "\n",
      "Notes:\n",
      " - To detect near-duplicate images, compute hashes for many files and compare Hamming distances.\n",
      " - Small distances indicate visually similar images; choose a threshold based on your tolerance for differences.\n",
      " - When comparing aHash and dHash directly, be careful: different algorithms may have different bit lengths/meanings.\n"
     ]
    }
   ],
   "source": [
    "# ImageProfiler example with clear, human-friendly explanations.\n",
    "\n",
    "ip = ImageProfiler()  # create an ImageProfiler instance for the example\n",
    "\n",
    "# We use the existing ImageProfiler instance `ip` (do not recreate it).\n",
    "if not _HAS_PIL:\n",
    "    print(\"Pillow is not available; skipping ImageProfiler example\")\n",
    "else:\n",
    "    print(\"Using existing ImageProfiler instance:\", ip)\n",
    "    with tempfile.TemporaryDirectory() as td:\n",
    "        img_path = os.path.join(td, \"img.png\")\n",
    "\n",
    "        # Create a small RGB image and save it to a temp file.\n",
    "        new_img = Image.new(\"RGB\", (32, 32), color=(10, 20, 30))\n",
    "        new_img.save(img_path)\n",
    "        print(\"\\nCreated a test image at:\", img_path)\n",
    "        print(\" Image size:\", new_img.size, \"mode:\", new_img.mode)\n",
    "\n",
    "        # Compute perceptual hashes via ImageProfiler.\n",
    "        ahash = ip.compute_ahash(img_path)\n",
    "        dhash = ip.compute_dhash(img_path)\n",
    "\n",
    "        # Explain what each hash is and show the values.\n",
    "        print(\"\\nComputed perceptual hashes:\")\n",
    "        print(\" - aHash (average hash):\", ahash)\n",
    "        print(\"   -> aHash summarizes average light/dark pattern; small visual edits usually produce small Hamming distances.\")\n",
    "        print(\" - dHash (difference hash):\", dhash)\n",
    "        print(\"   -> dHash captures gradient differences; complementary sensitivity to aHash.\")\n",
    "\n",
    "        # Quick verification: computing the same hash twice on the same file should give identical results.\n",
    "        ahash2 = ip.compute_ahash(img_path)\n",
    "        print(\"\\nVerification: recomputing aHash on the same file yields identical value:\", ahash2 == ahash)\n",
    "        print(\" Recomputed aHash:\", ahash2)\n",
    "\n",
    "        # Show how you'd compare two hashes (Hamming distance example).\n",
    "        # For demonstration we compare the aHash with itself (distance should be 0).\n",
    "        try:\n",
    "            xor = int(ahash, 16) ^ int(ahash2, 16)\n",
    "            ham_dist = xor.bit_count()\n",
    "            print(\"\\nExample Hamming distance (aHash vs same aHash):\", ham_dist, \"(0 means identical)\")\n",
    "        except Exception:\n",
    "            print(\"\\nCould not compute Hamming distance (unexpected hash format).\")\n",
    "\n",
    "        # Short guidance for next steps.\n",
    "        print(\"\\nNotes:\")\n",
    "        print(\" - To detect near-duplicate images, compute hashes for many files and compare Hamming distances.\")\n",
    "        print(\" - Small distances indicate visually similar images; choose a threshold based on your tolerance for differences.\")\n",
    "        print(\" - When comparing aHash and dHash directly, be careful: different algorithms may have different bit lengths/meanings.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "169bd78d",
   "metadata": {},
   "source": [
    "## DataFrame convenience: evaluate_duplicates()\n",
    "\n",
    "What does `DataFrame.evaluate_duplicates()` provide for deduplication?\n",
    "\n",
    "- `DataFrame.evaluate_duplicates()` is a convenience for quickly assessing duplicates in a DataFrame that contains file paths.\n",
    "- It's useful for exploratory data analysis and quick cleaning steps, but for production-scale deduplication you should prefer MinHash/LSH or a specialized image search index.\n",
    "- The method returns groups and can be used to export or mark duplicates for removal.\n",
    "\n",
    "`DataFrame.evaluate_duplicates()` scans the `path` column and prints a small Rich summary table. It returns the raw groups for programmatic use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d7dbdb47",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">         Duplicate Summary          </span>\n",
       "┏━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> Type  </span>┃<span style=\"font-weight: bold\"> Groups </span>┃<span style=\"font-weight: bold\"> Files In Groups </span>┃\n",
       "┡━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> exact </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> 0      </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> 0               </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> text  </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> 1      </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> 2               </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> image </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> 0      </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> 0               </span>│\n",
       "└───────┴────────┴─────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m         Duplicate Summary          \u001b[0m\n",
       "┏━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mType \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mGroups\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mFiles In Groups\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mexact\u001b[0m\u001b[1;36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m0     \u001b[0m\u001b[37m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m0              \u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mtext \u001b[0m\u001b[1;36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m1     \u001b[0m\u001b[37m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m2              \u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mimage\u001b[0m\u001b[1;36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m0     \u001b[0m\u001b[37m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m0              \u001b[0m\u001b[37m \u001b[0m│\n",
       "└───────┴────────┴─────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m2025-09-13 22:43:56.446\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mfiloma.dataframe\u001b[0m:\u001b[36mevaluate_duplicates\u001b[0m:\u001b[36m915\u001b[0m - \u001b[1mDuplicate summary: exact=0 groups (0 files), text=1 groups (2 files), image=0 groups (0 files)\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Returned groups:\n",
      "{'exact': [], 'text': [['/tmp/tmps6z965tp/a.txt', '/tmp/tmps6z965tp/b.txt']], 'image': []}\n",
      "DataFrame we'll evaluate (contains file paths):\n",
      "filoma.DataFrame with 2 rows\n",
      "shape: (2, 1)\n",
      "┌────────────────────────┐\n",
      "│ path                   │\n",
      "│ ---                    │\n",
      "│ str                    │\n",
      "╞════════════════════════╡\n",
      "│ /tmp/tmps6z965tp/a.txt │\n",
      "│ /tmp/tmps6z965tp/b.txt │\n",
      "└────────────────────────┘\n",
      "\n",
      "Plan:\n",
      "- Run df.evaluate_duplicates(text_k=3, text_threshold=0.4)\n",
      "- This checks for exact (SHA256) duplicates and text near-duplicates using k-word shingles + Jaccard.\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">         Duplicate Summary          </span>\n",
       "┏━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> Type  </span>┃<span style=\"font-weight: bold\"> Groups </span>┃<span style=\"font-weight: bold\"> Files In Groups </span>┃\n",
       "┡━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> exact </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> 1      </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> 2               </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> text  </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> 0      </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> 0               </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> image </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> 0      </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> 0               </span>│\n",
       "└───────┴────────┴─────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m         Duplicate Summary          \u001b[0m\n",
       "┏━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mType \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mGroups\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mFiles In Groups\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mexact\u001b[0m\u001b[1;36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m1     \u001b[0m\u001b[37m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m2              \u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mtext \u001b[0m\u001b[1;36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m0     \u001b[0m\u001b[37m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m0              \u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mimage\u001b[0m\u001b[1;36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m0     \u001b[0m\u001b[37m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m0              \u001b[0m\u001b[37m \u001b[0m│\n",
       "└───────┴────────┴─────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m2025-09-13 22:43:56.449\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mfiloma.dataframe\u001b[0m:\u001b[36mevaluate_duplicates\u001b[0m:\u001b[36m915\u001b[0m - \u001b[1mDuplicate summary: exact=1 groups (2 files), text=0 groups (0 files), image=0 groups (0 files)\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Raw returned groups:\n",
      "{'exact': [['/tmp/tmps6z965tp/a.txt', '/tmp/tmps6z965tp/b.txt']], 'text': [], 'image': []}\n",
      "\n",
      "Interpretation:\n",
      " - Exact duplicate groups (byte-for-byte identical):\n",
      "   > ['/tmp/tmps6z965tp/a.txt', '/tmp/tmps6z965tp/b.txt']\n",
      " - No text near-duplicate groups found (threshold not met).\n",
      "\n",
      "Summary: evaluate_duplicates scanned the 'path' column, computed exact hashes and text shingles,\n",
      "and grouped files whose shingle-based Jaccard met the threshold. To change sensitivity, adjust text_k or text_threshold.\n"
     ]
    }
   ],
   "source": [
    "# Friendly, step-by-step explanation of DataFrame.evaluate_duplicates()\n",
    "\n",
    "# Build a DataFrame from the two text files used earlier and run evaluation\n",
    "with tempfile.TemporaryDirectory() as td:\n",
    "    p1 = os.path.join(td, \"a.txt\")\n",
    "    p2 = os.path.join(td, \"b.txt\")\n",
    "    with open(p1, \"w\") as f:\n",
    "        f.write(\"the quick brown fox jumps over the lazy dog\")\n",
    "    with open(p2, \"w\") as f:\n",
    "        f.write(\"the quick brown fox jumped over the lazy dog\")\n",
    "\n",
    "    df = DataFrame([p1, p2])\n",
    "    groups = df.evaluate_duplicates(text_threshold=0.4, show_table=True)\n",
    "    print(\"Returned groups:\")\n",
    "    print(groups)\n",
    "\n",
    "\n",
    "print(\"DataFrame we'll evaluate (contains file paths):\")\n",
    "print(df)\n",
    "print()\n",
    "\n",
    "print(\"Plan:\")\n",
    "print(f\"- Run df.evaluate_duplicates(text_k={k}, text_threshold={text_threshold})\")\n",
    "print(\"- This checks for exact (SHA256) duplicates and text near-duplicates using k-word shingles + Jaccard.\")\n",
    "print()\n",
    "\n",
    "groups = df.evaluate_duplicates(text_k=k, text_threshold=text_threshold, show_table=True)\n",
    "print(\"\\nRaw returned groups:\")\n",
    "print(groups)\n",
    "print()\n",
    "\n",
    "# Human-friendly interpretation using the shingle/Jaccard info already computed in the notebook\n",
    "print(\"Interpretation:\")\n",
    "if groups.get(\"exact\"):\n",
    "    print(\" - Exact duplicate groups (byte-for-byte identical):\")\n",
    "    for grp in groups[\"exact\"]:\n",
    "        print(\"   >\", grp)\n",
    "else:\n",
    "    print(\" - No exact duplicates found.\")\n",
    "\n",
    "if groups.get(\"text\"):\n",
    "    print(\" - Text near-duplicate groups found:\")\n",
    "    print(f\"   The two files were grouped because their {k}-word Jaccard similarity = {jaccard:.3f} >= threshold {text_threshold}.\")\n",
    "    print(\"   Shingles for file A (example):\", s1)\n",
    "    print(\"   Shingles for file B (example):\", s2)\n",
    "    print(\"   Shared shingles (intersection):\", inter)\n",
    "    print(f\"   Jaccard = {len(inter)}/{len(union)} = {jaccard:.3f}\")\n",
    "else:\n",
    "    print(\" - No text near-duplicate groups found (threshold not met).\")\n",
    "\n",
    "if groups.get(\"image\"):\n",
    "    print(\" - Image groups (not relevant for these text files):\", groups[\"image\"])\n",
    "\n",
    "print(\"\\nSummary: evaluate_duplicates scanned the 'path' column, computed exact hashes and text shingles,\")\n",
    "print(\"and grouped files whose shingle-based Jaccard met the threshold. To change sensitivity, adjust text_k or text_threshold.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "baceef36",
   "metadata": {},
   "source": [
    "## Closing notes\n",
    "- For large datasets consider using `datasketch.MinHash` + LSH to scale text similarity.\n",
    "- For image deduping at scale consider using perceptual hashes + a nearest-neighbor index.\n",
    "- `DataFrame.evaluate_duplicates()` is intended as a quick way to get actionable groups; you can export the groups and apply cleaning workflows (drop, label, or move duplicates)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "filoma (3.11.11)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
