{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell_header",
   "metadata": {},
   "source": [
    "# Dedup & Survivorship\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lakelogic/LakeLogic/blob/main/examples/02_core_patterns/dedup_survivorship/playbook.ipynb) \n",
    "[![GitHub Repo](https://img.shields.io/badge/GitHub-Repo-blue?logo=github)](https://github.com/lakelogic/LakeLogic/blob/main/examples/02_core_patterns/dedup_survivorship/playbook.ipynb)\n",
    "\n",
    "## Business Scenario\n",
    "\n",
    "CRMs and event streams often produce multiple updates for the same customer.\n",
    "Without survivorship rules, metrics inflate and teams act on stale attributes.\n",
    "\n",
    "## Value Proposition\n",
    "\n",
    "- Enforce one canonical record per entity\n",
    "- Keep the most recent or highest-quality version\n",
    "- Reduce downstream noise in analytics\n",
    "\n",
    "---\n",
    "\n",
    "## Goals\n",
    "\n",
    "1. Sort updates by timestamp\n",
    "2. Keep the best record per customer\n",
    "3. Return a clean, deduplicated dataset\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_setup_heading",
   "metadata": {},
   "source": [
    "## 1. Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_setup",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ── Setup ─────────────────────────────────────────────────────────────────────\n",
    "# Single cell: installs lakelogic, clones the repo on Colab, and resolves paths.\n",
    "# Safe to re-run; skips work already done.\n",
    "\n",
    "import importlib.util\n",
    "import os\n",
    "import sys\n",
    "import csv\n",
    "from pathlib import Path\n",
    "\n",
    "# 1. Install lakelogic if missing\n",
    "if importlib.util.find_spec(\"lakelogic\") is None:\n",
    "    print(\"Installing lakelogic ...\")\n",
    "    import subprocess\n",
    "\n",
    "    subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"lakelogic\", \"-q\"], check=True)\n",
    "    print(\"lakelogic installed.\")\n",
    "else:\n",
    "    print(\"lakelogic ready.\")\n",
    "\n",
    "# 2. On Google Colab, clone the repo so YAML/CSV files are available\n",
    "if \"google.colab\" in sys.modules:\n",
    "    repo = Path(\"/content/LakeLogic\")\n",
    "    if not repo.exists():\n",
    "        print(\"Cloning LakeLogic repo ...\")\n",
    "        import subprocess\n",
    "\n",
    "        subprocess.run(\n",
    "            [\n",
    "                \"git\",\n",
    "                \"clone\",\n",
    "                \"--quiet\",\n",
    "                \"https://github.com/lakelogic/LakeLogic.git\",\n",
    "                str(repo),\n",
    "            ],\n",
    "            check=True,\n",
    "        )\n",
    "    example_dir = repo / \"examples\" / \"02_core_patterns\" / \"dedup_survivorship\"\n",
    "    os.chdir(example_dir)\n",
    "    print(f\"Working directory: {Path.cwd()}\")\n",
    "\n",
    "\n",
    "# 3. Path helper — resolves paths whether running locally or on Colab\n",
    "def get_path(*parts: str) -> Path:\n",
    "    \"\"\"Return an absolute path relative to this notebook's directory.\"\"\"\n",
    "    base = Path(__file__).parent if \"__file__\" in dir() else Path.cwd()\n",
    "    # Walk up until we find the dedup_survivorship folder\n",
    "    for candidate in [base] + list(base.parents):\n",
    "        target = candidate / \"dedup_survivorship\" if candidate.name != \"dedup_survivorship\" else candidate\n",
    "        if (target / \"contract.yaml\").exists():\n",
    "            return (target / Path(*parts)).resolve()\n",
    "    return (base / Path(*parts)).resolve()\n",
    "\n",
    "\n",
    "from lakelogic import DataProcessor\n",
    "\n",
    "print(\"Setup complete.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_how_it_works",
   "metadata": {},
   "source": [
    "## How It Works\n",
    "\n",
    "LakeLogic uses a **YAML contract** to describe dedup and survivorship rules — no custom\n",
    "Python required.\n",
    "\n",
    "### Key contract settings for this example\n",
    "\n",
    "| Setting | Value | What it does |\n",
    "|---|---|---|\n",
    "| `dedup key` | `customer_id` | Groups rows by this field |\n",
    "| `sort field` | `updated_at DESC` | Picks the latest update as the survivor |\n",
    "| `post-transform` | `status = 'active' AS is_active` | Derives a boolean column from `status` |\n",
    "| `quality rules` | `email LIKE '%@%'`, `status IN (...)` | Quarantines invalid rows |\n",
    "\n",
    "### Data flow\n",
    "\n",
    "```\n",
    "customer_updates.csv  (5 rows, 2 customers appear twice)\n",
    "        |\n",
    "        v\n",
    "  [pre-transform]  ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY updated_at DESC)\n",
    "        |            → keeps rn = 1 (most recent row per customer)\n",
    "        v\n",
    "  [post-transform] Derives is_active from status\n",
    "        |\n",
    "        v\n",
    "  [quality rules]  email format check, valid status check\n",
    "        |\n",
    "        +---------> good_df   (3 canonical records)\n",
    "        +---------> bad_df    (0 quarantined in this dataset)\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_run_heading",
   "metadata": {},
   "source": [
    "## 2. Run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_run",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the CSV and run the processor\n",
    "contract = get_path(\"contract.yaml\")\n",
    "csv_file = get_path(\"data\", \"customer_updates.csv\")\n",
    "\n",
    "with open(csv_file, newline=\"\", encoding=\"utf-8\") as f:\n",
    "    data = list(csv.DictReader(f))\n",
    "\n",
    "processor = DataProcessor(contract=contract)\n",
    "result = processor.run(data)\n",
    "\n",
    "raw_df = result.raw\n",
    "good_df = result.good\n",
    "bad_df = result.bad\n",
    "\n",
    "report = processor.last_report or {}\n",
    "counts = report.get(\"counts\", {})\n",
    "print(\n",
    "    f\"Source: {counts.get('source')}  |  \"\n",
    "    f\"Good: {counts.get('good')}  |  \"\n",
    "    f\"Dropped (dedup): {counts.get('pre_transform_dropped')}  |  \"\n",
    "    f\"Quarantined: {counts.get('quarantined')}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_raw_heading",
   "metadata": {},
   "source": [
    "## Raw Input Data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_raw",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"RAW DATA (5 rows, note duplicate customer_ids 1 and 2)\")\n",
    "display(raw_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_good_heading",
   "metadata": {},
   "source": [
    "## Good Data — Survivors\n",
    "One canonical record per `customer_id` — always the row with the latest `updated_at`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_good",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"GOOD DATA (deduplicated)\")\n",
    "display(good_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_bad_heading",
   "metadata": {},
   "source": [
    "## Quarantined Data\n",
    "Rows that failed quality rules (e.g. invalid email, unknown status) land here with error reasons attached.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_bad",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"QUARANTINED DATA\")\n",
    "display(bad_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_summary",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Summary\n",
    "\n",
    "| Metric | Value |\n",
    "|---|---|\n",
    "| Source records ingested | 5 |\n",
    "| Canonical records (good) | 3 |\n",
    "| Duplicate rows dropped | 2 |\n",
    "| Quarantined (bad) records | 0 |\n",
    "\n",
    "### Survivorship outcome\n",
    "\n",
    "| customer_id | Rows in | Survivor `updated_at` | Survivor `status` | `is_active` |\n",
    "|---|---|---|---|---|\n",
    "| 1 (Alice) | 2 | 2024-02-01 | inactive | false |\n",
    "| 2 (Bob) | 2 | 2024-01-05 | active | true |\n",
    "| 3 (Carla) | 1 | 2024-01-20 | active | true |\n",
    "\n",
    "### What LakeLogic did automatically\n",
    "\n",
    "- Ran a window function (`ROW_NUMBER`) to rank rows per customer by recency — **no Pandas/Polars code needed**\n",
    "- Derived `is_active` via a SQL post-transform\n",
    "- Added `_lakelogic_processed_at`, `_lakelogic_run_id` audit columns to every output row\n",
    "- Reported schema drift (`is_active` not in source) as a warning, not a failure\n",
    "\n",
    "---\n",
    "\n",
    "## Next Steps — Try It Yourself\n",
    "\n",
    "The two files that drive this entire playbook are small and easy to edit.\n",
    "\n",
    "### 1. Edit the input data\n",
    "\n",
    "Open `data/customer_updates.csv` and try:\n",
    "\n",
    "```\n",
    "customer_id,email,status,updated_at,last_login\n",
    "1,alice@example.com,active,2024-01-01,2024-01-10\n",
    "1,alice@example.com,inactive,2024-02-01,2024-02-05\n",
    "2,bob@example.com,active,2024-01-05,2024-02-01\n",
    "2,bob@example.com,active,2023-12-20,2023-12-30\n",
    "3,carla@example.com,active,2024-01-20,2024-01-25\n",
    "```\n",
    "\n",
    "**Ideas:**\n",
    "- Add a row with `status = \"unknown\"` → watch it land in `bad_df` (fails `valid_status` rule)\n",
    "- Add a row with `email = \"notanemail\"` → quarantined for failing `email_format` rule\n",
    "- Add a 4th customer with 3 competing rows and verify only the latest survives\n",
    "\n",
    "### 2. Edit the contract\n",
    "\n",
    "Open `contract.yaml` and try:\n",
    "\n",
    "```yaml\n",
    "# Change the survivorship sort field — e.g. keep the OLDEST record instead:\n",
    "transformations:\n",
    "  - sql: |\n",
    "      SELECT * FROM (\n",
    "        SELECT *, ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY updated_at ASC) AS rn\n",
    "        FROM source\n",
    "      ) AS t\n",
    "      WHERE rn = 1\n",
    "    phase: pre\n",
    "\n",
    "# Add a new quality rule — e.g. reject records with no last_login:\n",
    "quality:\n",
    "  row_rules:\n",
    "    - name: has_last_login\n",
    "      sql: \"last_login IS NOT NULL\"\n",
    "      category: completeness\n",
    "```\n",
    "\n",
    "**Key contract knobs:**\n",
    "| What to change | Where in contract.yaml |\n",
    "|---|---|\n",
    "| Dedup / partition key | `PARTITION BY <field>` in pre-transform SQL |\n",
    "| Survivorship order | `ORDER BY <field> DESC/ASC` in pre-transform SQL |\n",
    "| Derived columns | `phase: post` SQL `SELECT` list |\n",
    "| Quality rules | `quality.row_rules` |\n",
    "| Quarantine behaviour | `quarantine.include_error_reason` |\n",
    "\n",
    "### 3. Explore related playbooks\n",
    "\n",
    "- [`../medallion_architecture/`](../medallion_architecture/quickstart_tutorial.ipynb) — combine dedup with a Bronze → Silver pipeline\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv_lakelogic",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
