{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell_header",
   "metadata": {},
   "source": [
    "# Reference Joins\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lakelogic/LakeLogic/blob/main/examples/02_core_patterns/reference_joins/playbook.ipynb) \n",
    "[![GitHub Repo](https://img.shields.io/badge/GitHub-Repo-blue?logo=github)](https://github.com/lakelogic/LakeLogic/blob/main/examples/02_core_patterns/reference_joins/playbook.ipynb)\n",
    "\n",
    "## Business Scenario\n",
    "\n",
    "Customer records are often missing context like region or opt-out status.\n",
    "Without reference tables, analytics and compliance checks are inconsistent.\n",
    "You need a repeatable way to enrich records using trusted dimension tables —\n",
    "without writing custom join code in every pipeline.\n",
    "\n",
    "## Value Proposition\n",
    "\n",
    "- Declare reference tables once in the contract; reuse across pipelines\n",
    "- Enrich raw records with standard dimensions via SQL post-transforms\n",
    "- Validate referential integrity and quality in the same pass\n",
    "\n",
    "---\n",
    "\n",
    "## Goals\n",
    "\n",
    "1. Load source customer records and reference tables\n",
    "2. Join and enrich via SQL transforms declared in the contract\n",
    "3. Inspect enriched output and quarantined rejects\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_setup_heading",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_setup",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ── Setup ─────────────────────────────────────────────────────────────────────\n",
    "# Single cell: installs lakelogic, clones the repo on Colab, and resolves paths.\n",
    "# Safe to re-run; skips work already done.\n",
    "\n",
    "import importlib.util\n",
    "import os\n",
    "import sys\n",
    "import csv\n",
    "import polars as pl\n",
    "from pathlib import Path\n",
    "\n",
    "# 1. Install lakelogic if missing\n",
    "if importlib.util.find_spec(\"lakelogic\") is None:\n",
    "    print(\"Installing lakelogic ...\")\n",
    "    import subprocess\n",
    "\n",
    "    subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"lakelogic\", \"-q\"], check=True)\n",
    "    print(\"lakelogic installed.\")\n",
    "else:\n",
    "    print(\"lakelogic ready.\")\n",
    "\n",
    "# 2. On Google Colab, clone the repo so YAML/CSV files are available\n",
    "if \"google.colab\" in sys.modules:\n",
    "    repo = Path(\"/content/LakeLogic\")\n",
    "    if not repo.exists():\n",
    "        print(\"Cloning LakeLogic repo ...\")\n",
    "        import subprocess\n",
    "\n",
    "        subprocess.run(\n",
    "            [\n",
    "                \"git\",\n",
    "                \"clone\",\n",
    "                \"--quiet\",\n",
    "                \"https://github.com/lakelogic/LakeLogic.git\",\n",
    "                str(repo),\n",
    "            ],\n",
    "            check=True,\n",
    "        )\n",
    "    example_dir = repo / \"examples\" / \"02_core_patterns\" / \"reference_joins\"\n",
    "    os.chdir(example_dir)\n",
    "    print(f\"Working directory: {Path.cwd()}\")\n",
    "\n",
    "\n",
    "# 3. Path helper — resolves paths whether running locally or on Colab\n",
    "def get_path(*parts: str) -> Path:\n",
    "    \"\"\"Return an absolute path relative to this notebook's directory.\"\"\"\n",
    "    base = Path.cwd()\n",
    "    for candidate in [base] + list(base.parents):\n",
    "        target = candidate / \"reference_joins\" if candidate.name != \"reference_joins\" else candidate\n",
    "        if (target / \"contract.yaml\").exists():\n",
    "            return (target / Path(*parts)).resolve()\n",
    "    return (base / Path(*parts)).resolve()\n",
    "\n",
    "\n",
    "from lakelogic import DataProcessor\n",
    "\n",
    "print(\"Setup complete.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_how_it_works",
   "metadata": {},
   "source": [
    "## How It Works\n",
    "\n",
    "LakeLogic's **`links`** section in the YAML contract declares reference tables.\n",
    "They are loaded automatically alongside the source data and made available inside\n",
    "SQL transform blocks — no manual join code required.\n",
    "\n",
    "### Data flow\n",
    "\n",
    "```\n",
    "data/customers.csv          (9 rows — has nulls, duplicates, invalid tiers)\n",
    "        |\n",
    "        v  [pre-transforms]\n",
    "        |    1. rename   email_address --> email\n",
    "        |    2. filter   WHERE email IS NOT NULL          (drops row 4: NoEmail User)\n",
    "        |    3. dedup    on email, keep latest created_at (drops row 5: earliest duplicate)\n",
    "        |\n",
    "        v  [links loaded automatically]\n",
    "        |    dim_geography.csv      (country_id --> region:  1=USA, 2=UK, 3=Unknown)\n",
    "        |    marketing_opt_outs.csv (email --> opted_out:    opted@out.com=true)\n",
    "        |\n",
    "        v  [post-transform SQL]\n",
    "        |    LEFT JOIN dim_geography   --> region  (COALESCE to 'Unknown' if no match)\n",
    "        |    LEFT JOIN marketing_opt_outs --> opted_out (COALESCE to 'false' if no match)\n",
    "        |    CONCAT first_name + last_name --> full_name\n",
    "        |\n",
    "        v  [quality rules]\n",
    "        |    email_format:    email LIKE '%@%'          (rejects 'invalid-email')\n",
    "        |    valid_membership: IN ('GOLD','SILVER','BRONZE','PLATINUM') (rejects 'INVALID_TIER')\n",
    "        |\n",
    "        +---> good_df    enriched, validated records\n",
    "        +---> bad_df     quarantined rejects with _lakelogic_errors\n",
    "```\n",
    "\n",
    "### Key contract settings\n",
    "\n",
    "| Setting | Value | What it does |\n",
    "|---|---|---|\n",
    "| `links[0]` | `dim_geography.csv` | Maps `country_id` → `region` |\n",
    "| `links[1]` | `marketing_opt_outs.csv` | Maps `email` → `opted_out` |\n",
    "| `pre: rename` | `email_address` → `email` | Normalises field name from source |\n",
    "| `pre: filter` | `email IS NOT NULL` | Drops records with no email before dedup |\n",
    "| `pre: deduplicate` | on `email`, latest `created_at` | One canonical row per email |\n",
    "| `post: SQL JOIN` | LEFT JOIN both reference tables | Enriches each surviving row |\n",
    "| `quality: email_format` | `LIKE '%@%'` | Quarantines malformed emails |\n",
    "| `quality: valid_membership` | `IN (GOLD, SILVER, ...)` | Quarantines unknown tiers |\n",
    "| `quality: valid_rejoin` | `region IN ('USA','UK')` | Quarantines rows where country resolves to 'Unknown' |\n",
    "\n",
    "### Known issues in the raw data\n",
    "\n",
    "| Customer | Problem | Expected outcome |\n",
    "|---|---|---|\n",
    "| 4 — NoEmail User | `email_address` is empty | Dropped by pre-filter |\n",
    "| 5 — First Dup | Duplicate email (earlier `created_at`) | Dropped by dedup |\n",
    "| 6 — Second Dup | Duplicate email (later `created_at`) | Survives as canonical |\n",
    "| 7 — Bad Email | `email = \"invalid-email\"` | Quarantined (`email_format`) |\n",
    "| 8 — Old Timer | `membership_level = \"INVALID_TIER\"` | Quarantined (`valid_membership`) |\n",
    "| 3 — Billy Kid | `country_id = 1` (has match) | region = USA → passes `valid_rejoin` |\n",
    "| 9 — Opted Out | `email = opted@out.com` (in opt-out list) | `opted_out = true`, region = USA |\n",
    "| 4 — NoEmail User | `country_id = 3` (no match → 'Unknown') | Quarantined (`valid_rejoin`) after join |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_peek_heading",
   "metadata": {},
   "source": [
    "## 1. Preview Source Files\n",
    "\n",
    "A quick look at the raw records and the two reference tables before processing.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_peek",
   "metadata": {},
   "outputs": [],
   "source": [
    "customers_path = get_path(\"data\", \"customers.csv\")\n",
    "geography_path = get_path(\"data\", \"dim_geography.csv\")\n",
    "opt_outs_path = get_path(\"data\", \"marketing_opt_outs.csv\")\n",
    "\n",
    "# Keep raw dicts for the run step (DataProcessor expects list-of-dicts)\n",
    "with open(customers_path, newline=\"\", encoding=\"utf-8\") as f:\n",
    "    customers = list(csv.DictReader(f))\n",
    "\n",
    "print(\"customers.csv  —  9 rows\")\n",
    "display(pl.read_csv(customers_path))\n",
    "\n",
    "print(\"dim_geography.csv\")\n",
    "display(pl.read_csv(geography_path))\n",
    "\n",
    "print(\"marketing_opt_outs.csv\")\n",
    "display(pl.read_csv(opt_outs_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_run_heading",
   "metadata": {},
   "source": [
    "## 2. Run the Contract\n",
    "\n",
    "`DataProcessor` loads the source rows, resolves the `links` reference tables,\n",
    "applies all pre/post transforms, and enforces quality rules in one pass.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_run",
   "metadata": {},
   "outputs": [],
   "source": [
    "contract_path = get_path(\"contract.yaml\")\n",
    "\n",
    "processor = DataProcessor(contract=contract_path)\n",
    "result = processor.run(customers, source_path=str(customers_path))\n",
    "\n",
    "good_df = result.good\n",
    "bad_df = result.bad\n",
    "\n",
    "report = processor.last_report or {}\n",
    "counts = report.get(\"counts\", {})\n",
    "print(\n",
    "    f\"Source: {counts.get('source')}  |  \"\n",
    "    f\"Good: {counts.get('good')}  |  \"\n",
    "    f\"Pre-dropped: {counts.get('pre_transform_dropped')}  |  \"\n",
    "    f\"Quarantined: {counts.get('quarantined')}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_good_heading",
   "metadata": {},
   "source": [
    "## 3. Enriched Output (Good Records)\n",
    "\n",
    "Surviving records have `region`, `opted_out`, and `full_name` added by the\n",
    "reference joins and post-transform SQL.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_good",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"ENRICHED GOOD RECORDS:\")\n",
    "display(good_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_bad_heading",
   "metadata": {},
   "source": [
    "## 4. Quarantined Records\n",
    "\n",
    "Records that failed quality rules land here with `_lakelogic_errors` explaining why.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_bad",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"QUARANTINED RECORDS (with error reasons):\")\n",
    "display(bad_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_summary",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Summary\n",
    "\n",
    "| Metric | Value |\n",
    "|---|---|\n",
    "| Source rows loaded | 9 |\n",
    "| Dropped by pre-filter (null email) | 1 |\n",
    "| Dropped by dedup (older duplicate) | 1 |\n",
    "| Good (enriched output) | ~4 |\n",
    "| Quarantined (bad email / bad tier / unknown region) | ~3 |\n",
    "\n",
    "> Exact counts depend on rule execution order. Run the notebook to see live numbers.\n",
    "> The `valid_rejoin` rule (`region IN ('USA','UK')`) now also quarantines any customer\n",
    "> whose `country_id` has no match in `dim_geography` (resolved to `'Unknown'`).\n",
    "\n",
    "### Enrichment outcome per customer\n",
    "\n",
    "| customer_id | Email | region | opted_out | full_name | Outcome |\n",
    "|---|---|---|---|---|---|\n",
    "| 1 | john.doe@example.com | USA | false | John Doe | Good |\n",
    "| 2 | jane.smith@gmail.com | UK | false | Jane Smith | Good |\n",
    "| 3 | kid@outlook.com | USA | false | Billy Kid | Good |\n",
    "| 4 | *(empty)* | — | — | — | Dropped (null email pre-filter) |\n",
    "| 5 | duplicate@example.com | — | — | — | Dropped (older dedup) |\n",
    "| 6 | duplicate@example.com | USA | false | Second Dup | Good (latest) |\n",
    "| 7 | invalid-email | — | — | — | Quarantined (email_format) |\n",
    "| 8 | old@timer.com | USA | false | Old Timer | Quarantined (valid_membership) |\n",
    "| 9 | opted@out.com | USA | true | Opted Out | Good |\n",
    "| *(any country_id=3)* | *(no dim_geography match)* | Unknown | — | — | Quarantined (valid_rejoin) |\n",
    "\n",
    "### What LakeLogic did automatically\n",
    "\n",
    "- Renamed `email_address` → `email` in the pre-transform stage\n",
    "- Filtered null emails **before** dedup to avoid biasing the survivor selection\n",
    "- Resolved `dim_geography` and `marketing_opt_outs` as in-memory SQL tables\n",
    "  from the `links` section — no extra Python needed\n",
    "- Applied `COALESCE` for country_id = 3 (no geography match → `'Unknown'`)\n",
    "- Added `_lakelogic_processed_at`, `_lakelogic_run_id`, `_lakelogic_errors`\n",
    "  audit columns automatically\n",
    "\n",
    "---\n",
    "\n",
    "## Next Steps — Try It Yourself\n",
    "\n",
    "### 1. Edit the source customers\n",
    "\n",
    "Open `data/customers.csv` and try:\n",
    "\n",
    "```\n",
    "customer_id,email_address,first_name,last_name,birth_date,membership_level,country_id,created_at\n",
    "1,john.doe@example.com,John,Doe,1985-05-15,GOLD,1,2023-01-05\n",
    "2,jane.smith@gmail.com,Jane,Smith,1990-10-20,SILVER,2,2023-02-10\n",
    "3,kid@outlook.com,Billy,Kid,2015-01-01,BRONZE,1,2023-03-01\n",
    "4,,NoEmail,User,1970-01-01,PLATINUM,3,2022-12-15\n",
    "5,duplicate@example.com,First,Dup,1980-01-01,GOLD,1,2023-01-01\n",
    "6,duplicate@example.com,Second,Dup,1980-01-01,GOLD,1,2023-01-10\n",
    "7,invalid-email,Bad,Email,1990-01-01,BRONZE,2,2023-04-10\n",
    "8,old@timer.com,Old,Timer,1940-01-01,INVALID_TIER,1,2021-05-05\n",
    "9,opted@out.com,Opted,Out,1988-08-08,GOLD,1,2023-06-06\n",
    "```\n",
    "\n",
    "**Ideas:**\n",
    "- Add a new customer with `country_id = 3` → watch `region` become `'Unknown'`\n",
    "- Add a new email to `data/marketing_opt_outs.csv` → see `opted_out = 'true'` flow through\n",
    "- Add `country_id = 4` to `data/dim_geography.csv` as `CANADA` → then use it in a customer row\n",
    "\n",
    "### 2. Edit the reference tables\n",
    "\n",
    "Open `data/dim_geography.csv`:\n",
    "\n",
    "```\n",
    "country_id,region\n",
    "1,USA\n",
    "2,UK\n",
    "3,Canada\n",
    "```\n",
    "\n",
    "Open `data/marketing_opt_outs.csv`:\n",
    "\n",
    "```\n",
    "email,opted_out\n",
    "opted@out.com,true\n",
    "john.doe@example.com,true\n",
    "```\n",
    "\n",
    "### 3. Edit the contract\n",
    "\n",
    "Open `contract.yaml` and try:\n",
    "\n",
    "```yaml\n",
    "# Add a third reference link — e.g. a loyalty tier label lookup:\n",
    "links:\n",
    "  - name: dim_geography\n",
    "    path: ./data/dim_geography.csv\n",
    "    type: csv\n",
    "  - name: marketing_opt_outs\n",
    "    path: ./data/marketing_opt_outs.csv\n",
    "    type: csv\n",
    "  - name: tier_labels           # <-- new\n",
    "    path: ./data/tier_labels.csv\n",
    "    type: csv\n",
    "\n",
    "# Then use it in the post-transform SQL:\n",
    "transformations:\n",
    "  - sql: |\n",
    "      SELECT\n",
    "        src.*,\n",
    "        COALESCE(geo.region, 'Unknown') AS region,\n",
    "        COALESCE(opt.opted_out, 'false') AS opted_out,\n",
    "        COALESCE(tl.label, src.membership_level) AS tier_label,\n",
    "        src.first_name || ' ' || src.last_name AS full_name\n",
    "      FROM source src\n",
    "      LEFT JOIN dim_geography geo ON src.country_id = geo.country_id\n",
    "      LEFT JOIN marketing_opt_outs opt ON src.email = opt.email\n",
    "      LEFT JOIN tier_labels tl ON src.membership_level = tl.code\n",
    "    phase: post\n",
    "\n",
    "# Add a new quality rule — reject customers under 18:\n",
    "quality:\n",
    "  row_rules:\n",
    "    - name: adult_only\n",
    "      sql: \"DATEDIFF('year', birth_date, CURRENT_DATE) >= 18\"\n",
    "      category: business_logic\n",
    "```\n",
    "\n",
    "**Key contract knobs:**\n",
    "\n",
    "| What to change | Where in contract.yaml |\n",
    "|---|---|\n",
    "| Reference table paths | `links[*].path` |\n",
    "| Field rename | `transformations[rename].mappings` |\n",
    "| Pre-filter condition | `transformations[filter].sql` |\n",
    "| Dedup key / sort | `transformations[deduplicate].on` / `sort_by` |\n",
    "| Join SQL | `transformations[sql (phase: post)]` |\n",
    "| Quality rules | `quality.row_rules` |\n",
    "\n",
    "### 4. Explore related playbooks\n",
    "\n",
    "- [`../dedup_survivorship/`](../dedup_survivorship/playbook.ipynb) — focus on the dedup/survivorship pattern in isolation\n",
    "- [`../medallion_architecture/`](../medallion_architecture/quickstart_tutorial.ipynb) — combine joins with a full Bronze → Silver pipeline\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv_lakelogic",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
