{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell_header",
   "metadata": {},
   "source": [
    "# Slowly Changing Dimension Type 2 (SCD2)\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lakelogic/LakeLogic/blob/main/examples/02_core_patterns/scd2_dimension/playbook.ipynb) \n",
    "[![GitHub Repo](https://img.shields.io/badge/GitHub-Repo-blue?logo=github)](https://github.com/lakelogic/LakeLogic/blob/main/examples/02_core_patterns/scd2_dimension/playbook.ipynb)\n",
    "\n",
    "## Business Scenario\n",
    "\n",
    "Customer attributes change over time — a status flip, a name correction, a tier upgrade.\n",
    "Reporting and compliance teams need to know what was true **on a specific date**, not just\n",
    "the latest snapshot. Without versioned history, point-in-time queries are impossible.\n",
    "\n",
    "## Value Proposition\n",
    "\n",
    "- Preserve full history of every attribute change\n",
    "- Track active vs. expired record versions with `is_current` and `effective_to`\n",
    "- Enable time-travel analytics with zero custom code\n",
    "\n",
    "---\n",
    "\n",
    "## Goals\n",
    "\n",
    "1. Load Snapshot 1 — seed the dimension table with initial records\n",
    "2. Load Snapshot 2 — detect changes, close old rows, open new versions\n",
    "3. Inspect the SCD2 output (both `is_current = true` and closed-out history rows)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_setup_heading",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "cell_setup",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lakelogic ready.\n",
      "Cleared: D:\\Github\\_SaaS\\lakelogic\\examples\\02_core_patterns\\scd2_dimension\\output\n",
      "Setup complete.\n"
     ]
    }
   ],
   "source": [
    "# ── Setup ─────────────────────────────────────────────────────────────────────\n",
    "# Single cell: installs lakelogic, clones the repo on Colab, resolves paths,\n",
    "# and clears any previous output so the run is clean.\n",
    "# Safe to re-run.\n",
    "\n",
    "import importlib.util\n",
    "import os\n",
    "import sys\n",
    "import shutil\n",
    "import polars as pl\n",
    "from pathlib import Path\n",
    "\n",
    "# 1. Install lakelogic if missing\n",
    "if importlib.util.find_spec(\"lakelogic\") is None:\n",
    "    print(\"Installing lakelogic ...\")\n",
    "    import subprocess\n",
    "\n",
    "    subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"lakelogic\", \"-q\"], check=True)\n",
    "    print(\"lakelogic installed.\")\n",
    "else:\n",
    "    print(\"lakelogic ready.\")\n",
    "\n",
    "# 2. On Google Colab, clone the repo so YAML/CSV files are available\n",
    "if \"google.colab\" in sys.modules:\n",
    "    repo = Path(\"/content/LakeLogic\")\n",
    "    if not repo.exists():\n",
    "        print(\"Cloning LakeLogic repo ...\")\n",
    "        import subprocess\n",
    "\n",
    "        subprocess.run(\n",
    "            [\n",
    "                \"git\",\n",
    "                \"clone\",\n",
    "                \"--quiet\",\n",
    "                \"https://github.com/lakelogic/LakeLogic.git\",\n",
    "                str(repo),\n",
    "            ],\n",
    "            check=True,\n",
    "        )\n",
    "    example_dir = repo / \"examples\" / \"02_core_patterns\" / \"scd2_dimension\"\n",
    "    os.chdir(example_dir)\n",
    "    print(f\"Working directory: {Path.cwd()}\")\n",
    "\n",
    "\n",
    "# 3. Path helper — resolves paths whether running locally or on Colab\n",
    "def get_path(*parts: str) -> Path:\n",
    "    \"\"\"Return an absolute path relative to this notebook's directory.\"\"\"\n",
    "    base = Path.cwd()\n",
    "    for candidate in [base] + list(base.parents):\n",
    "        target = candidate / \"scd2_dimension\" if candidate.name != \"scd2_dimension\" else candidate\n",
    "        if (target / \"contract.yaml\").exists():\n",
    "            return (target / Path(*parts)).resolve()\n",
    "    return (base / Path(*parts)).resolve()\n",
    "\n",
    "\n",
    "# 4. Clear previous output so each full run starts clean\n",
    "output_dir = get_path(\"output\")\n",
    "if output_dir.exists():\n",
    "    shutil.rmtree(output_dir)\n",
    "    print(f\"Cleared: {output_dir}\")\n",
    "\n",
    "from lakelogic import DataProcessor\n",
    "\n",
    "print(\"Setup complete.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_how_it_works",
   "metadata": {},
   "source": [
    "## How It Works\n",
    "\n",
    "LakeLogic's **`scd2` materialization strategy** manages versioned dimension rows\n",
    "automatically. You declare field names and run snapshots — LakeLogic handles the\n",
    "`effective_from`, `effective_to`, and `is_current` bookkeeping.\n",
    "\n",
    "### SCD2 mechanics\n",
    "\n",
    "```\n",
    "Snapshot 1  (3 rows: Alice active, Bob active, Carla inactive)\n",
    "      |\n",
    "      v  DataProcessor.run_source() + materialize()\n",
    "      |\n",
    "      +---> gold_dim_customers/data.csv\n",
    "              customer_id | name  | status   | updated_at | effective_to | is_current\n",
    "              1           | Alice | active   | 2024-01-01 | 9999-12-31   | true\n",
    "              2           | Bob   | active   | 2024-01-01 | 9999-12-31   | true\n",
    "              3           | Carla | inactive | 2024-01-01 | 9999-12-31   | true\n",
    "\n",
    "\n",
    "Snapshot 2  (3 rows: Bobby renamed, Carla flipped, Dan new)\n",
    "      |\n",
    "      v  DataProcessor.run_source() + materialize()\n",
    "      |    For each row that CHANGED vs. current dimension:\n",
    "      |      - Close old version: set effective_to = new updated_at, is_current = false\n",
    "      |      - Open new version: effective_to = 9999-12-31, is_current = true\n",
    "      |    For NEW rows: insert with is_current = true\n",
    "      |    For UNCHANGED rows: leave as-is\n",
    "      |\n",
    "      +---> gold_dim_customers/data.csv\n",
    "              customer_id | name  | status   | updated_at | effective_to | is_current\n",
    "              1           | Alice | active   | 2024-01-01 | 9999-12-31   | true   (unchanged)\n",
    "              2           | Bob   | active   | 2024-01-01 | 2024-02-01   | false  (closed - name changed)\n",
    "              2           | Bobby | active   | 2024-02-01 | 9999-12-31   | true   (new version)\n",
    "              3           | Carla | inactive | 2024-01-01 | 2024-02-01   | false  (closed - status changed)\n",
    "              3           | Carla | active   | 2024-02-01 | 9999-12-31   | true   (new version)\n",
    "              4           | Dan   | active   | 2024-02-01 | 9999-12-31   | true   (new customer)\n",
    "```\n",
    "\n",
    "### Key contract settings\n",
    "\n",
    "| Setting | Value | What it does |\n",
    "|---|---|---|\n",
    "| `materialization.strategy` | `scd2` | Enables versioned history tracking |\n",
    "| `scd2.effective_from_field` | `updated_at` | Source field used as the version start date |\n",
    "| `scd2.effective_to_field` | `effective_to` | Added column: `9999-12-31` = still current |\n",
    "| `scd2.current_flag_field` | `is_current` | Added column: `true` = latest version |\n",
    "| `primary_key` | `customer_id` | Key used to match incoming rows to existing dimension |\n",
    "| `quality: valid_status` | `status IN ('active','inactive')` | Quarantines rows with unknown status |\n",
    "\n",
    "### Snapshot data\n",
    "\n",
    "**Snapshot 1** (`data/dim_customers_snapshot1.csv`):\n",
    "```\n",
    "customer_id,name,status,updated_at\n",
    "1,Alice,active,2024-01-01\n",
    "2,Bob,active,2024-01-01\n",
    "3,Carla,inactive,2024-01-01\n",
    "```\n",
    "\n",
    "**Snapshot 2** (`data/dim_customers_snapshot2.csv`):\n",
    "```\n",
    "customer_id,name,status,updated_at\n",
    "2,Bobby,active,2024-02-01     ← name changed (Bob → Bobby)\n",
    "3,Carla,active,2024-02-01     ← status changed (inactive → active)\n",
    "4,Dan,active,2024-02-01       ← new customer\n",
    "```\n",
    "Note: Alice (customer_id=1) is **absent** from Snapshot 2 — she is unchanged.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_peek_heading",
   "metadata": {},
   "source": [
    "## 1. Preview Input Snapshots\n",
    "\n",
    "A look at both input CSV files before we run the processor.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "cell_peek",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Snapshot 1 — initial load (3 customers)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr,\n",
       ".dataframe > tbody > tr {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (3, 4)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>customer_id</th><th>name</th><th>status</th><th>updated_at</th></tr><tr><td>i64</td><td>str</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>1</td><td>&quot;Alice&quot;</td><td>&quot;active&quot;</td><td>&quot;2024-01-01&quot;</td></tr><tr><td>2</td><td>&quot;Bob&quot;</td><td>&quot;active&quot;</td><td>&quot;2024-01-01&quot;</td></tr><tr><td>3</td><td>&quot;Carla&quot;</td><td>&quot;inactive&quot;</td><td>&quot;2024-01-01&quot;</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (3, 4)\n",
       "┌─────────────┬───────┬──────────┬────────────┐\n",
       "│ customer_id ┆ name  ┆ status   ┆ updated_at │\n",
       "│ ---         ┆ ---   ┆ ---      ┆ ---        │\n",
       "│ i64         ┆ str   ┆ str      ┆ str        │\n",
       "╞═════════════╪═══════╪══════════╪════════════╡\n",
       "│ 1           ┆ Alice ┆ active   ┆ 2024-01-01 │\n",
       "│ 2           ┆ Bob   ┆ active   ┆ 2024-01-01 │\n",
       "│ 3           ┆ Carla ┆ inactive ┆ 2024-01-01 │\n",
       "└─────────────┴───────┴──────────┴────────────┘"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Snapshot 2 — incremental update (Bob renamed, Carla flipped, Dan new)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr,\n",
       ".dataframe > tbody > tr {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (3, 4)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>customer_id</th><th>name</th><th>status</th><th>updated_at</th></tr><tr><td>i64</td><td>str</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>2</td><td>&quot;Bobby&quot;</td><td>&quot;active&quot;</td><td>&quot;2024-02-01&quot;</td></tr><tr><td>3</td><td>&quot;Carla&quot;</td><td>&quot;active&quot;</td><td>&quot;2024-02-01&quot;</td></tr><tr><td>4</td><td>&quot;Dan&quot;</td><td>&quot;active&quot;</td><td>&quot;2024-02-01&quot;</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (3, 4)\n",
       "┌─────────────┬───────┬────────┬────────────┐\n",
       "│ customer_id ┆ name  ┆ status ┆ updated_at │\n",
       "│ ---         ┆ ---   ┆ ---    ┆ ---        │\n",
       "│ i64         ┆ str   ┆ str    ┆ str        │\n",
       "╞═════════════╪═══════╪════════╪════════════╡\n",
       "│ 2           ┆ Bobby ┆ active ┆ 2024-02-01 │\n",
       "│ 3           ┆ Carla ┆ active ┆ 2024-02-01 │\n",
       "│ 4           ┆ Dan   ┆ active ┆ 2024-02-01 │\n",
       "└─────────────┴───────┴────────┴────────────┘"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "snap1_path = get_path(\"data\", \"dim_customers_snapshot1.csv\")\n",
    "snap2_path = get_path(\"data\", \"dim_customers_snapshot2.csv\")\n",
    "\n",
    "print(\"Snapshot 1 — initial load (3 customers)\")\n",
    "display(pl.read_csv(snap1_path))\n",
    "\n",
    "print(\"Snapshot 2 — incremental update (Bob renamed, Carla flipped, Dan new)\")\n",
    "display(pl.read_csv(snap2_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_snap1_heading",
   "metadata": {},
   "source": [
    "## 2. Run Snapshot 1 — Seed the Dimension\n",
    "\n",
    "First run: all rows are new, so every customer gets `is_current = true`\n",
    "and `effective_to = 9999-12-31`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "cell_snap1",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m2026-03-01 15:49:29.476\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mlakelogic.core.processor\u001b[0m:\u001b[36mrun_source\u001b[0m:\u001b[36m647\u001b[0m - \u001b[1mLoading source: D:\\Github\\_SaaS\\lakelogic\\examples\\02_core_patterns\\scd2_dimension\\data\\dim_customers_snapshot1.csv via polars\u001b[0m\n",
      "\u001b[32m2026-03-01 15:49:29.483\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mlakelogic.core.processor\u001b[0m:\u001b[36mrun\u001b[0m:\u001b[36m446\u001b[0m - \u001b[1mRun complete. Source: 3, Total (post-transform): 3, Good: 3, Quarantined: 0, Pre-Transform Dropped: 0, Ratio: 0.00%\u001b[0m\n",
      "\u001b[32m2026-03-01 15:49:29.491\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mlakelogic.core.materialization\u001b[0m:\u001b[36mmaterialize_dataframe\u001b[0m:\u001b[36m1539\u001b[0m - \u001b[1mMaterialized 3 rows to D:\\Github\\_SaaS\\lakelogic\\examples\\02_core_patterns\\scd2_dimension\\output\\gold_dim_customers\\data.csv\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source: 3  |  Good: 3  |  Quarantined: 0\n",
      "\n",
      "SCD2 TABLE AFTER SNAPSHOT 1:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr,\n",
       ".dataframe > tbody > tr {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (3, 10)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>customer_id</th><th>name</th><th>status</th><th>updated_at</th><th>_lakelogic_source</th><th>_lakelogic_processed_at</th><th>_lakelogic_run_id</th><th>effective_from</th><th>effective_to</th><th>is_current</th></tr><tr><td>i64</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>bool</td></tr></thead><tbody><tr><td>1</td><td>&quot;Alice&quot;</td><td>&quot;active&quot;</td><td>&quot;2024-01-01&quot;</td><td>&quot;D:\\Github\\_SaaS\\lakelogic\\exam…</td><td>&quot;2026-03-01 15:49:29+00:00&quot;</td><td>&quot;8220dae5-e86d-4d8e-98f8-d9d5d3…</td><td>&quot;2026-03-01T15:49:29+00:00&quot;</td><td>null</td><td>true</td></tr><tr><td>2</td><td>&quot;Bob&quot;</td><td>&quot;active&quot;</td><td>&quot;2024-01-01&quot;</td><td>&quot;D:\\Github\\_SaaS\\lakelogic\\exam…</td><td>&quot;2026-03-01 15:49:29+00:00&quot;</td><td>&quot;8220dae5-e86d-4d8e-98f8-d9d5d3…</td><td>&quot;2026-03-01T15:49:29+00:00&quot;</td><td>null</td><td>true</td></tr><tr><td>3</td><td>&quot;Carla&quot;</td><td>&quot;inactive&quot;</td><td>&quot;2024-01-01&quot;</td><td>&quot;D:\\Github\\_SaaS\\lakelogic\\exam…</td><td>&quot;2026-03-01 15:49:29+00:00&quot;</td><td>&quot;8220dae5-e86d-4d8e-98f8-d9d5d3…</td><td>&quot;2026-03-01T15:49:29+00:00&quot;</td><td>null</td><td>true</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (3, 10)\n",
       "┌────────────┬───────┬──────────┬────────────┬───┬────────────┬────────────┬───────────┬───────────┐\n",
       "│ customer_i ┆ name  ┆ status   ┆ updated_at ┆ … ┆ _lakelogic ┆ effective_ ┆ effective ┆ is_curren │\n",
       "│ d          ┆ ---   ┆ ---      ┆ ---        ┆   ┆ _run_id    ┆ from       ┆ _to       ┆ t         │\n",
       "│ ---        ┆ str   ┆ str      ┆ str        ┆   ┆ ---        ┆ ---        ┆ ---       ┆ ---       │\n",
       "│ i64        ┆       ┆          ┆            ┆   ┆ str        ┆ str        ┆ str       ┆ bool      │\n",
       "╞════════════╪═══════╪══════════╪════════════╪═══╪════════════╪════════════╪═══════════╪═══════════╡\n",
       "│ 1          ┆ Alice ┆ active   ┆ 2024-01-01 ┆ … ┆ 8220dae5-e ┆ 2026-03-01 ┆ null      ┆ true      │\n",
       "│            ┆       ┆          ┆            ┆   ┆ 86d-4d8e-9 ┆ T15:49:29+ ┆           ┆           │\n",
       "│            ┆       ┆          ┆            ┆   ┆ 8f8-d9d5d3 ┆ 00:00      ┆           ┆           │\n",
       "│            ┆       ┆          ┆            ┆   ┆ …          ┆            ┆           ┆           │\n",
       "│ 2          ┆ Bob   ┆ active   ┆ 2024-01-01 ┆ … ┆ 8220dae5-e ┆ 2026-03-01 ┆ null      ┆ true      │\n",
       "│            ┆       ┆          ┆            ┆   ┆ 86d-4d8e-9 ┆ T15:49:29+ ┆           ┆           │\n",
       "│            ┆       ┆          ┆            ┆   ┆ 8f8-d9d5d3 ┆ 00:00      ┆           ┆           │\n",
       "│            ┆       ┆          ┆            ┆   ┆ …          ┆            ┆           ┆           │\n",
       "│ 3          ┆ Carla ┆ inactive ┆ 2024-01-01 ┆ … ┆ 8220dae5-e ┆ 2026-03-01 ┆ null      ┆ true      │\n",
       "│            ┆       ┆          ┆            ┆   ┆ 86d-4d8e-9 ┆ T15:49:29+ ┆           ┆           │\n",
       "│            ┆       ┆          ┆            ┆   ┆ 8f8-d9d5d3 ┆ 00:00      ┆           ┆           │\n",
       "│            ┆       ┆          ┆            ┆   ┆ …          ┆            ┆           ┆           │\n",
       "└────────────┴───────┴──────────┴────────────┴───┴────────────┴────────────┴───────────┴───────────┘"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "contract_path = get_path(\"contract.yaml\")\n",
    "output_csv = get_path(\"output\", \"gold_dim_customers\", \"data.csv\")\n",
    "\n",
    "processor = DataProcessor(contract=contract_path)\n",
    "result1 = processor.run_source(snap1_path)\n",
    "processor.materialize(result1.good, result1.bad)\n",
    "\n",
    "report = processor.last_report or {}\n",
    "counts = report.get(\"counts\", {})\n",
    "print(f\"Source: {counts.get('source')}  |  Good: {counts.get('good')}  |  Quarantined: {counts.get('quarantined')}\")\n",
    "\n",
    "print(\"\\nSCD2 TABLE AFTER SNAPSHOT 1:\")\n",
    "display(pl.read_csv(output_csv))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_snap2_heading",
   "metadata": {},
   "source": [
    "## 3. Run Snapshot 2 — Apply Incremental Changes\n",
    "\n",
    "Second run: LakeLogic compares incoming rows to the current dimension.\n",
    "- **Changed rows** (Bob's name, Carla's status): old version closed, new version opened\n",
    "- **New row** (Dan): inserted as current\n",
    "- **Unchanged row** (Alice): untouched\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "cell_snap2",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m2026-03-01 15:49:29.513\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mlakelogic.core.processor\u001b[0m:\u001b[36mrun_source\u001b[0m:\u001b[36m647\u001b[0m - \u001b[1mLoading source: D:\\Github\\_SaaS\\lakelogic\\examples\\02_core_patterns\\scd2_dimension\\data\\dim_customers_snapshot2.csv via polars\u001b[0m\n",
      "\u001b[32m2026-03-01 15:49:29.520\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mlakelogic.core.processor\u001b[0m:\u001b[36mrun\u001b[0m:\u001b[36m446\u001b[0m - \u001b[1mRun complete. Source: 3, Total (post-transform): 3, Good: 3, Quarantined: 0, Pre-Transform Dropped: 0, Ratio: 0.00%\u001b[0m\n",
      "\u001b[32m2026-03-01 15:49:29.543\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mlakelogic.core.materialization\u001b[0m:\u001b[36mmaterialize_dataframe\u001b[0m:\u001b[36m1539\u001b[0m - \u001b[1mMaterialized 6 rows to D:\\Github\\_SaaS\\lakelogic\\examples\\02_core_patterns\\scd2_dimension\\output\\gold_dim_customers\\data.csv\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source: 3  |  Good: 3  |  Quarantined: 0\n",
      "\n",
      "SCD2 TABLE AFTER SNAPSHOT 2 (full history):\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr,\n",
       ".dataframe > tbody > tr {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (6, 10)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>customer_id</th><th>name</th><th>status</th><th>updated_at</th><th>_lakelogic_source</th><th>_lakelogic_processed_at</th><th>_lakelogic_run_id</th><th>effective_from</th><th>effective_to</th><th>is_current</th></tr><tr><td>i64</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>bool</td></tr></thead><tbody><tr><td>1</td><td>&quot;Alice&quot;</td><td>&quot;active&quot;</td><td>&quot;2024-01-01&quot;</td><td>&quot;D:\\Github\\_SaaS\\lakelogic\\exam…</td><td>&quot;2026-03-01 15:49:29+00:00&quot;</td><td>&quot;8220dae5-e86d-4d8e-98f8-d9d5d3…</td><td>&quot;2026-03-01T15:49:29+00:00&quot;</td><td>null</td><td>true</td></tr><tr><td>2</td><td>&quot;Bob&quot;</td><td>&quot;active&quot;</td><td>&quot;2024-01-01&quot;</td><td>&quot;D:\\Github\\_SaaS\\lakelogic\\exam…</td><td>&quot;2026-03-01 15:49:29+00:00&quot;</td><td>&quot;8220dae5-e86d-4d8e-98f8-d9d5d3…</td><td>&quot;2026-03-01T15:49:29+00:00&quot;</td><td>&quot;2026-03-01T15:49:29+00:00&quot;</td><td>false</td></tr><tr><td>2</td><td>&quot;Bobby&quot;</td><td>&quot;active&quot;</td><td>&quot;2024-02-01 00:00:00&quot;</td><td>&quot;D:\\Github\\_SaaS\\lakelogic\\exam…</td><td>&quot;2026-03-01 15:49:29+00:00&quot;</td><td>&quot;8220dae5-e86d-4d8e-98f8-d9d5d3…</td><td>&quot;2026-03-01T15:49:29+00:00&quot;</td><td>null</td><td>true</td></tr><tr><td>3</td><td>&quot;Carla&quot;</td><td>&quot;inactive&quot;</td><td>&quot;2024-01-01&quot;</td><td>&quot;D:\\Github\\_SaaS\\lakelogic\\exam…</td><td>&quot;2026-03-01 15:49:29+00:00&quot;</td><td>&quot;8220dae5-e86d-4d8e-98f8-d9d5d3…</td><td>&quot;2026-03-01T15:49:29+00:00&quot;</td><td>&quot;2026-03-01T15:49:29+00:00&quot;</td><td>false</td></tr><tr><td>3</td><td>&quot;Carla&quot;</td><td>&quot;active&quot;</td><td>&quot;2024-02-01 00:00:00&quot;</td><td>&quot;D:\\Github\\_SaaS\\lakelogic\\exam…</td><td>&quot;2026-03-01 15:49:29+00:00&quot;</td><td>&quot;8220dae5-e86d-4d8e-98f8-d9d5d3…</td><td>&quot;2026-03-01T15:49:29+00:00&quot;</td><td>null</td><td>true</td></tr><tr><td>4</td><td>&quot;Dan&quot;</td><td>&quot;active&quot;</td><td>&quot;2024-02-01 00:00:00&quot;</td><td>&quot;D:\\Github\\_SaaS\\lakelogic\\exam…</td><td>&quot;2026-03-01 15:49:29+00:00&quot;</td><td>&quot;8220dae5-e86d-4d8e-98f8-d9d5d3…</td><td>&quot;2026-03-01T15:49:29+00:00&quot;</td><td>null</td><td>true</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (6, 10)\n",
       "┌────────────┬───────┬──────────┬────────────┬───┬────────────┬────────────┬───────────┬───────────┐\n",
       "│ customer_i ┆ name  ┆ status   ┆ updated_at ┆ … ┆ _lakelogic ┆ effective_ ┆ effective ┆ is_curren │\n",
       "│ d          ┆ ---   ┆ ---      ┆ ---        ┆   ┆ _run_id    ┆ from       ┆ _to       ┆ t         │\n",
       "│ ---        ┆ str   ┆ str      ┆ str        ┆   ┆ ---        ┆ ---        ┆ ---       ┆ ---       │\n",
       "│ i64        ┆       ┆          ┆            ┆   ┆ str        ┆ str        ┆ str       ┆ bool      │\n",
       "╞════════════╪═══════╪══════════╪════════════╪═══╪════════════╪════════════╪═══════════╪═══════════╡\n",
       "│ 1          ┆ Alice ┆ active   ┆ 2024-01-01 ┆ … ┆ 8220dae5-e ┆ 2026-03-01 ┆ null      ┆ true      │\n",
       "│            ┆       ┆          ┆            ┆   ┆ 86d-4d8e-9 ┆ T15:49:29+ ┆           ┆           │\n",
       "│            ┆       ┆          ┆            ┆   ┆ 8f8-d9d5d3 ┆ 00:00      ┆           ┆           │\n",
       "│            ┆       ┆          ┆            ┆   ┆ …          ┆            ┆           ┆           │\n",
       "│ 2          ┆ Bob   ┆ active   ┆ 2024-01-01 ┆ … ┆ 8220dae5-e ┆ 2026-03-01 ┆ 2026-03-0 ┆ false     │\n",
       "│            ┆       ┆          ┆            ┆   ┆ 86d-4d8e-9 ┆ T15:49:29+ ┆ 1T15:49:2 ┆           │\n",
       "│            ┆       ┆          ┆            ┆   ┆ 8f8-d9d5d3 ┆ 00:00      ┆ 9+00:00   ┆           │\n",
       "│            ┆       ┆          ┆            ┆   ┆ …          ┆            ┆           ┆           │\n",
       "│ 2          ┆ Bobby ┆ active   ┆ 2024-02-01 ┆ … ┆ 8220dae5-e ┆ 2026-03-01 ┆ null      ┆ true      │\n",
       "│            ┆       ┆          ┆ 00:00:00   ┆   ┆ 86d-4d8e-9 ┆ T15:49:29+ ┆           ┆           │\n",
       "│            ┆       ┆          ┆            ┆   ┆ 8f8-d9d5d3 ┆ 00:00      ┆           ┆           │\n",
       "│            ┆       ┆          ┆            ┆   ┆ …          ┆            ┆           ┆           │\n",
       "│ 3          ┆ Carla ┆ inactive ┆ 2024-01-01 ┆ … ┆ 8220dae5-e ┆ 2026-03-01 ┆ 2026-03-0 ┆ false     │\n",
       "│            ┆       ┆          ┆            ┆   ┆ 86d-4d8e-9 ┆ T15:49:29+ ┆ 1T15:49:2 ┆           │\n",
       "│            ┆       ┆          ┆            ┆   ┆ 8f8-d9d5d3 ┆ 00:00      ┆ 9+00:00   ┆           │\n",
       "│            ┆       ┆          ┆            ┆   ┆ …          ┆            ┆           ┆           │\n",
       "│ 3          ┆ Carla ┆ active   ┆ 2024-02-01 ┆ … ┆ 8220dae5-e ┆ 2026-03-01 ┆ null      ┆ true      │\n",
       "│            ┆       ┆          ┆ 00:00:00   ┆   ┆ 86d-4d8e-9 ┆ T15:49:29+ ┆           ┆           │\n",
       "│            ┆       ┆          ┆            ┆   ┆ 8f8-d9d5d3 ┆ 00:00      ┆           ┆           │\n",
       "│            ┆       ┆          ┆            ┆   ┆ …          ┆            ┆           ┆           │\n",
       "│ 4          ┆ Dan   ┆ active   ┆ 2024-02-01 ┆ … ┆ 8220dae5-e ┆ 2026-03-01 ┆ null      ┆ true      │\n",
       "│            ┆       ┆          ┆ 00:00:00   ┆   ┆ 86d-4d8e-9 ┆ T15:49:29+ ┆           ┆           │\n",
       "│            ┆       ┆          ┆            ┆   ┆ 8f8-d9d5d3 ┆ 00:00      ┆           ┆           │\n",
       "│            ┆       ┆          ┆            ┆   ┆ …          ┆            ┆           ┆           │\n",
       "└────────────┴───────┴──────────┴────────────┴───┴────────────┴────────────┴───────────┴───────────┘"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "result2 = processor.run_source(snap2_path)\n",
    "processor.materialize(result2.good, result2.bad)\n",
    "\n",
    "report = processor.last_report or {}\n",
    "counts = report.get(\"counts\", {})\n",
    "print(f\"Source: {counts.get('source')}  |  Good: {counts.get('good')}  |  Quarantined: {counts.get('quarantined')}\")\n",
    "\n",
    "print(\"\\nSCD2 TABLE AFTER SNAPSHOT 2 (full history):\")\n",
    "display(pl.read_csv(output_csv).sort([\"customer_id\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_current_heading",
   "metadata": {},
   "source": [
    "## 4. Current Records Only\n",
    "\n",
    "Filter `is_current = true` to get the latest version of each customer —\n",
    "equivalent to a standard snapshot query.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "cell_current",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CURRENT RECORDS ONLY (is_current = true):\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr,\n",
       ".dataframe > tbody > tr {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (4, 10)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>customer_id</th><th>name</th><th>status</th><th>updated_at</th><th>_lakelogic_source</th><th>_lakelogic_processed_at</th><th>_lakelogic_run_id</th><th>effective_from</th><th>effective_to</th><th>is_current</th></tr><tr><td>i64</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>bool</td></tr></thead><tbody><tr><td>1</td><td>&quot;Alice&quot;</td><td>&quot;active&quot;</td><td>&quot;2024-01-01&quot;</td><td>&quot;D:\\Github\\_SaaS\\lakelogic\\exam…</td><td>&quot;2026-03-01 15:49:29+00:00&quot;</td><td>&quot;8220dae5-e86d-4d8e-98f8-d9d5d3…</td><td>&quot;2026-03-01T15:49:29+00:00&quot;</td><td>null</td><td>true</td></tr><tr><td>2</td><td>&quot;Bobby&quot;</td><td>&quot;active&quot;</td><td>&quot;2024-02-01 00:00:00&quot;</td><td>&quot;D:\\Github\\_SaaS\\lakelogic\\exam…</td><td>&quot;2026-03-01 15:49:29+00:00&quot;</td><td>&quot;8220dae5-e86d-4d8e-98f8-d9d5d3…</td><td>&quot;2026-03-01T15:49:29+00:00&quot;</td><td>null</td><td>true</td></tr><tr><td>3</td><td>&quot;Carla&quot;</td><td>&quot;active&quot;</td><td>&quot;2024-02-01 00:00:00&quot;</td><td>&quot;D:\\Github\\_SaaS\\lakelogic\\exam…</td><td>&quot;2026-03-01 15:49:29+00:00&quot;</td><td>&quot;8220dae5-e86d-4d8e-98f8-d9d5d3…</td><td>&quot;2026-03-01T15:49:29+00:00&quot;</td><td>null</td><td>true</td></tr><tr><td>4</td><td>&quot;Dan&quot;</td><td>&quot;active&quot;</td><td>&quot;2024-02-01 00:00:00&quot;</td><td>&quot;D:\\Github\\_SaaS\\lakelogic\\exam…</td><td>&quot;2026-03-01 15:49:29+00:00&quot;</td><td>&quot;8220dae5-e86d-4d8e-98f8-d9d5d3…</td><td>&quot;2026-03-01T15:49:29+00:00&quot;</td><td>null</td><td>true</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (4, 10)\n",
       "┌────────────┬───────┬────────┬────────────┬───┬────────────┬────────────┬────────────┬────────────┐\n",
       "│ customer_i ┆ name  ┆ status ┆ updated_at ┆ … ┆ _lakelogic ┆ effective_ ┆ effective_ ┆ is_current │\n",
       "│ d          ┆ ---   ┆ ---    ┆ ---        ┆   ┆ _run_id    ┆ from       ┆ to         ┆ ---        │\n",
       "│ ---        ┆ str   ┆ str    ┆ str        ┆   ┆ ---        ┆ ---        ┆ ---        ┆ bool       │\n",
       "│ i64        ┆       ┆        ┆            ┆   ┆ str        ┆ str        ┆ str        ┆            │\n",
       "╞════════════╪═══════╪════════╪════════════╪═══╪════════════╪════════════╪════════════╪════════════╡\n",
       "│ 1          ┆ Alice ┆ active ┆ 2024-01-01 ┆ … ┆ 8220dae5-e ┆ 2026-03-01 ┆ null       ┆ true       │\n",
       "│            ┆       ┆        ┆            ┆   ┆ 86d-4d8e-9 ┆ T15:49:29+ ┆            ┆            │\n",
       "│            ┆       ┆        ┆            ┆   ┆ 8f8-d9d5d3 ┆ 00:00      ┆            ┆            │\n",
       "│            ┆       ┆        ┆            ┆   ┆ …          ┆            ┆            ┆            │\n",
       "│ 2          ┆ Bobby ┆ active ┆ 2024-02-01 ┆ … ┆ 8220dae5-e ┆ 2026-03-01 ┆ null       ┆ true       │\n",
       "│            ┆       ┆        ┆ 00:00:00   ┆   ┆ 86d-4d8e-9 ┆ T15:49:29+ ┆            ┆            │\n",
       "│            ┆       ┆        ┆            ┆   ┆ 8f8-d9d5d3 ┆ 00:00      ┆            ┆            │\n",
       "│            ┆       ┆        ┆            ┆   ┆ …          ┆            ┆            ┆            │\n",
       "│ 3          ┆ Carla ┆ active ┆ 2024-02-01 ┆ … ┆ 8220dae5-e ┆ 2026-03-01 ┆ null       ┆ true       │\n",
       "│            ┆       ┆        ┆ 00:00:00   ┆   ┆ 86d-4d8e-9 ┆ T15:49:29+ ┆            ┆            │\n",
       "│            ┆       ┆        ┆            ┆   ┆ 8f8-d9d5d3 ┆ 00:00      ┆            ┆            │\n",
       "│            ┆       ┆        ┆            ┆   ┆ …          ┆            ┆            ┆            │\n",
       "│ 4          ┆ Dan   ┆ active ┆ 2024-02-01 ┆ … ┆ 8220dae5-e ┆ 2026-03-01 ┆ null       ┆ true       │\n",
       "│            ┆       ┆        ┆ 00:00:00   ┆   ┆ 86d-4d8e-9 ┆ T15:49:29+ ┆            ┆            │\n",
       "│            ┆       ┆        ┆            ┆   ┆ 8f8-d9d5d3 ┆ 00:00      ┆            ┆            │\n",
       "│            ┆       ┆        ┆            ┆   ┆ …          ┆            ┆            ┆            │\n",
       "└────────────┴───────┴────────┴────────────┴───┴────────────┴────────────┴────────────┴────────────┘"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "CLOSED HISTORY ROWS (is_current = false):\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr,\n",
       ".dataframe > tbody > tr {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (2, 10)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>customer_id</th><th>name</th><th>status</th><th>updated_at</th><th>_lakelogic_source</th><th>_lakelogic_processed_at</th><th>_lakelogic_run_id</th><th>effective_from</th><th>effective_to</th><th>is_current</th></tr><tr><td>i64</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>bool</td></tr></thead><tbody><tr><td>2</td><td>&quot;Bob&quot;</td><td>&quot;active&quot;</td><td>&quot;2024-01-01&quot;</td><td>&quot;D:\\Github\\_SaaS\\lakelogic\\exam…</td><td>&quot;2026-03-01 15:49:29+00:00&quot;</td><td>&quot;8220dae5-e86d-4d8e-98f8-d9d5d3…</td><td>&quot;2026-03-01T15:49:29+00:00&quot;</td><td>&quot;2026-03-01T15:49:29+00:00&quot;</td><td>false</td></tr><tr><td>3</td><td>&quot;Carla&quot;</td><td>&quot;inactive&quot;</td><td>&quot;2024-01-01&quot;</td><td>&quot;D:\\Github\\_SaaS\\lakelogic\\exam…</td><td>&quot;2026-03-01 15:49:29+00:00&quot;</td><td>&quot;8220dae5-e86d-4d8e-98f8-d9d5d3…</td><td>&quot;2026-03-01T15:49:29+00:00&quot;</td><td>&quot;2026-03-01T15:49:29+00:00&quot;</td><td>false</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (2, 10)\n",
       "┌────────────┬───────┬──────────┬────────────┬───┬────────────┬────────────┬───────────┬───────────┐\n",
       "│ customer_i ┆ name  ┆ status   ┆ updated_at ┆ … ┆ _lakelogic ┆ effective_ ┆ effective ┆ is_curren │\n",
       "│ d          ┆ ---   ┆ ---      ┆ ---        ┆   ┆ _run_id    ┆ from       ┆ _to       ┆ t         │\n",
       "│ ---        ┆ str   ┆ str      ┆ str        ┆   ┆ ---        ┆ ---        ┆ ---       ┆ ---       │\n",
       "│ i64        ┆       ┆          ┆            ┆   ┆ str        ┆ str        ┆ str       ┆ bool      │\n",
       "╞════════════╪═══════╪══════════╪════════════╪═══╪════════════╪════════════╪═══════════╪═══════════╡\n",
       "│ 2          ┆ Bob   ┆ active   ┆ 2024-01-01 ┆ … ┆ 8220dae5-e ┆ 2026-03-01 ┆ 2026-03-0 ┆ false     │\n",
       "│            ┆       ┆          ┆            ┆   ┆ 86d-4d8e-9 ┆ T15:49:29+ ┆ 1T15:49:2 ┆           │\n",
       "│            ┆       ┆          ┆            ┆   ┆ 8f8-d9d5d3 ┆ 00:00      ┆ 9+00:00   ┆           │\n",
       "│            ┆       ┆          ┆            ┆   ┆ …          ┆            ┆           ┆           │\n",
       "│ 3          ┆ Carla ┆ inactive ┆ 2024-01-01 ┆ … ┆ 8220dae5-e ┆ 2026-03-01 ┆ 2026-03-0 ┆ false     │\n",
       "│            ┆       ┆          ┆            ┆   ┆ 86d-4d8e-9 ┆ T15:49:29+ ┆ 1T15:49:2 ┆           │\n",
       "│            ┆       ┆          ┆            ┆   ┆ 8f8-d9d5d3 ┆ 00:00      ┆ 9+00:00   ┆           │\n",
       "│            ┆       ┆          ┆            ┆   ┆ …          ┆            ┆           ┆           │\n",
       "└────────────┴───────┴──────────┴────────────┴───┴────────────┴────────────┴───────────┴───────────┘"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "full_dim = pl.read_csv(output_csv)\n",
    "\n",
    "print(\"CURRENT RECORDS ONLY (is_current = true):\")\n",
    "display(full_dim.filter(pl.col(\"is_current\") == True))\n",
    "\n",
    "print(\"\\nCLOSED HISTORY ROWS (is_current = false):\")\n",
    "display(full_dim.filter(pl.col(\"is_current\") == False))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_summary",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Summary\n",
    "\n",
    "| Metric | Snapshot 1 | Snapshot 2 |\n",
    "|---|---|---|\n",
    "| Rows in source file | 3 | 3 |\n",
    "| New versions opened | 3 | 3 (2 updates + 1 new) |\n",
    "| Old versions closed | 0 | 2 (Bob + Carla) |\n",
    "| Total rows in dimension | 3 | 6 |\n",
    "| Current rows (`is_current = true`) | 3 | 4 |\n",
    "| Quarantined | 0 | 0 |\n",
    "\n",
    "### SCD2 outcome per customer\n",
    "\n",
    "| customer_id | Snapshot 1 | Snapshot 2 | Versions |\n",
    "|---|---|---|---|\n",
    "| 1 — Alice | active | *(absent — unchanged)* | 1 row, still current |\n",
    "| 2 — Bob | active `Bob` | name → `Bobby` | 2 rows: old closed, new current |\n",
    "| 3 — Carla | inactive | status → `active` | 2 rows: old closed, new current |\n",
    "| 4 — Dan | *(absent)* | new active | 1 row, current |\n",
    "\n",
    "### What LakeLogic did automatically\n",
    "\n",
    "- Compared each incoming row against the existing dimension using `primary_key: customer_id`\n",
    "- Detected attribute changes by comparing all non-key fields\n",
    "- Set `effective_to` on closed rows to the new snapshot's `updated_at` date\n",
    "- Set `effective_to = 9999-12-31` on all current rows\n",
    "- Added `_lakelogic_processed_at` and `_lakelogic_run_id` audit columns\n",
    "\n",
    "---\n",
    "\n",
    "## Next Steps — Try It Yourself\n",
    "\n",
    "### 1. Edit the snapshot files\n",
    "\n",
    "Open `data/dim_customers_snapshot1.csv`:\n",
    "\n",
    "```\n",
    "customer_id,name,status,updated_at\n",
    "1,Alice,active,2024-01-01\n",
    "2,Bob,active,2024-01-01\n",
    "3,Carla,inactive,2024-01-01\n",
    "```\n",
    "\n",
    "Open `data/dim_customers_snapshot2.csv`:\n",
    "\n",
    "```\n",
    "customer_id,name,status,updated_at\n",
    "2,Bobby,active,2024-02-01\n",
    "3,Carla,active,2024-02-01\n",
    "4,Dan,active,2024-02-01\n",
    "```\n",
    "\n",
    "**Ideas:**\n",
    "- Add a third snapshot file and call `processor.run_source(snap3_path)` — watch another\n",
    "  layer of history accumulate\n",
    "- Introduce a row with `status = \"suspended\"` → it will be quarantined by `valid_status`\n",
    "- Include Alice in Snapshot 2 but with a changed status → she will now have 2 versions too\n",
    "- Omit Bob from Snapshot 2 entirely (i.e. delete him) — observe that SCD2 leaves his\n",
    "  current row as-is (deletions are not tracked unless you add a soft-delete flag)\n",
    "\n",
    "### 2. Edit the contract\n",
    "\n",
    "Open `contract.yaml` and try:\n",
    "\n",
    "```yaml\n",
    "# Rename the SCD2 output columns to match your warehouse convention:\n",
    "materialization:\n",
    "  strategy: scd2\n",
    "  target_path: output/gold_dim_customers\n",
    "  format: csv\n",
    "  scd2:\n",
    "    effective_from_field: updated_at    # source field for version start\n",
    "    effective_to_field: valid_to        # rename column\n",
    "    current_flag_field: is_latest       # rename column\n",
    "\n",
    "# Add a new tracked attribute to the model:\n",
    "model:\n",
    "  fields:\n",
    "    - name: customer_id\n",
    "      type: int\n",
    "      required: true\n",
    "    - name: name\n",
    "      type: string\n",
    "    - name: status\n",
    "      type: string\n",
    "    - name: tier           # <-- new field — add to CSV and here\n",
    "      type: string\n",
    "    - name: updated_at\n",
    "      type: date\n",
    "\n",
    "# Add a new quality rule:\n",
    "quality:\n",
    "  row_rules:\n",
    "    - name: valid_status\n",
    "      sql: \"status IN ('active','inactive')\"\n",
    "      category: consistency\n",
    "    - name: name_not_empty\n",
    "      sql: \"name IS NOT NULL AND name != ''\"\n",
    "      category: completeness\n",
    "```\n",
    "\n",
    "**Key contract knobs:**\n",
    "\n",
    "| What to change | Where in contract.yaml |\n",
    "|---|---|\n",
    "| SCD2 column names | `materialization.scd2.*_field` |\n",
    "| Change-detection key | `primary_key` |\n",
    "| Version start timestamp | `scd2.effective_from_field` |\n",
    "| Output path / format | `materialization.target_path` / `format` |\n",
    "| Quality rules | `quality.row_rules` |\n",
    "\n",
    "### 3. Explore related playbooks\n",
    "\n",
    "- [`../dedup_survivorship/`](../dedup_survivorship/playbook.ipynb) — deduplicate before\n",
    "  feeding into a dimension\n",
    "- [`../medallion_architecture/`](../medallion_architecture/quickstart_tutorial.ipynb) —\n",
    "  build a full Bronze → Silver pipeline that feeds a Gold dimension\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv_lakelogic",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
