{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell_header",
   "metadata": {},
   "source": [
    "# Medallion Architecture — Bronze to Silver\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lakelogic/LakeLogic/blob/main/examples/02_core_patterns/medallion_architecture/quickstart_tutorial.ipynb) \n",
    "[![GitHub Repo](https://img.shields.io/badge/GitHub-Repo-blue?logo=github)](https://github.com/lakelogic/LakeLogic/blob/main/examples/02_core_patterns/medallion_architecture/quickstart_tutorial.ipynb)\n",
    "\n",
    "## Business Scenario\n",
    "\n",
    "A CRM export contains duplicates, invalid emails, missing fields, and bad spend values.\n",
    "You need a **Bronze** layer to preserve every raw row, and a **Silver** layer to apply\n",
    "quality rules and produce analytics-ready data — without losing sight of what was rejected.\n",
    "\n",
    "## Value Proposition\n",
    "\n",
    "- Separate raw ingestion from validated outputs\n",
    "- Make failures visible without stopping ingestion\n",
    "- Create a repeatable, auditable Bronze → Silver pipeline in a single contract file\n",
    "\n",
    "---\n",
    "\n",
    "## Goals\n",
    "\n",
    "1. Run Bronze ingestion — capture all raw rows, even bad ones\n",
    "2. Apply Silver validation — deduplicate, type-cast, enforce business rules\n",
    "3. Inspect clean outputs and quarantined rejects\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_setup_heading",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "cell_setup",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lakelogic ready.\n",
      "Setup complete.\n"
     ]
    }
   ],
   "source": [
    "# ── Setup ─────────────────────────────────────────────────────────────────────\n",
    "# Single cell: installs lakelogic, clones the repo on Colab, and resolves paths.\n",
    "# Safe to re-run; skips work already done.\n",
    "\n",
    "import importlib.util\n",
    "import os\n",
    "import sys\n",
    "import csv\n",
    "import shutil\n",
    "from pathlib import Path\n",
    "\n",
    "# 1. Install lakelogic if missing\n",
    "if importlib.util.find_spec(\"lakelogic\") is None:\n",
    "    print(\"Installing lakelogic ...\")\n",
    "    import subprocess\n",
    "\n",
    "    subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"lakelogic\", \"-q\"], check=True)\n",
    "    print(\"lakelogic installed.\")\n",
    "else:\n",
    "    print(\"lakelogic ready.\")\n",
    "\n",
    "# 2. On Google Colab, clone the repo so YAML/CSV files are available\n",
    "if \"google.colab\" in sys.modules:\n",
    "    repo = Path(\"/content/LakeLogic\")\n",
    "    if not repo.exists():\n",
    "        print(\"Cloning LakeLogic repo ...\")\n",
    "        import subprocess\n",
    "\n",
    "        subprocess.run(\n",
    "            [\n",
    "                \"git\",\n",
    "                \"clone\",\n",
    "                \"--quiet\",\n",
    "                \"https://github.com/lakelogic/LakeLogic.git\",\n",
    "                str(repo),\n",
    "            ],\n",
    "            check=True,\n",
    "        )\n",
    "    example_dir = repo / \"examples\" / \"02_core_patterns\" / \"medallion_architecture\"\n",
    "    os.chdir(example_dir)\n",
    "    print(f\"Working directory: {Path.cwd()}\")\n",
    "\n",
    "\n",
    "# 3. Path helper — resolves paths whether running locally or on Colab\n",
    "def get_path(*parts: str) -> Path:\n",
    "    \"\"\"Return an absolute path relative to this notebook's directory.\"\"\"\n",
    "    base = Path.cwd()\n",
    "    for candidate in [base] + list(base.parents):\n",
    "        target = candidate / \"medallion_architecture\" if candidate.name != \"medallion_architecture\" else candidate\n",
    "        if (target / \"contract.yaml\").exists():\n",
    "            return (target / Path(*parts)).resolve()\n",
    "    return (base / Path(*parts)).resolve()\n",
    "\n",
    "\n",
    "from lakelogic import DataProcessor\n",
    "\n",
    "print(\"Setup complete.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_how_it_works",
   "metadata": {},
   "source": [
    "## How It Works\n",
    "\n",
    "LakeLogic uses a **single YAML contract** to define both pipeline stages.\n",
    "No orchestration framework, no custom transform code.\n",
    "\n",
    "### Pipeline stages\n",
    "\n",
    "```\n",
    "data/crm_*.csv  (8 rows — duplicates, bad emails, negative spend, non-numeric values)\n",
    "      |\n",
    "      v  [Bronze stage]\n",
    "      |    - Reads all matching CSV files via glob pattern\n",
    "      |    - No type enforcement, no quality rules — capture everything\n",
    "      |    - Deduplication is OFF (Bronze preserves raw volume)\n",
    "      |    - Writes Parquet to data/bronze/bronze_customers.parquet\n",
    "      |\n",
    "      v  [Silver stage  (default)]\n",
    "      |    - Reads Bronze Parquet\n",
    "      |    - Renames plan_type --> tier\n",
    "      |    - Deduplicates on customer_id (keeps latest signup_date)\n",
    "      |    - Enforces model types (date, float, boolean)\n",
    "      |    - Applies quality rules: valid email regex, total_spend >= 0\n",
    "      |    - Merges into data/silver/silver_customers.parquet\n",
    "      |\n",
    "      +---> good_df    production-ready records\n",
    "      +---> bad_df     quarantined rejects with _lakelogic_errors\n",
    "```\n",
    "\n",
    "### Key contract settings\n",
    "\n",
    "| Setting | Bronze | Silver |\n",
    "|---|---|---|\n",
    "| Source | `data/crm_*.csv` (glob) | `data/bronze/bronze_customers.parquet` |\n",
    "| Quality rules | None | email regex + positive spend |\n",
    "| Deduplication | Off | `customer_id`, sort by `signup_date DESC` |\n",
    "| Output | Append Parquet | Merge Parquet |\n",
    "| Run logs | — | DuckDB (`logs/lakelogic_run_logs.duckdb`) |\n",
    "\n",
    "### Known issues in the raw data\n",
    "\n",
    "| Customer | Problem | Expected outcome |\n",
    "|---|---|---|\n",
    "| 102 Bob | Invalid email format | Quarantined at Silver |\n",
    "| 103 Charlie | Missing email | Quarantined at Silver |\n",
    "| 104 Dup User | Appears twice (2024-01-15 and 2024-01-18) | Latest row survives |\n",
    "| 105 Eve | Negative total_spend (-500) | Quarantined at Silver |\n",
    "| 106 Frank | `signup_date = \"not-a-date\"` | Quarantined at Silver |\n",
    "| 107 Grace | `total_spend = \"high\"` (non-numeric) | Quarantined at Silver |\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "cell_helpers",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ── Helper functions ──────────────────────────────────────────────────────────\n",
    "\n",
    "\n",
    "def reset_outputs():\n",
    "    \"\"\"Delete any previously written Bronze/Silver Parquet files for a clean run.\"\"\"\n",
    "    targets = [\n",
    "        get_path(\"data\", \"bronze\", \"bronze_customers.parquet\"),\n",
    "        get_path(\"data\", \"silver\", \"silver_customers.parquet\"),\n",
    "    ]\n",
    "    for t in targets:\n",
    "        if t.exists():\n",
    "            shutil.rmtree(t) if t.is_dir() else t.unlink()\n",
    "\n",
    "\n",
    "def preview_frame(df, title=None, limit=10):\n",
    "    \"\"\"Print up to `limit` rows from any DataFrame-like object or list of dicts.\"\"\"\n",
    "    if title:\n",
    "        print(title)\n",
    "    if df is None:\n",
    "        print(\"<empty>\")\n",
    "        return\n",
    "    # Polars / Spark / list support\n",
    "    for method, call in [\n",
    "        (\"show\", lambda: df.show(limit, truncate=False)),\n",
    "        (\"head\", lambda: display(df.head(limit))),\n",
    "        (\"limit\", lambda: display(df.limit(limit))),\n",
    "    ]:\n",
    "        if hasattr(df, method):\n",
    "            try:\n",
    "                call()\n",
    "                return\n",
    "            except Exception:\n",
    "                pass\n",
    "    if isinstance(df, list):\n",
    "        for row in df[:limit]:\n",
    "            print(row)\n",
    "        return\n",
    "    display(df)\n",
    "\n",
    "\n",
    "def print_run_summary(proc, title=None):\n",
    "    \"\"\"Print a compact run summary from a DataProcessor.\"\"\"\n",
    "    if title:\n",
    "        print(title)\n",
    "    report = proc.last_report or {}\n",
    "    counts = report.get(\"counts\") or {}\n",
    "    print(\n",
    "        {\n",
    "            \"stage\": report.get(\"stage\"),\n",
    "            \"source\": counts.get(\"source\"),\n",
    "            \"good\": counts.get(\"good\"),\n",
    "            \"quarantined\": counts.get(\"quarantined\"),\n",
    "        }\n",
    "    )\n",
    "\n",
    "\n",
    "contract_path = get_path(\"contract.yaml\")\n",
    "data_path = get_path(\"data\", \"crm_export.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_peek_heading",
   "metadata": {},
   "source": [
    "## 1. Peek at the Raw Data\n",
    "\n",
    "Let's confirm what's coming in before we run anything.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "cell_peek",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Raw rows: 8\n",
      "{'customer_id': '101', 'name': 'Alice Smith', 'email': 'alice.smith@example.com', 'signup_date': '2024-01-15', 'plan_type': 'premium', 'total_spend': '1250.50', 'is_active': 'true'}\n",
      "{'customer_id': '102', 'name': 'Bob Jones', 'email': 'bob.jones_invalid_email', 'signup_date': '2024-01-16', 'plan_type': 'basic', 'total_spend': '45.00', 'is_active': 'true'}\n",
      "{'customer_id': '103', 'name': 'Charlie Brown', 'email': '', 'signup_date': '2024-01-17', 'plan_type': 'free', 'total_spend': '0', 'is_active': 'true'}\n",
      "{'customer_id': '104', 'name': 'Duplicate User', 'email': 'dup@example.com', 'signup_date': '2024-01-15', 'plan_type': 'free', 'total_spend': '10.00', 'is_active': 'true'}\n",
      "{'customer_id': '104', 'name': 'Duplicate User', 'email': 'dup@example.com', 'signup_date': '2024-01-18', 'plan_type': 'free', 'total_spend': '25.00', 'is_active': 'true'}\n",
      "{'customer_id': '105', 'name': 'Eve Adams', 'email': 'eve@example.com', 'signup_date': '2024-02-01', 'plan_type': 'premium', 'total_spend': '-500', 'is_active': 'false'}\n",
      "{'customer_id': '106', 'name': 'Frank White', 'email': 'frank@example.com', 'signup_date': 'not-a-date', 'plan_type': 'basic', 'total_spend': '100', 'is_active': 'true'}\n",
      "{'customer_id': '107', 'name': 'Grace Lee', 'email': 'grace@example.com', 'signup_date': '2024-02-10', 'plan_type': 'gold', 'total_spend': 'high', 'is_active': 'true'}\n"
     ]
    }
   ],
   "source": [
    "if not data_path.exists():\n",
    "    raise FileNotFoundError(f\"Missing input: {data_path}\")\n",
    "\n",
    "with open(data_path, newline=\"\", encoding=\"utf-8\") as f:\n",
    "    rows = list(csv.DictReader(f))\n",
    "\n",
    "print(f\"Raw rows: {len(rows)}\")\n",
    "for row in rows:\n",
    "    print(row)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_bronze_heading",
   "metadata": {},
   "source": [
    "## 2. Bronze Ingestion\n",
    "\n",
    "Bronze captures **all** raw rows — no type enforcement, no quality gates.\n",
    "Its job is to land data exactly as received and write it to Parquet for Silver to consume.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "cell_bronze",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m2026-03-01 13:33:09.791\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mlakelogic.core.processor\u001b[0m:\u001b[36mrun_source\u001b[0m:\u001b[36m647\u001b[0m - \u001b[1mLoading source: data\\crm_*.csv via polars\u001b[0m\n",
      "\u001b[32m2026-03-01 13:33:09.798\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mlakelogic.core.processor\u001b[0m:\u001b[36mrun\u001b[0m:\u001b[36m451\u001b[0m - \u001b[1mRun complete. [domain=customer_analytics, system=crm_export] Source: 8, Total: 8, Pre-Transform Dropped: 0\u001b[0m\n",
      "\u001b[32m2026-03-01 13:33:09.868\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mlakelogic.core.run_log\u001b[0m:\u001b[36m_write_run_log_table\u001b[0m:\u001b[36m325\u001b[0m - \u001b[1mWrote run log to DuckDB table lakelogic_run_logs (D:\\Github\\_SaaS\\lakelogic\\examples\\02_core_patterns\\medallion_architecture\\logs\\lakelogic_run_logs.duckdb)\u001b[0m\n",
      "\u001b[32m2026-03-01 13:33:09.870\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mlakelogic.core.materialization\u001b[0m:\u001b[36mmaterialize_dataframe\u001b[0m:\u001b[36m1299\u001b[0m - \u001b[1mMaterialized 8 rows to D:\\Github\\_SaaS\\lakelogic\\examples\\02_core_patterns\\medallion_architecture\\data\\bronze\\bronze_customers.parquet\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bronze summary:\n",
      "{'stage': 'bronze', 'source': 8, 'good': 8, 'quarantined': 0}\n",
      "BRONZE OUTPUT (sample):\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr,\n",
       ".dataframe > tbody > tr {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (8, 12)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>customer_id</th><th>name</th><th>email</th><th>signup_date</th><th>plan_type</th><th>total_spend</th><th>is_active</th><th>_lakelogic_source</th><th>_lakelogic_processed_at</th><th>_lakelogic_run_id</th><th>_lakelogic_domain</th><th>_lakelogic_system</th></tr><tr><td>i64</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>bool</td><td>str</td><td>datetime[μs, UTC]</td><td>str</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>101</td><td>&quot;Alice Smith&quot;</td><td>&quot;alice.smith@example.com&quot;</td><td>&quot;2024-01-15&quot;</td><td>&quot;premium&quot;</td><td>&quot;1250.50&quot;</td><td>true</td><td>&quot;data\\crm_*.csv&quot;</td><td>2026-03-01 13:33:09 UTC</td><td>&quot;400d1be9-d908-48f9-b9b8-2c4079…</td><td>&quot;customer_analytics&quot;</td><td>&quot;crm_export&quot;</td></tr><tr><td>102</td><td>&quot;Bob Jones&quot;</td><td>&quot;bob.jones_invalid_email&quot;</td><td>&quot;2024-01-16&quot;</td><td>&quot;basic&quot;</td><td>&quot;45.00&quot;</td><td>true</td><td>&quot;data\\crm_*.csv&quot;</td><td>2026-03-01 13:33:09 UTC</td><td>&quot;400d1be9-d908-48f9-b9b8-2c4079…</td><td>&quot;customer_analytics&quot;</td><td>&quot;crm_export&quot;</td></tr><tr><td>103</td><td>&quot;Charlie Brown&quot;</td><td>null</td><td>&quot;2024-01-17&quot;</td><td>&quot;free&quot;</td><td>&quot;0&quot;</td><td>true</td><td>&quot;data\\crm_*.csv&quot;</td><td>2026-03-01 13:33:09 UTC</td><td>&quot;400d1be9-d908-48f9-b9b8-2c4079…</td><td>&quot;customer_analytics&quot;</td><td>&quot;crm_export&quot;</td></tr><tr><td>104</td><td>&quot;Duplicate User&quot;</td><td>&quot;dup@example.com&quot;</td><td>&quot;2024-01-15&quot;</td><td>&quot;free&quot;</td><td>&quot;10.00&quot;</td><td>true</td><td>&quot;data\\crm_*.csv&quot;</td><td>2026-03-01 13:33:09 UTC</td><td>&quot;400d1be9-d908-48f9-b9b8-2c4079…</td><td>&quot;customer_analytics&quot;</td><td>&quot;crm_export&quot;</td></tr><tr><td>104</td><td>&quot;Duplicate User&quot;</td><td>&quot;dup@example.com&quot;</td><td>&quot;2024-01-18&quot;</td><td>&quot;free&quot;</td><td>&quot;25.00&quot;</td><td>true</td><td>&quot;data\\crm_*.csv&quot;</td><td>2026-03-01 13:33:09 UTC</td><td>&quot;400d1be9-d908-48f9-b9b8-2c4079…</td><td>&quot;customer_analytics&quot;</td><td>&quot;crm_export&quot;</td></tr><tr><td>105</td><td>&quot;Eve Adams&quot;</td><td>&quot;eve@example.com&quot;</td><td>&quot;2024-02-01&quot;</td><td>&quot;premium&quot;</td><td>&quot;-500&quot;</td><td>false</td><td>&quot;data\\crm_*.csv&quot;</td><td>2026-03-01 13:33:09 UTC</td><td>&quot;400d1be9-d908-48f9-b9b8-2c4079…</td><td>&quot;customer_analytics&quot;</td><td>&quot;crm_export&quot;</td></tr><tr><td>106</td><td>&quot;Frank White&quot;</td><td>&quot;frank@example.com&quot;</td><td>&quot;not-a-date&quot;</td><td>&quot;basic&quot;</td><td>&quot;100&quot;</td><td>true</td><td>&quot;data\\crm_*.csv&quot;</td><td>2026-03-01 13:33:09 UTC</td><td>&quot;400d1be9-d908-48f9-b9b8-2c4079…</td><td>&quot;customer_analytics&quot;</td><td>&quot;crm_export&quot;</td></tr><tr><td>107</td><td>&quot;Grace Lee&quot;</td><td>&quot;grace@example.com&quot;</td><td>&quot;2024-02-10&quot;</td><td>&quot;gold&quot;</td><td>&quot;high&quot;</td><td>true</td><td>&quot;data\\crm_*.csv&quot;</td><td>2026-03-01 13:33:09 UTC</td><td>&quot;400d1be9-d908-48f9-b9b8-2c4079…</td><td>&quot;customer_analytics&quot;</td><td>&quot;crm_export&quot;</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (8, 12)\n",
       "┌───────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬──────────┐\n",
       "│ customer_ ┆ name      ┆ email     ┆ signup_da ┆ … ┆ _lakelogi ┆ _lakelogi ┆ _lakelogi ┆ _lakelog │\n",
       "│ id        ┆ ---       ┆ ---       ┆ te        ┆   ┆ c_process ┆ c_run_id  ┆ c_domain  ┆ ic_syste │\n",
       "│ ---       ┆ str       ┆ str       ┆ ---       ┆   ┆ ed_at     ┆ ---       ┆ ---       ┆ m        │\n",
       "│ i64       ┆           ┆           ┆ str       ┆   ┆ ---       ┆ str       ┆ str       ┆ ---      │\n",
       "│           ┆           ┆           ┆           ┆   ┆ datetime[ ┆           ┆           ┆ str      │\n",
       "│           ┆           ┆           ┆           ┆   ┆ μs, UTC]  ┆           ┆           ┆          │\n",
       "╞═══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪══════════╡\n",
       "│ 101       ┆ Alice     ┆ alice.smi ┆ 2024-01-1 ┆ … ┆ 2026-03-0 ┆ 400d1be9- ┆ customer_ ┆ crm_expo │\n",
       "│           ┆ Smith     ┆ th@exampl ┆ 5         ┆   ┆ 1         ┆ d908-48f9 ┆ analytics ┆ rt       │\n",
       "│           ┆           ┆ e.com     ┆           ┆   ┆ 13:33:09  ┆ -b9b8-2c4 ┆           ┆          │\n",
       "│           ┆           ┆           ┆           ┆   ┆ UTC       ┆ 079…      ┆           ┆          │\n",
       "│ 102       ┆ Bob Jones ┆ bob.jones ┆ 2024-01-1 ┆ … ┆ 2026-03-0 ┆ 400d1be9- ┆ customer_ ┆ crm_expo │\n",
       "│           ┆           ┆ _invalid_ ┆ 6         ┆   ┆ 1         ┆ d908-48f9 ┆ analytics ┆ rt       │\n",
       "│           ┆           ┆ email     ┆           ┆   ┆ 13:33:09  ┆ -b9b8-2c4 ┆           ┆          │\n",
       "│           ┆           ┆           ┆           ┆   ┆ UTC       ┆ 079…      ┆           ┆          │\n",
       "│ 103       ┆ Charlie   ┆ null      ┆ 2024-01-1 ┆ … ┆ 2026-03-0 ┆ 400d1be9- ┆ customer_ ┆ crm_expo │\n",
       "│           ┆ Brown     ┆           ┆ 7         ┆   ┆ 1         ┆ d908-48f9 ┆ analytics ┆ rt       │\n",
       "│           ┆           ┆           ┆           ┆   ┆ 13:33:09  ┆ -b9b8-2c4 ┆           ┆          │\n",
       "│           ┆           ┆           ┆           ┆   ┆ UTC       ┆ 079…      ┆           ┆          │\n",
       "│ 104       ┆ Duplicate ┆ dup@examp ┆ 2024-01-1 ┆ … ┆ 2026-03-0 ┆ 400d1be9- ┆ customer_ ┆ crm_expo │\n",
       "│           ┆ User      ┆ le.com    ┆ 5         ┆   ┆ 1         ┆ d908-48f9 ┆ analytics ┆ rt       │\n",
       "│           ┆           ┆           ┆           ┆   ┆ 13:33:09  ┆ -b9b8-2c4 ┆           ┆          │\n",
       "│           ┆           ┆           ┆           ┆   ┆ UTC       ┆ 079…      ┆           ┆          │\n",
       "│ 104       ┆ Duplicate ┆ dup@examp ┆ 2024-01-1 ┆ … ┆ 2026-03-0 ┆ 400d1be9- ┆ customer_ ┆ crm_expo │\n",
       "│           ┆ User      ┆ le.com    ┆ 8         ┆   ┆ 1         ┆ d908-48f9 ┆ analytics ┆ rt       │\n",
       "│           ┆           ┆           ┆           ┆   ┆ 13:33:09  ┆ -b9b8-2c4 ┆           ┆          │\n",
       "│           ┆           ┆           ┆           ┆   ┆ UTC       ┆ 079…      ┆           ┆          │\n",
       "│ 105       ┆ Eve Adams ┆ eve@examp ┆ 2024-02-0 ┆ … ┆ 2026-03-0 ┆ 400d1be9- ┆ customer_ ┆ crm_expo │\n",
       "│           ┆           ┆ le.com    ┆ 1         ┆   ┆ 1         ┆ d908-48f9 ┆ analytics ┆ rt       │\n",
       "│           ┆           ┆           ┆           ┆   ┆ 13:33:09  ┆ -b9b8-2c4 ┆           ┆          │\n",
       "│           ┆           ┆           ┆           ┆   ┆ UTC       ┆ 079…      ┆           ┆          │\n",
       "│ 106       ┆ Frank     ┆ frank@exa ┆ not-a-dat ┆ … ┆ 2026-03-0 ┆ 400d1be9- ┆ customer_ ┆ crm_expo │\n",
       "│           ┆ White     ┆ mple.com  ┆ e         ┆   ┆ 1         ┆ d908-48f9 ┆ analytics ┆ rt       │\n",
       "│           ┆           ┆           ┆           ┆   ┆ 13:33:09  ┆ -b9b8-2c4 ┆           ┆          │\n",
       "│           ┆           ┆           ┆           ┆   ┆ UTC       ┆ 079…      ┆           ┆          │\n",
       "│ 107       ┆ Grace Lee ┆ grace@exa ┆ 2024-02-1 ┆ … ┆ 2026-03-0 ┆ 400d1be9- ┆ customer_ ┆ crm_expo │\n",
       "│           ┆           ┆ mple.com  ┆ 0         ┆   ┆ 1         ┆ d908-48f9 ┆ analytics ┆ rt       │\n",
       "│           ┆           ┆           ┆           ┆   ┆ 13:33:09  ┆ -b9b8-2c4 ┆           ┆          │\n",
       "│           ┆           ┆           ┆           ┆   ┆ UTC       ┆ 079…      ┆           ┆          │\n",
       "└───────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴──────────┘"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "RESET_OUTPUTS = True  # Set False to skip re-writing outputs on reruns\n",
    "\n",
    "if RESET_OUTPUTS:\n",
    "    reset_outputs()\n",
    "\n",
    "bronze_proc = DataProcessor(contract=contract_path, stage=\"bronze\")\n",
    "bronze_result = bronze_proc.run_source()\n",
    "bronze_proc.materialize(bronze_result.good, bronze_result.bad)\n",
    "\n",
    "print_run_summary(bronze_proc, \"Bronze summary:\")\n",
    "preview_frame(bronze_result.good, \"BRONZE OUTPUT (sample):\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_silver_heading",
   "metadata": {},
   "source": [
    "## 3. Silver Quality Gate\n",
    "\n",
    "Silver reads the Bronze Parquet, deduplicates, enforces types, and applies\n",
    "business quality rules. Records that fail any rule go to the quarantine table.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "cell_silver",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m2026-03-01 13:33:22.195\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mlakelogic.core.processor\u001b[0m:\u001b[36mrun_source\u001b[0m:\u001b[36m647\u001b[0m - \u001b[1mLoading source: D:\\Github\\_SaaS\\lakelogic\\examples\\02_core_patterns\\medallion_architecture\\data\\bronze\\bronze_customers.parquet via polars\u001b[0m\n",
      "\u001b[32m2026-03-01 13:33:22.215\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mlakelogic.core.processor\u001b[0m:\u001b[36mrun\u001b[0m:\u001b[36m446\u001b[0m - \u001b[1mRun complete. [domain=customer_analytics, system=crm_export] Source: 8, Total (post-transform): 7, Good: 2, Quarantined: 5, Pre-Transform Dropped: 1, Ratio: 71.43%\u001b[0m\n",
      "\u001b[32m2026-03-01 13:33:22.280\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mlakelogic.core.run_log\u001b[0m:\u001b[36m_write_run_log_table\u001b[0m:\u001b[36m325\u001b[0m - \u001b[1mWrote run log to DuckDB table lakelogic_run_logs (D:\\Github\\_SaaS\\lakelogic\\examples\\02_core_patterns\\medallion_architecture\\logs\\lakelogic_run_logs.duckdb)\u001b[0m\n",
      "\u001b[32m2026-03-01 13:33:22.286\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mlakelogic.core.materialization\u001b[0m:\u001b[36mmaterialize_dataframe\u001b[0m:\u001b[36m1460\u001b[0m - \u001b[1mMaterialized 2 rows to D:\\Github\\_SaaS\\lakelogic\\examples\\02_core_patterns\\medallion_architecture\\data\\silver\\silver_customers.parquet\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Silver summary:\n",
      "{'stage': 'default', 'source': 8, 'good': 2, 'quarantined': 5}\n"
     ]
    }
   ],
   "source": [
    "silver_proc = DataProcessor(contract=contract_path)  # no stage = default (Silver)\n",
    "silver_result = silver_proc.run_source()\n",
    "silver_proc.materialize(silver_result.good, silver_result.bad)\n",
    "\n",
    "print_run_summary(silver_proc, \"Silver summary:\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_results_heading",
   "metadata": {},
   "source": [
    "## 4. Inspect Results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "cell_good",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PRODUCTION-READY RECORDS:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr,\n",
       ".dataframe > tbody > tr {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (2, 12)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>customer_id</th><th>name</th><th>email</th><th>signup_date</th><th>tier</th><th>total_spend</th><th>is_active</th><th>_lakelogic_source</th><th>_lakelogic_processed_at</th><th>_lakelogic_run_id</th><th>_lakelogic_domain</th><th>_lakelogic_system</th></tr><tr><td>i64</td><td>str</td><td>str</td><td>date</td><td>str</td><td>f64</td><td>bool</td><td>str</td><td>datetime[μs, UTC]</td><td>str</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>104</td><td>&quot;Duplicate User&quot;</td><td>&quot;dup@example.com&quot;</td><td>2024-01-18</td><td>&quot;free&quot;</td><td>25.0</td><td>true</td><td>&quot;D:\\Github\\_SaaS\\lakelogic\\exam…</td><td>2026-03-01 13:33:22 UTC</td><td>&quot;544724ec-5fff-4036-9025-0c5259…</td><td>&quot;customer_analytics&quot;</td><td>&quot;crm_export&quot;</td></tr><tr><td>101</td><td>&quot;Alice Smith&quot;</td><td>&quot;alice.smith@example.com&quot;</td><td>2024-01-15</td><td>&quot;premium&quot;</td><td>1250.5</td><td>true</td><td>&quot;D:\\Github\\_SaaS\\lakelogic\\exam…</td><td>2026-03-01 13:33:22 UTC</td><td>&quot;544724ec-5fff-4036-9025-0c5259…</td><td>&quot;customer_analytics&quot;</td><td>&quot;crm_export&quot;</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (2, 12)\n",
       "┌───────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬──────────┐\n",
       "│ customer_ ┆ name      ┆ email     ┆ signup_da ┆ … ┆ _lakelogi ┆ _lakelogi ┆ _lakelogi ┆ _lakelog │\n",
       "│ id        ┆ ---       ┆ ---       ┆ te        ┆   ┆ c_process ┆ c_run_id  ┆ c_domain  ┆ ic_syste │\n",
       "│ ---       ┆ str       ┆ str       ┆ ---       ┆   ┆ ed_at     ┆ ---       ┆ ---       ┆ m        │\n",
       "│ i64       ┆           ┆           ┆ date      ┆   ┆ ---       ┆ str       ┆ str       ┆ ---      │\n",
       "│           ┆           ┆           ┆           ┆   ┆ datetime[ ┆           ┆           ┆ str      │\n",
       "│           ┆           ┆           ┆           ┆   ┆ μs, UTC]  ┆           ┆           ┆          │\n",
       "╞═══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪══════════╡\n",
       "│ 104       ┆ Duplicate ┆ dup@examp ┆ 2024-01-1 ┆ … ┆ 2026-03-0 ┆ 544724ec- ┆ customer_ ┆ crm_expo │\n",
       "│           ┆ User      ┆ le.com    ┆ 8         ┆   ┆ 1         ┆ 5fff-4036 ┆ analytics ┆ rt       │\n",
       "│           ┆           ┆           ┆           ┆   ┆ 13:33:22  ┆ -9025-0c5 ┆           ┆          │\n",
       "│           ┆           ┆           ┆           ┆   ┆ UTC       ┆ 259…      ┆           ┆          │\n",
       "│ 101       ┆ Alice     ┆ alice.smi ┆ 2024-01-1 ┆ … ┆ 2026-03-0 ┆ 544724ec- ┆ customer_ ┆ crm_expo │\n",
       "│           ┆ Smith     ┆ th@exampl ┆ 5         ┆   ┆ 1         ┆ 5fff-4036 ┆ analytics ┆ rt       │\n",
       "│           ┆           ┆ e.com     ┆           ┆   ┆ 13:33:22  ┆ -9025-0c5 ┆           ┆          │\n",
       "│           ┆           ┆           ┆           ┆   ┆ UTC       ┆ 259…      ┆           ┆          │\n",
       "└───────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴──────────┘"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "preview_frame(silver_result.good, \"PRODUCTION-READY RECORDS:\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "cell_bad",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "QUARANTINED RECORDS (with error reasons):\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr,\n",
       ".dataframe > tbody > tr {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 16)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>customer_id</th><th>name</th><th>email</th><th>signup_date</th><th>tier</th><th>total_spend</th><th>is_active</th><th>quarantine_state</th><th>quarantine_reprocessed</th><th>_lakelogic_source</th><th>_lakelogic_processed_at</th><th>_lakelogic_run_id</th><th>_lakelogic_domain</th><th>_lakelogic_system</th><th>_lakelogic_errors</th><th>_lakelogic_categories</th></tr><tr><td>i64</td><td>str</td><td>str</td><td>date</td><td>str</td><td>f64</td><td>bool</td><td>str</td><td>bool</td><td>str</td><td>datetime[μs, UTC]</td><td>str</td><td>str</td><td>str</td><td>list[str]</td><td>list[str]</td></tr></thead><tbody><tr><td>106</td><td>&quot;Frank White&quot;</td><td>&quot;frank@example.com&quot;</td><td>null</td><td>&quot;basic&quot;</td><td>100.0</td><td>true</td><td>&quot;active&quot;</td><td>false</td><td>&quot;D:\\Github\\_SaaS\\lakelogic\\exam…</td><td>2026-03-01 13:33:22 UTC</td><td>&quot;544724ec-5fff-4036-9025-0c5259…</td><td>&quot;customer_analytics&quot;</td><td>&quot;crm_export&quot;</td><td>[&quot;Rule failed: signup_date_required (&quot;signup_date&quot; IS NOT NULL)&quot;]</td><td>[&quot;completeness&quot;]</td></tr><tr><td>107</td><td>&quot;Grace Lee&quot;</td><td>&quot;grace@example.com&quot;</td><td>2024-02-10</td><td>&quot;gold&quot;</td><td>null</td><td>true</td><td>&quot;active&quot;</td><td>false</td><td>&quot;D:\\Github\\_SaaS\\lakelogic\\exam…</td><td>2026-03-01 13:33:22 UTC</td><td>&quot;544724ec-5fff-4036-9025-0c5259…</td><td>&quot;customer_analytics&quot;</td><td>&quot;crm_export&quot;</td><td>[&quot;Rule failed: positive_spend (total_spend &gt;= 0)&quot;]</td><td>[&quot;correctness&quot;]</td></tr><tr><td>105</td><td>&quot;Eve Adams&quot;</td><td>&quot;eve@example.com&quot;</td><td>2024-02-01</td><td>&quot;premium&quot;</td><td>-500.0</td><td>false</td><td>&quot;active&quot;</td><td>false</td><td>&quot;D:\\Github\\_SaaS\\lakelogic\\exam…</td><td>2026-03-01 13:33:22 UTC</td><td>&quot;544724ec-5fff-4036-9025-0c5259…</td><td>&quot;customer_analytics&quot;</td><td>&quot;crm_export&quot;</td><td>[&quot;Rule failed: positive_spend (total_spend &gt;= 0)&quot;]</td><td>[&quot;correctness&quot;]</td></tr><tr><td>103</td><td>&quot;Charlie Brown&quot;</td><td>null</td><td>2024-01-17</td><td>&quot;free&quot;</td><td>0.0</td><td>true</td><td>&quot;active&quot;</td><td>false</td><td>&quot;D:\\Github\\_SaaS\\lakelogic\\exam…</td><td>2026-03-01 13:33:22 UTC</td><td>&quot;544724ec-5fff-4036-9025-0c5259…</td><td>&quot;customer_analytics&quot;</td><td>&quot;crm_export&quot;</td><td>[&quot;Rule failed: email_required (&quot;email&quot; IS NOT NULL)&quot;, &quot;Rule failed: email_regex_match (REGEXP_LIKE(&quot;email&quot;, &#x27;^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+$&#x27;))&quot;]</td><td>[&quot;completeness&quot;, &quot;correctness&quot;]</td></tr><tr><td>102</td><td>&quot;Bob Jones&quot;</td><td>&quot;bob.jones_invalid_email&quot;</td><td>2024-01-16</td><td>&quot;basic&quot;</td><td>45.0</td><td>true</td><td>&quot;active&quot;</td><td>false</td><td>&quot;D:\\Github\\_SaaS\\lakelogic\\exam…</td><td>2026-03-01 13:33:22 UTC</td><td>&quot;544724ec-5fff-4036-9025-0c5259…</td><td>&quot;customer_analytics&quot;</td><td>&quot;crm_export&quot;</td><td>[&quot;Rule failed: email_regex_match (REGEXP_LIKE(&quot;email&quot;, &#x27;^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+$&#x27;))&quot;]</td><td>[&quot;correctness&quot;]</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 16)\n",
       "┌───────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬──────────┐\n",
       "│ customer_ ┆ name      ┆ email     ┆ signup_da ┆ … ┆ _lakelogi ┆ _lakelogi ┆ _lakelogi ┆ _lakelog │\n",
       "│ id        ┆ ---       ┆ ---       ┆ te        ┆   ┆ c_domain  ┆ c_system  ┆ c_errors  ┆ ic_categ │\n",
       "│ ---       ┆ str       ┆ str       ┆ ---       ┆   ┆ ---       ┆ ---       ┆ ---       ┆ ories    │\n",
       "│ i64       ┆           ┆           ┆ date      ┆   ┆ str       ┆ str       ┆ list[str] ┆ ---      │\n",
       "│           ┆           ┆           ┆           ┆   ┆           ┆           ┆           ┆ list[str │\n",
       "│           ┆           ┆           ┆           ┆   ┆           ┆           ┆           ┆ ]        │\n",
       "╞═══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪══════════╡\n",
       "│ 106       ┆ Frank     ┆ frank@exa ┆ null      ┆ … ┆ customer_ ┆ crm_expor ┆ [\"Rule    ┆ [\"comple │\n",
       "│           ┆ White     ┆ mple.com  ┆           ┆   ┆ analytics ┆ t         ┆ failed:   ┆ teness\"] │\n",
       "│           ┆           ┆           ┆           ┆   ┆           ┆           ┆ signup_da ┆          │\n",
       "│           ┆           ┆           ┆           ┆   ┆           ┆           ┆ te_req…   ┆          │\n",
       "│ 107       ┆ Grace Lee ┆ grace@exa ┆ 2024-02-1 ┆ … ┆ customer_ ┆ crm_expor ┆ [\"Rule    ┆ [\"correc │\n",
       "│           ┆           ┆ mple.com  ┆ 0         ┆   ┆ analytics ┆ t         ┆ failed:   ┆ tness\"]  │\n",
       "│           ┆           ┆           ┆           ┆   ┆           ┆           ┆ positive_ ┆          │\n",
       "│           ┆           ┆           ┆           ┆   ┆           ┆           ┆ spend …   ┆          │\n",
       "│ 105       ┆ Eve Adams ┆ eve@examp ┆ 2024-02-0 ┆ … ┆ customer_ ┆ crm_expor ┆ [\"Rule    ┆ [\"correc │\n",
       "│           ┆           ┆ le.com    ┆ 1         ┆   ┆ analytics ┆ t         ┆ failed:   ┆ tness\"]  │\n",
       "│           ┆           ┆           ┆           ┆   ┆           ┆           ┆ positive_ ┆          │\n",
       "│           ┆           ┆           ┆           ┆   ┆           ┆           ┆ spend …   ┆          │\n",
       "│ 103       ┆ Charlie   ┆ null      ┆ 2024-01-1 ┆ … ┆ customer_ ┆ crm_expor ┆ [\"Rule    ┆ [\"comple │\n",
       "│           ┆ Brown     ┆           ┆ 7         ┆   ┆ analytics ┆ t         ┆ failed:   ┆ teness\", │\n",
       "│           ┆           ┆           ┆           ┆   ┆           ┆           ┆ email_req ┆ \"correct │\n",
       "│           ┆           ┆           ┆           ┆   ┆           ┆           ┆ uired …   ┆ ness\"…   │\n",
       "│ 102       ┆ Bob Jones ┆ bob.jones ┆ 2024-01-1 ┆ … ┆ customer_ ┆ crm_expor ┆ [\"Rule    ┆ [\"correc │\n",
       "│           ┆           ┆ _invalid_ ┆ 6         ┆   ┆ analytics ┆ t         ┆ failed:   ┆ tness\"]  │\n",
       "│           ┆           ┆ email     ┆           ┆   ┆           ┆           ┆ email_reg ┆          │\n",
       "│           ┆           ┆           ┆           ┆   ┆           ┆           ┆ ex_mat…   ┆          │\n",
       "└───────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴──────────┘"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "preview_frame(silver_result.bad, \"QUARANTINED RECORDS (with error reasons):\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "cell_paths",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Silver Parquet: D:\\Github\\_SaaS\\lakelogic\\examples\\02_core_patterns\\medallion_architecture\\data\\silver\\silver_customers.parquet\n",
      "Exists: True\n"
     ]
    }
   ],
   "source": [
    "silver_out = get_path(\"data\", \"silver\", \"silver_customers.parquet\")\n",
    "print(f\"Silver Parquet: {silver_out}\")\n",
    "print(f\"Exists: {silver_out.exists()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_summary",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Summary\n",
    "\n",
    "| Metric | Bronze | Silver |\n",
    "|---|---|---|\n",
    "| Source rows | 8 (all raw) | 8 (from Bronze Parquet) |\n",
    "| Good (output) rows | 8 | ~2 |\n",
    "| Duplicate rows dropped | 0 (Bronze keeps all) | 1 (customer 104) |\n",
    "| Quarantined rows | 0 | ~5 (bad email / spend / date / type) |\n",
    "\n",
    "> Exact quarantine counts depend on which rules fire first. Run the notebook to see live numbers.\n",
    "\n",
    "### What LakeLogic did automatically\n",
    "\n",
    "- **Bronze**: Glob-matched `data/crm_*.csv`, wrote all 8 rows to Parquet — no code needed\n",
    "- **Silver**: Renamed `plan_type` → `tier`, deduplicated on `customer_id` (latest record wins),\n",
    "  cast types, applied email regex + spend rules\n",
    "- **Audit columns**: `_lakelogic_processed_at`, `_lakelogic_run_id`, `_lakelogic_errors`\n",
    "  added to every output row automatically\n",
    "- **Run logs**: Execution history written to `logs/lakelogic_run_logs.duckdb` for watermarking\n",
    "  and incremental load tracking\n",
    "\n",
    "---\n",
    "\n",
    "## Next Steps — Try It Yourself\n",
    "\n",
    "The two files that drive this entire pipeline are small and easy to edit.\n",
    "\n",
    "### 1. Edit the input data\n",
    "\n",
    "Open `data/crm_export.csv` and try:\n",
    "\n",
    "```\n",
    "customer_id,name,email,signup_date,plan_type,total_spend,is_active\n",
    "101,Alice Smith,alice.smith@example.com,2024-01-15,premium,1250.50,true\n",
    "102,Bob Jones,bob.jones_invalid_email,2024-01-16,basic,45.00,true\n",
    "103,Charlie Brown,,2024-01-17,free,0,true\n",
    "104,Duplicate User,dup@example.com,2024-01-15,free,10.00,true\n",
    "104,Duplicate User,dup@example.com,2024-01-18,free,25.00,true\n",
    "105,Eve Adams,eve@example.com,2024-02-01,premium,-500,false\n",
    "106,Frank White,frank@example.com,not-a-date,basic,100,true\n",
    "107,Grace Lee,grace@example.com,2024-02-10,gold,high,true\n",
    "```\n",
    "\n",
    "**Ideas:**\n",
    "- Add a new valid row (e.g. customer 108) and watch it flow cleanly through both layers\n",
    "- Add a second CSV file (`data/crm_export_v2.csv`) — Bronze picks it up automatically via the glob `data/crm_*.csv`\n",
    "- Fix Bob's email to `bob.jones@example.com` and confirm he moves from quarantine to good\n",
    "\n",
    "### 2. Edit the contract\n",
    "\n",
    "Open `contract.yaml` and try:\n",
    "\n",
    "```yaml\n",
    "# Add a new quality rule — reject free-tier customers with zero spend:\n",
    "quality:\n",
    "  row_rules:\n",
    "    - name: paying_or_trial\n",
    "      sql: \"tier != 'free' OR total_spend >= 0\"\n",
    "      category: business_logic\n",
    "\n",
    "# Change Bronze to incremental loading (only new files since last run):\n",
    "stages:\n",
    "  bronze:\n",
    "    source:\n",
    "      type: raw_landing\n",
    "      path: \"data/crm_*.csv\"\n",
    "      load_mode: incremental   # <-- uncomment this line\n",
    "\n",
    "# Change dedup sort direction (keep the oldest record instead of latest):\n",
    "transformations:\n",
    "  - deduplicate:\n",
    "      on: [\"customer_id\"]\n",
    "      sort_by: [\"signup_date\"]\n",
    "      order: \"asc\"             # <-- change desc to asc\n",
    "    phase: \"pre\"\n",
    "```\n",
    "\n",
    "**Key contract knobs:**\n",
    "\n",
    "| What to change | Where in contract.yaml |\n",
    "|---|---|\n",
    "| Bronze source glob | `stages.bronze.source.path` |\n",
    "| Bronze output path | `stages.bronze.materialization.target_path` |\n",
    "| Silver dedup key | `transformations[deduplicate].on` |\n",
    "| Survivorship order | `transformations[deduplicate].order` |\n",
    "| Quality rules | `quality.row_rules` |\n",
    "| Run log location | `metadata.run_log_database` |\n",
    "\n",
    "### 3. Explore related playbooks\n",
    "\n",
    "- [`../dedup_survivorship/`](../dedup_survivorship/playbook.ipynb) — dive deeper into survivorship rules in isolation\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv_lakelogic",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
