{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell_header",
   "metadata": {},
   "source": [
    "# Database Governance Quickstart\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lakelogic/LakeLogic/blob/main/examples/01_quickstart/02_database_governance.ipynb) \n",
    "[![GitHub Repo](https://img.shields.io/badge/GitHub-Repo-blue?logo=github)](https://github.com/lakelogic/LakeLogic/blob/main/examples/01_quickstart/02_database_governance.ipynb)\n",
    "\n",
    "## Business Scenario\n",
    "\n",
    "Most organizations have critical data locked in traditional SQL databases (ERP, CRM, legacy apps).\n",
    "To unlock analytics and ML, this data must move to a modern Data Lakehouse — but database records\n",
    "are often messy: null emails, invalid IDs, duplicates. Copying dirty data into your lake builds\n",
    "a **Data Swamp**.\n",
    "\n",
    "LakeLogic acts as a **Governance Gate**: extract from the database, validate against a contract,\n",
    "route clean rows to Silver, and quarantine bad ones for review — automatically.\n",
    "\n",
    "## Value Proposition\n",
    "\n",
    "- Only validated records reach your analytics layer\n",
    "- Bad records are isolated with an error reason column — no silent data loss\n",
    "- Rules live in a YAML contract, not scattered across ETL scripts\n",
    "\n",
    "---\n",
    "\n",
    "## Goals\n",
    "\n",
    "1. Seed a local SQLite database with intentionally dirty data (1 good + 2 bad)\n",
    "2. Define quality rules in a contract\n",
    "3. Run the governance pipeline — inspect what passed and what was quarantined\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_setup_heading",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_setup",
   "metadata": {},
   "outputs": [],
   "source": [
    "import importlib.util\n",
    "import os\n",
    "import sys\n",
    "import sqlite3\n",
    "from pathlib import Path\n",
    "\n",
    "if importlib.util.find_spec(\"lakelogic\") is None:\n",
    "    import subprocess\n",
    "\n",
    "    subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"lakelogic\", \"-q\"], check=True)\n",
    "    print(\"lakelogic installed.\")\n",
    "else:\n",
    "    print(\"lakelogic ready.\")\n",
    "\n",
    "if \"google.colab\" in sys.modules:\n",
    "    repo = Path(\"/content/LakeLogic\")\n",
    "    if not repo.exists():\n",
    "        import subprocess\n",
    "\n",
    "        subprocess.run(\n",
    "            [\n",
    "                \"git\",\n",
    "                \"clone\",\n",
    "                \"--quiet\",\n",
    "                \"https://github.com/lakelogic/LakeLogic.git\",\n",
    "                str(repo),\n",
    "            ],\n",
    "            check=True,\n",
    "        )\n",
    "    os.chdir(repo / \"examples\" / \"01_quickstart\")\n",
    "    print(f\"Working directory: {Path.cwd()}\")\n",
    "\n",
    "\n",
    "def get_path(filename):\n",
    "    \"\"\"Resolve path relative to this notebook's directory, Colab-safe.\"\"\"\n",
    "    cwd = Path.cwd()\n",
    "    for base in [cwd] + list(cwd.parents):\n",
    "        p = base / filename\n",
    "        if p.exists():\n",
    "            return str(p.resolve())\n",
    "    return str((cwd / filename).resolve())\n",
    "\n",
    "\n",
    "from lakelogic import DataProcessor\n",
    "\n",
    "print(\"Setup complete.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_how_it_works",
   "metadata": {},
   "source": [
    "## How It Works\n",
    "\n",
    "```\n",
    "SQLite DB (3 rows: 1 good, 2 bad)\n",
    "      |\n",
    "      v  DataProcessor.run(rows)\n",
    "      |    Applies quality rules from contract.yaml row-by-row\n",
    "      |\n",
    "      ├── result.good  →  Analytics-ready Silver layer\n",
    "      |                    (only Alice — valid email, valid id)\n",
    "      |\n",
    "      └── result.bad   →  Quarantine\n",
    "                           Bob    — failed Valid Email rule  (no @ in email)\n",
    "                           Charlie — failed Valid ID rule    (id < 0)\n",
    "```\n",
    "\n",
    "### Quality rules in `users_contract.yaml`\n",
    "\n",
    "| Rule name | SQL expression | Catches |\n",
    "|---|---|---|\n",
    "| Valid Email | `email LIKE '%@%'` | Any row without `@` in the email field |\n",
    "| Valid ID | `id > 0` | Any row with a zero or negative ID |\n",
    "\n",
    "### Known data seeded into the database\n",
    "\n",
    "| id | name | email | Expected outcome |\n",
    "|---|---|---|---|\n",
    "| 1 | Alice | alice@example.com | ✅ Good — passes both rules |\n",
    "| 2 | Bob | bob-invalid | ❌ Bad — fails Valid Email |\n",
    "| -99 | Charlie | charlie@example.com | ❌ Bad — fails Valid ID |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_seed_heading",
   "metadata": {},
   "source": [
    "## 1. Seed the Database with Dirty Data\n",
    "\n",
    "We use SQLite to simulate a source ERP/CRM database. One good record, two intentionally bad ones.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_seed",
   "metadata": {},
   "outputs": [],
   "source": [
    "db_file = \"example.db\"\n",
    "conn = sqlite3.connect(db_file)\n",
    "c = conn.cursor()\n",
    "c.execute(\"CREATE TABLE IF NOT EXISTS users (id INTEGER, name TEXT, email TEXT)\")\n",
    "c.execute(\"DELETE FROM users\")\n",
    "c.execute('INSERT INTO users VALUES (1,   \"Alice\",   \"alice@example.com\")')  # good\n",
    "c.execute('INSERT INTO users VALUES (2,   \"Bob\",     \"bob-invalid-email\")')  # bad: no @\n",
    "c.execute('INSERT INTO users VALUES (-99, \"Charlie\", \"charlie@example.com\")')  # bad: id < 0\n",
    "conn.commit()\n",
    "conn.close()\n",
    "print(\"Database seeded: 1 good + 2 bad records.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_contract_heading",
   "metadata": {},
   "source": [
    "## 2. Review the Contract\n",
    "\n",
    "The `users_contract.yaml` file defines quality rules. It lives in Git — no code changes needed\n",
    "when rules change, just update the YAML.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_contract",
   "metadata": {},
   "outputs": [],
   "source": [
    "contract_path = get_path(\"users_contract.yaml\")\n",
    "print(f\"Contract: {contract_path}\")\n",
    "print(\"---\")\n",
    "print(Path(contract_path).read_text())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_run_heading",
   "metadata": {},
   "source": [
    "## 3. Run the Governance Pipeline\n",
    "\n",
    "Extract from SQLite, pass rows to `DataProcessor.run()`. LakeLogic validates each row\n",
    "and splits into `good` / `bad` automatically.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_run",
   "metadata": {},
   "outputs": [],
   "source": [
    "conn = sqlite3.connect(\"example.db\")\n",
    "conn.row_factory = sqlite3.Row\n",
    "rows = [dict(r) for r in conn.execute(\"SELECT * FROM users\").fetchall()]\n",
    "conn.close()\n",
    "\n",
    "processor = DataProcessor(contract=contract_path)\n",
    "result = processor.run(rows)\n",
    "\n",
    "print(f\"Engine           : {processor.engine_name}\")\n",
    "print(f\"Records validated: {len(result.good)}\")\n",
    "print(f\"Records quarantined: {len(result.bad)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_results_heading",
   "metadata": {},
   "source": [
    "## 4. Inspect Results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_good",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"CLEAN SILVER DATA (analytics-ready):\")\n",
    "display(result.good)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_bad",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"QUARANTINED ROWS (failed quality rules):\")\n",
    "display(result.bad)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_summary",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Summary\n",
    "\n",
    "| Row | Name | Outcome | Reason |\n",
    "|---|---|---|---|\n",
    "| 1 | Alice | ✅ Good | Passes both rules |\n",
    "| 2 | Bob | ❌ Quarantined | Failed Valid Email (no `@`) |\n",
    "| -99 | Charlie | ❌ Quarantined | Failed Valid ID (negative id) |\n",
    "\n",
    "### What LakeLogic did automatically\n",
    "\n",
    "- Evaluated each database row against every rule in the contract\n",
    "- Routed passing rows to `result.good` — zero custom filtering code\n",
    "- Routed failing rows to `result.bad` with an `_lakelogic_error_reason` column\n",
    "- Added `_lakelogic_processed_at` and `_lakelogic_run_id` audit columns\n",
    "\n",
    "---\n",
    "\n",
    "## Next Steps — Try It Yourself\n",
    "\n",
    "### 1. Change the seeded data\n",
    "\n",
    "```python\n",
    "# Add a new bad record — null name:\n",
    "c.execute('INSERT INTO users VALUES (4, NULL, \"diana@example.com\")')\n",
    "\n",
    "# Then add a rule to catch it:\n",
    "```\n",
    "\n",
    "### 2. Edit the contract\n",
    "\n",
    "Open `users_contract.yaml` and add new rules:\n",
    "\n",
    "```yaml\n",
    "version: 1.0.0\n",
    "dataset: users\n",
    "source:\n",
    "  type: landing\n",
    "\n",
    "quality:\n",
    "  row_rules:\n",
    "    - name: Valid Email\n",
    "      sql: \"email LIKE '%@%'\"\n",
    "    - name: Valid ID\n",
    "      sql: \"id > 0\"              # already here — catches Charlie\n",
    "    - name: Name Required        # <-- add this\n",
    "      sql: \"name IS NOT NULL AND name != ''\"\n",
    "    - name: Known Domain         # <-- add this\n",
    "      sql: \"email LIKE '%@example.com'\"\n",
    "```\n",
    "\n",
    "**Key contract knobs:**\n",
    "\n",
    "| What to change | Where in `contract.yaml` | Effect |\n",
    "|---|---|---|\n",
    "| Quality rules | `quality.row_rules` | Any SQL expression — rows that fail are quarantined |\n",
    "| Rule severity | `severity: warning` | Mark rule as warning instead of error |\n",
    "| Required fields | `model.fields[].required: true` | Quarantine rows with null in required columns |\n",
    "| Write output | `materialization.target_path` + `format` | Persist good rows to Parquet/CSV/Delta |\n",
    "\n",
    "### 3. Explore related quickstarts\n",
    "\n",
    "- [`01_hello_world.ipynb`](01_hello_world.ipynb) — same governance applied to a remote URL\n",
    "- [`../02_core_patterns/scd2_dimension/`](../02_core_patterns/scd2_dimension/playbook.ipynb) — add history tracking downstream\n",
    "- [`../02_core_patterns/soft_delete/`](../02_core_patterns/soft_delete/soft_delete_pattern.ipynb) — flag deletes instead of removing rows\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv_lakelogic",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}