{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell_header",
   "metadata": {},
   "source": [
    "# Bronze Quality Gate\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lakelogic/LakeLogic/blob/main/examples/02_core_patterns/bronze_quality_gate/playbook.ipynb) \n",
    "[![GitHub Repo](https://img.shields.io/badge/GitHub-Repo-blue?logo=github)](https://github.com/lakelogic/LakeLogic/blob/main/examples/02_core_patterns/bronze_quality_gate/playbook.ipynb)\n",
    "\n",
    "## Business Scenario\n",
    "\n",
    "Raw signups and web leads arrive with missing emails, invalid ages, and inconsistent formats.\n",
    "Before this data hits Silver, you need a fast quality gate to stop obvious issues at ingestion.\n",
    "Catching errors early prevents bad data from silently polluting downstream analytics.\n",
    "\n",
    "## Value Proposition\n",
    "\n",
    "- Stop bad data early without breaking the pipeline\n",
    "- Quarantine invalid rows with a clear, auditable error reason\n",
    "- Keep Bronze ingestion lightweight and contract-driven — no custom filtering code\n",
    "- Full reconciliation: every row is either in `good` or `bad`, nothing is silently dropped\n",
    "\n",
    "---\n",
    "\n",
    "## Goals\n",
    "\n",
    "1. Load raw web signups from CSV\n",
    "2. Validate schema fields and row-level quality rules from `contract.yaml`\n",
    "3. Inspect the good (Silver-ready) and quarantined (bad) rows\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_setup_heading",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_setup",
   "metadata": {},
   "outputs": [],
   "source": [
    "import importlib.util\n",
    "import os\n",
    "import sys\n",
    "from pathlib import Path\n",
    "\n",
    "if importlib.util.find_spec(\"lakelogic\") is None:\n",
    "    import subprocess\n",
    "\n",
    "    subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"lakelogic\", \"-q\"], check=True)\n",
    "    print(\"lakelogic installed.\")\n",
    "else:\n",
    "    print(\"lakelogic ready.\")\n",
    "\n",
    "if \"google.colab\" in sys.modules:\n",
    "    repo = Path(\"/content/LakeLogic\")\n",
    "    if not repo.exists():\n",
    "        import subprocess\n",
    "\n",
    "        subprocess.run(\n",
    "            [\n",
    "                \"git\",\n",
    "                \"clone\",\n",
    "                \"--quiet\",\n",
    "                \"https://github.com/lakelogic/LakeLogic.git\",\n",
    "                str(repo),\n",
    "            ],\n",
    "            check=True,\n",
    "        )\n",
    "    os.chdir(repo / \"examples\" / \"02_core_patterns\" / \"bronze_quality_gate\")\n",
    "    print(f\"Working directory: {Path.cwd()}\")\n",
    "\n",
    "\n",
    "def get_path(*parts):\n",
    "    \"\"\"Resolve a path relative to the bronze_quality_gate directory, Colab-safe.\"\"\"\n",
    "    cwd = Path.cwd()\n",
    "    for base in [cwd] + list(cwd.parents):\n",
    "        target = base if base.name == \"bronze_quality_gate\" else base / \"bronze_quality_gate\"\n",
    "        if (target / \"contract.yaml\").exists():\n",
    "            return (target / Path(*parts)).resolve()\n",
    "    return (cwd / Path(*parts)).resolve()\n",
    "\n",
    "\n",
    "from lakelogic import DataProcessor\n",
    "\n",
    "print(\"Setup complete.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_how_it_works",
   "metadata": {},
   "source": [
    "## How It Works\n",
    "\n",
    "LakeLogic reads the CSV, applies the contract's schema rules and quality rules row-by-row,\n",
    "and returns a `ValidationResult` with three DataFrames:\n",
    "\n",
    "### Data flow\n",
    "\n",
    "```\n",
    "data/raw_signups.csv\n",
    "      |\n",
    "      v  DataProcessor.run_source()\n",
    "      |    1. Schema validation  (required fields, type coercion)\n",
    "      |    2. Row rules          (email_format, age_positive)\n",
    "      |\n",
    "      ├── result.good  →  Silver-ready rows (all rules passed)\n",
    "      └── result.bad   →  Quarantine       (failed rows + error reason)\n",
    "```\n",
    "\n",
    "### Quality rules in `contract.yaml`\n",
    "\n",
    "| Rule | SQL | Catches |\n",
    "|---|---|---|\n",
    "| `email_format` | `email LIKE '%@%'` | Missing or malformed email addresses |\n",
    "| `age_positive` | `age IS NULL OR age >= 0` | Negative age values |\n",
    "\n",
    "### Schema fields enforced\n",
    "\n",
    "| Field | Type | Required |\n",
    "|---|---|---|\n",
    "| `signup_id` | string | ✅ yes |\n",
    "| `email` | string | ✅ yes |\n",
    "| `event_date` | date | ✅ yes |\n",
    "| `source` | string | no |\n",
    "| `age` | int | no |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_run_heading",
   "metadata": {},
   "source": [
    "## 1. Run the Bronze Quality Gate\n",
    "\n",
    "Point `DataProcessor` at `contract.yaml` and run it against the raw signups CSV.\n",
    "LakeLogic handles schema coercion, rule evaluation, and quarantine routing automatically.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_run",
   "metadata": {},
   "outputs": [],
   "source": [
    "contract_path = get_path(\"contract.yaml\")\n",
    "data_path = get_path(\"data\", \"raw_signups.csv\")\n",
    "\n",
    "processor = DataProcessor(contract=contract_path)\n",
    "result = processor.run_source(data_path)\n",
    "\n",
    "print(f\"Source rows  : {len(result.raw)}\")\n",
    "print(f\"Good rows    : {len(result.good)}\")\n",
    "print(f\"Bad rows     : {len(result.bad)}\")\n",
    "print(f\"Reconciled   : {len(result.raw)} = {len(result.good)} + {len(result.bad)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_raw_heading",
   "metadata": {},
   "source": [
    "## 2. Inspect Raw Data\n",
    "\n",
    "Every row exactly as read from the CSV, before any validation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_raw",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"RAW DATA (all source rows):\")\n",
    "display(result.raw)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_good_heading",
   "metadata": {},
   "source": [
    "## 3. Inspect Good Data (Silver-Ready)\n",
    "\n",
    "Rows that passed all schema checks and quality rules — safe to promote to Silver.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_good",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"GOOD DATA (passed all rules):\")\n",
    "display(result.good)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_bad_heading",
   "metadata": {},
   "source": [
    "## 4. Inspect Quarantined Data\n",
    "\n",
    "Rows that failed one or more rules — routed to quarantine with `_lakelogic_error_reason`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_bad",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"BAD DATA (quarantined rows + reason):\")\n",
    "display(result.bad)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_summary",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Summary\n",
    "\n",
    "| What happened | Detail |\n",
    "|---|---|\n",
    "| Source | Raw CSV read — schema coercion applied automatically |\n",
    "| `email_format` rule | Rows without `@` in email → quarantine |\n",
    "| `age_positive` rule | Rows with negative age → quarantine |\n",
    "| `result.good` | Silver-ready rows |\n",
    "| `result.bad` | Quarantined with `_lakelogic_error_reason` |\n",
    "\n",
    "### What LakeLogic did automatically\n",
    "\n",
    "- Read and coerced the raw CSV against the schema definition\n",
    "- Evaluated each row against every quality rule in the contract\n",
    "- Routed passing rows to `result.good` with zero custom code\n",
    "- Routed failing rows to `result.bad` with an error reason column\n",
    "- Added `_lakelogic_processed_at` and `_lakelogic_run_id` audit columns\n",
    "\n",
    "---\n",
    "\n",
    "## Next Steps — Try It Yourself\n",
    "\n",
    "### 1. Edit the source data\n",
    "\n",
    "Open `data/raw_signups.csv` and add a new row with a bad email or negative age:\n",
    "\n",
    "```csv\n",
    "signup_id,email,event_date,source,age\n",
    "S999,not-an-email,2024-01-15,organic,-5\n",
    "```\n",
    "\n",
    "Re-run the notebook — both rules should fire on that row.\n",
    "\n",
    "### 2. Edit the contract\n",
    "\n",
    "```yaml\n",
    "quality:\n",
    "  row_rules:\n",
    "    - name: email_format\n",
    "      sql: \"email LIKE '%@%'\"\n",
    "    - name: age_positive\n",
    "      sql: \"age IS NULL OR age >= 0\"\n",
    "    - name: age_max          # <-- add this\n",
    "      sql: \"age IS NULL OR age <= 120\"\n",
    "    - name: known_source     # <-- add this\n",
    "      sql: \"source IN ('organic', 'paid', 'referral')\"\n",
    "      severity: warning      # warning = quarantine but still counted as good\n",
    "```\n",
    "\n",
    "**Key contract knobs:**\n",
    "\n",
    "| What to change | Where in `contract.yaml` | Effect |\n",
    "|---|---|---|\n",
    "| Quality rules | `quality.row_rules` | Any SQL expression — failures route to `result.bad` |\n",
    "| Rule severity | `severity: warning` | Flag row but still route to `result.good` |\n",
    "| Required fields | `model.fields[].required: true` | Null values in required columns are quarantined |\n",
    "| Unknown column handling | `schema_policy.unknown_fields` | `drop` or `error` |\n",
    "\n",
    "### 3. Explore related playbooks\n",
    "\n",
    "- [`../dedup_survivorship/`](../dedup_survivorship/playbook.ipynb) — deduplicate records after the quality gate\n",
    "- [`../scd2_dimension/`](../scd2_dimension/playbook.ipynb) — add full history tracking downstream\n",
    "- [`../../04_compliance_governance/hipaa_pii_masking/`](../../04_compliance_governance/hipaa_pii_masking/tutorial_hipaa_gdpr_compliance.ipynb) — layer in PII masking and HIPAA/GDPR policy packs\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv_lakelogic",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}