{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "qs-dbt-0001",
   "metadata": {},
   "source": [
    "# From dbt Schema to GDPR-Compliant Pipeline — in 5 Minutes\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lakelogic/LakeLogic/blob/main/examples/01_quickstart/03_dbt_pii_quality.ipynb)\n",
    "[![GitHub](https://img.shields.io/badge/GitHub-Repo-blue?logo=github)](https://github.com/lakelogic/LakeLogic)\n",
    "\n",
    "## The Scenario\n",
    "\n",
    "You have a **dbt project** for your `customers` model — schema defined, tests written,\n",
    "PII columns annotated. Now you need:\n",
    "\n",
    "- ✅ **Quality enforcement** — run the same checks that live in `schema.yml` at ingestion time\n",
    "- 🔒 **GDPR compliance** — honour erasure requests and mask PII for dev/test environments  \n",
    "- 🧪 **Test data** — generate realistic synthetic customers that obey the same contract\n",
    "\n",
    "Normally these would be three separate tools. With LakeLogic + `from_dbt()`, they all\n",
    "share the same schema file you already maintain.\n",
    "\n",
    "---\n",
    "\n",
    "## What we cover\n",
    "\n",
    "| Step | What happens |\n",
    "|---|---|\n",
    "| **1** | Import `dbt_schema.yml` → LakeLogic contract (inspect what was mapped) |\n",
    "| **2** | Generate synthetic customers from the dbt schema |\n",
    "| **3** | Run quality validation — see quarantined rows |\n",
    "| **4** | GDPR: erase a specific customer (nullify / hash / redact strategies) |\n",
    "| **5** | GDPR: mask all PII for a dev/test copy |\n",
    "| **6** | Batch erasure + audit report |\n",
    "| **7** | CLI reference |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "qs-dbt-0002",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "qs-dbt-0003",
   "metadata": {},
   "outputs": [],
   "source": [
    "# SETUP: Install lakelogic if not already present\n",
    "from pathlib import Path\n",
    "import os\n",
    "import importlib.util\n",
    "import sys\n",
    "\n",
    "# --- Package check (works on Colab, Databricks, local) ---\n",
    "if importlib.util.find_spec(\"lakelogic\") is None:\n",
    "    print(\"📦 Installing lakelogic...\")\n",
    "    !pip install lakelogic\n",
    "    print(\"✅ lakelogic installed.\")\n",
    "else:\n",
    "    print(\"✅ lakelogic is already installed.\")\n",
    "\n",
    "# --- Colab: clone repo so contract files (YAML, etc.) are available ---\n",
    "if \"google.colab\" in sys.modules:\n",
    "    repo_dir = Path(\"/content/LakeLogic\")\n",
    "    if not repo_dir.exists():\n",
    "        print(\"📂 Cloning LakeLogic repo for example files...\")\n",
    "        !git clone --quiet https://github.com/lakelogic/LakeLogic.git /content/LakeLogic\n",
    "    os.chdir(repo_dir / \"examples\" / \"01_quickstart\")\n",
    "    print(f\"📍 Working directory set to: {Path.cwd()}\")\n",
    "\n",
    "\n",
    "def find_example_dir(name: str) -> Path:\n",
    "    cwd = Path.cwd()\n",
    "    for base in [cwd] + list(cwd.parents):\n",
    "        for candidate in [\n",
    "            base / name,\n",
    "            base / \"examples\",\n",
    "            base / \"lakelogic\" / \"examples\",\n",
    "        ]:\n",
    "            if candidate.exists():\n",
    "                return candidate\n",
    "    return cwd\n",
    "\n",
    "\n",
    "# Global Path Resolver\n",
    "def get_full_path(filename: str) -> str:\n",
    "    path_obj = find_example_dir(filename)\n",
    "    if path_obj.is_dir():\n",
    "        return str((path_obj / filename).resolve())\n",
    "    return str(path_obj.resolve())\n",
    "\n",
    "\n",
    "SCHEMA_FILE = get_full_path(\"dbt_schema.yml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "qs-dbt-0004",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Step 1 — Import dbt Schema → LakeLogic Contract\n",
    "\n",
    "`DbtAdapter` reads the dbt YAML and maps:\n",
    "- `meta: {pii: true}` → `field.pii = True`\n",
    "- `not_null` → `field.required = True`\n",
    "- `unique` → dataset-level uniqueness rule (`SELECT COUNT(*) - COUNT(DISTINCT ...) FROM ...`)\n",
    "- `accepted_values` → `row_rule` IN check\n",
    "- `expression_is_true` → `row_rule` raw SQL\n",
    "\n",
    "**No manual YAML editing. No new files to maintain.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "qs-dbt-0005",
   "metadata": {},
   "outputs": [],
   "source": [
    "from lakelogic import DataProcessor, DataGenerator, DbtAdapter\n",
    "from lakelogic.core.gdpr import (\n",
    "    forget_subjects,\n",
    "    mask_pii_columns,\n",
    "    generate_erasure_report,\n",
    ")\n",
    "\n",
    "adapter = DbtAdapter(SCHEMA_FILE)\n",
    "contract = adapter.model_to_contract(\"customers\")\n",
    "\n",
    "# ── What got mapped? ────────────────────────────────────────────────\n",
    "fields = contract.model.fields if contract.model else []\n",
    "pii_fields = [f.name for f in fields if f.pii]\n",
    "req_fields = [f.name for f in fields if f.required]\n",
    "row_rules = contract.quality.row_rules if contract.quality else []\n",
    "ds_rules = contract.quality.dataset_rules if contract.quality else []\n",
    "\n",
    "print(f\"Contract    : {contract.dataset}\")\n",
    "print(f\"Fields      : {len(fields)}\")\n",
    "print(f\"Required    : {req_fields}\")\n",
    "print(f\"PII fields  : {pii_fields}  ← auto-detected from meta.pii\")\n",
    "print(f\"Row rules   : {len(row_rules)}\")\n",
    "print(f\"Dataset rules: {len(ds_rules)}\")\n",
    "print()\n",
    "for r in row_rules:\n",
    "    print(f\"  row rule  ▸ {r.name}: {r.sql}\")\n",
    "for r in ds_rules:\n",
    "    print(f\"  dset rule ▸ {r.name}: {r.sql}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "qs-dbt-0006",
   "metadata": {},
   "source": [
    "### Preview the generated LakeLogic contract YAML\n",
    "\n",
    "You can also export it to disk — once, if you want to add LakeLogic-specific extras\n",
    "(streaming config, materialization targets, SLOs) on top of what dbt provides."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "qs-dbt-0007",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preview the YAML that would be written\n",
    "yaml_text = adapter.to_yaml(\"customers\")\n",
    "print(yaml_text[:2000])  # first 2000 chars\n",
    "\n",
    "# To write it → contracts/customers.yaml:\n",
    "#   adapter.to_yaml(\"customers\", output=EXAMPLE_DIR / \"contracts\" / \"customers.yaml\")\n",
    "#\n",
    "# Or via CLI:\n",
    "#   lakelogic import-dbt --schema dbt_schema.yml --model customers --output contracts/ --dry-run"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "qs-dbt-0008",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Step 2 — Generate Synthetic Customers\n",
    "\n",
    "`DataGenerator.from_dbt()` reads the same schema file and generates rows that:\n",
    "- Respect field types (boolean stays boolean, integer stays integer)\n",
    "- Honour `accepted_values` — `country` and `tier` are always from the allowed set\n",
    "- Honour `expression_is_true` — `total_spend` is always ≥ 0 for valid rows\n",
    "- Inject a controlled ratio of **intentionally invalid rows** to test quarantine\n",
    "\n",
    "No data file needed. Unblock your pipeline on day one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "qs-dbt-0009",
   "metadata": {},
   "outputs": [],
   "source": [
    "gen = DataGenerator.from_dbt(SCHEMA_FILE, model=\"customers\", seed=42)\n",
    "\n",
    "# 200 rows total: 90% valid, 10% intentionally bad (to exercise quarantine)\n",
    "df = gen.generate(rows=200, invalid_ratio=0.10)\n",
    "\n",
    "print(f\"Generated  : {len(df)} rows\")\n",
    "print(f\"Schema     : {df.schema}\")\n",
    "print()\n",
    "display(df.head(6))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "qs-dbt-0010",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verify field rules were injected from dbt tests\n",
    "fr = gen._build_field_rules()\n",
    "print(f\"country accepted values : {fr.get('country', {}).get('accepted_values')}\")\n",
    "print(f\"tier    accepted values : {fr.get('tier', {}).get('accepted_values')}\")\n",
    "print(f\"total_spend min         : {fr.get('total_spend', {}).get('min')}\")\n",
    "print()\n",
    "\n",
    "# Spot-check correct types and valid constrained values\n",
    "print(\"Sample PII fields:\")\n",
    "display(df.select([\"customer_id\", \"email\", \"full_name\", \"tier\", \"country\"]).head(4))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "qs-dbt-0011",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Step 3 — Quality Validation\n",
    "\n",
    "`DataProcessor.from_dbt()` uses the same schema file and runs all the quality rules\n",
    "that were mapped from dbt tests — at ingestion time, before the data reaches Silver.\n",
    "\n",
    "The result splits cleanly into **good** (passes all rules) and **quarantined** (fails any rule).\n",
    "\n",
    "> ℹ️ With `invalid_ratio=0.10` you should see ~90% good, ~10% quarantined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "qs-dbt-0012",
   "metadata": {},
   "outputs": [],
   "source": [
    "proc = DataProcessor.from_dbt(SCHEMA_FILE, model=\"customers\")\n",
    "\n",
    "good_df, bad_df = proc.run(df)\n",
    "\n",
    "n_total = len(df)\n",
    "n_good = len(good_df) if good_df is not None else 0\n",
    "n_bad = len(bad_df) if bad_df is not None else 0\n",
    "\n",
    "print(f\"Input rows      : {n_total}\")\n",
    "print(f\"Passed (good)   : {n_good}  ({n_good / n_total:.1%})\")\n",
    "print(f\"Quarantined     : {n_bad}   ({n_bad / n_total:.1%})\")\n",
    "\n",
    "# Rules that were active:\n",
    "print()\n",
    "print(\"Rules that were active:\")\n",
    "q = proc.contract.quality\n",
    "if q:\n",
    "    for r in q.row_rules or []:\n",
    "        print(f\"  [row]  {r.name}: {r.sql}\")\n",
    "    for r in q.dataset_rules or []:\n",
    "        print(f\"  [data] {r.name}: {r.sql}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "qs-dbt-0013",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Peek at why bad rows were quarantined\n",
    "# _lakelogic_errors holds the list of rule names that each row failed\n",
    "if bad_df is not None and len(bad_df) > 0:\n",
    "    print(\"Sample quarantined rows:\")\n",
    "    diag_cols = [\n",
    "        c\n",
    "        for c in [\n",
    "            \"customer_id\",\n",
    "            \"email\",\n",
    "            \"tier\",\n",
    "            \"country\",\n",
    "            \"total_spend\",\n",
    "            \"_lakelogic_errors\",\n",
    "            \"_lakelogic_categories\",\n",
    "        ]\n",
    "        if c in bad_df.columns\n",
    "    ]\n",
    "    display(bad_df.select(diag_cols).head(5))\n",
    "else:\n",
    "    print(\"No bad rows — all invalid rows may have been caught by required-field rules only.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "qs-dbt-0014",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Step 4 — GDPR: Right-to-Erasure\n",
    "\n",
    "A customer contacts support: *\"Please delete all my personal data.\"*\n",
    "\n",
    "`forget_subjects()` knows which columns are PII **because the dbt schema said so**\n",
    "via `meta: {pii: true}`. No extra config. Three strategies available:\n",
    "\n",
    "| Strategy | What happens to PII values | Referential integrity |\n",
    "|---|---|---|\n",
    "| `nullify` | Replaced with `null` | Lost |\n",
    "| `hash` | SHA-256, deterministic | Preserved — can still JOIN on hashed values |\n",
    "| `redact` | Replaced with `***REDACTED***` | Lost (human-readable audit trail) |\n",
    "\n",
    "> **Note**: `forget_subjects` casts `customer_id` to string internally for matching,\n",
    "> so passing `subject_id` as a string works even though the column is `Int64`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "qs-dbt-0015",
   "metadata": {},
   "outputs": [],
   "source": [
    "import polars as pl\n",
    "\n",
    "# Pick a real customer_id from the good dataset to erase\n",
    "subject_id = str(good_df[\"customer_id\"][0])\n",
    "print(f\"Erasing customer_id = {subject_id}\")\n",
    "print()\n",
    "\n",
    "# Show before\n",
    "before = good_df.filter(pl.col(\"customer_id\").cast(pl.Utf8) == subject_id)\n",
    "print(\"Before erasure:\")\n",
    "display(before.select([\"customer_id\", \"email\", \"full_name\", \"phone\"]))\n",
    "print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "qs-dbt-0016",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Strategy 1: NULLIFY — PII fields become null\n",
    "df_nullified = forget_subjects(\n",
    "    good_df,\n",
    "    proc.contract,\n",
    "    subject_column=\"customer_id\",\n",
    "    subject_ids=[subject_id],\n",
    "    erasure_strategy=\"nullify\",\n",
    ")\n",
    "after_null = df_nullified.filter(pl.col(\"customer_id\").cast(pl.Utf8) == subject_id)\n",
    "print(\"After NULLIFY:\")\n",
    "display(after_null.select([\"customer_id\", \"email\", \"full_name\", \"phone\"]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "qs-dbt-0017",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Strategy 2: HASH — PII replaced with SHA-256, references intact\n",
    "df_hashed = forget_subjects(\n",
    "    good_df,\n",
    "    proc.contract,\n",
    "    subject_column=\"customer_id\",\n",
    "    subject_ids=[subject_id],\n",
    "    erasure_strategy=\"hash\",\n",
    "    hash_salt=\"my-secret-salt-2026\",\n",
    ")\n",
    "after_hash = df_hashed.filter(pl.col(\"customer_id\").cast(pl.Utf8) == subject_id)\n",
    "print(\"After HASH (PII is deterministically hashed — can still JOIN):\")\n",
    "display(after_hash.select([\"customer_id\", \"email\", \"full_name\", \"phone\"]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "qs-dbt-0018",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Strategy 3: REDACT — human-readable, best for audit logs\n",
    "df_redacted = forget_subjects(\n",
    "    good_df,\n",
    "    proc.contract,\n",
    "    subject_column=\"customer_id\",\n",
    "    subject_ids=[subject_id],\n",
    "    erasure_strategy=\"redact\",\n",
    ")\n",
    "after_redact = df_redacted.filter(pl.col(\"customer_id\").cast(pl.Utf8) == subject_id)\n",
    "print(\"After REDACT:\")\n",
    "display(after_redact.select([\"customer_id\", \"email\", \"full_name\", \"phone\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "qs-dbt-0019",
   "metadata": {},
   "source": [
    "### Generate a compliance-ready audit report\n",
    "\n",
    "This is what you send to the DPO or attach to a DSAR (Data Subject Access Request) response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "qs-dbt-0020",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "report = generate_erasure_report(\n",
    "    contract=proc.contract,\n",
    "    subject_column=\"customer_id\",\n",
    "    subject_ids=[subject_id],\n",
    "    erasure_strategy=\"nullify\",\n",
    "    affected_rows=len(before),\n",
    ")\n",
    "\n",
    "print(json.dumps(report, indent=2, default=str))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "qs-dbt-0021",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Step 5 — GDPR: Mask All PII for Dev / Test\n",
    "\n",
    "You want to give the data engineering team a copy of production data to test with —\n",
    "but with all PII stripped. `mask_pii_columns()` blanket-anonymises every field\n",
    "marked `pii: true` in the contract (which we got from `meta: {pii: true}` in dbt)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "qs-dbt-0022",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_masked = mask_pii_columns(\n",
    "    good_df,\n",
    "    proc.contract,\n",
    "    strategy=\"redact\",  # or \"nullify\" or \"hash\"\n",
    ")\n",
    "\n",
    "print(\"PII fields in dev copy:\")\n",
    "pii_preview = [\"customer_id\", \"email\", \"full_name\", \"phone\", \"date_of_birth\"]\n",
    "display(df_masked.select([c for c in pii_preview if c in df_masked.columns]).head(5))\n",
    "print()\n",
    "print(\"Non-PII fields are untouched:\")\n",
    "display(df_masked.select([\"customer_id\", \"tier\", \"country\", \"total_spend\", \"is_active\"]).head(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "qs-dbt-0023",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Step 6 — Batch Erasure + Audit Report\n",
    "\n",
    "In practice, erasure requests come in batches. One call handles them all."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "qs-dbt-0024",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Simulate a batch of 5 erasure requests received this week\n",
    "erasure_ids = [str(good_df[\"customer_id\"][i]) for i in range(5)]\n",
    "print(f\"Batch erasure for {len(erasure_ids)} subjects: {erasure_ids}\")\n",
    "\n",
    "df_batch_erased = forget_subjects(\n",
    "    good_df,\n",
    "    proc.contract,\n",
    "    subject_column=\"customer_id\",\n",
    "    subject_ids=erasure_ids,\n",
    "    erasure_strategy=\"hash\",\n",
    "    hash_salt=\"weekly-erasure-salt\",\n",
    ")\n",
    "\n",
    "# Verify erased rows are hashed\n",
    "erased_sample = df_batch_erased.filter(pl.col(\"customer_id\").cast(pl.Utf8).is_in(erasure_ids))\n",
    "print(f\"\\nErased rows: {len(erased_sample)}\")\n",
    "display(erased_sample.select([\"customer_id\", \"email\", \"full_name\"]).head(5))\n",
    "\n",
    "# DPO audit report for the batch\n",
    "batch_report = generate_erasure_report(\n",
    "    contract=proc.contract,\n",
    "    subject_column=\"customer_id\",\n",
    "    subject_ids=erasure_ids,\n",
    "    erasure_strategy=\"hash\",\n",
    "    affected_rows=len(erased_sample),\n",
    ")\n",
    "print()\n",
    "import json\n",
    "\n",
    "print(json.dumps(batch_report, indent=2, default=str))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "qs-dbt-0025",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Step 7 — CLI Reference\n",
    "\n",
    "Everything above is also available from the terminal — no notebook required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "qs-dbt-0026",
   "metadata": {},
   "outputs": [],
   "source": [
    "schema_path = str(SCHEMA_FILE)\n",
    "\n",
    "print(\"# ── Import dbt schema → LakeLogic contract ─────────────────────────\")\n",
    "print(f\"lakelogic import-dbt --schema {schema_path} --model customers --output contracts/\")\n",
    "print()\n",
    "print(\"# ── Dry-run preview (no files written) ──────────────────────────────\")\n",
    "print(f\"lakelogic import-dbt --schema {schema_path} --model customers --dry-run\")\n",
    "print()\n",
    "print(\"# ── Import every model in the schema file ────────────────────────────\")\n",
    "print(f\"lakelogic import-dbt --schema {schema_path} --output contracts/\")\n",
    "print()\n",
    "print(\"# ── Run validation against a data file ──────────────────────────────\")\n",
    "print(\"lakelogic run --contract contracts/customers.yaml --source data/customers.csv\")\n",
    "print()\n",
    "print(\"# ── Generate 500 synthetic test rows ─────────────────────────────────\")\n",
    "print(\n",
    "    \"lakelogic generate --contract contracts/customers.yaml --rows 500 --invalid-ratio 0.1 --output data/test_customers.parquet\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "qs-dbt-0027",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Summary\n",
    "\n",
    "```\n",
    "dbt_schema.yml          ← the ONLY schema file you maintain\n",
    "         │\n",
    "         ├─ DbtAdapter()              → DataContract (PII, required, rules all mapped)\n",
    "         │\n",
    "         ├─ DataGenerator.from_dbt()  → synthetic test data  (Step 2)\n",
    "         ├─ DataProcessor.from_dbt()  → quality validation    (Step 3)\n",
    "         ├─ forget_subjects()         → GDPR right-to-erasure (Step 4)\n",
    "         ├─ mask_pii_columns()        → dev/test PII masking  (Step 5)\n",
    "         └─ generate_erasure_report() → DPO audit trail       (Steps 4 & 6)\n",
    "```\n",
    "\n",
    "**One schema file → zero duplication → full compliance.**\n",
    "\n",
    "### Next steps\n",
    "\n",
    "- **`02_core_patterns`** — Medallion architecture and reference joins\n",
    "- **`03_advanced_workflows/polars_streaming`** — Process 50 GB files without OOM\n",
    "- **`03_advanced_workflows/gdpr_compliance`** — Full GDPR demo with audit reports\n",
    "- **`03_advanced_workflows/shared_governance_scale`** — 26 contracts in parallel with shared templates\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv_lakelogic",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
