{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell_header",
   "metadata": {},
   "source": [
    "# HIPAA & GDPR Compliance with LakeLogic\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lakelogic/LakeLogic/blob/main/examples/03_compliance_governance/hipaa_gdpr_pii_masking/tutorial_hipaa_gdpr_compliance.ipynb) \n",
    "[![GitHub Repo](https://img.shields.io/badge/GitHub-Repo-blue?logo=github)](https://github.com/lakelogic/LakeLogic/blob/main/examples/03_compliance_governance/hipaa_gdpr_pii_masking/tutorial_hipaa_gdpr_compliance.ipynb)\n",
    "\n",
    "## Business Scenario\n",
    "\n",
    "A global healthcare and e-commerce company operates across the **United States** and **European Union**.\n",
    "They ingest two types of sensitive data:\n",
    "\n",
    "1. **US Patient Records** — containing **Protected Health Information (PHI)** under **HIPAA**\n",
    "   (names, SSNs, diagnoses)\n",
    "2. **EU Customer Records** — containing **Personally Identifiable Information (PII)** under **GDPR**\n",
    "   (names, emails, phone numbers, consent status)\n",
    "\n",
    "Both regulations require strict controls, but they differ in important ways:\n",
    "\n",
    "| Requirement | HIPAA (US) | GDPR (EU) |\n",
    "| :--- | :--- | :--- |\n",
    "| **Scope** | Protected Health Information (PHI) | All personal data of EU/EEA residents |\n",
    "| **Consent** | Implied for treatment | Must be explicit and recorded (Art. 7) |\n",
    "| **Lawful Basis** | Not required per record | Must be one of six types (Art. 6) |\n",
    "| **Right to Erasure** | Limited HIPAA exceptions | Full right to be forgotten (Art. 17) |\n",
    "| **Data Retention** | Minimum 6 years | Storage limitation required (Art. 5(1)(e)) |\n",
    "| **PII Masking** | De-identification of 18 PHI identifiers | Pseudonymisation or anonymisation |\n",
    "\n",
    "## Value Proposition\n",
    "\n",
    "- **Multi-Regulation Compliance** — apply HIPAA and GDPR Policy Packs to different tables using the same framework\n",
    "- **Automated PII Protection** — mask sensitive fields at ingestion, not as a downstream afterthought\n",
    "- **Audit-Ready Quarantine** — every failed record is quarantined with the exact rule violation\n",
    "- **Portable Governance** — same YAML contracts work on Polars (dev), Spark (prod), DuckDB (CI/CD)\n",
    "\n",
    "---\n",
    "\n",
    "## Goals\n",
    "\n",
    "1. Apply HIPAA Policy Pack to US patient records — validate format, mask PHI\n",
    "2. Apply GDPR Policy Pack to EU customer records — enforce consent, lawful basis, retention\n",
    "3. Compare results side-by-side to understand the regulatory differences\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_setup_heading",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_setup",
   "metadata": {},
   "outputs": [],
   "source": [
    "import importlib.util\n",
    "import os\n",
    "import sys\n",
    "import yaml\n",
    "import polars as pl\n",
    "from pathlib import Path\n",
    "\n",
    "if importlib.util.find_spec(\"lakelogic\") is None:\n",
    "    import subprocess\n",
    "\n",
    "    subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"lakelogic\", \"-q\"], check=True)\n",
    "    print(\"lakelogic installed.\")\n",
    "else:\n",
    "    print(\"lakelogic ready.\")\n",
    "\n",
    "if \"google.colab\" in sys.modules:\n",
    "    repo = Path(\"/content/LakeLogic\")\n",
    "    if not repo.exists():\n",
    "        import subprocess\n",
    "\n",
    "        subprocess.run(\n",
    "            [\n",
    "                \"git\",\n",
    "                \"clone\",\n",
    "                \"--quiet\",\n",
    "                \"https://github.com/lakelogic/LakeLogic.git\",\n",
    "                str(repo),\n",
    "            ],\n",
    "            check=True,\n",
    "        )\n",
    "    os.chdir(repo / \"examples\" / \"03_compliance_governance\" / \"hipaa_gdpr_pii_masking\")\n",
    "    print(f\"Working directory: {Path.cwd()}\")\n",
    "\n",
    "\n",
    "def resolve_example_dir(name: str) -> Path:\n",
    "    cwd = Path.cwd()\n",
    "    for base in [cwd] + list(cwd.parents):\n",
    "        for candidate in [\n",
    "            base / name,\n",
    "            base / \"examples\" / \"03_compliance_governance\" / name,\n",
    "        ]:\n",
    "            if candidate.exists():\n",
    "                return candidate\n",
    "    return cwd / name\n",
    "\n",
    "\n",
    "def resolve_repo_root(start: Path) -> Path:\n",
    "    for base in [start] + list(start.parents):\n",
    "        if (base / \"policy_packs\" / \"hipaa_compliance.yaml\").exists():\n",
    "            return base\n",
    "    return start\n",
    "\n",
    "\n",
    "base_path = resolve_example_dir(\"hipaa_gdpr_pii_masking\")\n",
    "repo_root = resolve_repo_root(base_path)\n",
    "\n",
    "print(f\"Example dir: {base_path}\")\n",
    "print(f\"Repo root  : {repo_root}\")\n",
    "print(\"Setup complete.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_how_it_works",
   "metadata": {},
   "source": [
    "## How It Works\n",
    "\n",
    "LakeLogic's **Policy Pack** system lets you define compliance rules once and apply them\n",
    "across any number of contracts. Each regulation gets its own pack:\n",
    "\n",
    "```\n",
    "policy_packs/\n",
    "  hipaa_compliance.yaml   ←  SSN format, email validation, PHI de-identification\n",
    "  gdpr_compliance.yaml    ←  Consent (Art.7), lawful basis (Art.6), retention (Art.5)\n",
    "\n",
    "contracts/\n",
    "  medical_records.yaml    ←  references hipaa_compliance + pii_masking hook\n",
    "  eu_customers.yaml       ←  references gdpr_compliance + pii_masking hook\n",
    "```\n",
    "\n",
    "### HIPAA vs GDPR — rules at a glance\n",
    "\n",
    "| Pack | Key rules | PII treatment |\n",
    "|---|---|---|\n",
    "| HIPAA | SSN format, email validation | Fields → `[PROTECTED]` |\n",
    "| GDPR | Consent required, lawful basis, retention date, email | Fields → `[GDPR_REDACTED]` |\n",
    "\n",
    "### Source data overview\n",
    "\n",
    "**US patient records** (`phi_records.csv`): Bob — invalid email; Edward — malformed SSN\n",
    "\n",
    "**EU customer records** (`eu_customers.csv`):\n",
    "\n",
    "| Customer | Violation | GDPR Article |\n",
    "| :--- | :--- | :--- |\n",
    "| Sofia García | Invalid email format | Art. 5 (Accuracy) |\n",
    "| Lars Eriksson | Missing retention date, empty phone | Art. 5(1)(e) (Storage Limitation) |\n",
    "| Maria Rossi | Missing consent status | Art. 7 (Consent) |\n",
    "| Hans Weber | Invalid lawful basis | Art. 6 (Lawful Basis) |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_part1_heading",
   "metadata": {},
   "source": [
    "---\n",
    "## Part 1 — HIPAA Compliance (US Healthcare)\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_hipaa_policy_heading",
   "metadata": {},
   "source": [
    "## 1. Load HIPAA Policy Pack\n",
    "\n",
    "The HIPAA pack enforces SSN format, email validation, and strict schema evolution\n",
    "across every contract that references it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_hipaa_policy",
   "metadata": {},
   "outputs": [],
   "source": [
    "hipaa_pack_path = repo_root / \"policy_packs\" / \"hipaa_compliance.yaml\"\n",
    "\n",
    "with open(hipaa_pack_path, \"r\") as f:\n",
    "    hipaa_pack = yaml.safe_load(f)\n",
    "\n",
    "print(\"HIPAA Policy Pack rules:\")\n",
    "print(f\"  Framework   : {hipaa_pack.get('defaults', {}).get('metadata', {}).get('compliance_framework', 'N/A')}\")\n",
    "print(f\"  Schema policy: {hipaa_pack.get('defaults', {}).get('schema_policy', {}).get('evolution', 'N/A')}\")\n",
    "print()\n",
    "for rule in hipaa_pack.get(\"defaults\", {}).get(\"quality\", {}).get(\"row_rules\", []):\n",
    "    print(f\"  - {rule.get('name')}: {rule.get('description')}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_hipaa_raw_heading",
   "metadata": {},
   "source": [
    "## 2. Inspect Raw Patient Records (PHI)\n",
    "\n",
    "The source CSV contains Protected Health Information — patient names, SSNs, and emails.\n",
    "Bob has an invalid email; Edward has a malformed SSN.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_hipaa_raw",
   "metadata": {},
   "outputs": [],
   "source": [
    "phi_data_path = base_path / \"data\" / \"phi_records.csv\"\n",
    "raw_phi = pl.read_csv(phi_data_path)\n",
    "\n",
    "print(\"Raw Patient Records (contains PHI):\")\n",
    "display(raw_phi)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_hipaa_run_heading",
   "metadata": {},
   "source": [
    "## 3. Run HIPAA Validation with PII Masking\n",
    "\n",
    "Two layers activate in one `run_source()` call:\n",
    "1. **Quality gates** — SSN format and email validation from the HIPAA Policy Pack\n",
    "2. **PII masking** — names, SSNs, and emails are replaced with `[PROTECTED]` before Silver\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_hipaa_run",
   "metadata": {},
   "outputs": [],
   "source": [
    "from lakelogic import DataProcessor\n",
    "\n",
    "hipaa_contract = base_path / \"contracts\" / \"medical_records.yaml\"\n",
    "hipaa_processor = DataProcessor(contract=hipaa_contract)\n",
    "hipaa_result = hipaa_processor.run_source(phi_data_path)\n",
    "\n",
    "print(\"HIPAA Results:\")\n",
    "print(f\"  Source records : {len(hipaa_result.raw)}\")\n",
    "print(f\"  Valid (Silver) : {len(hipaa_result.good)}\")\n",
    "print(f\"  Quarantined    : {len(hipaa_result.bad)}\")\n",
    "print(f\"  Reconciled     : {len(hipaa_result.raw)} = {len(hipaa_result.good)} + {len(hipaa_result.bad)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_hipaa_quarantine_heading",
   "metadata": {},
   "source": [
    "## 4. Inspect HIPAA Quarantine\n",
    "\n",
    "Bob and Edward failed the HIPAA Policy Pack rules. Each quarantined record includes\n",
    "the exact failure reason for compliance audit.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_hipaa_quarantine",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"QUARANTINE — Failed HIPAA Rules:\")\n",
    "display(hipaa_result.bad)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_hipaa_silver_heading",
   "metadata": {},
   "source": [
    "## 5. Inspect Masked Silver Layer (HIPAA)\n",
    "\n",
    "Alice, Charlie, and Diane passed validation. Their `patient_name`, `ssn`, and `email`\n",
    "are replaced with `[PROTECTED]` — safe for analytics teams, no raw PHI in Silver.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_hipaa_silver",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"SILVER LAYER — Clean & Masked (HIPAA Safe):\")\n",
    "display(hipaa_result.good)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_part2_heading",
   "metadata": {},
   "source": [
    "---\n",
    "## Part 2 — GDPR Compliance (EU Data Protection)\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_gdpr_policy_heading",
   "metadata": {},
   "source": [
    "## 6. Load GDPR Policy Pack\n",
    "\n",
    "GDPR requires different controls than HIPAA. The GDPR pack adds:\n",
    "- **Consent tracking** (Art. 7) — every record must have explicit consent\n",
    "- **Lawful basis** (Art. 6) — one of six legal justifications must be documented\n",
    "- **Storage limitation** (Art. 5(1)(e)) — a retention expiry date must be set\n",
    "- **Data accuracy** (Art. 5) — email must be valid\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_gdpr_policy",
   "metadata": {},
   "outputs": [],
   "source": [
    "gdpr_pack_path = repo_root / \"policy_packs\" / \"gdpr_compliance.yaml\"\n",
    "\n",
    "with open(gdpr_pack_path, \"r\") as f:\n",
    "    gdpr_pack = yaml.safe_load(f)\n",
    "\n",
    "print(\"GDPR Policy Pack rules:\")\n",
    "print(f\"  Framework: {gdpr_pack.get('defaults', {}).get('metadata', {}).get('compliance_framework', 'N/A')}\")\n",
    "print(f\"  Regions  : {gdpr_pack.get('defaults', {}).get('metadata', {}).get('applicable_regions', 'N/A')}\")\n",
    "print()\n",
    "for rule in gdpr_pack.get(\"defaults\", {}).get(\"quality\", {}).get(\"row_rules\", []):\n",
    "    print(f\"  - {rule.get('name')}: {rule.get('description')}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_gdpr_raw_heading",
   "metadata": {},
   "source": [
    "## 7. Inspect Raw EU Customer Data\n",
    "\n",
    "Four records have intentional GDPR violations across different articles.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_gdpr_raw",
   "metadata": {},
   "outputs": [],
   "source": [
    "eu_data_path = base_path / \"data\" / \"eu_customers.csv\"\n",
    "raw_eu = pl.read_csv(eu_data_path)\n",
    "\n",
    "print(\"Raw EU Customer Records (contains PII):\")\n",
    "display(raw_eu)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_gdpr_run_heading",
   "metadata": {},
   "source": [
    "## 8. Run GDPR Validation with PII Masking\n",
    "\n",
    "The EU customer contract validates consent, lawful basis, retention date, and email format,\n",
    "then pseudonymises PII fields in passing records.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_gdpr_run",
   "metadata": {},
   "outputs": [],
   "source": [
    "gdpr_contract = base_path / \"contracts\" / \"eu_customers.yaml\"\n",
    "gdpr_processor = DataProcessor(contract=gdpr_contract)\n",
    "gdpr_result = gdpr_processor.run_source(eu_data_path)\n",
    "\n",
    "print(\"GDPR Results:\")\n",
    "print(f\"  Source records : {len(gdpr_result.raw)}\")\n",
    "print(f\"  Valid (Silver) : {len(gdpr_result.good)}\")\n",
    "print(f\"  Quarantined    : {len(gdpr_result.bad)}\")\n",
    "print(f\"  Reconciled     : {len(gdpr_result.raw)} = {len(gdpr_result.good)} + {len(gdpr_result.bad)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_gdpr_quarantine_heading",
   "metadata": {},
   "source": [
    "## 9. Inspect GDPR Quarantine\n",
    "\n",
    "Each quarantined record includes the specific GDPR article violation —\n",
    "exactly what Data Protection Officers (DPOs) need for audit evidence.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_gdpr_quarantine",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"QUARANTINE — Failed GDPR Rules:\")\n",
    "display(gdpr_result.bad)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_gdpr_silver_heading",
   "metadata": {},
   "source": [
    "## 10. Inspect GDPR-Compliant Silver Layer\n",
    "\n",
    "Valid records have PII pseudonymised (`[GDPR_REDACTED]`). Notice:\n",
    "- `full_name`, `email`, and `phone` are masked\n",
    "- `consent_given`, `lawful_basis`, and `retention_expires_at` are **preserved** (needed for compliance tracking)\n",
    "- Non-PII fields (`country_code`, `id`) remain untouched\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_gdpr_silver",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"SILVER LAYER — GDPR-Compliant & Pseudonymised:\")\n",
    "display(gdpr_result.good)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_part3_heading",
   "metadata": {},
   "source": [
    "---\n",
    "## Part 3 — Comparison & Summary\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_compare_heading",
   "metadata": {},
   "source": [
    "## 11. Side-by-Side Results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_compare",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"COMPLIANCE SUMMARY\")\n",
    "print(\"=\" * 60)\n",
    "print(\"  HIPAA (US Healthcare)\")\n",
    "print(\n",
    "    f\"    Source: {len(hipaa_result.raw):>3}  |  Valid: {len(hipaa_result.good):>3}  |  Quarantined: {len(hipaa_result.bad):>3}\"\n",
    ")\n",
    "print(\"    Masking: patient_name, ssn, email -> [PROTECTED]\")\n",
    "print(\"    Rules: SSN format (XXX-XX-XXXX), email validation\")\n",
    "print()\n",
    "print(\"  GDPR (EU Data Protection)\")\n",
    "print(\n",
    "    f\"    Source: {len(gdpr_result.raw):>3}  |  Valid: {len(gdpr_result.good):>3}  |  Quarantined: {len(gdpr_result.bad):>3}\"\n",
    ")\n",
    "print(\"    Masking: full_name, email, phone -> [GDPR_REDACTED]\")\n",
    "print(\"    Rules: consent (Art.7), lawful basis (Art.6), retention (Art.5), email\")\n",
    "print(\"=\" * 60)\n",
    "print(\"  Key difference:\")\n",
    "print(\"  HIPAA focuses on data FORMAT  (SSN patterns, email syntax)\")\n",
    "print(\"  GDPR  focuses on data RIGHTS  (consent, legal basis, retention)\")\n",
    "print(\"  LakeLogic handles BOTH with the same contract framework.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_summary",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Summary\n",
    "\n",
    "### What LakeLogic did automatically\n",
    "\n",
    "1. **Policy Packs as Compliance Templates** — HIPAA and GDPR rules are defined once and\n",
    "   applied across any number of contracts. Compliance teams manage the pack; data engineers\n",
    "   just reference it.\n",
    "\n",
    "2. **Automated PII Masking at Ingestion** — PII is masked *before* data reaches Silver.\n",
    "   Analysts never see raw identifiers.\n",
    "\n",
    "3. **Audit-Ready Quarantine** — every failed record includes the exact rule name,\n",
    "   the regulatory article reference, and the failure reason.\n",
    "\n",
    "4. **100% Reconciliation** — `source = valid + quarantined`. No silent data loss.\n",
    "\n",
    "5. **Engine-Agnostic** — same contracts run identically on Polars (dev), Spark (prod),\n",
    "   or DuckDB (CI/CD). No compliance logic rewrite when changing platforms.\n",
    "\n",
    "---\n",
    "\n",
    "## Next Steps — Try It Yourself\n",
    "\n",
    "### 1. Add a new patient record with a violation\n",
    "\n",
    "```csv\n",
    "# data/phi_records.csv\n",
    "P006,Frank,999-88-7777,frank-no-at-sign,Flu\n",
    "```\n",
    "\n",
    "### 2. Add a GDPR rule to an existing policy pack\n",
    "\n",
    "```yaml\n",
    "# policy_packs/gdpr_compliance.yaml\n",
    "defaults:\n",
    "  quality:\n",
    "    row_rules:\n",
    "      - name: Phone Not Empty       # <-- add this\n",
    "        sql: \"phone IS NOT NULL AND phone != ''\"\n",
    "        description: \"Art. 5 — contact data must be complete\"\n",
    "```\n",
    "\n",
    "**Key knobs:**\n",
    "\n",
    "| What to change | Where | Effect |\n",
    "|---|---|---|\n",
    "| Add a PHI column to mask | `hooks.post_validate[pii_masking].columns` | That column becomes `[PROTECTED]` in Silver |\n",
    "| Swap policy pack | `policy_pack:` in contract | Switch between HIPAA / GDPR / custom |\n",
    "| Add a quality rule | `defaults.quality.row_rules` in policy pack | Applies to all contracts using that pack |\n",
    "\n",
    "### 3. Explore related playbooks\n",
    "\n",
    "- [`tutorial_hipaa_compliance.ipynb`](tutorial_hipaa_compliance.ipynb) — HIPAA only, simpler walkthrough\n",
    "- [`../../02_core_patterns/bronze_quality_gate/`](../../02_core_patterns/bronze_quality_gate/playbook.ipynb) — quality gating without compliance layer\n",
    "- [`../../notifications_and_secrets/`](../notifications_and_secrets/notifications_secrets.ipynb) — alert on quarantine thresholds\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv_lakelogic",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
