{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell_header",
   "metadata": {},
   "source": [
    "# HIPAA Compliance & PII Masking\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lakelogic/LakeLogic/blob/main/examples/03_compliance_governance/hipaa_gdpr_pii_masking/tutorial_hipaa_compliance.ipynb) \n",
    "[![GitHub Repo](https://img.shields.io/badge/GitHub-Repo-blue?logo=github)](https://github.com/lakelogic/LakeLogic/blob/main/examples/03_compliance_governance/hipaa_gdpr_pii_masking/tutorial_hipaa_compliance.ipynb)\n",
    "\n",
    "## Business Scenario\n",
    "\n",
    "A healthcare provider is ingesting patient medical records into their Data Lakehouse.\n",
    "These records contain **Protected Health Information (PHI)** and **Personally Identifiable\n",
    "Information (PII)** — patient names, Social Security Numbers, and email addresses.\n",
    "\n",
    "The organization has two strict requirements:\n",
    "1. **Governance Consistency** — every medical dataset must adhere to the institutional HIPAA\n",
    "   policy (SSN formats, email validation) without re-specifying rules in each contract.\n",
    "2. **Risk Mitigation** — all PII must be masked automatically before it reaches the Silver\n",
    "   layer, so data scientists can work with the data without exposure to raw identifiers.\n",
    "\n",
    "## Value Proposition\n",
    "\n",
    "- **Zero-Day Compliance** — apply shared HIPAA Policy Packs across hundreds of tables instantly\n",
    "- **Automated Protection** — shift-left security: PII is masked at ingestion, not as an afterthought\n",
    "- **Audit-Ready Quarantine** — every failed record is tagged with lineage and the exact rule violation\n",
    "\n",
    "---\n",
    "\n",
    "## Goals\n",
    "\n",
    "1. Set up paths to the contract, HIPAA Policy Pack, and source data\n",
    "2. Inspect the HIPAA Policy Pack rules\n",
    "3. Run ingestion with PII masking\n",
    "4. Inspect quarantined rows (governance failures)\n",
    "5. Inspect the masked Silver layer (safe for analytics)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_setup_heading",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_setup",
   "metadata": {},
   "outputs": [],
   "source": [
    "import importlib.util\n",
    "import os\n",
    "import sys\n",
    "from pathlib import Path\n",
    "\n",
    "if importlib.util.find_spec(\"lakelogic\") is None:\n",
    "    import subprocess\n",
    "\n",
    "    subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"lakelogic\", \"-q\"], check=True)\n",
    "    print(\"lakelogic installed.\")\n",
    "else:\n",
    "    print(\"lakelogic ready.\")\n",
    "\n",
    "if \"google.colab\" in sys.modules:\n",
    "    repo = Path(\"/content/LakeLogic\")\n",
    "    if not repo.exists():\n",
    "        import subprocess\n",
    "\n",
    "        subprocess.run(\n",
    "            [\n",
    "                \"git\",\n",
    "                \"clone\",\n",
    "                \"--quiet\",\n",
    "                \"https://github.com/lakelogic/LakeLogic.git\",\n",
    "                str(repo),\n",
    "            ],\n",
    "            check=True,\n",
    "        )\n",
    "    os.chdir(repo / \"examples\" / \"03_compliance_governance\" / \"hipaa_gdpr_pii_masking\")\n",
    "    print(f\"Working directory: {Path.cwd()}\")\n",
    "\n",
    "\n",
    "def resolve_example_dir(name: str) -> Path:\n",
    "    cwd = Path.cwd()\n",
    "    for base in [cwd] + list(cwd.parents):\n",
    "        for candidate in [\n",
    "            base / name,\n",
    "            base / \"examples\" / \"03_compliance_governance\" / name,\n",
    "        ]:\n",
    "            if candidate.exists():\n",
    "                return candidate\n",
    "    return cwd / name\n",
    "\n",
    "\n",
    "def resolve_repo_root(start: Path) -> Path:\n",
    "    for base in [start] + list(start.parents):\n",
    "        if (base / \"policy_packs\" / \"hipaa_compliance.yaml\").exists():\n",
    "            return base\n",
    "    return start\n",
    "\n",
    "\n",
    "base_path = resolve_example_dir(\"hipaa_gdpr_pii_masking\")\n",
    "repo_root = resolve_repo_root(base_path)\n",
    "contract_path = base_path / \"contracts\" / \"medical_records.yaml\"\n",
    "policy_pack_path = repo_root / \"policy_packs\" / \"hipaa_compliance.yaml\"\n",
    "data_path = base_path / \"data\" / \"phi_records.csv\"\n",
    "\n",
    "print(f\"Contract   : {contract_path}\")\n",
    "print(f\"Policy Pack: {policy_pack_path}\")\n",
    "print(f\"Data       : {data_path}\")\n",
    "print(\"Setup complete.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_how_it_works",
   "metadata": {},
   "source": [
    "## How It Works\n",
    "\n",
    "LakeLogic applies two layers of protection in a single pass:\n",
    "\n",
    "```\n",
    "phi_records.csv  (raw PHI: names, SSNs, emails)\n",
    "      |\n",
    "      v  DataProcessor.run_source()\n",
    "      |    Layer 1: Quality gates from HIPAA Policy Pack\n",
    "      |             - SSN format (XXX-XX-XXXX)\n",
    "      |             - Email validation (must contain @)\n",
    "      |\n",
    "      |    Layer 2: PII masking hook\n",
    "      |             - patient_name → [PROTECTED]\n",
    "      |             - ssn         → [PROTECTED]\n",
    "      |             - email       → [PROTECTED]\n",
    "      |\n",
    "      ├── result.good  →  Silver layer (masked, safe for analytics)\n",
    "      └── result.bad   →  Quarantine  (failed HIPAA rules + reason)\n",
    "```\n",
    "\n",
    "### HIPAA Policy Pack rules\n",
    "\n",
    "| Rule | What it enforces | HIPAA relevance |\n",
    "|---|---|---|\n",
    "| SSN format | `XXX-XX-XXXX` pattern | De-identification standard |\n",
    "| Email validation | Must contain `@` | Data accuracy requirement |\n",
    "\n",
    "### Known source data\n",
    "\n",
    "| Patient | SSN | Email | Expected outcome |\n",
    "|---|---|---|---|\n",
    "| Alice | 123-45-6789 | alice@example.com | ✅ Valid — PII masked in Silver |\n",
    "| Bob | 111-22-3333 | bob@wrong-format | ❌ Quarantined — failed email rule |\n",
    "| Charlie | 222-33-4444 | charlie@example.com | ✅ Valid — PII masked in Silver |\n",
    "| Diane | 333-44-5555 | diane@example.com | ✅ Valid — PII masked in Silver |\n",
    "| Edward | 555-555-555 | edward@example.com | ❌ Quarantined — malformed SSN |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_policy_heading",
   "metadata": {},
   "source": [
    "## 1. Load and Inspect the HIPAA Policy Pack\n",
    "\n",
    "A **Policy Pack** acts as an organisational baseline. The HIPAA pack enforces rules across\n",
    "every contract that references it — the compliance team manages it once; data engineers just point to it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_policy",
   "metadata": {},
   "outputs": [],
   "source": [
    "import yaml\n",
    "\n",
    "with open(policy_pack_path, \"r\") as f:\n",
    "    policy_pack = yaml.safe_load(f)\n",
    "\n",
    "print(\"HIPAA Policy Pack rules:\")\n",
    "for rule in policy_pack.get(\"quality\", {}).get(\"row_rules\", []):\n",
    "    print(f\"  - {rule.get('name')}: {rule.get('description')}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_run_heading",
   "metadata": {},
   "source": [
    "## 2. Run Ingestion with PII Masking\n",
    "\n",
    "Pass the contract to `DataProcessor`. The contract references the HIPAA Policy Pack and\n",
    "activates the `pii_masking` hook — both are applied automatically during `run_source()`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_run",
   "metadata": {},
   "outputs": [],
   "source": [
    "from lakelogic import DataProcessor\n",
    "\n",
    "processor = DataProcessor(contract=contract_path)\n",
    "result = processor.run_source(data_path)\n",
    "\n",
    "print(f\"Source records : {len(result.raw)}\")\n",
    "print(f\"Valid (Silver) : {len(result.good)}\")\n",
    "print(f\"Quarantined    : {len(result.bad)}\")\n",
    "print(f\"Reconciled     : {len(result.raw)} = {len(result.good)} + {len(result.bad)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_quarantine_heading",
   "metadata": {},
   "source": [
    "## 3. Inspect the Quarantine (Governance Failures)\n",
    "\n",
    "Bob (invalid email) and Edward (malformed SSN) were caught by the shared HIPAA rules\n",
    "and diverted to quarantine — with the exact failure reason attached for compliance audit.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_quarantine",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"QUARANTINE — Failed HIPAA Rules:\")\n",
    "display(result.bad)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_silver_heading",
   "metadata": {},
   "source": [
    "## 4. Inspect the Masked Silver Layer\n",
    "\n",
    "For the valid records (Alice, Charlie, Diane), `patient_name`, `ssn`, and `email`\n",
    "have been replaced with `[PROTECTED]`. This data is safe for analytics teams — no raw PHI exposure.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_silver",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"SILVER LAYER — Clean & Masked (HIPAA Safe):\")\n",
    "display(result.good)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_summary",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Summary\n",
    "\n",
    "| Record | Outcome | Reason |\n",
    "|---|---|---|\n",
    "| Alice | ✅ Silver (masked) | Passes all HIPAA rules |\n",
    "| Bob | ❌ Quarantined | Failed email validation |\n",
    "| Charlie | ✅ Silver (masked) | Passes all HIPAA rules |\n",
    "| Diane | ✅ Silver (masked) | Passes all HIPAA rules |\n",
    "| Edward | ❌ Quarantined | Failed SSN format check |\n",
    "\n",
    "### What LakeLogic did automatically\n",
    "\n",
    "- Applied the HIPAA Policy Pack rules to every row — no per-contract repetition\n",
    "- Executed the `pii_masking` hook in-flight during ingestion\n",
    "- Replaced `patient_name`, `ssn`, and `email` in `result.good` with `[PROTECTED]`\n",
    "- Quarantined failing rows to `result.bad` with the exact rule violation reason\n",
    "- Added `_lakelogic_processed_at` and `_lakelogic_run_id` audit columns\n",
    "\n",
    "---\n",
    "\n",
    "## Next Steps — Try It Yourself\n",
    "\n",
    "### 1. Edit the source data\n",
    "\n",
    "Open `data/phi_records.csv` and add a row:\n",
    "\n",
    "```csv\n",
    "patient_id,patient_name,ssn,email,diagnosis\n",
    "P006,Frank,999-88-7777,frank-no-at-sign,Flu\n",
    "```\n",
    "\n",
    "Re-run — Frank should land in quarantine (invalid email format).\n",
    "\n",
    "### 2. Edit the contract\n",
    "\n",
    "```yaml\n",
    "# contracts/medical_records.yaml\n",
    "policy_pack: hipaa_compliance       # references the shared HIPAA rules\n",
    "\n",
    "hooks:\n",
    "  post_validate:\n",
    "    - name: pii_masking\n",
    "      columns: [patient_name, ssn, email]   # <-- add or remove columns here\n",
    "      replacement: \"[PROTECTED]\"\n",
    "```\n",
    "\n",
    "**Key contract knobs:**\n",
    "\n",
    "| What to change | Where | Effect |\n",
    "|---|---|---|\n",
    "| Columns to mask | `hooks.post_validate[pii_masking].columns` | Add any PHI field |\n",
    "| Mask replacement string | `hooks.post_validate[pii_masking].replacement` | Defaults to `[PROTECTED]` |\n",
    "| Policy Pack used | `policy_pack` | Swap to `gdpr_compliance` for EU data |\n",
    "\n",
    "### 3. Explore related tutorials\n",
    "\n",
    "- [`tutorial_hipaa_gdpr_compliance.ipynb`](tutorial_hipaa_gdpr_compliance.ipynb) — HIPAA + GDPR side-by-side comparison\n",
    "- [`../../02_core_patterns/bronze_quality_gate/`](../../02_core_patterns/bronze_quality_gate/playbook.ipynb) — quality gating without PII concerns\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
