{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell_header",
   "metadata": {},
   "source": [
    "# Soft Delete Pattern\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lakelogic/LakeLogic/blob/main/examples/02_core_patterns/soft_delete/soft_delete_pattern.ipynb) \n",
    "[![GitHub Repo](https://img.shields.io/badge/GitHub-Repo-blue?logo=github)](https://github.com/lakelogic/LakeLogic/blob/main/examples/02_core_patterns/soft_delete/soft_delete_pattern.ipynb)\n",
    "\n",
    "## Business Scenario\n",
    "\n",
    "In regulated environments (GDPR, HIPAA, SOX), hard-deleting rows removes audit history and\n",
    "makes recovery impossible. You need a consistent strategy that flags records as deleted,\n",
    "stamps *when* and *why*, and keeps every row available for compliance queries — without\n",
    "writing custom merge logic in every pipeline.\n",
    "\n",
    "## Value Proposition\n",
    "\n",
    "- Preserve full audit trails — deleted rows stay in the table, just flagged\n",
    "- Capture deletion timestamp and reason automatically\n",
    "- Support easy recovery: flip `_lakelogic_is_deleted` back to `false`\n",
    "- Standardize delete metadata across all datasets via a single contract setting\n",
    "\n",
    "---\n",
    "\n",
    "## Goals\n",
    "\n",
    "1. Load an initial batch of active users\n",
    "2. Process a CDC delete signal — watch LakeLogic flag the row instead of removing it\n",
    "3. Filter active vs. soft-deleted rows\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_setup_heading",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_setup",
   "metadata": {},
   "outputs": [],
   "source": [
    "import importlib.util\n",
    "import os\n",
    "import sys\n",
    "import shutil\n",
    "import polars as pl\n",
    "from pathlib import Path\n",
    "\n",
    "if importlib.util.find_spec(\"lakelogic\") is None:\n",
    "    import subprocess\n",
    "\n",
    "    subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"lakelogic\", \"-q\"], check=True)\n",
    "    print(\"lakelogic installed.\")\n",
    "else:\n",
    "    print(\"lakelogic ready.\")\n",
    "\n",
    "if \"google.colab\" in sys.modules:\n",
    "    repo = Path(\"/content/LakeLogic\")\n",
    "    if not repo.exists():\n",
    "        import subprocess\n",
    "\n",
    "        subprocess.run(\n",
    "            [\n",
    "                \"git\",\n",
    "                \"clone\",\n",
    "                \"--quiet\",\n",
    "                \"https://github.com/lakelogic/LakeLogic.git\",\n",
    "                str(repo),\n",
    "            ],\n",
    "            check=True,\n",
    "        )\n",
    "    os.chdir(repo / \"examples\" / \"02_core_patterns\" / \"soft_delete\")\n",
    "\n",
    "\n",
    "def get_path(*parts):\n",
    "    base = Path.cwd()\n",
    "    for candidate in [base] + list(base.parents):\n",
    "        target = candidate / \"soft_delete\" if candidate.name != \"soft_delete\" else candidate\n",
    "        if (target / \"contract.yaml\").exists():\n",
    "            return (target / Path(*parts)).resolve()\n",
    "    return (base / Path(*parts)).resolve()\n",
    "\n",
    "\n",
    "# Clear previous output for a clean run\n",
    "output_dir = get_path(\"data\", \"users_silver\")\n",
    "if output_dir.exists():\n",
    "    shutil.rmtree(output_dir)\n",
    "\n",
    "from lakelogic import DataProcessor\n",
    "\n",
    "print(\"Setup complete.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_how_it_works",
   "metadata": {},
   "source": [
    "## How It Works\n",
    "\n",
    "LakeLogic's **`merge` strategy** with soft-delete columns intercepts CDC delete signals\n",
    "and, instead of removing the row, stamps it with configurable metadata columns.\n",
    "\n",
    "### Data flow\n",
    "\n",
    "```\n",
    "Batch 1  (Alice I, Bob I)\n",
    "  → merge into silver table\n",
    "  → _lakelogic_is_deleted = false for both\n",
    "\n",
    "Batch 2  (Bob D  ← CDC delete signal)\n",
    "  → cdc_op_field detects op = \"D\"\n",
    "  → instead of removing Bob:\n",
    "       _lakelogic_is_deleted  = true\n",
    "       _lakelogic_deleted_at  = <current UTC timestamp>\n",
    "       _lakelogic_delete_reason = \"cdc_delete_signal\"\n",
    "  → Alice untouched\n",
    "```\n",
    "\n",
    "### Key contract settings\n",
    "\n",
    "| Setting | Value | What it does |\n",
    "|---|---|---|\n",
    "| `source.cdc_op_field` | `op` | Column that signals the operation type |\n",
    "| `source.cdc_delete_values` | `[\"D\"]` | Values in `op` that mean delete |\n",
    "| `materialization.strategy` | `merge` | Upserts by primary key; applies soft-delete logic |\n",
    "| `soft_delete_column` | `_lakelogic_is_deleted` | Boolean flag set to `true` when deleted |\n",
    "| `soft_delete_time_column` | `_lakelogic_deleted_at` | Auto-stamped with current UTC timestamp |\n",
    "| `soft_delete_reason_column` | `_lakelogic_delete_reason` | Reason string — defaults to `cdc_delete_signal` |\n",
    "\n",
    "### Known data\n",
    "\n",
    "| user_id | Name | Batch | `op` | Expected outcome |\n",
    "|---|---|---|---|---|\n",
    "| 1 | Alice | 1 | `I` | Active — `is_deleted = false` |\n",
    "| 2 | Bob | 1 | `I` | Active — `is_deleted = false` |\n",
    "| 2 | Bob | 2 | `D` | Soft-deleted — `is_deleted = true`, timestamp set |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_load1_heading",
   "metadata": {},
   "source": [
    "## 1. Initial Load — Insert Active Users\n",
    "\n",
    "Two users arrive with `op = \"I\"`. Both are merged as active records.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_load1",
   "metadata": {},
   "outputs": [],
   "source": [
    "contract_path = get_path(\"contract.yaml\")\n",
    "output_path = get_path(\"data\", \"users_silver\", \"data.parquet\")\n",
    "\n",
    "processor = DataProcessor(contract=contract_path)\n",
    "processor.run(\n",
    "    [\n",
    "        {\"user_id\": 1, \"name\": \"Alice\", \"op\": \"I\"},\n",
    "        {\"user_id\": 2, \"name\": \"Bob\", \"op\": \"I\"},\n",
    "    ],\n",
    "    source_path=\"initial_load\",\n",
    "    materialize=True,\n",
    ")\n",
    "\n",
    "print(\"SILVER TABLE — BATCH 1 (initial load):\")\n",
    "display(\n",
    "    pl.read_parquet(output_path).select(\n",
    "        [\n",
    "            \"user_id\",\n",
    "            \"name\",\n",
    "            \"_lakelogic_is_deleted\",\n",
    "            \"_lakelogic_deleted_at\",\n",
    "            \"_lakelogic_delete_reason\",\n",
    "        ]\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_delete_heading",
   "metadata": {},
   "source": [
    "## 2. Soft Delete — Process a CDC Delete Signal\n",
    "\n",
    "Bob arrives with `op = \"D\"`. LakeLogic stamps the row in-place; no hard delete.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_delete",
   "metadata": {},
   "outputs": [],
   "source": [
    "processor.run(\n",
    "    [{\"user_id\": 2, \"name\": \"Bob\", \"op\": \"D\"}],\n",
    "    source_path=\"cdc_batch_2\",\n",
    "    materialize=True,\n",
    ")\n",
    "\n",
    "print(\"SILVER TABLE — BATCH 2 (soft delete applied):\")\n",
    "display(\n",
    "    pl.read_parquet(output_path).select(\n",
    "        [\n",
    "            \"user_id\",\n",
    "            \"name\",\n",
    "            \"_lakelogic_is_deleted\",\n",
    "            \"_lakelogic_deleted_at\",\n",
    "            \"_lakelogic_delete_reason\",\n",
    "        ]\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_filter_heading",
   "metadata": {},
   "source": [
    "## 3. Filter — Active vs. Deleted\n",
    "\n",
    "Both rows are preserved. Use `_lakelogic_is_deleted` to split them for downstream views.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_filter",
   "metadata": {},
   "outputs": [],
   "source": [
    "full_table = pl.read_parquet(output_path)\n",
    "\n",
    "print(\"ACTIVE (_lakelogic_is_deleted = false):\")\n",
    "display(full_table.filter(pl.col(\"_lakelogic_is_deleted\") == False).select([\"user_id\", \"name\"]))\n",
    "\n",
    "print(\"SOFT-DELETED (_lakelogic_is_deleted = true):\")\n",
    "display(\n",
    "    full_table.filter(pl.col(\"_lakelogic_is_deleted\") == True).select(\n",
    "        [\"user_id\", \"name\", \"_lakelogic_deleted_at\", \"_lakelogic_delete_reason\"]\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_summary",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Summary\n",
    "\n",
    "| Metric | Batch 1 | Batch 2 |\n",
    "|---|---|---|\n",
    "| Records processed | 2 inserts | 1 delete signal |\n",
    "| Active rows after | 2 | 1 (Alice) |\n",
    "| Soft-deleted rows after | 0 | 1 (Bob) |\n",
    "| Hard deletes | 0 | 0 — row preserved |\n",
    "\n",
    "### What LakeLogic did automatically\n",
    "\n",
    "- Intercepted `op = \"D\"` via `cdc_op_field` / `cdc_delete_values` — no conditional code needed\n",
    "- Stamped `_lakelogic_is_deleted = true` on the matched row (by `primary_key: user_id`)\n",
    "- Auto-filled `_lakelogic_deleted_at` with the current UTC timestamp\n",
    "- Auto-filled `_lakelogic_delete_reason = \"cdc_delete_signal\"`\n",
    "- Left Alice unchanged\n",
    "\n",
    "---\n",
    "\n",
    "## Next Steps — Try It Yourself\n",
    "\n",
    "### 1. Edit the source data\n",
    "\n",
    "```python\n",
    "# Add a third user and supply a custom delete reason:\n",
    "v1_rows = [\n",
    "    {\"user_id\": 1, \"name\": \"Alice\", \"op\": \"I\"},\n",
    "    {\"user_id\": 2, \"name\": \"Bob\",   \"op\": \"I\"},\n",
    "    {\"user_id\": 3, \"name\": \"Carol\", \"op\": \"I\"},   # <-- new\n",
    "]\n",
    "\n",
    "v2_rows = [\n",
    "    {\"user_id\": 1, \"name\": \"Alice\", \"op\": \"D\",\n",
    "     \"_lakelogic_delete_reason\": \"gdpr_erasure_request\"},  # custom reason\n",
    "]\n",
    "```\n",
    "\n",
    "**Ideas:**\n",
    "- Delete a user never inserted → tombstone row created as already-deleted\n",
    "- Mix inserts and deletes in the same batch\n",
    "- Add `_lakelogic_delete_reason` to the source row to override the default reason\n",
    "\n",
    "### 2. Edit the contract\n",
    "\n",
    "```yaml\n",
    "source:\n",
    "  cdc_op_field: \"op\"              # column that signals the operation type\n",
    "  cdc_delete_values: [\"D\"]        # → try [\"DELETE\"] or [\"d\", \"D\"] for your CDC\n",
    "\n",
    "primary_key: [\"user_id\"]          # key used to match incoming row to existing record\n",
    "\n",
    "materialization:\n",
    "  strategy: merge\n",
    "  soft_delete_column: \"_lakelogic_is_deleted\"      # boolean flag column\n",
    "  soft_delete_value: true                           # value written when deleted\n",
    "  soft_delete_time_column: \"_lakelogic_deleted_at\" # auto-stamped UTC timestamp\n",
    "  soft_delete_reason_column: \"_lakelogic_delete_reason\" # overridable per-row\n",
    "```\n",
    "\n",
    "**Key contract knobs:**\n",
    "\n",
    "| What to change | Where in `contract.yaml` | Effect |\n",
    "|---|---|---|\n",
    "| CDC operation column | `source.cdc_op_field` | Which column signals insert / delete |\n",
    "| Delete signal values | `source.cdc_delete_values` | e.g. `[\"D\"]`, `[\"DELETE\"]` |\n",
    "| Deleted flag column | `soft_delete_column` | Boolean column written on deleted rows |\n",
    "| Deletion timestamp column | `soft_delete_time_column` | Auto-stamped; omit to skip |\n",
    "| Deletion reason column | `soft_delete_reason_column` | Overridable per-row in source |\n",
    "| Match key | `primary_key` | Column(s) used to find the existing row |\n",
    "\n",
    "### 3. Explore related playbooks\n",
    "\n",
    "- [`../scd2_dimension/`](../scd2_dimension/playbook.ipynb) — preserve full history with versioned rows\n",
    "- [`../reference_joins/`](../reference_joins/playbook.ipynb) — enrich records in the same contract\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv_lakelogic",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}