{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "header",
   "metadata": {},
   "source": [
    "# Pipeline Quickstart: Bronze → Silver → Gold with DAG\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lakelogic/LakeLogic/blob/main/examples/02_pipeline_quickstart/02_pipeline_quickstart.ipynb)\n",
    "[![GitHub Repo](https://img.shields.io/badge/GitHub-Repo-blue?logo=github)](https://github.com/lakelogic/LakeLogic/blob/main/examples/02_pipeline_quickstart/02_pipeline_quickstart.ipynb)\n",
    "\n",
    "## Business Scenario\n",
    "\n",
    "You need to build a **medallion lakehouse pipeline** that:\n",
    "\n",
    "1. **Bronze** — Ingest raw employee CSV with minimal validation\n",
    "2. **Silver** — Cleanse, enrich with tenure categories, enforce business rules\n",
    "3. **Gold** — Aggregate into a department headcount summary for dashboards\n",
    "\n",
    "LakeLogic does this with **YAML contracts** and a **single `pipeline.run()` call**.\n",
    "\n",
    "---\n",
    "\n",
    "## What You'll Learn\n",
    "\n",
    "| Section | Feature |\n",
    "|---|---|\n",
    "| 1 | `_system.yaml` — the pipeline registry |\n",
    "| 2 | DAG visualization — see contract dependencies |\n",
    "| 3 | Pipeline execution — bronze → silver → gold in one call |\n",
    "| 4 | Per-layer controls — schema evolution, materialization defaults |\n",
    "| 5 | Run summary — what happened across the full pipeline |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "setup_heading",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "setup",
   "metadata": {},
   "outputs": [],
   "source": [
    "import subprocess, sys, os\n",
    "from pathlib import Path\n",
    "\n",
    "# Install/upgrade lakelogic with polars engine (v1.5+ required for pipeline features)\n",
    "subprocess.run(\n",
    "    [sys.executable, \"-m\", \"pip\", \"install\", \"--upgrade\", \"-q\",\n",
    "     \"lakelogic[polars]>=1.5.0\"],\n",
    "    check=True,\n",
    ")\n",
    "\n",
    "import lakelogic\n",
    "print(f\"lakelogic v{lakelogic.__version__} ready.\")\n",
    "\n",
    "# On Colab: clone repo for contract files and sample data\n",
    "if \"google.colab\" in sys.modules:\n",
    "    repo = Path(\"/content/LakeLogic\")\n",
    "    if repo.exists():\n",
    "        import shutil\n",
    "        shutil.rmtree(repo)  # fresh clone to get latest contracts\n",
    "    subprocess.run(\n",
    "        [\"git\", \"clone\", \"--quiet\",\n",
    "         \"https://github.com/lakelogic/LakeLogic.git\", str(repo)],\n",
    "        check=True,\n",
    "    )\n",
    "    os.chdir(repo / \"examples\" / \"02_pipeline_quickstart\")\n",
    "    print(f\"Working directory: {Path.cwd()}\")\n",
    "\n",
    "print(\"Setup complete.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "section1_heading",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 1. The Pipeline Registry (`_system.yaml`)\n",
    "\n",
    "A `_system.yaml` file defines your entire pipeline:\n",
    "\n",
    "- **Contracts** — which entities to process at each layer\n",
    "- **Dependencies** — `depends_on` determines execution order\n",
    "- **Per-layer defaults** — schema evolution, materialization strategy\n",
    "\n",
    "```yaml\n",
    "contracts:\n",
    "  - layer: bronze\n",
    "    entity: raw_employees\n",
    "    path: contracts/bronze/bronze_employees_v1.0.yaml\n",
    "\n",
    "  - layer: silver\n",
    "    entity: clean_employees\n",
    "    path: contracts/silver/silver_employees_v1.0.yaml\n",
    "    depends_on: [raw_employees]    # ← waits for bronze\n",
    "\n",
    "  - layer: gold\n",
    "    entity: headcount\n",
    "    path: contracts/gold/gold_headcount_v1.0.yaml\n",
    "    depends_on: [clean_employees]  # ← waits for silver\n",
    "\n",
    "server_defaults:\n",
    "  bronze:\n",
    "    schema_evolution: append      # source schemas may change\n",
    "  silver:\n",
    "    schema_evolution: strict      # curated — no surprises\n",
    "  gold:\n",
    "    schema_evolution: strict      # business layer — locked down\n",
    "```\n",
    "\n",
    "> **Key rule:** Each entity name must be unique across the pipeline.\n",
    "> `depends_on` references entity names, not layer names.\n",
    "\n",
    "Let's load it:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "load_registry",
   "metadata": {},
   "outputs": [],
   "source": [
    "from lakelogic.core.registry import DomainRegistry\n",
    "from lakelogic.pipeline import LakehousePipeline\n",
    "\n",
    "registry = DomainRegistry.from_yaml(\"_system.yaml\")\n",
    "\n",
    "print(f\"Domain  : {registry.domain}\")\n",
    "print(f\"System  : {registry.system}\")\n",
    "print(f\"Contracts: {len(registry.contracts)}\")\n",
    "print()\n",
    "for c in registry.contracts:\n",
    "    deps = c.depends_on or []\n",
    "    print(f\"  [{c.layer}] {c.entity} {'→ depends_on: ' + str(deps) if deps else '(root)'}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "section2_heading",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 2. DAG Visualization\n",
    "\n",
    "LakeLogic resolves contract dependencies into an execution DAG.\n",
    "Visualize it to see the processing order:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dag_viz",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = LakehousePipeline(registry, engine=\"polars\")\n",
    "\n",
    "# Generate the DAG as interactive HTML\n",
    "dag_html = pipeline.visualize_dag()\n",
    "\n",
    "if dag_html:\n",
    "    from IPython.display import HTML\n",
    "    display(HTML(dag_html))\n",
    "    print(\"\\n↑ Interactive DAG — hover over nodes for details.\")\n",
    "else:\n",
    "    # Fallback: text-based DAG\n",
    "    print(\"DAG Execution Order:\")\n",
    "    print(\"  bronze.raw_employees  →  silver.clean_employees  →  gold.headcount\")\n",
    "    print(\"  (ingest raw)             (cleanse+enrich)           (aggregate)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "section3_heading",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 3. Pipeline Execution\n",
    "\n",
    "Run the entire medallion pipeline with a single call.\n",
    "LakeLogic processes contracts in DAG order: bronze → silver → gold.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "pipeline_run",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ── Run all layers ────────────────────────────────────────────────────\n",
    "summary = pipeline.run(\n",
    "    target_layers=\"bronze,silver,gold\",\n",
    "    environment=\"dev\",\n",
    ")\n",
    "\n",
    "# ── Display summary ───────────────────────────────────────────────────\n",
    "print(\"\\n\" + \"=\" * 65)\n",
    "print(\"PIPELINE RUN SUMMARY\")\n",
    "print(\"=\" * 65)\n",
    "print(f\"  Pipeline run : {summary.run_id}\")\n",
    "print(f\"  Environment  : {summary.environment}\")\n",
    "print()\n",
    "print(f\"  {'Contract':<30} {'Layer':<8} {'Status':<15} Rows\")\n",
    "print(f\"  {'-'*30} {'-'*8} {'-'*15} ----\")\n",
    "\n",
    "for r in summary.results:\n",
    "    contract = r.get(\"contract\", \"?\")\n",
    "    layer = r.get(\"layer\", \"?\")\n",
    "    status = r.get(\"status\", \"?\")\n",
    "    rows = r.get(\"rows\", \"-\")\n",
    "    error = r.get(\"error\", \"\")\n",
    "    print(f\"  {contract:<30} {layer:<8} {status:<15} {rows}\")\n",
    "    if error:\n",
    "        print(f\"    ↳ {error}\")\n",
    "\n",
    "print(\"=\" * 65)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "section4_heading",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 4. Per-Layer Controls\n",
    "\n",
    "LakeLogic applies different behaviors per layer, all configured in `_system.yaml`:\n",
    "\n",
    "| Layer | Schema Evolution | Materialization | Purpose |\n",
    "|---|---|---|---|\n",
    "| **Bronze** | `append` — auto-add new columns | `append` — never lose raw data | Fidelity |\n",
    "| **Silver** | `strict` — fail on schema change | `overwrite` — always fresh | Quality |\n",
    "| **Gold** | `strict` — locked down | `overwrite` — rebuild aggregates | Trust |\n",
    "\n",
    "### Schema Evolution Options\n",
    "\n",
    "```yaml\n",
    "server_defaults:\n",
    "  bronze:\n",
    "    schema_evolution: append      # auto-add new columns (Delta mergeSchema)\n",
    "    allow_schema_drift: true      # log drift but don't fail\n",
    "    cast_to_string: false         # set true for 'all strings' bronze pattern\n",
    "  silver:\n",
    "    schema_evolution: strict      # fail on schema mismatch (production safety)\n",
    "  gold:\n",
    "    schema_evolution: strict\n",
    "```\n",
    "\n",
    "Individual contracts can **override** any server default:\n",
    "\n",
    "```yaml\n",
    "# In a specific contract:\n",
    "server:\n",
    "  type: table\n",
    "  path: \"catalog.schema.table\"\n",
    "  schema_evolution: merge    # ← overrides the layer default\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "section5_heading",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 5. Inspect the Output\n",
    "\n",
    "Let's verify what was materialized at each layer:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "inspect_output",
   "metadata": {},
   "outputs": [],
   "source": [
    "output_dir = Path(\"output\")\n",
    "\n",
    "if output_dir.exists():\n",
    "    for layer_dir in sorted(output_dir.iterdir()):\n",
    "        if layer_dir.is_dir():\n",
    "            files = list(layer_dir.rglob(\"*\"))\n",
    "            data_files = [f for f in files if f.is_file() and not f.name.startswith(\".\")]\n",
    "            total_size = sum(f.stat().st_size for f in data_files)\n",
    "            print(f\"📁 {layer_dir.name}\")\n",
    "            print(f\"   Files: {len(data_files)} | Size: {total_size:,} bytes\")\n",
    "            for f in data_files[:3]:\n",
    "                print(f\"   └── {f.name}\")\n",
    "            print()\n",
    "else:\n",
    "    print(\"No output directory — run the pipeline first.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "features_heading",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Pipeline Features Reference\n",
    "\n",
    "### Selective Layer Runs\n",
    "\n",
    "```python\n",
    "# Run only bronze\n",
    "pipeline.run(target_layers=\"bronze\")\n",
    "\n",
    "# Run silver + gold (skip bronze)\n",
    "pipeline.run(target_layers=\"silver,gold\")\n",
    "\n",
    "# Filter to specific entities\n",
    "pipeline.run(target_layers=\"bronze\", entity_filter=\"raw_employees\")\n",
    "```\n",
    "\n",
    "### Reprocessing\n",
    "\n",
    "Fix historical data without rebuilding the full pipeline:\n",
    "\n",
    "```python\n",
    "pipeline.run(\n",
    "    target_layers=\"silver\",\n",
    "    reprocess_from=\"2024-01-01\",\n",
    "    reprocess_to=\"2024-03-31\",\n",
    "    reprocess_column=\"hire_date\",\n",
    ")\n",
    "```\n",
    "\n",
    "### Reset & Reload\n",
    "\n",
    "```python\n",
    "# DROP and recreate tables (dev only)\n",
    "pipeline.run(reset_layers=\"silver,gold\")\n",
    "\n",
    "# TRUNCATE tables but keep structure\n",
    "pipeline.run(reload_layers=\"silver\")\n",
    "```\n",
    "\n",
    "### Dry Run\n",
    "\n",
    "```python\n",
    "# Validate contracts without writing anything\n",
    "pipeline.run(dry_run=True)\n",
    "```\n",
    "\n",
    "### GDPR / Right to Delete\n",
    "\n",
    "```python\n",
    "pipeline.run(\n",
    "    forget_column=\"email\",\n",
    "    forget_values=\"frank@company.com\",\n",
    "    forget_strategy=\"soft_delete\",  # or 'hard_delete' or 'hash'\n",
    ")\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Summary\n",
    "\n",
    "| Concept | What You Saw |\n",
    "|---|---|\n",
    "| **`_system.yaml`** | Single file defines contracts, dependencies, per-layer defaults |\n",
    "| **DAG** | Visual dependency graph — LakeLogic resolves execution order |\n",
    "| **`pipeline.run()`** | One call processes all layers in order |\n",
    "| **Server defaults** | Schema evolution, drift controls per layer |\n",
    "| **Reprocessing** | Fix historical data surgically |\n",
    "\n",
    "### Next Steps\n",
    "\n",
    "- **Scale to Databricks** — Deploy with `databricks.yml` bundles\n",
    "- **Add SCD2** — Track slowly changing dimensions\n",
    "- **External logic** — Custom Python processors for gold aggregations\n",
    "- **Parallel execution** — `pipeline.run(parallel=True, max_workers=4)`\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv_lakelogic",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
