{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell_header",
   "metadata": {},
   "source": [
    "# Hello World: From Raw CSV to Governed Data\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lakelogic/LakeLogic/blob/main/examples/01_quickstart/01_hello_world.ipynb) \n",
    "[![GitHub Repo](https://img.shields.io/badge/GitHub-Repo-blue?logo=github)](https://github.com/lakelogic/LakeLogic/blob/main/examples/01_quickstart/01_hello_world.ipynb)\n",
    "\n",
    "## Business Scenario\n",
    "\n",
    "You have a raw CSV of employee data and need to:\n",
    "\n",
    "1. **Enforce a schema** — validate column types and required fields\n",
    "2. **Apply quality rules** — catch invalid emails, out-of-range salaries, unknown departments\n",
    "3. **Add derived columns** — enrich with computed fields like tenure category\n",
    "4. **Materialize output** — write validated data to Parquet\n",
    "5. **Inspect the run report** — see exactly what LakeLogic tracked\n",
    "\n",
    "All of this with **one contract** and **three lines of code**.\n",
    "\n",
    "---\n",
    "\n",
    "## What You'll Learn\n",
    "\n",
    "| Section | Concept |\n",
    "|---|---|\n",
    "| 1 | In-memory contract — schema + quality + transformations |\n",
    "| 2 | YAML file contract — production-ready, Git-versioned |\n",
    "| 3 | Materialization — write validated data to Parquet |\n",
    "| 4 | Run Report — what LakeLogic tracked automatically |\n",
    "| 5 | From notebook to production — next steps |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_setup_heading",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_setup",
   "metadata": {},
   "outputs": [],
   "source": [
    "import subprocess, sys, os, urllib.request\n",
    "from pathlib import Path\n",
    "\n",
    "# Install/upgrade lakelogic with polars engine (v1.5+ required)\n",
    "subprocess.run(\n",
    "    [sys.executable, \"-m\", \"pip\", \"install\", \"--upgrade\", \"-q\",\n",
    "     \"lakelogic[polars]>=1.5.0\"],\n",
    "    check=True,\n",
    ")\n",
    "\n",
    "import lakelogic\n",
    "print(f\"lakelogic v{lakelogic.__version__} ready.\")\n",
    "\n",
    "# On Colab: clone repo for contract files and sample data\n",
    "if \"google.colab\" in sys.modules:\n",
    "    repo = Path(\"/content/LakeLogic\")\n",
    "    if repo.exists():\n",
    "        import shutil\n",
    "        shutil.rmtree(repo)\n",
    "    subprocess.run(\n",
    "        [\"git\", \"clone\", \"--quiet\",\n",
    "         \"https://github.com/lakelogic/LakeLogic.git\", str(repo)],\n",
    "        check=True,\n",
    "    )\n",
    "    os.chdir(repo / \"examples\" / \"01_quickstart\")\n",
    "    print(f\"Working directory: {Path.cwd()}\")\n",
    "\n",
    "from lakelogic import DataProcessor\n",
    "\n",
    "REMOTE_URL = (\n",
    "    \"https://raw.githubusercontent.com/lakelogic/LakeLogic/main/examples/01_quickstart/files/excel/data/employees.csv\"\n",
    ")\n",
    "print(f\"Source URL: {REMOTE_URL}\")\n",
    "print(\"Setup complete.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_engine_heading",
   "metadata": {},
   "source": [
    "## Engine Selection\n",
    "\n",
    "LakeLogic supports multiple engines from the same contract. Choose your engine below.\n",
    "\n",
    "> **Spark note:** Spark cannot read files directly from HTTPS URLs (the Hadoop `HttpsFileSystem`\n",
    "> driver doesn't implement `listStatus`). When `engine = 'spark'`, the cell below automatically\n",
    "> downloads the file to `/tmp/` first and passes the local path to LakeLogic.\n",
    "> All other engines (Polars, DuckDB, Pandas) read the remote URL directly.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_engine_select",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ── Choose your engine ──────────────────────────────────────────────────\n",
    "# Options: 'polars' | 'duckdb' | 'pandas' | 'spark'\n",
    "ENGINE = \"polars\"  # change this to try a different engine\n",
    "\n",
    "# ── Resolve source path ──────────────────────────────────────────────────\n",
    "# Spark can't read from HTTPS URLs directly — download locally first.\n",
    "if ENGINE == \"spark\":\n",
    "    LOCAL_FILE = \"/tmp/lakelogic_employees.csv\"\n",
    "    if not os.path.exists(LOCAL_FILE):\n",
    "        print(f\"Downloading {REMOTE_URL} for Spark...\")\n",
    "        urllib.request.urlretrieve(REMOTE_URL, LOCAL_FILE)\n",
    "    SOURCE = LOCAL_FILE\n",
    "    print(f\"Spark engine: using local file → {SOURCE}\")\n",
    "else:\n",
    "    SOURCE = REMOTE_URL\n",
    "    print(f\"Engine: {ENGINE} | Source: {SOURCE}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_section1_heading",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 1. In-Memory Contract: Schema + Quality + Transformations\n",
    "\n",
    "Define everything in a Python dict — great for prototyping. This contract:\n",
    "\n",
    "- **Schema:** Enforces column types and required fields\n",
    "- **Quality rules:** Validates email format, salary range, department values, and active status\n",
    "- **Transformations:** Derives `department_upper` and `tenure_category` columns\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_dict_contract",
   "metadata": {},
   "outputs": [],
   "source": [
    "contract_dict = {\n",
    "    \"version\": \"1.0.0\",\n",
    "    \"dataset\": \"employees\",\n",
    "    \"source\": {\"type\": \"landing\"},\n",
    "    \n",
    "    # ── Schema: define expected columns and types ──────────────────────\n",
    "    \"model\": {\n",
    "        \"fields\": [\n",
    "            {\"name\": \"id\",         \"type\": \"integer\", \"required\": True},\n",
    "            {\"name\": \"name\",       \"type\": \"string\",  \"required\": True},\n",
    "            {\"name\": \"email\",      \"type\": \"string\",  \"required\": True},\n",
    "            {\"name\": \"department\", \"type\": \"string\"},\n",
    "            {\"name\": \"salary\",     \"type\": \"integer\"},\n",
    "            {\"name\": \"hire_date\",  \"type\": \"string\"},\n",
    "            {\"name\": \"status\",     \"type\": \"string\"},\n",
    "        ]\n",
    "    },\n",
    "\n",
    "    # ── Transformations: enrich the data ──────────────────────────────\n",
    "    \"transformations\": [\n",
    "        {\n",
    "            \"phase\": \"pre\",\n",
    "            \"derive\": {\n",
    "                \"field\": \"department_upper\",\n",
    "                \"sql\": \"UPPER(department)\",\n",
    "            },\n",
    "        },\n",
    "        {\n",
    "            \"phase\": \"pre\",\n",
    "            \"derive\": {\n",
    "                \"field\": \"tenure_category\",\n",
    "                \"sql\": \"CASE WHEN hire_date >= '2024-01-01' THEN 'New Hire' \"\n",
    "                       \"WHEN hire_date >= '2023-01-01' THEN 'Mid-Tenure' \"\n",
    "                       \"ELSE 'Veteran' END\",\n",
    "            },\n",
    "        },\n",
    "    ],\n",
    "\n",
    "    # ── Quality rules: validate every row ─────────────────────────────\n",
    "    \"quality\": {\n",
    "        \"row_rules\": [\n",
    "            {\"name\": \"valid_email\",      \"sql\": \"email LIKE '%@%'\",        \"category\": \"validity\"},\n",
    "            {\"name\": \"has_name\",         \"sql\": \"name IS NOT NULL AND name != ''\", \"category\": \"completeness\"},\n",
    "            {\"name\": \"valid_salary\",     \"sql\": \"salary > 0 AND salary < 500000\",  \"category\": \"validity\"},\n",
    "            {\"name\": \"valid_department\", \"sql\": \"department IN ('Engineering', 'Sales', 'HR', 'Marketing', 'Finance')\", \"category\": \"validity\"},\n",
    "            {\"name\": \"is_active\",        \"sql\": \"status = 'active'\",       \"category\": \"validity\"},\n",
    "        ]\n",
    "    },\n",
    "}\n",
    "\n",
    "print(\"Contract defined with:\")\n",
    "print(f\"  Schema fields:    {len(contract_dict['model']['fields'])}\")\n",
    "print(f\"  Transformations:  {len(contract_dict['transformations'])}\")\n",
    "print(f\"  Quality rules:    {len(contract_dict['quality']['row_rules'])}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_dict_run",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ── Run the contract ─────────────────────────────────────────────────\n",
    "processor = DataProcessor(contract=contract_dict, engine=ENGINE)\n",
    "result = processor.run_source(SOURCE)\n",
    "\n",
    "print(f\"Engine      : {ENGINE}\")\n",
    "print(f\"Source rows : {result.source_count}\")\n",
    "print(f\"Good rows   : {result.good_count}\")\n",
    "print(f\"Bad rows    : {result.bad_count}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_dict_inspect",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ── Inspect results ──────────────────────────────────────────────────\n",
    "def show_df(label, df):\n",
    "    \"\"\"Display a DataFrame — handles Spark vs Polars/Pandas/DuckDB.\"\"\"\n",
    "    print(f\"\\n{label}\")\n",
    "    print(\"─\" * 60)\n",
    "    if hasattr(df, \"toPandas\"):  # Spark DataFrame\n",
    "        df.show(20, truncate=False)\n",
    "    else:\n",
    "        try:\n",
    "            display(df)  # Jupyter display for Polars/Pandas\n",
    "        except Exception:\n",
    "            print(df)\n",
    "\n",
    "show_df(\"✅ GOOD rows (passed all rules):\", result.good)\n",
    "show_df(\"❌ BAD rows (quarantined):\", result.bad)\n",
    "\n",
    "# Notice the derived columns: department_upper, tenure_category\n",
    "print(\"\\n📋 Output columns (including derived):\")\n",
    "if hasattr(result.good, 'columns'):\n",
    "    print(f\"  {list(result.good.columns)}\")\n",
    "elif hasattr(result.good, 'schema'):\n",
    "    print(f\"  {result.good.schema.names}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_section2_heading",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 2. YAML File Contract — Production-Ready\n",
    "\n",
    "In production, contracts live in `.yaml` files tracked in Git. The code stays the same;\n",
    "the contract controls everything.\n",
    "\n",
    "Here's what our production contract looks like:\n",
    "\n",
    "```yaml\n",
    "version: 1.0.0\n",
    "dataset: employees\n",
    "\n",
    "info:\n",
    "  title: Employee Directory (Quickstart)\n",
    "  version: 1.0.0\n",
    "  description: >\n",
    "    Validates and cleanses employee CSV data.\n",
    "\n",
    "model:\n",
    "  fields:\n",
    "    - name: id\n",
    "      type: integer\n",
    "      required: true\n",
    "    - name: email\n",
    "      type: string\n",
    "      required: true\n",
    "    # ... (see employees_contract.yaml)\n",
    "\n",
    "transformations:\n",
    "  - phase: pre\n",
    "    derive:\n",
    "      field: tenure_category\n",
    "      sql: \"CASE WHEN hire_date >= '2024-01-01' THEN 'New Hire' ... END\"\n",
    "\n",
    "quality:\n",
    "  row_rules:\n",
    "    - name: valid_email\n",
    "      sql: \"email LIKE '%@%'\"\n",
    "\n",
    "materialization:\n",
    "  target_path: ./output/employees_validated\n",
    "  format: parquet\n",
    "  strategy: overwrite\n",
    "```\n",
    "\n",
    "> **Key difference:** The YAML contract includes `materialization` — it will write the validated\n",
    "> data to Parquet automatically. The code is identical.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_yaml_run",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ── Same code, YAML contract ─────────────────────────────────────────\n",
    "processor_yaml = DataProcessor(contract=\"employees_contract.yaml\", engine=ENGINE)\n",
    "result_yaml = processor_yaml.run_source(SOURCE)\n",
    "\n",
    "print(f\"YAML contract results:\")\n",
    "print(f\"  Good: {result_yaml.good_count}  |  Bad: {result_yaml.bad_count}\")\n",
    "\n",
    "show_df(\"✅ GOOD rows:\", result_yaml.good)\n",
    "show_df(\"❌ BAD rows:\", result_yaml.bad)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_section3_heading",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 3. Materialize — Write Validated Data to Parquet\n",
    "\n",
    "The YAML contract includes `materialization.target_path`, so LakeLogic can write\n",
    "the validated `good` rows directly to Parquet. No extra code needed.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_materialize",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ── Materialize the validated data ────────────────────────────────────\n",
    "mat_result = processor_yaml.materialize(result_yaml.good, result_yaml.bad)\n",
    "\n",
    "print(f\"Materialization result: {mat_result}\")\n",
    "\n",
    "# ── Verify the output ────────────────────────────────────────────────\n",
    "output_path = Path(\"./output/employees_validated\")\n",
    "if output_path.exists():\n",
    "    output_files = list(output_path.rglob(\"*\"))\n",
    "    print(f\"\\n📁 Output directory: {output_path}\")\n",
    "    print(f\"   Files written: {len([f for f in output_files if f.is_file()])}\")\n",
    "    for f in output_files:\n",
    "        if f.is_file():\n",
    "            print(f\"   └── {f.name} ({f.stat().st_size:,} bytes)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_section4_heading",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 4. Run Report — What LakeLogic Tracked\n",
    "\n",
    "Every run produces a detailed report with row counts, timing, rule results,\n",
    "and lineage metadata — automatically.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell_run_report",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ── Inspect the run report ────────────────────────────────────────────\n",
    "report = processor_yaml.last_report\n",
    "\n",
    "if report:\n",
    "    print(\"📊 Run Report\")\n",
    "    print(\"═\" * 60)\n",
    "    print(f\"  Contract      : {report.get('contract', '?')}\")\n",
    "    print(f\"  Run ID        : {report.get('run_id', '?')}\")\n",
    "    print(f\"  Timestamp     : {report.get('timestamp', '?')}\")\n",
    "    print(f\"  Engine        : {report.get('engine', '?')}\")\n",
    "    print(f\"  Dataset       : {report.get('dataset', '?')}\")\n",
    "    print(f\"  Domain        : {report.get('domain', '?')}\")\n",
    "    print(f\"  Data Layer    : {report.get('data_layer', '?')}\")\n",
    "    print()\n",
    "    \n",
    "    # Row counts\n",
    "    counts = report.get('counts', {})\n",
    "    print(\"  📈 Row Counts:\")\n",
    "    print(f\"    Source       : {counts.get('source', '?')}\")\n",
    "    print(f\"    Total (good) : {counts.get('total', '?')}\")\n",
    "    print(f\"    Quarantined  : {counts.get('quarantined', '?')}\")\n",
    "    print()\n",
    "    \n",
    "    # Quality rule failures\n",
    "    failures = report.get('row_rule_failures', [])\n",
    "    if failures:\n",
    "        print(\"  ❌ Quality Rule Failures:\")\n",
    "        for r in failures:\n",
    "            if isinstance(r, dict):\n",
    "                print(f\"    • {r.get('rule', r.get('name', '?'))}: {r.get('failed_count', r.get('count', '?'))} rows\")\n",
    "            else:\n",
    "                print(f\"    • {r}\")\n",
    "    else:\n",
    "        print(\"  ✅ No quality rule failures in report!\")\n",
    "    print()\n",
    "    \n",
    "    # Schema drift\n",
    "    drift = report.get('schema_drift', {})\n",
    "    if drift:\n",
    "        print(f\"  🔄 Schema Drift: {drift}\")\n",
    "    \n",
    "    print(\"═\" * 60)\n",
    "    \n",
    "    # Full report for exploration\n",
    "    print(\"\\n🔍 Full report keys:\", list(report.keys()))\n",
    "else:\n",
    "    print(\"No report available — run the processor first.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell_section5_heading",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 5. Summary\n",
    "\n",
    "### What LakeLogic Did Automatically\n",
    "\n",
    "| Step | What happened |\n",
    "|---|---|\n",
    "| **Schema** | Validated column types and required fields |\n",
    "| **Transformations** | Derived `department_upper` and `tenure_category` |\n",
    "| **Quality** | Applied 5 rules, split data into good / bad |\n",
    "| **Materialization** | Wrote validated rows to Parquet |\n",
    "| **Audit** | Added `_lakelogic_processed_at` and `_lakelogic_run_id` columns |\n",
    "| **Run Report** | Captured row counts, timing, rule results, drift |\n",
    "\n",
    "### Dict vs YAML — When to Use Each\n",
    "\n",
    "| Format | Best for | Key difference |\n",
    "|---|---|---|\n",
    "| Python `dict` | Prototyping, dynamic generation, notebooks | Quick iteration, inline with code |\n",
    "| YAML file | Production, Git versioning, team governance | Declarative, reviewable, version-controlled |\n",
    "\n",
    "Both produce identical results — switch between them with zero code change.\n",
    "\n",
    "---\n",
    "\n",
    "### From Notebook to Production\n",
    "\n",
    "This quickstart showed a single contract processing one file. In production, LakeLogic scales to:\n",
    "\n",
    "| Capability | How |\n",
    "|---|---|\n",
    "| **Multi-layer pipelines** | Chain bronze → silver → gold with `_system.yaml` |\n",
    "| **DAG visualization** | See contract dependencies as an interactive graph |\n",
    "| **Incremental processing** | Only process new data with `load_mode: incremental` |\n",
    "| **Schema evolution** | Control per-layer with `server_defaults` |\n",
    "| **Reprocessing** | Fix historical data without rebuilding the full pipeline |\n",
    "| **GDPR compliance** | Soft-delete or hash PII with `forget_*` parameters |\n",
    "\n",
    "### Next Quickstarts\n",
    "\n",
    "- **[`02_pipeline_quickstart`](../02_pipeline_quickstart/02_pipeline_quickstart.ipynb)** — Build a bronze → silver → gold pipeline with DAG visualization\n",
    "- **[`02_database_governance.ipynb`](02_database_governance.ipynb)** — Quality-gate pattern applied to a SQLite database\n",
    "- **[`SCD2 dimension`](../02_core_patterns/scd2_dimension/playbook.ipynb)** — SCD2 history tracking\n",
    "- **[`Soft delete`](../02_core_patterns/soft_delete/soft_delete_pattern.ipynb)** — Flag deletes instead of removing rows\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv_lakelogic",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
