{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9d3b9d91eab9490685026ec5a91ccf4b",
   "metadata": {},
   "source": [
    "# Excel Data Ingestion with LakeLogic \n",
    "\n",
    "[![\"Open In Colab\"](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lakelogic/LakeLogic/blob/main/examples/03_data_sources/files/excel/excel_example.ipynb)\n",
    "[![GitHub Repo](https://img.shields.io/badge/GitHub-Repo-blue?logo=github)](https://github.com/lakelogic/LakeLogic/blob/main/examples/03_data_sources/files/excel/excel_example.ipynb)\n",
    "\n",
    "## Business Scenario\n",
    "Your finance team sends you monthly employee data reports in Excel format. These spreadsheets contain:\n",
    "- Employee contact information\n",
    "- Salary data\n",
    "- Department assignments\n",
    "- Employment status\n",
    "\n",
    "**The Problem**: Manual Excel uploads often contain data quality issues that break downstream reports.\n",
    "\n",
    "**The Solution**: Use LakeLogic to automatically validate Excel files before they reach your data warehouse.\n",
    "\n",
    "## Value Proposition\n",
    "- **Zero Manual Parsing**: Direct `.xlsx` and `.xls` file reading\n",
    "- **Instant Validation**: Catch errors before they impact analytics\n",
    "- **User-Friendly Error Reports**: Send clear feedback to data providers\n",
    "- **Automated Pipeline**: Transform manual uploads into governed data flows"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd31171f2db74614b1ce26187c13b5f8",
   "metadata": {},
   "source": [
    "## 1. Setup\n",
    "Install LakeLogic with Excel support and set up your environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b40ef6200e2a4ad4ad361883b2a768ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "from lakelogic import DataProcessor\n",
    "\n",
    "BASE = Path.cwd()\n",
    "contract_path = BASE / \"excel_contract.yaml\"\n",
    "if not contract_path.exists():\n",
    "    candidate = BASE / \"examples\" / \"03_data_sources\" / \"files\" / \"excel\" / \"excel_contract.yaml\"\n",
    "    if candidate.exists():\n",
    "        BASE = candidate.parent\n",
    "        contract_path = candidate\n",
    "\n",
    "excel_path = BASE / \"data\" / \"employees.xlsx\"\n",
    "csv_fallback = BASE / \"data\" / \"employees.csv\"\n",
    "result = None\n",
    "\n",
    "\n",
    "def preview_frame(df, title=None, limit=5):\n",
    "    if title:\n",
    "        print(title)\n",
    "    if df is None:\n",
    "        print(\"<empty>\")\n",
    "        return\n",
    "    if hasattr(df, \"head\"):\n",
    "        try:\n",
    "            print(df.head(limit))\n",
    "            return\n",
    "        except Exception:\n",
    "            pass\n",
    "    if hasattr(df, \"limit\"):\n",
    "        try:\n",
    "            sample = df.limit(limit)\n",
    "            print(sample)\n",
    "            return\n",
    "        except Exception:\n",
    "            pass\n",
    "    print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4b57b854e5f425f91439be93c9b18f0",
   "metadata": {},
   "source": [
    "## 2. Preview Excel File Metadata\n",
    "Let's check what sheets and data the Excel file contains."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd6de359bf5041959cabe1b0f32ecf65",
   "metadata": {},
   "outputs": [],
   "source": [
    "if not excel_path.exists():\n",
    "    print(f\"Missing Excel file: {excel_path.resolve()}\")\n",
    "    if csv_fallback.exists():\n",
    "        print(f\"Sample CSV is available at: {csv_fallback.resolve()}\")\n",
    "        print(\"Run data/create_excel.py to generate the Excel file, then re-run this cell.\")\n",
    "    else:\n",
    "        print(\"Sample data file not found. Run data/create_excel.py to generate it.\")\n",
    "else:\n",
    "    processor = DataProcessor(contract=contract_path)\n",
    "    try:\n",
    "        result = processor.run_source(excel_path)\n",
    "    except Exception as exc:\n",
    "        print(f\"Excel ingestion failed: {exc}\")\n",
    "        result = None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8abe913321ac4f15a8560f1412eccc81",
   "metadata": {},
   "source": [
    "## 3. Run LakeLogic Validation\n",
    "Now let's use LakeLogic to validate the Excel data against our contract."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb61939e748c42b7920d67c4c0cfc521",
   "metadata": {},
   "outputs": [],
   "source": [
    "if result is None:\n",
    "    print(\"No results to summarize.\")\n",
    "else:\n",
    "    print(result)\n",
    "    print(f\"Raw: {len(result.raw)} | Good: {len(result.good)} | Bad: {len(result.bad)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cd68a64e41f4529a3d26422a3218e79",
   "metadata": {},
   "source": [
    "## 4. Access Results Using Named Attributes\n",
    "LakeLogic returns a `ValidationResult` object with clear, named attributes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbed0577eeb8400ca657200293127904",
   "metadata": {},
   "outputs": [],
   "source": [
    "if result is None:\n",
    "    print(\"No validated data to display.\")\n",
    "else:\n",
    "    preview_frame(result.good, \"VALIDATED EMPLOYEES (sample):\")\n",
    "    preview_frame(result.bad, \"QUARANTINED EMPLOYEES (sample):\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be25dfee33554c6190c3bcde41cdbd95",
   "metadata": {},
   "source": [
    "## 5. Inspect Validated Data\n",
    "Let's see the clean data that passed all validation rules."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "347a9160376a4f708e0e0fe0a8a242d1",
   "metadata": {},
   "source": [
    "## 6. Review Data Quality Issues\n",
    "For quarantined records, let's see exactly what went wrong."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88865189cf454f8ea08be60791492d48",
   "metadata": {},
   "source": [
    "## 7. Generate User-Friendly Error Report\n",
    "Create a simple report to send back to your data providers."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ec0693a2395467ab7399f7957a6e3d0",
   "metadata": {},
   "source": [
    "## 8. Calculate Data Quality Metrics\n",
    "Track your data quality over time with key metrics."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "977fa60478444d1bb8cb03675a914f91",
   "metadata": {},
   "source": [
    "## 9. Multi-Sheet Excel Files (Advanced)\n",
    "Working with Excel files that have multiple sheets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96b9e8f809b547f4af8a3556edf25dcb",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "In this tutorial, we demonstrated:\n",
    "\n",
    "1.  **Native Excel Reading**: Direct `.xlsx` file ingestion without manual conversion\n",
    "2.  **Named Attributes**: Clean API with `result.good`, `result.bad`, `result.raw`\n",
    "3.  **Validation Rules**: Email formats, salary ranges, and status checks\n",
    "4.  **Error Reporting**: Generate user-friendly reports for data providers\n",
    "5.  **Quality Metrics**: Track data quality with actionable scorecards\n",
    "\n",
    "### Production Use Cases\n",
    "\n",
    "**Finance Team Uploads**\n",
    "```python\n",
    "result = processor.run_source(\"monthly_budget.xlsx\")\n",
    "if len(result.bad) > 0:\n",
    "    send_email_to_finance(result.bad)\n",
    "else:\n",
    "    warehouse.write_table(\"finance.budget\", result.good)\n",
    "```\n",
    "\n",
    "**HR Data Integration**\n",
    "```python\n",
    "result = processor.run_source(\"employee_updates.xlsx\")\n",
    "metrics = {\"pass_rate\": len(result.good) / len(result.raw) * 100}\n",
    "dashboard.update_quality_metrics(metrics)\n",
    "```\n",
    "\n",
    "### Next Steps\n",
    "\n",
    "- **Customize Validation**: Edit `excel_contract.yaml` to match your business rules\n",
    "- **Automate**: Build a scheduled job to process Excel uploads from a shared folder\n",
    "- **Materialize**: Use LakeLogic's `materialize` option to save validated data\n",
    "- **Monitor**: Track data quality trends over time"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}