{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "855597ee0f384ba09ded2e9a3f6fe822",
   "metadata": {},
   "source": [
    "# XML Data Ingestion with LakeLogic \n",
    "\n",
    "[![\"Open In Colab\"](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lakelogic/LakeLogic/blob/main/examples/03_data_sources/files/xml/xml_example.ipynb)\n",
    "[![GitHub Repo](https://img.shields.io/badge/GitHub-Repo-blue?logo=github)](https://github.com/lakelogic/LakeLogic/blob/main/examples/03_data_sources/files/xml/xml_example.ipynb)\n",
    "\n",
    "## Business Scenario\n",
    "Your organization receives employee data exports from legacy HR systems in XML format. These files contain employee records with critical information like emails, salaries, and employment status.\n",
    "\n",
    "Common data quality issues:\n",
    "- Invalid email formats (missing @ symbol)\n",
    "- Negative or zero salaries (data entry errors)\n",
    "- Inactive employees included in active reports\n",
    "\n",
    "## Value Proposition\n",
    "- **Native XML Support**: No manual parsing required - point LakeLogic directly at `.xml` files\n",
    "- **Engine Agnostic**: Works with Polars, Pandas, DuckDB, or Spark\n",
    "- **Automatic Validation**: Catch data quality issues before they reach your warehouse\n",
    "- **Clear Error Tracking**: Every quarantined record includes specific failure reasons"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79ed2a5b7a0144279ae3d35151a818cc",
   "metadata": {},
   "source": [
    "## 1. Setup\n",
    "First, we'll install LakeLogic (if needed) and import the necessary libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c94aae70741041998764898ccfc8a01b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "from lakelogic import DataProcessor\n",
    "\n",
    "BASE = Path.cwd()\n",
    "contract_path = BASE / \"xml_contract.yaml\"\n",
    "if not contract_path.exists():\n",
    "    candidate = BASE / \"examples\" / \"03_data_sources\" / \"files\" / \"xml\" / \"xml_contract.yaml\"\n",
    "    if candidate.exists():\n",
    "        BASE = candidate.parent\n",
    "        contract_path = candidate\n",
    "\n",
    "xml_path = BASE / \"data\" / \"employees.xml\"\n",
    "result = None\n",
    "\n",
    "\n",
    "def preview_frame(df, title=None, limit=5):\n",
    "    if title:\n",
    "        print(title)\n",
    "    if df is None:\n",
    "        print(\"<empty>\")\n",
    "        return\n",
    "    if hasattr(df, \"head\"):\n",
    "        try:\n",
    "            print(df.head(limit))\n",
    "            return\n",
    "        except Exception:\n",
    "            pass\n",
    "    if hasattr(df, \"limit\"):\n",
    "        try:\n",
    "            sample = df.limit(limit)\n",
    "            print(sample)\n",
    "            return\n",
    "        except Exception:\n",
    "            pass\n",
    "    print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ac340389c2f46d29f0635e3be0d842a",
   "metadata": {},
   "source": [
    "## 2. Preview the XML Data\n",
    "Let's take a quick look at the raw XML file to see what we're working with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b735c6a6ca564e828c4170cc86f4dfc3",
   "metadata": {},
   "outputs": [],
   "source": [
    "if not xml_path.exists():\n",
    "    raise FileNotFoundError(f\"Missing XML file: {xml_path.resolve()}\")\n",
    "\n",
    "with open(xml_path, \"r\", encoding=\"utf-8\") as handle:\n",
    "    content = handle.read(800)\n",
    "    print(content)\n",
    "    print(\"...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42ab1c659cbd41ef978f6ba126834263",
   "metadata": {},
   "source": [
    "## 3. Run LakeLogic with XML Source\n",
    "Now we'll use the Data Contract to ingest, validate, and transform the XML data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8567ec6758c74c5b83c40b49c95f6ea6",
   "metadata": {},
   "outputs": [],
   "source": [
    "processor = DataProcessor(contract=contract_path)\n",
    "try:\n",
    "    result = processor.run_source(xml_path)\n",
    "except Exception as exc:\n",
    "    print(f\"XML ingestion failed: {exc}\")\n",
    "    result = None\n",
    "\n",
    "if result is not None:\n",
    "    print(result)\n",
    "    print(f\"Raw: {len(result.raw)} | Good: {len(result.good)} | Bad: {len(result.bad)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "517ea4f5b82a4f90859101262beeee55",
   "metadata": {},
   "source": [
    "## 4. Inspect Raw XML Data\n",
    "Let's see what the data looks like after XML parsing (before validation)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9158e214e1ea4eff9ed135be2384b4f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "if result is None:\n",
    "    print(\"No results to display.\")\n",
    "else:\n",
    "    preview_frame(result.good, \"VALIDATED DATA (sample):\")\n",
    "    preview_frame(result.bad, \"QUARANTINED DATA (sample):\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afc765f4c681440e994ff748ec43a6d7",
   "metadata": {},
   "source": [
    "## 5. Inspect Validated Records (Silver Layer)\n",
    "These records passed all validation rules and transformations."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef523114bdfe4c1fa31fd4afc438fbcc",
   "metadata": {},
   "source": [
    "## 6. Inspect Quarantined Records\n",
    "These records failed validation - let's see why."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0e727de466e46c38e1fcd0d5c3a60b5",
   "metadata": {},
   "source": [
    "## 7. Data Quality Metrics\n",
    "Let's calculate some key metrics about our data quality."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c221af7b952478c929f24d2bea794d1",
   "metadata": {},
   "source": [
    "## 8. Try Different Engines\n",
    "LakeLogic supports multiple engines. Let's try switching engines to see the flexibility."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87d8d838e3e24906a004e5d6b1e948fd",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "In this tutorial, we demonstrated:\n",
    "\n",
    "1.  **Native XML Ingestion**: No manual parsing - LakeLogic automatically detected and read the `.xml` file\n",
    "2.  **Automatic Validation**: Email formats, salary ranges, and employment status were validated\n",
    "3.  **Clear Error Tracking**: Each quarantined record includes specific reasons for failure\n",
    "4.  **Data Enrichment**: Added `seniority_level` based on hire date\n",
    "5.  **Engine Flexibility**: Same contract works with Polars, Pandas, DuckDB, or Spark\n",
    "\n",
    "### Next Steps\n",
    "\n",
    "- **Modify the contract**: Try adding your own validation rules to `xml_contract.yaml`\n",
    "- **Change the data**: Edit `employees.xml` to test different validation scenarios\n",
    "- **Materialize results**: Use LakeLogic's materialization feature to persist validated data\n",
    "- **Production deployment**: Integrate this into your data pipeline for automated XML ingestion"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}