{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fb6d2d77",
   "metadata": {},
   "source": [
    "# DuckGuard 3.0 - Complete Guide\n",
    "\n",
    "A comprehensive walkthrough of **every** feature in DuckGuard 3.0 -- the Python-native data quality toolkit built on DuckDB.\n",
    "\n",
    "Topics covered:\n",
    "\n",
    "| # | Section | # | Section |\n",
    "|---|---------|---|---------|\n",
    "| 1 | Introduction & Setup | 12 | Multi-Column Checks |\n",
    "| 2 | Connecting to Data | 13 | Query-Based Checks |\n",
    "| 3 | Exploring Columns | 14 | Distributional Checks |\n",
    "| 4 | Column Validation | 15 | Anomaly Detection |\n",
    "| 5 | Row-Level Error Debugging | 16 | Auto-Profiling & Semantic Types |\n",
    "| 6 | Quality Scoring | 17 | YAML Rules & Data Contracts |\n",
    "| 7 | Cross-Dataset Validation | 18 | Reports & Notifications |\n",
    "| 8 | Reconciliation | 19 | Freshness, Schema & History |\n",
    "| 9 | Distribution Drift Detection | 20 | Integrations |\n",
    "| 10 | Group-By Validation | 21 | Enhanced Error Messages |\n",
    "| 11 | Conditional Checks | 22 | Quick Reference |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f43410c",
   "metadata": {},
   "source": [
    "## 1. Introduction & Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35f01d11",
   "metadata": {},
   "outputs": [],
   "source": "!pip install duckguard"
  },
  {
   "cell_type": "markdown",
   "id": "43eacf94",
   "metadata": {},
   "source": [
    "### Create sample data\n",
    "\n",
    "We write an in-memory CSV string to a file so every cell is self-contained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09a1016a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os, csv, io\n",
    "\n",
    "ORDERS_CSV = \"\"\"order_id,customer_id,product_name,quantity,unit_price,subtotal,tax,shipping,total_amount,status,country,email,phone,created_at,ship_date\n",
    "ORD-001,C100,Widget A,2,25.00,50.00,4.50,5.00,59.50,shipped,US,alice@example.com,+12125551001,2024-01-15,2024-01-17\n",
    "ORD-002,C101,Widget B,1,49.99,49.99,4.50,5.00,59.49,shipped,US,bob@example.com,+12125551002,2024-01-16,2024-01-18\n",
    "ORD-003,C102,Gadget X,3,15.00,45.00,4.05,5.00,54.05,delivered,UK,carol@example.co.uk,+442071234567,2024-01-16,2024-01-19\n",
    "ORD-004,C103,Widget A,1,25.00,25.00,2.25,5.00,32.25,pending,US,dave@example.com,+12125551004,2024-01-17,\n",
    "ORD-005,C100,Gadget Y,500,10.00,5000.00,450.00,5.00,5455.00,shipped,CA,alice@example.com,+12125551001,2024-01-17,2024-01-19\n",
    "ORD-006,C104,Widget C,2,35.00,70.00,6.30,5.00,81.30,shipped,US,eve@example.com,,2024-01-18,2024-01-20\n",
    "ORD-007,C105,Gadget X,1,15.00,15.00,1.35,5.00,21.35,cancelled,DE,frank@example.de,+4930123456,2024-01-18,\n",
    "ORD-008,C106,Widget B,4,49.99,199.96,18.00,5.00,222.96,delivered,US,grace@example.com,+12125551008,2024-01-19,2024-01-22\n",
    "ORD-009,C107,Premium Z,1,999.99,999.99,90.00,0.00,1089.99,shipped,JP,hiro@example.jp,+81312345678,2024-01-19,2024-01-23\n",
    "ORD-010,C108,Widget A,10,25.00,250.00,22.50,5.00,277.50,pending,US,ivan@example.com,+12125551010,2024-01-20,\n",
    "ORD-011,C109,Gadget Y,2,10.00,20.00,1.80,5.00,26.80,shipped,UK,jane@example.co.uk,+442079876543,2024-01-20,2024-01-22\n",
    "ORD-012,C110,Widget C,1,35.00,35.00,3.15,5.00,43.15,delivered,US,karl@example.com,+12125551012,2024-01-21,2024-01-23\n",
    "ORD-013,C111,Premium Z,2,999.99,1999.98,180.00,0.00,2179.98,pending,CA,liam@example.ca,+14165551013,2024-01-21,\n",
    "ORD-014,C112,Widget A,3,25.00,75.00,6.75,5.00,86.75,shipped,US,,+12125551014,2024-01-22,2024-01-24\n",
    "ORD-015,C113,Gadget X,1,15.00,15.00,1.35,5.00,21.35,shipped,DE,nina@example.de,+4930234567,2024-01-22,2024-01-25\n",
    "ORD-016,C114,Widget B,2,49.99,99.98,9.00,5.00,113.98,delivered,US,oscar@example.com,+12125551016,2024-01-23,2024-01-25\n",
    "ORD-017,C115,Gadget Y,5,10.00,50.00,4.50,5.00,59.50,shipped,UK,pat@example.co.uk,+442071239999,2024-01-23,2024-01-26\n",
    "ORD-018,C116,Widget C,1,35.00,35.00,3.15,5.00,43.15,pending,US,quinn@example.com,+12125551018,2024-01-24,\n",
    "ORD-019,C117,Premium Z,1,999.99,999.99,90.00,0.00,1089.99,cancelled,JP,rina@example.jp,,2024-01-24,\n",
    "ORD-020,C118,Widget A,2,25.00,50.00,4.50,5.00,59.50,shipped,CA,sam@example.ca,+14165551020,2024-01-25,2024-01-27\n",
    "ORD-021,C119,Gadget X,4,15.00,60.00,5.40,5.00,70.40,delivered,US,tom@example.com,+12125551021,2024-01-25,2024-01-28\n",
    "ORD-022,,Widget B,1,49.99,49.99,4.50,5.00,59.49,pending,US,,+12125551022,2024-01-26,\n",
    "ORD-023,C121,Gadget Y,-2,10.00,-20.00,-1.80,5.00,-16.80,returned,DE,uma@example.de,+4930345678,2024-01-26,\n",
    "ORD-024,C122,Widget A,1,25.00,25.00,2.25,5.00,32.25,shipped,US,vera@example.com,+12125551024,2024-01-27,2024-01-29\n",
    "ORD-025,C123,Premium Z,1,999.99,999.99,90.00,0.00,1089.99,shipped,UK,will@example.co.uk,+442071231111,2024-01-27,2024-01-30\n",
    "ORD-026,C124,Widget C,3,35.00,105.00,9.45,5.00,119.45,delivered,CA,xena@example.ca,+14165551026,2024-01-28,2024-01-30\n",
    "ORD-027,C125,Gadget X,2,15.00,30.00,2.70,5.00,37.70,shipped,US,yuri@example.com,+12125551027,2024-01-28,2024-01-31\n",
    "ORD-028,C126,Widget B,1,49.99,49.99,4.50,5.00,59.49,pending,JP,zoe@example.jp,+81312349999,2024-01-29,\n",
    "ORD-029,C127,Gadget Y,0,10.00,0.00,0.00,5.00,5.00,cancelled,US,adam@example.com,+12125551029,2024-01-29,\n",
    "ORD-030,C128,Widget A,1,25.00,25.00,2.25,5.00,32.25,shipped,DE,beth@example.de,+4930456789,2024-01-30,2024-02-01\n",
    "\"\"\"\n",
    "\n",
    "# Write to file\n",
    "os.makedirs(\"sample_data\", exist_ok=True)\n",
    "with open(\"sample_data/orders.csv\", \"w\", newline=\"\") as f:\n",
    "    f.write(ORDERS_CSV.strip())\n",
    "\n",
    "print(\"[OK] sample_data/orders.csv written (\" + str(len(ORDERS_CSV.strip().splitlines())-1) + \" data rows)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c9759fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Also create a duckguard.yaml rules file for later use\n",
    "RULES_YAML = \"\"\"\n",
    "name: orders_validation\n",
    "description: Quality checks for the orders dataset\n",
    "\n",
    "columns:\n",
    "  order_id:\n",
    "    checks:\n",
    "      - type: not_null\n",
    "      - type: unique\n",
    "\n",
    "  customer_id:\n",
    "    checks:\n",
    "      - type: not_null\n",
    "\n",
    "  quantity:\n",
    "    checks:\n",
    "      - type: between\n",
    "        value: [1, 1000]\n",
    "\n",
    "  status:\n",
    "    checks:\n",
    "      - type: allowed_values\n",
    "        value: [pending, shipped, delivered, cancelled, returned]\n",
    "\"\"\"\n",
    "\n",
    "with open(\"sample_data/duckguard.yaml\", \"w\") as f:\n",
    "    f.write(RULES_YAML.strip())\n",
    "\n",
    "print(\"[OK] sample_data/duckguard.yaml written\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ba66319",
   "metadata": {},
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89d17b4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from duckguard import (\n",
    "    connect,\n",
    "    AutoProfiler,\n",
    "    SemanticAnalyzer,\n",
    "    detect_type,\n",
    "    detect_types_for_dataset,\n",
    "    load_rules_from_string,\n",
    "    execute_rules,\n",
    "    generate_rules,\n",
    "    RuleSet,\n",
    "    generate_contract,\n",
    "    validate_contract,\n",
    "    diff_contracts,\n",
    "    detect_anomalies,\n",
    "    AnomalyDetector,\n",
    "    ColumnNotFoundError,\n",
    "    ValidationError,\n",
    "    UnsupportedConnectorError,\n",
    ")\n",
    "from duckguard.contracts import contract_to_yaml\n",
    "from duckguard.anomaly import BaselineMethod, KSTestMethod, SeasonalMethod\n",
    "from duckguard.freshness import FreshnessMonitor\n",
    "from duckguard.schema_history import SchemaTracker, SchemaChangeAnalyzer\n",
    "from duckguard.history import HistoryStorage, TrendAnalyzer\n",
    "from duckguard.notifications import (\n",
    "    EmailNotifier,\n",
    "    SlackNotifier,\n",
    "    TeamsNotifier,\n",
    "    format_results_text,\n",
    "    format_results_markdown,\n",
    ")\n",
    "from duckguard.reports import generate_html_report\n",
    "\n",
    "import tempfile, os, shutil\n",
    "\n",
    "print(\"[OK] All imports successful - DuckGuard\", end=\" \")\n",
    "from duckguard import __version__\n",
    "print(__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91d7f218",
   "metadata": {},
   "source": [
    "## 2. Connecting to Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7bc6fd70",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Connect to the CSV file\n",
    "orders = connect(\"sample_data/orders.csv\")\n",
    "\n",
    "# Dataset metadata\n",
    "print(\"Row count  :\", orders.row_count)\n",
    "print(\"Columns    :\", orders.columns)\n",
    "print(\"Col count  :\", orders.column_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5dbf4c8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preview the first 5 rows\n",
    "for row in orders.head(5):\n",
    "    print(row)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73f00039",
   "metadata": {},
   "outputs": [],
   "source": [
    "# --- Alternative connection methods (commented) --------------------------\n",
    "#\n",
    "# Parquet\n",
    "# orders = connect(\"data/orders.parquet\")\n",
    "#\n",
    "# JSON / NDJSON\n",
    "# orders = connect(\"data/orders.json\")\n",
    "#\n",
    "# Amazon S3\n",
    "# orders = connect(\"s3://my-bucket/orders.parquet\")\n",
    "#\n",
    "# PostgreSQL\n",
    "# orders = connect(\"postgresql://user:pass@localhost:5432/mydb\", table=\"orders\")\n",
    "#\n",
    "# Snowflake\n",
    "# orders = connect(\"snowflake://user:pass@account/db/schema\", table=\"orders\")\n",
    "#\n",
    "# pandas DataFrame\n",
    "# import pandas as pd\n",
    "# df = pd.read_csv(\"orders.csv\")\n",
    "# orders = connect(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74835801",
   "metadata": {},
   "source": [
    "## 3. Exploring Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac0ee6dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Access columns via dot notation or bracket notation\n",
    "col_status = orders.status          # dot notation\n",
    "col_email  = orders[\"email\"]        # bracket notation\n",
    "\n",
    "# String column statistics\n",
    "print(\"=== status column ===\")\n",
    "print(\"Null count   :\", col_status.null_count)\n",
    "print(\"Unique count :\", col_status.unique_count)\n",
    "\n",
    "# Numeric column statistics\n",
    "qty = orders.quantity\n",
    "print(\"\\n=== quantity column ===\")\n",
    "print(\"Min    :\", qty.min)\n",
    "print(\"Max    :\", qty.max)\n",
    "print(\"Mean   :\", qty.mean)\n",
    "print(\"Median :\", qty.median)\n",
    "print(\"Stddev :\", qty.stddev)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5d87d73",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Value counts - top values by frequency\n",
    "print(\"Status value counts:\")\n",
    "for val, cnt in orders.status.get_value_counts().items():\n",
    "    print(f\"  {val}: {cnt}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f617d1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Distinct values\n",
    "print(\"Distinct countries:\", orders.country.get_distinct_values())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32f8ecff",
   "metadata": {},
   "source": [
    "## 4. Column Validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f461e7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# is_not_null - check that a column has no nulls\n",
    "result = orders.order_id.is_not_null()\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", \"order_id is_not_null:\", result.message)\n",
    "\n",
    "# is_unique - check that all values are unique\n",
    "result = orders.order_id.is_unique()\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", \"order_id is_unique:\", result.message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f2ead64",
   "metadata": {},
   "outputs": [],
   "source": [
    "# between - range check\n",
    "result = orders.unit_price.between(1, 2000)\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", \"unit_price between(1,2000):\", result.message)\n",
    "\n",
    "# greater_than / less_than\n",
    "result = orders.unit_price.greater_than(0)\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", \"unit_price > 0:\", result.message)\n",
    "\n",
    "result = orders.unit_price.less_than(5000)\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", \"unit_price < 5000:\", result.message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed487466",
   "metadata": {},
   "outputs": [],
   "source": [
    "# matches - regex pattern check\n",
    "result = orders.order_id.matches(r\"^ORD-\\d{3}$\")\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", \"order_id pattern:\", result.message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02ed981b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# isin - enum / allowed values check\n",
    "allowed = [\"pending\", \"shipped\", \"delivered\", \"cancelled\", \"returned\"]\n",
    "result = orders.status.isin(allowed)\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", \"status isin:\", result.message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2e89b8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# has_no_duplicates\n",
    "result = orders.order_id.has_no_duplicates()\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", \"order_id no duplicates:\", result.message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3339ce30",
   "metadata": {},
   "outputs": [],
   "source": [
    "# value_lengths_between - string length range\n",
    "result = orders.order_id.value_lengths_between(5, 10)\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", \"order_id length [5,10]:\", result.message)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "974cbc09",
   "metadata": {},
   "source": [
    "## 5. Row-Level Error Debugging\n",
    "\n",
    "When a validation fails, DuckGuard captures the offending rows so you can debug immediately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2dc0c60",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Intentionally trigger a failure: quantity should be between 1 and 100\n",
    "result = orders.quantity.between(1, 100)\n",
    "print(\"Passed:\", result.passed)\n",
    "print()\n",
    "print(result.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f59d4e84",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Iterate over individual failed rows\n",
    "for row in result.failed_rows:\n",
    "    print(f\"  Row {row.row_number}: value={row.value}, expected={row.expected}, reason={row.reason}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c02ca35b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convenience helpers\n",
    "print(\"Failed values       :\", result.get_failed_values())\n",
    "print(\"Failed row indices  :\", result.get_failed_row_indices())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64557983",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Disable failure capture for performance on large datasets\n",
    "result_fast = orders.quantity.between(1, 100, capture_failures=False)\n",
    "print(\"Passed:\", result_fast.passed, \"| failed_rows captured:\", len(result_fast.failed_rows))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfbe8478",
   "metadata": {},
   "source": [
    "## 6. Quality Scoring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22b53a64",
   "metadata": {},
   "outputs": [],
   "source": [
    "score = orders.score()\n",
    "print(\"Overall score :\", score.overall)\n",
    "print(\"Grade         :\", score.grade)\n",
    "print(\"Completeness  :\", score.completeness)\n",
    "print(\"Uniqueness    :\", score.uniqueness)\n",
    "print(\"Validity      :\", score.validity)\n",
    "print(\"Consistency   :\", score.consistency)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fa12977",
   "metadata": {},
   "source": [
    "## 7. Cross-Dataset Validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "34e759d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create temporary CSV files for cross-dataset checks\n",
    "tmpdir = tempfile.mkdtemp(prefix=\"dg_cross_\")\n",
    "\n",
    "customers_csv = \"\"\"customer_id,name,email\n",
    "C100,Alice,alice@example.com\n",
    "C101,Bob,bob@example.com\n",
    "C102,Carol,carol@example.co.uk\n",
    "C103,Dave,dave@example.com\n",
    "C104,Eve,eve@example.com\n",
    "C105,Frank,frank@example.de\n",
    "C106,Grace,grace@example.com\n",
    "C107,Hiro,hiro@example.jp\n",
    "C108,Ivan,ivan@example.com\n",
    "C109,Jane,jane@example.co.uk\n",
    "C110,Karl,karl@example.com\n",
    "\"\"\"\n",
    "\n",
    "orders_orphans_csv = \"\"\"order_id,customer_id\n",
    "ORD-100,C100\n",
    "ORD-101,C999\n",
    "ORD-102,C888\n",
    "ORD-103,C101\n",
    "ORD-104,\n",
    "\"\"\"\n",
    "\n",
    "cust_path = os.path.join(tmpdir, \"customers.csv\")\n",
    "orp_path  = os.path.join(tmpdir, \"orders_orphans.csv\")\n",
    "\n",
    "with open(cust_path, \"w\", newline=\"\") as f:\n",
    "    f.write(customers_csv.strip())\n",
    "with open(orp_path, \"w\", newline=\"\") as f:\n",
    "    f.write(orders_orphans_csv.strip())\n",
    "\n",
    "customers = connect(cust_path)\n",
    "orders_orp = connect(orp_path)\n",
    "\n",
    "print(\"[OK] Temporary datasets created\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1492918b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# exists_in -- foreign-key check\n",
    "result = orders_orp.customer_id.exists_in(customers.customer_id)\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", result.message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db1fca01",
   "metadata": {},
   "outputs": [],
   "source": [
    "# references() with allow_nulls\n",
    "result_allow = orders_orp.customer_id.references(customers.customer_id, allow_nulls=True)\n",
    "print(\"allow_nulls=True  =>\", \"[PASS]\" if result_allow.passed else \"[FAIL]\", result_allow.message)\n",
    "\n",
    "result_strict = orders_orp.customer_id.references(customers.customer_id, allow_nulls=False)\n",
    "print(\"allow_nulls=False =>\", \"[PASS]\" if result_strict.passed else \"[FAIL]\", result_strict.message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52f4f087",
   "metadata": {},
   "outputs": [],
   "source": [
    "# find_orphans - quick list of orphan values\n",
    "orphans = orders_orp.customer_id.find_orphans(customers.customer_id)\n",
    "print(\"Orphan customer IDs:\", orphans)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52cb24a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# matches_values - compare distinct value sets\n",
    "result = orders_orp.customer_id.matches_values(customers.customer_id)\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", result.message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60e06308",
   "metadata": {},
   "outputs": [],
   "source": [
    "# row_count_matches - compare row counts with tolerance\n",
    "result = orders.row_count_matches(customers, tolerance=25)\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", result.message)\n",
    "\n",
    "# Clean up temp files\n",
    "shutil.rmtree(tmpdir, ignore_errors=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa75c5ae",
   "metadata": {},
   "source": [
    "## 8. Reconciliation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc6444ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "tmpdir = tempfile.mkdtemp(prefix=\"dg_recon_\")\n",
    "\n",
    "source_csv = \"\"\"id,name,amount,status\n",
    "1,Alpha,100.00,active\n",
    "2,Beta,200.00,active\n",
    "3,Gamma,300.00,inactive\n",
    "4,Delta,400.00,active\n",
    "5,Epsilon,500.00,active\n",
    "\"\"\"\n",
    "\n",
    "target_csv = \"\"\"id,name,amount,status\n",
    "1,Alpha,100.00,active\n",
    "2,Beta,210.00,active\n",
    "3,Gamma,300.00,active\n",
    "5,Epsilon,500.00,active\n",
    "6,Zeta,600.00,active\n",
    "\"\"\"\n",
    "\n",
    "src_path = os.path.join(tmpdir, \"source.csv\")\n",
    "tgt_path = os.path.join(tmpdir, \"target.csv\")\n",
    "\n",
    "with open(src_path, \"w\", newline=\"\") as f:\n",
    "    f.write(source_csv.strip())\n",
    "with open(tgt_path, \"w\", newline=\"\") as f:\n",
    "    f.write(target_csv.strip())\n",
    "\n",
    "source_ds = connect(src_path)\n",
    "target_ds = connect(tgt_path)\n",
    "\n",
    "recon = source_ds.reconcile(\n",
    "    target_ds,\n",
    "    key_columns=[\"id\"],\n",
    "    compare_columns=[\"name\", \"amount\", \"status\"],\n",
    ")\n",
    "\n",
    "print(\"Match %          :\", recon.match_percentage)\n",
    "print(\"Missing in target:\", recon.missing_in_target)\n",
    "print(\"Extra in target  :\", recon.extra_in_target)\n",
    "print(\"Value mismatches :\", recon.value_mismatches)\n",
    "print()\n",
    "print(recon.summary())\n",
    "\n",
    "shutil.rmtree(tmpdir, ignore_errors=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9997ad6",
   "metadata": {},
   "source": [
    "## 9. Distribution Drift Detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad40114f",
   "metadata": {},
   "outputs": [],
   "source": [
    "tmpdir = tempfile.mkdtemp(prefix=\"dg_drift_\")\n",
    "\n",
    "# Baseline: normal-ish amounts\n",
    "baseline_csv = \"amount\\n\" + \"\\n\".join(str(v) for v in [\n",
    "    100, 105, 98, 110, 95, 102, 108, 97, 103, 99,\n",
    "    101, 106, 94, 112, 96, 104, 107, 93, 111, 100,\n",
    "])\n",
    "\n",
    "# Drifted: shifted higher\n",
    "drifted_csv = \"amount\\n\" + \"\\n\".join(str(v) for v in [\n",
    "    200, 210, 195, 220, 205, 215, 198, 225, 202, 208,\n",
    "    190, 230, 197, 212, 207, 218, 193, 222, 201, 209,\n",
    "])\n",
    "\n",
    "bl_path = os.path.join(tmpdir, \"baseline.csv\")\n",
    "dr_path = os.path.join(tmpdir, \"drifted.csv\")\n",
    "\n",
    "with open(bl_path, \"w\", newline=\"\") as f:\n",
    "    f.write(baseline_csv)\n",
    "with open(dr_path, \"w\", newline=\"\") as f:\n",
    "    f.write(drifted_csv)\n",
    "\n",
    "baseline_ds = connect(bl_path)\n",
    "drifted_ds  = connect(dr_path)\n",
    "\n",
    "drift = drifted_ds.amount.detect_drift(baseline_ds.amount)\n",
    "\n",
    "print(\"P-value   :\", drift.p_value)\n",
    "print(\"Statistic :\", drift.statistic)\n",
    "print(\"Is drifted:\", drift.is_drifted)\n",
    "print(\"Message   :\", drift.message)\n",
    "\n",
    "shutil.rmtree(tmpdir, ignore_errors=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3120c8d8",
   "metadata": {},
   "source": [
    "## 10. Group-By Validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e86fcf95",
   "metadata": {},
   "outputs": [],
   "source": [
    "tmpdir = tempfile.mkdtemp(prefix=\"dg_grp_\")\n",
    "\n",
    "grp_csv = \"\"\"region,sales\n",
    "North,100\n",
    "North,110\n",
    "North,95\n",
    "South,200\n",
    "South,210\n",
    "East,50\n",
    "East,55\n",
    "East,60\n",
    "East,45\n",
    "West,300\n",
    "\"\"\"\n",
    "\n",
    "grp_path = os.path.join(tmpdir, \"grouped.csv\")\n",
    "with open(grp_path, \"w\", newline=\"\") as f:\n",
    "    f.write(grp_csv.strip())\n",
    "\n",
    "grp_ds = connect(grp_path)\n",
    "grouped = grp_ds.group_by(\"region\")\n",
    "\n",
    "# Group metadata\n",
    "print(\"Groups      :\", grouped.groups)\n",
    "print(\"Group count :\", grouped.group_count)\n",
    "\n",
    "# Stats per group\n",
    "print(\"\\nStats:\")\n",
    "for s in grouped.stats():\n",
    "    print(f\"  {s}\")\n",
    "\n",
    "# Validate each group has at least 2 rows\n",
    "result = grouped.row_count_greater_than(2)\n",
    "print(\"\\nAll groups > 2 rows?\", result.passed)\n",
    "for g in result.get_failed_groups():\n",
    "    print(f\"  [FAIL] {g.key_string}: {g.row_count} rows\")\n",
    "\n",
    "shutil.rmtree(tmpdir, ignore_errors=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1aae7555",
   "metadata": {},
   "source": [
    "## 11. Conditional Checks\n",
    "\n",
    "Validate columns only when a condition is true."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8bfe58d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# not_null_when: email must not be null when status = shipped\n",
    "result = orders.email.not_null_when(\"status = 'shipped'\")\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", \"email not_null_when shipped:\", result.message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a47265e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# unique_when: order_id must be unique among shipped orders\n",
    "result = orders.order_id.unique_when(\"status = 'shipped'\")\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", \"order_id unique_when shipped:\", result.message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13963bbf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# between_when: quantity between 1-100 when country = US\n",
    "result = orders.quantity.between_when(1, 100, \"country = 'US'\")\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", \"quantity between_when US:\", result.message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b5bcadf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# isin_when: status must be shipped/delivered when country = UK\n",
    "result = orders.status.isin_when(['shipped', 'delivered'], \"country = 'UK'\")\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", \"status isin_when UK:\", result.message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2e7984a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# matches_when: email must match pattern when status = delivered\n",
    "result = orders.email.matches_when(r'^[\\\\w\\\\.\\\\-]+@[\\\\w\\\\.\\\\-]+\\\\.[a-zA-Z]{2,}$', \"status = 'delivered'\")\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", \"email matches_when delivered:\", result.message)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a552c91",
   "metadata": {},
   "source": [
    "## 12. Multi-Column Checks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c884202",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Column pair satisfy: ship_date >= created_at\n",
    "# (only rows where both are non-null are checked)\n",
    "result = orders.expect_column_pair_satisfy(\n",
    "    column_a=\"ship_date\",\n",
    "    column_b=\"created_at\",\n",
    "    expression=\"ship_date >= created_at\",\n",
    ")\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", \"ship_date >= created_at:\", result.message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f501b821",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Composite key uniqueness: (order_id, customer_id)\n",
    "result = orders.expect_columns_unique(columns=[\"order_id\", \"customer_id\"])\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", \"composite key unique:\", result.message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2c00711",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Multi-column sum: subtotal + tax + shipping = total_amount\n",
    "# Note: this checks the row-level sum per row against a fixed expected value.\n",
    "# We pick a known row value to demonstrate -- ORD-001: 50 + 4.50 + 5 = 59.50\n",
    "# For a general \"all rows\" check, use expect_query_to_return_no_rows instead.\n",
    "result = orders.expect_multicolumn_sum_to_equal(\n",
    "    columns=[\"subtotal\", \"tax\", \"shipping\"],\n",
    "    expected_sum=59.50,\n",
    "    threshold=0.01,\n",
    ")\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", \"multicolumn sum:\", result.message)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3d6e019",
   "metadata": {},
   "source": [
    "## 13. Query-Based Checks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da73de3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# expect_query_to_return_no_rows: find violations\n",
    "# Find rows where quantity is negative (should have none for valid orders)\n",
    "result = orders.expect_query_to_return_no_rows(\n",
    "    \"SELECT * FROM table WHERE quantity < 0\"\n",
    ")\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", \"no negative qty:\", result.message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d57b8554",
   "metadata": {},
   "outputs": [],
   "source": [
    "# expect_query_to_return_rows: ensure data exists\n",
    "result = orders.expect_query_to_return_rows(\n",
    "    \"SELECT * FROM table WHERE status = 'shipped'\"\n",
    ")\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", \"shipped rows exist:\", result.message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b50abd19",
   "metadata": {},
   "outputs": [],
   "source": [
    "# expect_query_result_to_equal: exact value check\n",
    "result = orders.expect_query_result_to_equal(\n",
    "    \"SELECT COUNT(*) FROM table\",\n",
    "    expected=30,\n",
    ")\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", \"row count = 30:\", result.message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd575570",
   "metadata": {},
   "outputs": [],
   "source": [
    "# expect_query_result_to_be_between: range check on aggregate\n",
    "result = orders.expect_query_result_to_be_between(\n",
    "    \"SELECT AVG(unit_price) FROM table\",\n",
    "    min_value=10.0,\n",
    "    max_value=500.0,\n",
    ")\n",
    "print(\"[PASS]\" if result.passed else \"[FAIL]\", \"avg unit_price in [10,500]:\", result.message)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc2efd64",
   "metadata": {},
   "source": [
    "## 14. Distributional Checks\n",
    "\n",
    "These checks require **scipy**. They are wrapped in try/except so the notebook\n",
    "runs even if scipy is not installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "beba4787",
   "metadata": {},
   "outputs": [],
   "source": [
    "# expect_distribution_normal\n",
    "try:\n",
    "    result = orders.unit_price.expect_distribution_normal()\n",
    "    print(\"[PASS]\" if result.passed else \"[FAIL]\", \"normal distribution:\", result.message)\n",
    "except ImportError:\n",
    "    print(\"[SKIP] scipy not installed -- pip install scipy\")\n",
    "except Exception as e:\n",
    "    print(\"[INFO]\", type(e).__name__, str(e)[:120])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4afcadaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# expect_ks_test\n",
    "try:\n",
    "    result = orders.quantity.expect_ks_test(distribution=\"norm\")\n",
    "    print(\"[PASS]\" if result.passed else \"[FAIL]\", \"KS test (norm):\", result.message)\n",
    "except ImportError:\n",
    "    print(\"[SKIP] scipy not installed\")\n",
    "except Exception as e:\n",
    "    print(\"[INFO]\", type(e).__name__, str(e)[:120])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c3356f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# expect_chi_square_test\n",
    "try:\n",
    "    result = orders.status.expect_chi_square_test()\n",
    "    print(\"[PASS]\" if result.passed else \"[FAIL]\", \"chi-square test:\", result.message)\n",
    "except ImportError:\n",
    "    print(\"[SKIP] scipy not installed\")\n",
    "except Exception as e:\n",
    "    print(\"[INFO]\", type(e).__name__, str(e)[:120])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69b924b9",
   "metadata": {},
   "source": [
    "## 15. Anomaly Detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ff0bf2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# detect_anomalies (high-level, z-score)\n",
    "report = detect_anomalies(orders, method=\"zscore\", columns=[\"quantity\", \"unit_price\", \"total_amount\"])\n",
    "print(\"Has anomalies:\", report.has_anomalies)\n",
    "print(\"Anomaly count:\", report.anomaly_count)\n",
    "for a in report.anomalies:\n",
    "    tag = \"[!]\" if a.is_anomaly else \"[ ]\"\n",
    "    print(f\"  {tag} {a.column}: score={a.score:.2f}, threshold={a.threshold}, msg={a.message}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65cc4f1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# AnomalyDetector with IQR method\n",
    "detector = AnomalyDetector(method=\"iqr\", threshold=1.5)\n",
    "iqr_report = detector.detect(orders, columns=[\"quantity\", \"total_amount\"])\n",
    "print(\"IQR anomaly count:\", iqr_report.anomaly_count)\n",
    "for a in iqr_report.get_anomalies():\n",
    "    print(f\"  [!] {a.column}: {a.message}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7318526f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# BaselineMethod: fit + score\n",
    "bl = BaselineMethod(sensitivity=2.0)\n",
    "bl.fit([100, 102, 98, 105, 97, 103, 108, 96, 104, 99])\n",
    "\n",
    "print(\"Baseline mean:\", bl.baseline_mean)\n",
    "print(\"Baseline std :\", bl.baseline_std)\n",
    "\n",
    "sc_normal = bl.score(101)\n",
    "sc_outlier = bl.score(250)\n",
    "print(f\"Score 101 -> anomaly={sc_normal.is_anomaly}, score={sc_normal.score:.2f}\")\n",
    "print(f\"Score 250 -> anomaly={sc_outlier.is_anomaly}, score={sc_outlier.score:.2f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "509d90da",
   "metadata": {},
   "outputs": [],
   "source": [
    "# KSTestMethod: compare_distributions\n",
    "ks = KSTestMethod(p_value_threshold=0.05)\n",
    "ks.fit([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])\n",
    "\n",
    "comp = ks.compare_distributions([5, 6, 7, 8, 9, 10, 11, 12, 13, 14])\n",
    "print(\"Is drift :\", comp.is_drift)\n",
    "print(\"P-value  :\", comp.p_value)\n",
    "print(\"Message  :\", comp.message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f4081dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# SeasonalMethod: basic demo (without timestamps falls back to global stats)\n",
    "sm = SeasonalMethod(period=\"daily\", sensitivity=2.0)\n",
    "sm.fit([10, 12, 11, 13, 9, 14, 10, 11, 12, 10])\n",
    "\n",
    "sc = sm.score(50)  # way outside normal range\n",
    "print(\"Score 50 -> anomaly:\", sc.is_anomaly, \"score:\", f\"{sc.score:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3214f5ba",
   "metadata": {},
   "source": [
    "### Anomaly Methods Reference\n",
    "\n",
    "| Method | Class | Best For |\n",
    "|--------|-------|----------|\n",
    "| `zscore` | `ZScoreMethod` | Normally distributed data |\n",
    "| `iqr` | `IQRMethod` | Robust to outliers |\n",
    "| `percent_change` | `PercentChangeMethod` | Monitoring metrics over time |\n",
    "| `modified_zscore` | `ModifiedZScoreMethod` | Non-normal distributions |\n",
    "| `baseline` | `BaselineMethod` | ML-based baseline comparison |\n",
    "| `ks_test` | `KSTestMethod` | Distribution drift detection |\n",
    "| `seasonal` | `SeasonalMethod` | Seasonal pattern detection |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6169605",
   "metadata": {},
   "source": [
    "## 16. Auto-Profiling & Semantic Types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd2a5194",
   "metadata": {},
   "outputs": [],
   "source": "# AutoProfiler — quality scoring, pattern detection, and rule suggestions\nfrom duckguard.profiler import AutoProfiler\n\nprofiler = AutoProfiler()\nprofile = profiler.profile(orders)\n\n# Summary with quality grade\nprint(f\"Profiled {profile.column_count} columns, {profile.row_count} rows\")\nprint(f\"Overall quality: {profile.overall_quality_grade} ({profile.overall_quality_score:.1f}/100)\")\nprint()\n\n# Per-column quality grades and percentiles\nprint(\"Column quality grades:\")\nfor col in profile.columns:\n    grade_str = f\"  {col.name:20s} grade={col.quality_grade}  score={col.quality_score:.0f}\"\n    if col.median_value is not None:\n        grade_str += f\"  median={col.median_value}  p25={col.p25_value}  p75={col.p75_value}\"\n    print(grade_str)\nprint()\n\n# Suggested rules (25+ pattern types including email, SSN, UUID, etc.)\nprint(f\"Suggested rules ({len(profile.suggested_rules)}):\")\nfor rule in profile.suggested_rules[:10]:\n    print(f\"  {rule}\")\nif len(profile.suggested_rules) > 10:\n    print(f\"  ... and {len(profile.suggested_rules) - 10} more\")\nprint()\n\n# Deep profiling: distribution analysis + outlier detection (requires scipy)\ntry:\n    deep_profiler = AutoProfiler(deep=True)\n    deep_profile = deep_profiler.profile(orders)\n    print(\"Deep profiling results (numeric columns):\")\n    for col in deep_profile.columns:\n        if col.distribution_type:\n            print(f\"  {col.name}: dist={col.distribution_type}, skew={col.skewness:.2f}, \"\n                  f\"kurtosis={col.kurtosis:.2f}, normal={col.is_normal}\")\n        if col.outlier_count is not None:\n            print(f\"    outliers: {col.outlier_count} ({col.outlier_percentage:.1f}%)\")\nexcept ImportError:\n    print(\"[SKIP] Deep profiling requires scipy -- pip install scipy\")\nprint()\n\n# Configurable thresholds\nstrict = AutoProfiler(null_threshold=0.0, unique_threshold=100.0, pattern_min_confidence=95.0)\nstrict_profile = strict.profile(orders)\nprint(f\"Strict profiler: {len(strict_profile.suggested_rules)} rules \"\n      f\"(vs {len(profile.suggested_rules)} with defaults)\")"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6314a1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# detect_type - individual column\n",
    "sem = detect_type(orders, \"email\")\n",
    "print(\"email semantic type:\", sem)\n",
    "\n",
    "sem2 = detect_type(orders, \"country\")\n",
    "print(\"country semantic type:\", sem2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8df34edf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# detect_types_for_dataset - all columns at once\n",
    "type_map = detect_types_for_dataset(orders)\n",
    "for col, stype in type_map.items():\n",
    "    print(f\"  {col:20s} -> {stype}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d69a668b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# SemanticAnalyzer - full analysis with PII detection\n",
    "analyzer = SemanticAnalyzer()\n",
    "analysis = analyzer.analyze(orders)\n",
    "\n",
    "print(\"PII columns found:\", analysis.pii_columns)\n",
    "for col in analysis.columns[:5]:\n",
    "    print(f\"  {col.name:20s} type={col.semantic_type.value:15s} conf={col.confidence:.2f} pii={col.is_pii}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c2a241d",
   "metadata": {},
   "source": [
    "### Supported Semantic Types (selection)\n",
    "\n",
    "| Category | Types |\n",
    "|----------|-------|\n",
    "| Identity | `primary_key`, `foreign_key`, `uuid`, `id` |\n",
    "| Contact | `email`, `phone`, `url`, `ip_address` |\n",
    "| PII | `ssn`, `credit_card`, `person_name`, `address` |\n",
    "| Location | `country`, `state`, `city`, `zipcode`, `latitude`, `longitude` |\n",
    "| Date/Time | `date`, `datetime`, `timestamp`, `time`, `year`, `month` |\n",
    "| Numeric | `currency`, `percentage`, `quantity`, `age` |\n",
    "| Categorical | `boolean`, `enum`, `status`, `category`, `gender` |\n",
    "| Text | `text`, `description`, `title`, `slug`, `code` |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50b84bb9",
   "metadata": {},
   "source": [
    "## 17. YAML Rules & Data Contracts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22fe124a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load_rules_from_string - define rules inline\n",
    "yaml_str = \"\"\"\n",
    "name: inline_orders\n",
    "description: Inline validation rules\n",
    "\n",
    "columns:\n",
    "  order_id:\n",
    "    checks:\n",
    "      - type: not_null\n",
    "      - type: unique\n",
    "  quantity:\n",
    "    checks:\n",
    "      - type: between\n",
    "        value: [0, 1000]\n",
    "  status:\n",
    "    checks:\n",
    "      - type: allowed_values\n",
    "        value: [pending, shipped, delivered, cancelled, returned]\n",
    "\"\"\"\n",
    "\n",
    "rules = load_rules_from_string(yaml_str)\n",
    "print(\"Loaded rules:\", rules.name)\n",
    "print(\"Columns with rules:\", list(rules.columns.keys()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54630798",
   "metadata": {},
   "outputs": [],
   "source": [
    "# execute_rules\n",
    "exec_result = execute_rules(rules, \"sample_data/orders.csv\")\n",
    "print(\"Passed    :\", exec_result.passed)\n",
    "print(\"Total     :\", exec_result.total_checks)\n",
    "print(\"Passed    :\", exec_result.passed_count)\n",
    "print(\"Failed    :\", exec_result.failed_count)\n",
    "for r in exec_result.results:\n",
    "    tag = \"[PASS]\" if r.passed else \"[FAIL]\"\n",
    "    print(f\"  {tag} {r.message}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f17376f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# generate_rules - auto-generate from data\n",
    "auto_yaml = generate_rules(orders, dataset_name=\"orders\")\n",
    "print(\"Auto-generated YAML rules:\")\n",
    "print(auto_yaml[:600] if isinstance(auto_yaml, str) else str(auto_yaml))\n",
    "print(\"...\" if isinstance(auto_yaml, str) and len(auto_yaml) > 600 else \"\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a6a3da1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# RuleSet programmatic usage\n",
    "from duckguard.rules.schema import RuleSet, ColumnRules, Check, CheckType\n",
    "\n",
    "rs = RuleSet(\n",
    "    name=\"programmatic_rules\",\n",
    "    columns={\n",
    "        \"order_id\": ColumnRules(\n",
    "            name=\"order_id\",\n",
    "            checks=[\n",
    "                Check(type=CheckType.NOT_NULL),\n",
    "                Check(type=CheckType.UNIQUE),\n",
    "            ]\n",
    "        )\n",
    "    }\n",
    ")\n",
    "print(\"RuleSet:\", rs.name, \"| columns:\", list(rs.columns.keys()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5b196a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# generate_contract - create a data contract from a live dataset\n",
    "try:\n",
    "    contract = generate_contract(orders, name=\"orders_contract\", owner=\"data-team\")\n",
    "    print(\"Contract:\", contract.name)\n",
    "    print(\"Version :\", contract.version)\n",
    "    print(\"Fields  :\", len(contract.schema))\n",
    "    for field in contract.schema[:5]:\n",
    "        print(f\"  {field.name}: {field.type}\")\n",
    "except TypeError:\n",
    "    # CSV date columns stored as strings may cause contract generation issues\n",
    "    contract = None\n",
    "    print(\"[NOTE] Contract generation encountered a type error on date columns.\")\n",
    "    print(\"This is expected when CSV date columns are stored as strings.\")\n",
    "    print(\"Works best with typed sources (Parquet, databases).\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4ad1576",
   "metadata": {},
   "outputs": [],
   "source": [
    "# validate_contract\n",
    "if contract is not None:\n",
    "    validation = validate_contract(contract, \"sample_data/orders.csv\")\n",
    "    print(\"Contract valid:\", validation.passed)\n",
    "    if not validation.passed:\n",
    "        for v in validation.violations[:5]:\n",
    "            print(f\"  - {v}\")\n",
    "else:\n",
    "    print(\"[SKIP] Contract not generated (see note above)\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "920b510c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# contract_to_yaml\n",
    "if contract is not None:\n",
    "    yaml_out = contract_to_yaml(contract)\n",
    "    print(yaml_out[:500])\n",
    "    print(\"...\" if len(yaml_out) > 500 else \"\")\n",
    "else:\n",
    "    print(\"[SKIP] Contract not generated (see note above)\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d09de406",
   "metadata": {},
   "outputs": [],
   "source": [
    "# diff_contracts - detect breaking changes between two contract versions\n",
    "try:\n",
    "    contract_v1 = generate_contract(orders, name=\"orders_v1\")\n",
    "    import copy\n",
    "    contract_v2 = copy.deepcopy(contract_v1)\n",
    "    contract_v2.version = \"2.0.0\"\n",
    "    if len(contract_v2.schema) > 2:\n",
    "        removed = contract_v2.schema.pop()\n",
    "        print(\"Simulated removing field:\", removed.name)\n",
    "    diff = diff_contracts(contract_v1, contract_v2)\n",
    "    print(\"Has breaking changes:\", diff.has_breaking_changes)\n",
    "    for c in diff.changes[:5]:\n",
    "        print(f\"  - {c}\")\n",
    "except TypeError:\n",
    "    print(\"[NOTE] Contract diff skipped due to date column type issue.\")\n",
    "    print(\"Works best with typed sources (Parquet, databases).\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbe3f94f",
   "metadata": {},
   "source": [
    "## 18. Reports & Notifications"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae18af0f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate an HTML report from rule execution results\n",
    "tmpdir = tempfile.mkdtemp(prefix=\"dg_report_\")\n",
    "report_path = os.path.join(tmpdir, \"quality_report.html\")\n",
    "\n",
    "try:\n",
    "    result_path = generate_html_report(exec_result, report_path)\n",
    "    print(\"[OK] HTML report generated:\", result_path)\n",
    "    # File size\n",
    "    size = os.path.getsize(str(result_path))\n",
    "    print(f\"     Size: {size:,} bytes\")\n",
    "except Exception as e:\n",
    "    print(\"[INFO] Report generation:\", e)\n",
    "\n",
    "shutil.rmtree(tmpdir, ignore_errors=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96411526",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Notifier configuration examples (NOT sending -- these need real credentials)\n",
    "\n",
    "# Email notifier\n",
    "email_notifier = EmailNotifier(\n",
    "    smtp_host=\"smtp.example.com\",\n",
    "    smtp_port=587,\n",
    "    smtp_user=\"alerts@example.com\",\n",
    "    smtp_password=\"app_password_here\",\n",
    "    to_addresses=[\"team@example.com\"],\n",
    "    from_address=\"duckguard@example.com\",\n",
    ")\n",
    "print(\"[OK] EmailNotifier configured (not sending)\")\n",
    "\n",
    "# Slack notifier\n",
    "slack_notifier = SlackNotifier(webhook_url=\"https://hooks.slack.com/services/XXX/YYY/ZZZ\")\n",
    "print(\"[OK] SlackNotifier configured (not sending)\")\n",
    "\n",
    "# Teams notifier\n",
    "teams_notifier = TeamsNotifier(webhook_url=\"https://outlook.office.com/webhook/XXX\")\n",
    "print(\"[OK] TeamsNotifier configured (not sending)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7e2ebf6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Format results as text / markdown (useful for custom integrations)\n",
    "text_output = format_results_text(exec_result)\n",
    "print(\"=== Text Output (first 300 chars) ===\")\n",
    "print(text_output[:300])\n",
    "\n",
    "md_output = format_results_markdown(exec_result)\n",
    "print(\"\\n=== Markdown Output (first 300 chars) ===\")\n",
    "print(md_output[:300])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1306f42",
   "metadata": {},
   "source": [
    "**Note:** To actually send notifications, replace the dummy credentials above\n",
    "with real SMTP / webhook settings and call `notifier.send_results(exec_result)`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbd65c07",
   "metadata": {},
   "source": [
    "## 19. Freshness & Schema Evolution & History"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98a2cb6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Freshness -- how old is the data?\n",
    "freshness = orders.freshness\n",
    "print(\"Last modified:\", freshness.last_modified)\n",
    "print(\"Age (human) :\", freshness.age_human)\n",
    "print(\"Is fresh    :\", freshness.is_fresh)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "980cfbb5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# is_fresh with custom threshold\n",
    "from datetime import timedelta\n",
    "\n",
    "print(\"Fresh (24h)?  :\", orders.is_fresh(timedelta(hours=24)))\n",
    "print(\"Fresh (1 min)?:\", orders.is_fresh(timedelta(minutes=1)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3fc2fcd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# FreshnessMonitor\n",
    "monitor = FreshnessMonitor(threshold=timedelta(hours=1))\n",
    "result = monitor.check(orders)\n",
    "print(\"Monitor result:\", result.is_fresh, \"|\", result.age_human)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7954d878",
   "metadata": {},
   "outputs": [],
   "source": [
    "# SchemaTracker - capture snapshots\n",
    "from duckguard.schema_history import SchemaTracker, SchemaChangeAnalyzer\n",
    "from duckguard.history import HistoryStorage\n",
    "\n",
    "tracker = SchemaTracker()\n",
    "snapshot = tracker.capture(orders)\n",
    "print(\"Snapshot columns:\", len(snapshot.columns))\n",
    "for cs in snapshot.columns[:5]:\n",
    "    print(f\"  {cs.name}: {cs.dtype}\")\n",
    "\n",
    "# Get history of snapshots (pass source path, not Dataset)\n",
    "history = tracker.get_history(orders.source)\n",
    "print()\n",
    "print(f\"Schema history entries: {len(history)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a826972c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# SchemaChangeAnalyzer\n",
    "change_analyzer = SchemaChangeAnalyzer()\n",
    "report = change_analyzer.detect_changes(orders)\n",
    "print(\"Has breaking changes:\", report.has_breaking_changes)\n",
    "print(\"Changes:\", len(report.changes))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3a85018",
   "metadata": {},
   "outputs": [],
   "source": [
    "# HistoryStorage - store and query validation runs\n",
    "storage = HistoryStorage()\n",
    "storage.store(exec_result)\n",
    "\n",
    "runs = storage.get_runs(\"sample_data/orders.csv\", limit=5)\n",
    "print(\"Stored runs:\", len(runs))\n",
    "for run in runs:\n",
    "    print(f\"  {run.run_id}: passed={run.passed}, checks={run.total_checks}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c94853a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TrendAnalyzer\n",
    "trend_analyzer = TrendAnalyzer(storage)\n",
    "trends = trend_analyzer.analyze(\"sample_data/orders.csv\", days=30)\n",
    "print(trends.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "981f5e1d",
   "metadata": {},
   "source": [
    "## 20. Integrations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afab6ccf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# dbt integration - convert rules to dbt tests\n",
    "try:\n",
    "    from duckguard.integrations.dbt import rules_to_dbt_tests\n",
    "    dbt_tests = rules_to_dbt_tests(rules)\n",
    "    import json as _json\n",
    "    print(_json.dumps(dbt_tests, indent=2)[:500])\n",
    "except ImportError as e:\n",
    "    print(\"[SKIP] dbt integration requires yaml:\", e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "859a67a7",
   "metadata": {},
   "source": [
    "### Airflow DAG Example\n",
    "\n",
    "```python\n",
    "from airflow import DAG\n",
    "from airflow.operators.python import PythonOperator\n",
    "from datetime import datetime\n",
    "\n",
    "def validate_orders():\n",
    "    from duckguard import connect, load_rules, execute_rules\n",
    "    rules = load_rules(\"duckguard.yaml\")\n",
    "    result = execute_rules(rules, \"s3://bucket/orders.parquet\")\n",
    "    if not result.passed:\n",
    "        raise Exception(f\"Quality check failed: {result.failed_count} failures\")\n",
    "\n",
    "dag = DAG('data_quality', schedule_interval='@daily', start_date=datetime(2024, 1, 1))\n",
    "task = PythonOperator(task_id='validate', python_callable=validate_orders, dag=dag)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a57153f6",
   "metadata": {},
   "source": [
    "### GitHub Actions Example\n",
    "\n",
    "```yaml\n",
    "name: Data Quality\n",
    "on: [push]\n",
    "jobs:\n",
    "  quality-check:\n",
    "    runs-on: ubuntu-latest\n",
    "    steps:\n",
    "      - uses: actions/checkout@v4\n",
    "      - uses: actions/setup-python@v5\n",
    "        with:\n",
    "          python-version: '3.11'\n",
    "      - run: pip install duckguard\n",
    "      - run: duckguard check data/orders.csv --rules duckguard.yaml\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8cb04d5",
   "metadata": {},
   "source": [
    "### pytest Example\n",
    "\n",
    "```python\n",
    "# tests/test_data_quality.py\n",
    "from duckguard import connect\n",
    "\n",
    "def test_orders_quality():\n",
    "    orders = connect(\"data/orders.csv\")\n",
    "    assert orders.row_count > 0\n",
    "    assert orders.order_id.is_not_null()\n",
    "    assert orders.order_id.is_unique()\n",
    "    assert orders.quantity.between(0, 10000)\n",
    "    assert orders.status.isin([\"pending\", \"shipped\", \"delivered\", \"cancelled\"])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2e5292a",
   "metadata": {},
   "source": "### CLI Commands Reference\n\n```bash\n# Run checks from YAML rules\nduckguard check data.csv --rules duckguard.yaml\n\n# Auto-discover rules from data\nduckguard discover data.csv --output duckguard.yaml\n\n# Generate a data contract\nduckguard contract generate data.csv --output contract.yaml\n\n# Validate against a contract\nduckguard contract validate data.csv --contract contract.yaml\n\n# Profile dataset with quality scoring\nduckguard profile data.csv\nduckguard profile data.csv --deep --format json\n```"
  },
  {
   "cell_type": "markdown",
   "id": "45a0851d",
   "metadata": {},
   "source": [
    "## 21. Enhanced Error Messages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "467c0cf4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ColumnNotFoundError - includes suggestions\n",
    "try:\n",
    "    _ = orders.nonexistent_column\n",
    "except (AttributeError, ColumnNotFoundError) as e:\n",
    "    print(\"[ColumnNotFoundError]\")\n",
    "    print(str(e)[:200])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15bd0e19",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ValidationError\n",
    "try:\n",
    "    raise ValidationError(\n",
    "        check_name=\"between\",\n",
    "        column=\"quantity\",\n",
    "        actual_value=500,\n",
    "        expected_value=\"[1, 100]\",\n",
    "    )\n",
    "except ValidationError as e:\n",
    "    print(\"[ValidationError]\")\n",
    "    print(str(e)[:200])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6face11",
   "metadata": {},
   "outputs": [],
   "source": [
    "# UnsupportedConnectorError\n",
    "try:\n",
    "    raise UnsupportedConnectorError(source=\"ftp://data.example.com/file.xyz\")\n",
    "except UnsupportedConnectorError as e:\n",
    "    print(\"[UnsupportedConnectorError]\")\n",
    "    print(str(e)[:300])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b342cbff",
   "metadata": {},
   "source": [
    "## 22. Quick Reference"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bccf849",
   "metadata": {},
   "source": [
    "### Validation Methods\n",
    "\n",
    "| Method | Description | Returns |\n",
    "|--------|-------------|---------|\n",
    "| `col.is_not_null()` | Check nulls below threshold | `ValidationResult` |\n",
    "| `col.is_unique()` | Check uniqueness above threshold | `ValidationResult` |\n",
    "| `col.between(min, max)` | Range check (inclusive) | `ValidationResult` |\n",
    "| `col.greater_than(val)` | Minimum check (exclusive) | `ValidationResult` |\n",
    "| `col.less_than(val)` | Maximum check (exclusive) | `ValidationResult` |\n",
    "| `col.matches(regex)` | Regex pattern check | `ValidationResult` |\n",
    "| `col.isin(values)` | Enum / allowed values | `ValidationResult` |\n",
    "| `col.has_no_duplicates()` | Uniqueness check | `ValidationResult` |\n",
    "| `col.value_lengths_between(min, max)` | String length check | `ValidationResult` |\n",
    "| `col.exists_in(ref_col)` | Foreign key check | `ValidationResult` |\n",
    "| `col.references(ref_col)` | FK with null handling | `ValidationResult` |\n",
    "| `col.find_orphans(ref_col)` | List orphan values | `list` |\n",
    "| `col.matches_values(other_col)` | Compare value sets | `ValidationResult` |\n",
    "| `col.detect_drift(ref_col)` | Distribution drift | `DriftResult` |\n",
    "| `col.not_null_when(cond)` | Conditional not-null | `ValidationResult` |\n",
    "| `col.unique_when(cond)` | Conditional uniqueness | `ValidationResult` |\n",
    "| `col.between_when(min, max, cond)` | Conditional range | `ValidationResult` |\n",
    "| `col.isin_when(vals, cond)` | Conditional enum | `ValidationResult` |\n",
    "| `col.matches_when(pat, cond)` | Conditional pattern | `ValidationResult` |\n",
    "| `col.expect_distribution_normal()` | Normality test | `ValidationResult` |\n",
    "| `col.expect_ks_test()` | KS distribution test | `ValidationResult` |\n",
    "| `col.expect_chi_square_test()` | Chi-square test | `ValidationResult` |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a7707a9",
   "metadata": {},
   "source": [
    "### Dataset-Level Methods\n",
    "\n",
    "| Method | Description |\n",
    "|--------|-------------|\n",
    "| `ds.score()` | Quality score (completeness, uniqueness, validity, consistency) |\n",
    "| `ds.reconcile(target, keys, cols)` | Full reconciliation |\n",
    "| `ds.row_count_matches(other, tolerance)` | Row count comparison |\n",
    "| `ds.group_by(cols)` | Group-level validation |\n",
    "| `ds.expect_column_pair_satisfy(a, b, expr)` | Column pair check |\n",
    "| `ds.expect_columns_unique(cols)` | Composite key uniqueness |\n",
    "| `ds.expect_multicolumn_sum_to_equal(cols, sum)` | Multi-column sum |\n",
    "| `ds.expect_query_to_return_no_rows(sql)` | Custom SQL -- no violations |\n",
    "| `ds.expect_query_to_return_rows(sql)` | Custom SQL -- data exists |\n",
    "| `ds.expect_query_result_to_equal(sql, val)` | Custom SQL -- exact value |\n",
    "| `ds.expect_query_result_to_be_between(sql, min, max)` | Custom SQL -- range |\n",
    "| `ds.freshness` / `ds.is_fresh(max_age)` | Data freshness |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2577c0a0",
   "metadata": {},
   "source": [
    "### Next Steps\n",
    "\n",
    "- **Documentation**: [github.com/XDataHubAI/duckguard](https://github.com/XDataHubAI/duckguard)\n",
    "- **PyPI**: `pip install duckguard`\n",
    "- **CLI**: `duckguard --help`\n",
    "- **dbt integration**: `from duckguard.integrations.dbt import rules_to_dbt_tests`\n",
    "- **Notifications**: Slack, Teams, Email -- `from duckguard.notifications import SlackNotifier`\n",
    "\n",
    "Happy data quality checking!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}