{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DuckGuard - Data Quality in 60 Seconds\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/XDataHubAI/duckguard/blob/main/examples/colab_quickstart.ipynb)\n",
    "[![PyPI](https://img.shields.io/pypi/v/duckguard.svg)](https://pypi.org/project/duckguard/)\n",
    "\n",
    "**DuckGuard** is a Python-native data quality tool built on DuckDB. 10x faster than pandas-based tools.\n",
    "\n",
    "Features:\n",
    "- Quality Scoring (A-F grades)\n",
    "- YAML-based Rules\n",
    "- Semantic Type Detection (PII, emails, etc.)\n",
    "- Data Contracts\n",
    "- Anomaly Detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install DuckGuard\n",
    "!pip install duckguard -q\n",
    "print(\"DuckGuard installed!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create sample data\n",
    "import pandas as pd\n",
    "\n",
    "df = pd.DataFrame({\n",
    "    'order_id': ['ORD-001', 'ORD-002', 'ORD-003', 'ORD-004', 'ORD-005'],\n",
    "    'customer_id': ['CUST-001', 'CUST-002', None, 'CUST-004', 'CUST-005'],\n",
    "    'email': ['john@example.com', 'jane@example.com', 'bob@example.com', 'invalid-email', 'alice@example.com'],\n",
    "    'amount': [99.99, 149.50, 75.00, -10.00, 200.00],\n",
    "    'status': ['delivered', 'shipped', 'pending', 'unknown', 'delivered']\n",
    "})\n",
    "\n",
    "df.to_csv('orders.csv', index=False)\n",
    "print(\"Sample data created!\")\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Connect and Explore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from duckguard import connect\n",
    "\n",
    "# Connect to data\n",
    "orders = connect(\"orders.csv\")\n",
    "\n",
    "print(f\"Rows: {orders.row_count}\")\n",
    "print(f\"Columns: {orders.columns}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Quality Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get instant quality score\n",
    "result = orders.score()\n",
    "\n",
    "print(f\"Quality Score: {result.overall:.1f}/100\")\n",
    "print(f\"Grade: {result.grade}\")\n",
    "print(\"\\nDimensions:\")\n",
    "print(f\"  Completeness: {result.completeness:.1f}\")\n",
    "print(f\"  Uniqueness: {result.uniqueness:.1f}\")\n",
    "print(f\"  Validity: {result.validity:.1f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Column Statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check column quality\n",
    "print(f\"customer_id null %: {orders.customer_id.null_percent:.1f}%\")\n",
    "print(f\"order_id unique %: {orders.order_id.unique_percent:.1f}%\")\n",
    "print(f\"amount min: {orders.amount.min}\")\n",
    "print(f\"amount max: {orders.amount.max}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Semantic Type Detection (PII)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from duckguard import detect_types_for_dataset\n",
    "from duckguard.semantic import SemanticAnalyzer\n",
    "\n",
    "# Detect semantic types\n",
    "types = detect_types_for_dataset(orders)\n",
    "for col, sem_type in types.items():\n",
    "    print(f\"{col}: {sem_type.value if sem_type else 'generic'}\")\n",
    "\n",
    "# Check for PII\n",
    "analysis = SemanticAnalyzer().analyze(orders)\n",
    "if analysis.pii_columns:\n",
    "    print(f\"\\n⚠️ PII detected in: {analysis.pii_columns}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. YAML Rules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from duckguard import execute_rules, load_rules_from_string\n",
    "\n",
    "yaml_rules = \"\"\"\n",
    "dataset: orders\n",
    "rules:\n",
    "  - order_id is not null\n",
    "  - order_id is unique\n",
    "  - customer_id null_percent < 50\n",
    "  - amount >= 0\n",
    "  - status in ['pending', 'shipped', 'delivered']\n",
    "\"\"\"\n",
    "\n",
    "rules = load_rules_from_string(yaml_rules)\n",
    "result = execute_rules(rules, dataset=orders)\n",
    "\n",
    "print(f\"Passed: {result.passed_count}/{result.total_checks}\")\n",
    "print(\"\\nResults:\")\n",
    "for r in result.results:\n",
    "    status = \"✓\" if r.passed else \"✗\"\n",
    "    print(f\"  {status} {r.check.expression}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Anomaly Detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from duckguard import detect_anomalies\n",
    "\n",
    "report = detect_anomalies(orders, method=\"zscore\", threshold=2.0)\n",
    "\n",
    "print(f\"Anomalies found: {report.anomaly_count}\")\n",
    "for a in report.anomalies:\n",
    "    if a.is_anomaly:\n",
    "        print(f\"  ⚠️ {a.column}: {a.message}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Data Contracts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from duckguard import generate_contract, validate_contract\n",
    "\n",
    "# Generate contract from data\n",
    "contract = generate_contract(orders, name=\"orders_contract\")\n",
    "\n",
    "print(f\"Contract: {contract.name}\")\n",
    "print(\"Schema:\")\n",
    "for field in contract.schema:\n",
    "    print(f\"  {field.name}: {field.type.value}\")\n",
    "\n",
    "# Validate\n",
    "result = validate_contract(contract, orders)\n",
    "print(f\"\\nValid: {result.is_valid}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Steps\n",
    "\n",
    "- **GitHub**: https://github.com/XDataHubAI/duckguard\n",
    "- **PyPI**: https://pypi.org/project/duckguard/\n",
    "- **Full docs**: See `examples/getting_started.ipynb`\n",
    "\n",
    "```bash\n",
    "# CLI usage\n",
    "duckguard check data.csv\n",
    "duckguard discover data.csv --output rules.yaml\n",
    "duckguard anomaly data.csv\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Real-world validation workflow\n",
    "print(\"=\" * 60)\n",
    "print(\"DUCKGUARD 3.0 - COMPREHENSIVE VALIDATION\")\n",
    "print(\"=\" * 60)\n",
    "\n",
    "checks = {}\n",
    "\n",
    "# 1. Conditional: High-value orders need customer info\n",
    "checks['conditional'] = orders.customer_id.not_null_when(\"amount >= 100\")\n",
    "print(f\"\\n✓ Conditional check: {checks['conditional'].passed}\")\n",
    "\n",
    "# 2. Multi-column: Validate relationships\n",
    "checks['multicolumn'] = orders.expect_column_pair_satisfy(\n",
    "    column_a=\"amount\",\n",
    "    column_b=\"status\",\n",
    "    expression=\"amount >= 0 OR status = 'unknown'\",\n",
    "    threshold=1.0\n",
    ")\n",
    "print(f\"✓ Multi-column check: {checks['multicolumn'].passed}\")\n",
    "\n",
    "# 3. Query-based: Business logic validation\n",
    "checks['query'] = orders.expect_query_to_return_no_rows(\n",
    "    query=\"SELECT * FROM table WHERE status = 'delivered' AND customer_id IS NULL\"\n",
    ")\n",
    "print(f\"✓ Query-based check: {checks['query'].passed}\")\n",
    "\n",
    "# Summary\n",
    "passed = sum(1 for c in checks.values() if c.passed)\n",
    "print(f\"\\n{'='*60}\")\n",
    "print(f\"RESULTS: {passed}/{len(checks)} checks passed\")\n",
    "print(f\"{'='*60}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## 🎯 Putting It All Together: 3.0 Power Move\n\nCombine all 3.0 features for sophisticated validation:"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**Distributional test methods:**\n- `expect_distribution_normal()` - Test for normal distribution\n- `expect_distribution_uniform()` - Test for uniform distribution  \n- `expect_ks_test(distribution)` - Kolmogorov-Smirnov test for any distribution\n- `expect_chi_square_test(expected_frequencies)` - Chi-square goodness-of-fit test\n\nThese are perfect for validating ML features or detecting data drift!"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install scipy for distributional tests\n",
    "!pip install scipy -q\n",
    "\n",
    "# Test if amount follows a specific distribution\n",
    "try:\n",
    "    result = orders.amount.expect_ks_test(\n",
    "        distribution='norm',  # Normal distribution\n",
    "        significance_level=0.05\n",
    "    )\n",
    "\n",
    "    print(f\"Normality test: {result.passed}\")\n",
    "    print(f\"P-value: {result.details.get('pvalue', 0):.4f}\")\n",
    "    print(\"\\nInterpretation:\")\n",
    "    if result.passed:\n",
    "        print(\"  Data appears to follow a normal distribution\")\n",
    "    else:\n",
    "        print(\"  Data does not follow a normal distribution\")\n",
    "\n",
    "except ImportError:\n",
    "    print(\"scipy not available - skipping distributional tests\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## 🆕 New in 3.0: Distributional Testing\n\nTest if your data follows specific statistical distributions (requires scipy):"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**Query-based check methods:**\n- `expect_query_to_return_no_rows(query)` - Query should return empty result\n- `expect_query_to_return_rows(query)` - Query should return at least one row\n- `expect_query_result_to_equal(query, expected)` - Query result equals specific value\n- `expect_query_result_to_be_between(query, min, max)` - Query result in range\n\nAll queries use the keyword `table` to reference your dataset."
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check 1: Verify no invalid orders (query should return 0 rows)\n",
    "result = orders.expect_query_to_return_no_rows(\n",
    "    query=\"SELECT * FROM table WHERE status = 'unknown' AND amount > 0\"\n",
    ")\n",
    "\n",
    "print(f\"No invalid orders: {result.passed}\")\n",
    "print(f\"Message: {result.message}\")\n",
    "\n",
    "# Check 2: Verify we have data (query should return rows)\n",
    "result2 = orders.expect_query_to_return_rows(\n",
    "    query=\"SELECT * FROM table WHERE status = 'delivered'\"\n",
    ")\n",
    "\n",
    "print(f\"\\nHave delivered orders: {result2.passed}\")\n",
    "\n",
    "# Check 3: Verify specific metric equals expected value\n",
    "result3 = orders.expect_query_result_to_equal(\n",
    "    query=\"SELECT COUNT(*) FROM table WHERE amount < 0\",\n",
    "    expected=1  # We have exactly 1 negative amount in sample data\n",
    ")\n",
    "\n",
    "print(f\"\\nExact count check: {result3.passed}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## 🆕 New in 3.0: Query-Based Checks\n\nWrite custom SQL queries to validate complex business logic:"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example: Verify amount is always positive\n",
    "result = orders.expect_column_pair_satisfy(\n",
    "    column_a=\"amount\",\n",
    "    column_b=\"amount\",  # Can reference same column\n",
    "    expression=\"amount >= 0\",\n",
    "    threshold=1.0  # 100% must pass\n",
    ")\n",
    "\n",
    "print(f\"Check: {result.message}\")\n",
    "print(f\"Passed: {result.passed}\")\n",
    "\n",
    "# More complex: Check that negative amounts only occur in specific status\n",
    "result2 = orders.expect_column_pair_satisfy(\n",
    "    column_a=\"amount\",\n",
    "    column_b=\"status\",\n",
    "    expression=\"amount >= 0 OR status = 'unknown'\",\n",
    "    threshold=1.0\n",
    ")\n",
    "\n",
    "print(f\"\\nComplex check: {result2.passed}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## 🆕 New in 3.0: Multi-Column Validation\n\nValidate relationships between multiple columns using SQL expressions:"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "**Other conditional checks available:**\n- `not_null_when(condition)` - Column must not be null when condition is true\n- `unique_when(condition)` - Column must be unique when condition is true\n- `between_when(min, max, condition)` - Column must be in range when condition is true\n- `isin_when(values, condition)` - Column must be in list when condition is true\n- `pattern_when(regex, condition)` - Column must match pattern when condition is true"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Conditional validation: customer_id required when amount > 100\n",
    "result = orders.customer_id.not_null_when(\"amount >= 100\")\n",
    "\n",
    "print(f\"Check passed: {result.passed}\")\n",
    "print(f\"Message: {result.message}\")\n",
    "print(\"\\nDetails:\")\n",
    "print(f\"  Matching rows: {result.details.get('matching_rows', 0)}\")\n",
    "print(f\"  Violations: {result.details.get('violations', 0)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## 🆕 New in 3.0: Conditional Validation\n\nSometimes you need to validate a column only when certain conditions are met. DuckGuard 3.0 introduces **conditional expectations** with the `when` clause:"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}