{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Complete PyCharter Data Journey: Aircraft Configuration\n",
    "\n",
    "This notebook demonstrates the **complete data journey** with PyCharter, from contract specification to quality assurance.\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook walks through all stages of the PyCharter data journey:\n",
    "\n",
    "1. **Contract Creation** - Define a data contract with schema, metadata, and rules\n",
    "2. **Contract Parsing** - Decompose contract into components\n",
    "3. **Metadata Storage** - Store components in PostgreSQL with versioning\n",
    "4. **Contract Reconstruction** - Build complete contract from stored components\n",
    "5. **Pydantic Model Generation** - Generate comprehensive models with all rules\n",
    "6. **Runtime Validation** - Validate data records against the contract\n",
    "7. **Quality Assurance** - Check data quality, calculate metrics, and track violations\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "Before running this notebook, ensure you have:\n",
    "\n",
    "1. **PostgreSQL running** and accessible\n",
    "2. **Database initialized**: Run `pycharter db init postgresql://user:pass@localhost/dbname`\n",
    "3. **PyCharter installed**: `pip install pycharter[postgres]`\n",
    "\n",
    "## Setup\n",
    "\n",
    "We'll use dummy aircraft configuration data to demonstrate all PyCharter features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully saved 20 documents to ../data/examples/aircraft_configuration/aircraft_configuration.json\n"
     ]
    }
   ],
   "source": [
    "# Step 0: Setup and Import Dependencies\n",
    "\n",
    "import json\n",
    "import os\n",
    "from datetime import datetime\n",
    "from typing import Optional, Dict, Any, List\n",
    "import sys\n",
    "\n",
    "from pydantic import BaseModel, Field\n",
    "\n",
    "# Import PyCharter services\n",
    "from pycharter import (\n",
    "    # Contract Parser\n",
    "    parse_contract_file,\n",
    "    ContractMetadata,\n",
    "    # Metadata Store\n",
    "    PostgresMetadataStore,\n",
    "    # JSON Schema Converter\n",
    "    to_dict,\n",
    "    # Runtime Validator\n",
    "    get_model_from_contract,\n",
    "    validate_batch,\n",
    "    validate_with_store,\n",
    "    validate_with_contract,\n",
    "    # Contract Builder\n",
    "    build_contract_from_store,\n",
    "    # Quality Assurance\n",
    "    QualityCheck,\n",
    "    QualityCheckOptions,\n",
    "    QualityThresholds,\n",
    ")\n",
    "from pycharter.db.models.base import get_session\n",
    "\n",
    "# Setup paths\n",
    "ROOT_DIR = os.path.join('..')\n",
    "DATA_DIR = os.path.join(ROOT_DIR, 'data')\n",
    "EXAMPLES_DIR = os.path.join(DATA_DIR, 'examples')\n",
    "AIRCRAFT_CONFIGURATION_DIR = os.path.join(EXAMPLES_DIR, 'aircraft_configuration')\n",
    "AIRCRAFT_CONFIGURATION_CONTRACT = os.path.join(AIRCRAFT_CONFIGURATION_DIR, 'aircraft_configuration_contract.yaml')\n",
    "AIRCRAFT_CONFIGURATION_JSON = os.path.join(AIRCRAFT_CONFIGURATION_DIR, 'aircraft_configuration.json')\n",
    "\n",
    "print(\"✓ Imports successful\")\n",
    "print(\"✓ PyCharter services loaded\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Create Sample Data\n",
    "\n",
    "We'll create dummy aircraft configuration data that matches our contract structure. This demonstrates how PyCharter validates real-world data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✓ Loaded 20 records from JSON file\n"
     ]
    }
   ],
   "source": [
    "# Step 1: Load and Clean Data from JSON File\n",
    "\n",
    "def convert_mongodb_json(obj):\n",
    "    \"\"\"Convert MongoDB-style JSON ($oid, $date) to Python objects.\"\"\"\n",
    "    if isinstance(obj, dict):\n",
    "        if '$oid' in obj:\n",
    "            return obj['$oid']  # Return as string\n",
    "        elif '$date' in obj:\n",
    "            # Convert ISO date string to datetime, then to string for validation\n",
    "            date_str = obj['$date']\n",
    "            if isinstance(date_str, str):\n",
    "                try:\n",
    "                    dt = datetime.fromisoformat(date_str.replace('Z', '+00:00'))\n",
    "                    return dt.isoformat()  # Return as ISO string for validation\n",
    "                except:\n",
    "                    return date_str\n",
    "            return date_str\n",
    "        else:\n",
    "            return {k: convert_mongodb_json(v) for k, v in obj.items()}\n",
    "    elif isinstance(obj, list):\n",
    "        return [convert_mongodb_json(item) for item in obj]\n",
    "    elif isinstance(obj, datetime):\n",
    "        # Convert datetime objects to ISO format strings\n",
    "        return obj.isoformat()\n",
    "    else:\n",
    "        return obj\n",
    "\n",
    "def clean_data_for_validation(record: Dict[str, Any]) -> Dict[str, Any]:\n",
    "    \"\"\"Clean and prepare a record for validation.\"\"\"\n",
    "    # Create a copy to avoid modifying original\n",
    "    cleaned = record.copy()\n",
    "    \n",
    "    # Remove MongoDB _id field if present\n",
    "    cleaned.pop('_id', None)\n",
    "    \n",
    "    # Add metadata field if missing (required by contract)\n",
    "    if 'metadata' not in cleaned:\n",
    "        cleaned['metadata'] = {}\n",
    "    \n",
    "    # Convert all datetime objects to ISO format strings recursively\n",
    "    def convert_datetimes(obj):\n",
    "        if isinstance(obj, datetime):\n",
    "            return obj.isoformat()\n",
    "        elif isinstance(obj, dict):\n",
    "            return {k: convert_datetimes(v) for k, v in obj.items()}\n",
    "        elif isinstance(obj, list):\n",
    "            return [convert_datetimes(item) for item in obj]\n",
    "        else:\n",
    "            return obj\n",
    "    \n",
    "    cleaned = convert_datetimes(cleaned)\n",
    "    \n",
    "    return cleaned\n",
    "\n",
    "# Load sample data from JSON file\n",
    "sample_aircraft_data = []\n",
    "if os.path.exists(AIRCRAFT_CONFIGURATION_JSON):\n",
    "    with open(AIRCRAFT_CONFIGURATION_JSON, 'r') as f:\n",
    "        raw_data = json.load(f)\n",
    "        # Convert MongoDB-style JSON to Python objects\n",
    "        converted_data = convert_mongodb_json(raw_data)\n",
    "        # Clean each record for validation\n",
    "        sample_aircraft_data = [clean_data_for_validation(record) for record in converted_data]\n",
    "    print(f\"✓ Loaded {len(sample_aircraft_data)} records from JSON file\")\n",
    "    print(f\"  Sample registrations: {[r.get('registration', 'N/A') for r in sample_aircraft_data[:5]]}\")\n",
    "else:\n",
    "    print(\"⚠ JSON file not found, creating minimal sample data...\")\n",
    "    # Create minimal sample data with all required fields including metadata\n",
    "    sample_aircraft_data = [\n",
    "    {\n",
    "        \"registration\": \"N123AB\",\n",
    "        \"valid_since\": \"2020-01-01T00:00:00Z\",\n",
    "        \"valid_until\": \"2025-12-31T23:59:59Z\",\n",
    "        \"ac_operator\": \"AA\",\n",
    "        \"ac_owner\": \"AA\",\n",
    "        \"ac_subtype\": \"737-800\",\n",
    "        \"ac_logical_no\": 1,\n",
    "        \"ac_state\": \"A\",\n",
    "        \"dry_operating_wgt\": 41500,\n",
    "        \"max_takeoff_wgt\": 79000,\n",
    "        \"cargo_capacity\": 45.5,\n",
    "        \"fuel_capacity\": 26020,\n",
    "        \"avg_fuel_consump\": 2400,\n",
    "        \"ac_index\": 100,\n",
    "        \"crewsize_cockpit\": 2,\n",
    "        \"crewsize_cabin\": 4,\n",
    "        \"std_version\": \"STD\",\n",
    "        \"ap_restriction\": \"N\",\n",
    "        \"ac_owner_name\": \"American Airlines\",\n",
    "        \"ac_subtype_name\": \"Boeing 737-800\",\n",
    "        \"ac_category\": \"N\",\n",
    "        \"remark\": \"Standard configuration\",\n",
    "        \"fuel_measure_unit\": \"kg\",\n",
    "        \"ac_callsign\": \"AAL123\",\n",
    "        \"radio\": \"VHF\",\n",
    "        \"noise\": \"Stage 3\",\n",
    "        \"phone\": None,\n",
    "        \"special_equipment\": None,\n",
    "        \"acars\": \"Y\",\n",
    "        \"alt_reg\": None,\n",
    "        \"record_id\": 1001,\n",
    "        \"last_update\": \"2024-01-15T10:30:00Z\",\n",
    "        \"std_version_alt_1\": None,\n",
    "        \"std_version_alt_2\": None,\n",
    "        \"std_version_alt_3\": None,\n",
    "        \"std_version_alt_4\": None,\n",
    "        \"ils_equipment\": \"I\",\n",
    "        \"autoland\": \"Y\",\n",
    "        \"homebase\": \"KDFW\",\n",
    "        \"aircraft_compartment\": [\n",
    "            {\n",
    "                \"compartment\": \"F\",\n",
    "                \"valid_since\": \"2020-01-01T00:00:00Z\",\n",
    "                \"valid_until\": \"2025-12-31T23:59:59Z\",\n",
    "                \"seats\": 16,\n",
    "                \"code\": \"F\",\n",
    "                \"priority\": 1,\n",
    "                \"name\": \"First Class\",\n",
    "                \"booking_classes\": \"PFA\",\n",
    "                \"three_comp_code\": \"F\",\n",
    "                \"curtain_version\": \"\"\n",
    "            },\n",
    "            {\n",
    "                \"compartment\": \"J\",\n",
    "                \"valid_since\": \"2020-01-01T00:00:00Z\",\n",
    "                \"valid_until\": \"2025-12-31T23:59:59Z\",\n",
    "                \"seats\": 30,\n",
    "                \"code\": \"J\",\n",
    "                \"priority\": 2,\n",
    "                \"name\": \"Business Class\",\n",
    "                \"booking_classes\": \"CJ\",\n",
    "                \"three_comp_code\": \"J\",\n",
    "                \"curtain_version\": \"\"\n",
    "            }\n",
    "        ]\n",
    "    },\n",
    "    {\n",
    "        \"registration\": \"N456CD\",\n",
    "        \"valid_since\": \"2021-06-01T00:00:00Z\",\n",
    "        \"valid_until\": \"2026-05-31T23:59:59Z\",\n",
    "        \"ac_operator\": \"UA\",\n",
    "        \"ac_owner\": \"UA\",\n",
    "        \"ac_subtype\": \"A320\",\n",
    "        \"ac_logical_no\": 2,\n",
    "        \"ac_state\": \"A\",\n",
    "        \"dry_operating_wgt\": 42400,\n",
    "        \"max_takeoff_wgt\": 78000,\n",
    "        \"cargo_capacity\": 42.0,\n",
    "        \"fuel_capacity\": 23800,\n",
    "        \"avg_fuel_consump\": 2200,\n",
    "        \"ac_index\": 101,\n",
    "        \"crewsize_cockpit\": 2,\n",
    "        \"crewsize_cabin\": 4,\n",
    "        \"std_version\": \"STD\",\n",
    "        \"ap_restriction\": \"N\",\n",
    "        \"ac_owner_name\": \"United Airlines\",\n",
    "        \"ac_subtype_name\": \"Airbus A320\",\n",
    "        \"ac_category\": \"N\",\n",
    "        \"remark\": \"Winglet equipped\",\n",
    "        \"fuel_measure_unit\": \"kg\",\n",
    "        \"ac_callsign\": \"UAL456\",\n",
    "        \"radio\": \"VHF\",\n",
    "        \"noise\": \"Stage 4\",\n",
    "        \"phone\": None,\n",
    "        \"special_equipment\": \"WiFi\",\n",
    "        \"acars\": \"Y\",\n",
    "        \"alt_reg\": None,\n",
    "        \"record_id\": 1002,\n",
    "        \"last_update\": \"2024-02-20T14:15:00Z\",\n",
    "        \"std_version_alt_1\": None,\n",
    "        \"std_version_alt_2\": None,\n",
    "        \"std_version_alt_3\": None,\n",
    "        \"std_version_alt_4\": None,\n",
    "        \"ils_equipment\": \"I\",\n",
    "        \"autoland\": \"Y\",\n",
    "        \"homebase\": \"KORD\",\n",
    "        \"aircraft_compartment\": [\n",
    "            {\n",
    "                \"compartment\": \"Y\",\n",
    "                \"valid_since\": \"2021-06-01T00:00:00Z\",\n",
    "                \"valid_until\": \"2026-05-31T23:59:59Z\",\n",
    "                \"seats\": 150,\n",
    "                \"code\": \"Y\",\n",
    "                \"priority\": 3,\n",
    "                \"name\": \"Economy Class\",\n",
    "                \"booking_classes\": \"YBMHK\",\n",
    "                \"three_comp_code\": \"Y\",\n",
    "                \"curtain_version\": \"\"\n",
    "            }\n",
    "        ]\n",
    "    },\n",
    "    {\n",
    "        \"registration\": \"N789EF\",\n",
    "        \"valid_since\": \"2022-03-15T00:00:00Z\",\n",
    "        \"valid_until\": \"2027-03-14T23:59:59Z\",\n",
    "        \"ac_operator\": \"DL\",\n",
    "        \"ac_owner\": \"DL\",\n",
    "        \"ac_subtype\": \"757-300\",\n",
    "        \"ac_logical_no\": 3,\n",
    "        \"ac_state\": \"A\",\n",
    "        \"dry_operating_wgt\": 58000,\n",
    "        \"max_takeoff_wgt\": 124000,\n",
    "        \"cargo_capacity\": 68.0,\n",
    "        \"fuel_capacity\": 43400,\n",
    "        \"avg_fuel_consump\": 3800,\n",
    "        \"ac_index\": 102,\n",
    "        \"crewsize_cockpit\": 2,\n",
    "        \"crewsize_cabin\": 6,\n",
    "        \"std_version\": \"STD\",\n",
    "        \"ap_restriction\": \"N\",\n",
    "        \"ac_owner_name\": \"Delta Air Lines\",\n",
    "        \"ac_subtype_name\": \"Boeing 757-300\",\n",
    "        \"ac_category\": \"N\",\n",
    "        \"remark\": \"Extended range\",\n",
    "        \"fuel_measure_unit\": \"kg\",\n",
    "        \"ac_callsign\": \"DAL789\",\n",
    "        \"radio\": \"VHF\",\n",
    "        \"noise\": \"Stage 4\",\n",
    "        \"phone\": None,\n",
    "        \"special_equipment\": None,\n",
    "        \"acars\": \"Y\",\n",
    "        \"alt_reg\": None,\n",
    "        \"record_id\": 1003,\n",
    "        \"last_update\": \"2024-03-10T09:45:00Z\",\n",
    "        \"std_version_alt_1\": None,\n",
    "        \"std_version_alt_2\": None,\n",
    "        \"std_version_alt_3\": None,\n",
    "        \"std_version_alt_4\": None,\n",
    "        \"ils_equipment\": \"I\",\n",
    "        \"autoland\": \"Y\",\n",
    "        \"homebase\": \"KATL\",\n",
    "        \"aircraft_compartment\": [\n",
    "            {\n",
    "                \"compartment\": \"F\",\n",
    "                \"valid_since\": \"2022-03-15T00:00:00Z\",\n",
    "                \"valid_until\": \"2027-03-14T23:59:59Z\",\n",
    "                \"seats\": 24,\n",
    "                \"code\": \"F\",\n",
    "                \"priority\": 1,\n",
    "                \"name\": \"First Class\",\n",
    "                \"booking_classes\": \"PFA\",\n",
    "                \"three_comp_code\": \"F\",\n",
    "                \"curtain_version\": \"\"\n",
    "            },\n",
    "            {\n",
    "                \"compartment\": \"J\",\n",
    "                \"valid_since\": \"2022-03-15T00:00:00Z\",\n",
    "                \"valid_until\": \"2027-03-14T23:59:59Z\",\n",
    "                \"seats\": 48,\n",
    "                \"code\": \"J\",\n",
    "                \"priority\": 2,\n",
    "                \"name\": \"Business Class\",\n",
    "                \"booking_classes\": \"CJ\",\n",
    "                \"three_comp_code\": \"J\",\n",
    "                \"curtain_version\": \"\"\n",
    "            },\n",
    "            {\n",
    "                \"compartment\": \"Y\",\n",
    "                \"valid_since\": \"2022-03-15T00:00:00Z\",\n",
    "                \"valid_until\": \"2027-03-14T23:59:59Z\",\n",
    "                \"seats\": 204,\n",
    "                \"code\": \"Y\",\n",
    "                \"priority\": 3,\n",
    "                \"name\": \"Economy Class\",\n",
    "                \"booking_classes\": \"YBMHK\",\n",
    "                \"three_comp_code\": \"Y\",\n",
    "                \"curtain_version\": \"\"\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "]\n",
    "\n",
    "    print(f\"✓ Created {len(sample_aircraft_data)} sample records\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "======================================================================\n",
      "Step 1: Convert Pydantic Model to JSON Schema\n",
      "======================================================================\n",
      "✓ Converted AircraftConfigurationApi model to JSON Schema\n",
      "  Schema title: AircraftConfigurationApi\n",
      "  Schema version: 1.0.0\n",
      "  Number of properties: 57\n"
     ]
    }
   ],
   "source": [
    "## Step 2: Parse the Data Contract\n",
    "\n",
    "The contract file contains the complete specification including:\n",
    "- **Schema**: Data structure and types\n",
    "- **Metadata**: Business context and documentation\n",
    "- **Ownership**: Who owns and manages this data\n",
    "- **Governance Rules**: Data policies and compliance requirements\n",
    "- **Coercion Rules**: How to transform data types\n",
    "- **Validation Rules**: Business and technical constraints\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "======================================================================\n",
      "Step 2: Parse Contract YAML and Decompose into Components\n",
      "======================================================================\n",
      "✓ Parsed contract file: ../data/examples/aircraft_configuration/aircraft_configuration_contract.yaml\n",
      "  Schema version: 1.0.0\n",
      "  Metadata version: 1.0.0\n",
      "  Ownership: operations-team / data-engineering\n",
      "  Coercion rules: 40 fields\n",
      "  Validation rules: 34 fields\n",
      "  Governance rules: Yes\n"
     ]
    }
   ],
   "source": [
    "# Parse the contract file\n",
    "# This decomposes the contract into its component parts\n",
    "contract_metadata = parse_contract_file(AIRCRAFT_CONFIGURATION_CONTRACT)\n",
    "\n",
    "# Extract components\n",
    "schema = contract_metadata.schema\n",
    "metadata = contract_metadata.metadata\n",
    "ownership = contract_metadata.ownership\n",
    "governance_rules = contract_metadata.governance_rules\n",
    "versions = contract_metadata.versions\n",
    "\n",
    "print(\"=\" * 70)\n",
    "print(\"Contract Parsing Results\")\n",
    "print(\"=\" * 70)\n",
    "print(f\"✓ Parsed contract file: {AIRCRAFT_CONFIGURATION_CONTRACT}\")\n",
    "print(f\"\\n  Schema Information:\")\n",
    "print(f\"    Title: {schema.get('title', 'N/A')}\")\n",
    "print(f\"    Version: {versions.get('schema', schema.get('version', 'N/A'))}\")\n",
    "print(f\"    Properties: {len(schema.get('properties', {}))} fields\")\n",
    "print(f\"\\n  Metadata Information:\")\n",
    "print(f\"    Description: {metadata.get('description', 'N/A')[:60]}...\" if metadata and metadata.get('description') else \"    Description: N/A\")\n",
    "print(f\"    Version: {versions.get('metadata', 'N/A')}\")\n",
    "print(f\"\\n  Ownership:\")\n",
    "print(f\"    Owner: {ownership.get('owner', 'N/A')}\")\n",
    "print(f\"    Team: {ownership.get('team', 'N/A')}\")\n",
    "print(f\"\\n  Governance Rules:\")\n",
    "print(f\"    Present: {'Yes' if governance_rules else 'No'}\")\n",
    "if governance_rules:\n",
    "    print(f\"    Rules: {list(governance_rules.keys())[:3]}...\" if len(governance_rules) > 3 else f\"    Rules: {list(governance_rules.keys())}\")\n",
    "\n",
    "# Extract coercion and validation rules from schema properties\n",
    "coercion_rules = {}\n",
    "validation_rules = {}\n",
    "\n",
    "if \"properties\" in schema:\n",
    "    for field_name, field_schema in schema[\"properties\"].items():\n",
    "        if isinstance(field_schema, dict):\n",
    "            if \"coercion\" in field_schema:\n",
    "                coercion_rules[field_name] = field_schema[\"coercion\"]\n",
    "            if \"validations\" in field_schema:\n",
    "                validation_rules[field_name] = field_schema[\"validations\"]\n",
    "\n",
    "print(f\"\\n  Rules Extracted:\")\n",
    "print(f\"    Coercion rules: {len(coercion_rules)} fields\")\n",
    "print(f\"    Validation rules: {len(validation_rules)} fields\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "======================================================================\n",
      "Step 3: Store All Components in PostgreSQL Metadata Store\n",
      "======================================================================\n",
      "✓ Connected to PostgreSQL metadata store\n",
      "✓ Stored schema (ID: b66e6e79-8e04-4ad4-859e-8808d28be5fb, version: 1.0.0)\n",
      "✓ Stored metadata (including ownership and governance rules)\n",
      "✓ Stored coercion rules (version: 1.0.0)\n",
      "✓ Stored validation rules (version: 1.0.0)\n",
      "\n",
      "✓ All components stored successfully (Schema ID: b66e6e79-8e04-4ad4-859e-8808d28be5fb)\n"
     ]
    }
   ],
   "source": [
    "## Step 3: Store Components in Metadata Store\n",
    "\n",
    "Now we'll store all contract components in PostgreSQL. This enables:\n",
    "- **Versioning**: Track changes over time\n",
    "- **Centralized Management**: Single source of truth\n",
    "- **Reconstruction**: Rebuild complete contracts from stored components\n",
    "- **Audit Trail**: Track who created/modified what and when\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "======================================================================\n",
      "Step 4: Extract from Metadata Store and Reconstruct Contract\n",
      "======================================================================\n",
      "✓ Reconstructed contract from metadata store\n",
      "  Contract keys: ['schema', 'metadata', 'versions']\n",
      "  Schema title: Aircraft Configuration\n",
      "  Schema version: 1.0.0\n",
      "  Metadata present: Yes\n"
     ]
    }
   ],
   "source": [
    "# Connect to PostgreSQL metadata store\n",
    "# Update this connection string to match your PostgreSQL setup\n",
    "connection_string = \"postgresql://postgres:1234567890@localhost:5432/postgres\"\n",
    "\n",
    "store = PostgresMetadataStore(connection_string=connection_string)\n",
    "store.connect()\n",
    "\n",
    "print(\"=\" * 70)\n",
    "print(\"Storing Contract Components\")\n",
    "print(\"=\" * 70)\n",
    "print(\"✓ Connected to PostgreSQL metadata store\")\n",
    "\n",
    "# Store schema (creates data contract record)\n",
    "schema_version = versions.get(\"schema\", schema.get(\"version\", \"1.0.0\"))\n",
    "schema_id = store.store_schema(\n",
    "    schema_name=\"aircraft_configuration\",\n",
    "    schema=schema,\n",
    "    version=schema_version\n",
    ")\n",
    "print(f\"✓ Stored schema\")\n",
    "print(f\"    Schema ID: {schema_id}\")\n",
    "print(f\"    Version: {schema_version}\")\n",
    "\n",
    "# Merge ownership and governance into metadata (they're part of metadata)\n",
    "metadata_dict = metadata.copy() if metadata else {}\n",
    "if ownership:\n",
    "    metadata_dict[\"business_owners\"] = [ownership.get(\"owner\", \"unknown\")] if ownership.get(\"owner\") else []\n",
    "if governance_rules:\n",
    "    metadata_dict[\"governance_rules\"] = governance_rules\n",
    "\n",
    "# Store metadata with ownership and governance\n",
    "if metadata_dict:\n",
    "    store.store_metadata(\n",
    "        schema_id=schema_id,\n",
    "        metadata=metadata_dict,\n",
    "        version=schema_version\n",
    "    )\n",
    "    print(f\"✓ Stored metadata (with ownership and governance)\")\n",
    "\n",
    "# Store coercion rules\n",
    "if coercion_rules:\n",
    "    coercion_version = versions.get(\"coercion_rules\", \"1.0.0\")\n",
    "    store.store_coercion_rules(\n",
    "        schema_id=schema_id,\n",
    "        coercion_rules=coercion_rules,\n",
    "        version=coercion_version\n",
    "    )\n",
    "    print(f\"✓ Stored coercion rules ({len(coercion_rules)} fields)\")\n",
    "\n",
    "# Store validation rules\n",
    "if validation_rules:\n",
    "    validation_version = versions.get(\"validation_rules\", \"1.0.0\")\n",
    "    store.store_validation_rules(\n",
    "        schema_id=schema_id,\n",
    "        validation_rules=validation_rules,\n",
    "        version=validation_version\n",
    "    )\n",
    "    print(f\"✓ Stored validation rules ({len(validation_rules)} fields)\")\n",
    "\n",
    "print(f\"\\n✓ All components stored successfully!\")\n",
    "print(f\"  Schema ID: {schema_id}\")\n",
    "print(f\"  Contract Name: aircraft_configuration\")\n",
    "print(f\"  Version: {schema_version}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "======================================================================\n",
      "Step 5: Create Pydantic Model from Reconstructed Contract\n",
      "======================================================================\n",
      "✓ Generated Pydantic model: AircraftConfiguration\n",
      "\n",
      "  Model Details:\n",
      "    Model Name: AircraftConfiguration\n",
      "    Properties: 43\n",
      "    Required Fields: 23\n",
      "\n",
      "  Testing model with sample record:\n",
      "    ⚠ Validation issues found in sample record:\n",
      "      - {'type': 'missing', 'loc': ('metadata',), 'msg': 'Field required', 'input': {'registration': 'BHUR', 'valid_since': '2017-07-27T00:00:00+00:00', 'valid_until': '2018-08-31T00:00:00+00:00', 'ac_operator': 'CX', 'ac_owner': 'CX', 'ac_subtype': '7CF', 'ac_logical_no': 3, 'ac_state': 'R', 'dry_operating_wgt': 1, 'max_takeoff_wgt': 1, 'cargo_capacity': 0, 'fuel_capacity': 0, 'avg_fuel_consump': 0, 'ac_index': 190, 'crewsize_cockpit': 0, 'crewsize_cabin': 0, 'std_version': 'STN', 'ap_restriction': 'N', 'ac_owner_name': 'CX', 'ac_subtype_name': '7CF', 'ac_category': 'J', 'remark': '-', 'fuel_measure_unit': 'kg', 'ac_callsign': None, 'radio': None, 'noise': None, 'phone': None, 'special_equipment': None, 'acars': 'S', 'alt_reg': None, 'record_id': 309080, 'last_update': '2022-03-02T10:36:42+00:00', 'std_version_alt_1': None, 'std_version_alt_2': None, 'std_version_alt_3': None, 'std_version_alt_4': None, 'ils_equipment': 'I', 'autoland': 'Y', 'homebase': None, 'aircraft_compartment': [{'curtain_version': '', 'compartment': 'F', 'valid_since': datetime.datetime(2008, 1, 1, 0, 0, tzinfo=datetime.timezone.utc), 'valid_until': datetime.datetime(2035, 12, 31, 0, 0, tzinfo=datetime.timezone.utc), 'seats': 0, 'code': 'F', 'priority': 1, 'name': 'First', 'booking_classes': 'PFA', 'three_comp_code': 'F'}, {'curtain_version': '', 'compartment': 'J', 'valid_since': datetime.datetime(2008, 1, 1, 0, 0, tzinfo=datetime.timezone.utc), 'valid_until': datetime.datetime(2035, 12, 31, 0, 0, tzinfo=datetime.timezone.utc), 'seats': 0, 'code': 'J', 'priority': 2, 'name': 'Business', 'booking_classes': 'JCDIZ', 'three_comp_code': 'C'}, {'curtain_version': '', 'compartment': 'Y', 'valid_since': datetime.datetime(2008, 1, 1, 0, 0, tzinfo=datetime.timezone.utc), 'valid_until': datetime.datetime(2035, 12, 31, 0, 0, tzinfo=datetime.timezone.utc), 'seats': 0, 'code': 'Y', 'priority': 4, 'name': 'Economy', 'booking_classes': 'SYBHKLMNQTVXGUE', 'three_comp_code': 'Y'}], 'aircraft_restriction': [], 'aircraft_owner': 'CX', 'aircraft_subtype': '74Y', 'fleet_print_code': '74Y', 'rotational_flag': 'Y', 'standard_configuration': 'STN', 'payload': 0, 'mtom': 0, 'max_range': 0, 'noise_category': None, 'average_speed': 0, 'aircraft_group': '74F', 'icao_code': '74Y', 'name': 'Boeing 747-400 Freighter', 'body_type': 'W', 'category': 'J', 'is_active_aircraft': True}, 'url': 'https://errors.pydantic.dev/2.12/v/missing'}\n",
      "      - {'type': 'string_type', 'loc': ('valid_since',), 'msg': 'Input should be a valid string', 'input': datetime.datetime(2017, 7, 27, 0, 0, tzinfo=datetime.timezone.utc), 'url': 'https://errors.pydantic.dev/2.12/v/string_type'}\n",
      "      - {'type': 'string_type', 'loc': ('valid_until',), 'msg': 'Input should be a valid string', 'input': datetime.datetime(2018, 8, 31, 0, 0, tzinfo=datetime.timezone.utc), 'url': 'https://errors.pydantic.dev/2.12/v/string_type'}\n",
      "\n",
      "✓ Model generation completed - ready for runtime validation\n"
     ]
    }
   ],
   "source": [
    "## Step 4: Reconstruct Contract from Store\n",
    "\n",
    "We can now reconstruct the complete contract from stored components. This demonstrates how PyCharter maintains a single source of truth while allowing components to be stored separately.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "======================================================================\n",
      "Step 6: Runtime Validation - Validate All Data Records\n",
      "======================================================================\n",
      "✓ Prepared 20 records for validation\n",
      "\n",
      "Validating data using generated Pydantic model...\n",
      "\n",
      "  Validation Results:\n",
      "    Total Records: 20\n",
      "    Valid Records: 0\n",
      "    Invalid Records: 20\n",
      "    Success Rate: 0.00%\n",
      "\n",
      "  Example Invalid Records (showing first 3):\n",
      "\n",
      "    Record 1:\n",
      "      - {'type': 'missing', 'loc': ('metadata',), 'msg': 'Field required', 'input': {'registration': 'BHUR', 'valid_since': '2017-07-27T00:00:00+00:00', 'valid_until': '2018-08-31T00:00:00+00:00', 'ac_operator': 'CX', 'ac_owner': 'CX', 'ac_subtype': '7CF', 'ac_logical_no': 3, 'ac_state': 'R', 'dry_operating_wgt': 1, 'max_takeoff_wgt': 1, 'cargo_capacity': 0, 'fuel_capacity': 0, 'avg_fuel_consump': 0, 'ac_index': 190, 'crewsize_cockpit': 0, 'crewsize_cabin': 0, 'std_version': 'STN', 'ap_restriction': 'N', 'ac_owner_name': 'CX', 'ac_subtype_name': '7CF', 'ac_category': 'J', 'remark': '-', 'fuel_measure_unit': 'kg', 'ac_callsign': None, 'radio': None, 'noise': None, 'phone': None, 'special_equipment': None, 'acars': 'S', 'alt_reg': None, 'record_id': 309080, 'last_update': '2022-03-02T10:36:42+00:00', 'std_version_alt_1': None, 'std_version_alt_2': None, 'std_version_alt_3': None, 'std_version_alt_4': None, 'ils_equipment': 'I', 'autoland': 'Y', 'homebase': None, 'aircraft_compartment': [{'curtain_version': '', 'compartment': 'F', 'valid_since': datetime.datetime(2008, 1, 1, 0, 0, tzinfo=datetime.timezone.utc), 'valid_until': datetime.datetime(2035, 12, 31, 0, 0, tzinfo=datetime.timezone.utc), 'seats': 0, 'code': 'F', 'priority': 1, 'name': 'First', 'booking_classes': 'PFA', 'three_comp_code': 'F'}, {'curtain_version': '', 'compartment': 'J', 'valid_since': datetime.datetime(2008, 1, 1, 0, 0, tzinfo=datetime.timezone.utc), 'valid_until': datetime.datetime(2035, 12, 31, 0, 0, tzinfo=datetime.timezone.utc), 'seats': 0, 'code': 'J', 'priority': 2, 'name': 'Business', 'booking_classes': 'JCDIZ', 'three_comp_code': 'C'}, {'curtain_version': '', 'compartment': 'Y', 'valid_since': datetime.datetime(2008, 1, 1, 0, 0, tzinfo=datetime.timezone.utc), 'valid_until': datetime.datetime(2035, 12, 31, 0, 0, tzinfo=datetime.timezone.utc), 'seats': 0, 'code': 'Y', 'priority': 4, 'name': 'Economy', 'booking_classes': 'SYBHKLMNQTVXGUE', 'three_comp_code': 'Y'}], 'aircraft_restriction': [], 'aircraft_owner': 'CX', 'aircraft_subtype': '74Y', 'fleet_print_code': '74Y', 'rotational_flag': 'Y', 'standard_configuration': 'STN', 'payload': 0, 'mtom': 0, 'max_range': 0, 'noise_category': None, 'average_speed': 0, 'aircraft_group': '74F', 'icao_code': '74Y', 'name': 'Boeing 747-400 Freighter', 'body_type': 'W', 'category': 'J', 'is_active_aircraft': True}, 'url': 'https://errors.pydantic.dev/2.12/v/missing'}\n",
      "      - {'type': 'string_type', 'loc': ('valid_since',), 'msg': 'Input should be a valid string', 'input': datetime.datetime(2017, 7, 27, 0, 0, tzinfo=datetime.timezone.utc), 'url': 'https://errors.pydantic.dev/2.12/v/string_type'}\n",
      "      - {'type': 'string_type', 'loc': ('valid_until',), 'msg': 'Input should be a valid string', 'input': datetime.datetime(2018, 8, 31, 0, 0, tzinfo=datetime.timezone.utc), 'url': 'https://errors.pydantic.dev/2.12/v/string_type'}\n",
      "\n",
      "    Record 2:\n",
      "      - {'type': 'missing', 'loc': ('metadata',), 'msg': 'Field required', 'input': {'registration': 'N497', 'valid_since': '2017-06-01T00:00:00+00:00', 'valid_until': '2035-12-31T00:00:00+00:00', 'ac_operator': '5Y', 'ac_owner': '5Y', 'ac_subtype': '5Y4', 'ac_logical_no': 15, 'ac_state': 'R', 'dry_operating_wgt': 0, 'max_takeoff_wgt': 0, 'cargo_capacity': 0, 'fuel_capacity': 0, 'avg_fuel_consump': 0, 'ac_index': 230, 'crewsize_cockpit': 0, 'crewsize_cabin': 0, 'std_version': 'STN', 'ap_restriction': 'N', 'ac_owner_name': '5Y', 'ac_subtype_name': '5Y4', 'ac_category': 'J', 'remark': '-', 'fuel_measure_unit': 'kg', 'ac_callsign': None, 'radio': None, 'noise': None, 'phone': None, 'special_equipment': None, 'acars': None, 'alt_reg': None, 'record_id': 310651, 'last_update': '2022-03-02T10:36:42+00:00', 'std_version_alt_1': None, 'std_version_alt_2': None, 'std_version_alt_3': None, 'std_version_alt_4': None, 'ils_equipment': 'I', 'autoland': 'Y', 'homebase': None, 'aircraft_compartment': [{'curtain_version': '', 'compartment': 'F', 'valid_since': datetime.datetime(2017, 1, 1, 0, 0, tzinfo=datetime.timezone.utc), 'valid_until': datetime.datetime(2035, 12, 31, 0, 0, tzinfo=datetime.timezone.utc), 'seats': 0, 'code': 'F', 'priority': 1, 'name': 'First', 'booking_classes': 'PFA', 'three_comp_code': 'F'}, {'curtain_version': '', 'compartment': 'J', 'valid_since': datetime.datetime(2017, 1, 1, 0, 0, tzinfo=datetime.timezone.utc), 'valid_until': datetime.datetime(2035, 12, 31, 0, 0, tzinfo=datetime.timezone.utc), 'seats': 0, 'code': 'J', 'priority': 2, 'name': 'Business', 'booking_classes': 'JCDIZ', 'three_comp_code': 'C'}, {'curtain_version': '', 'compartment': 'W', 'valid_since': datetime.datetime(2017, 1, 1, 0, 0, tzinfo=datetime.timezone.utc), 'valid_until': datetime.datetime(2035, 12, 31, 0, 0, tzinfo=datetime.timezone.utc), 'seats': 0, 'code': 'W', 'priority': 3, 'name': 'Premium Eco', 'booking_classes': 'W', 'three_comp_code': 'Y'}, {'curtain_version': '', 'compartment': 'Y', 'valid_since': datetime.datetime(2017, 1, 1, 0, 0, tzinfo=datetime.timezone.utc), 'valid_until': datetime.datetime(2035, 12, 31, 0, 0, tzinfo=datetime.timezone.utc), 'seats': 0, 'code': 'Y', 'priority': 4, 'name': 'Economy', 'booking_classes': 'SYBHKLMNQTVXGUE', 'three_comp_code': 'Y'}], 'aircraft_restriction': [], 'aircraft_owner': '5Y', 'aircraft_subtype': '74Y', 'fleet_print_code': '74Y', 'rotational_flag': 'Y', 'standard_configuration': 'STN', 'payload': 0, 'mtom': 0, 'max_range': 0, 'noise_category': None, 'average_speed': 0, 'aircraft_group': '74F', 'icao_code': '74Y', 'name': 'Boeing 747-400 Freighter', 'body_type': 'W', 'category': 'J', 'is_active_aircraft': True}, 'url': 'https://errors.pydantic.dev/2.12/v/missing'}\n",
      "      - {'type': 'string_type', 'loc': ('valid_since',), 'msg': 'Input should be a valid string', 'input': datetime.datetime(2017, 6, 1, 0, 0, tzinfo=datetime.timezone.utc), 'url': 'https://errors.pydantic.dev/2.12/v/string_type'}\n",
      "      - {'type': 'string_type', 'loc': ('valid_until',), 'msg': 'Input should be a valid string', 'input': datetime.datetime(2035, 12, 31, 0, 0, tzinfo=datetime.timezone.utc), 'url': 'https://errors.pydantic.dev/2.12/v/string_type'}\n",
      "\n",
      "    Record 3:\n",
      "      - {'type': 'missing', 'loc': ('metadata',), 'msg': 'Field required', 'input': {'registration': 'N499', 'valid_since': '2017-06-01T00:00:00+00:00', 'valid_until': '2035-12-31T00:00:00+00:00', 'ac_operator': '5Y', 'ac_owner': '5Y', 'ac_subtype': '5Y4', 'ac_logical_no': 16, 'ac_state': 'R', 'dry_operating_wgt': 0, 'max_takeoff_wgt': 0, 'cargo_capacity': 0, 'fuel_capacity': 0, 'avg_fuel_consump': 0, 'ac_index': 230, 'crewsize_cockpit': 0, 'crewsize_cabin': 0, 'std_version': 'STN', 'ap_restriction': 'N', 'ac_owner_name': '5Y', 'ac_subtype_name': '5Y4', 'ac_category': 'J', 'remark': '-', 'fuel_measure_unit': 'kg', 'ac_callsign': None, 'radio': None, 'noise': None, 'phone': None, 'special_equipment': None, 'acars': None, 'alt_reg': None, 'record_id': 310652, 'last_update': '2022-03-02T10:36:43+00:00', 'std_version_alt_1': None, 'std_version_alt_2': None, 'std_version_alt_3': None, 'std_version_alt_4': None, 'ils_equipment': 'I', 'autoland': 'Y', 'homebase': None, 'aircraft_compartment': [{'curtain_version': '', 'compartment': 'F', 'valid_since': datetime.datetime(2017, 1, 1, 0, 0, tzinfo=datetime.timezone.utc), 'valid_until': datetime.datetime(2035, 12, 31, 0, 0, tzinfo=datetime.timezone.utc), 'seats': 0, 'code': 'F', 'priority': 1, 'name': 'First', 'booking_classes': 'PFA', 'three_comp_code': 'F'}, {'curtain_version': '', 'compartment': 'J', 'valid_since': datetime.datetime(2017, 1, 1, 0, 0, tzinfo=datetime.timezone.utc), 'valid_until': datetime.datetime(2035, 12, 31, 0, 0, tzinfo=datetime.timezone.utc), 'seats': 0, 'code': 'J', 'priority': 2, 'name': 'Business', 'booking_classes': 'JCDIZ', 'three_comp_code': 'C'}, {'curtain_version': '', 'compartment': 'W', 'valid_since': datetime.datetime(2017, 1, 1, 0, 0, tzinfo=datetime.timezone.utc), 'valid_until': datetime.datetime(2035, 12, 31, 0, 0, tzinfo=datetime.timezone.utc), 'seats': 0, 'code': 'W', 'priority': 3, 'name': 'Premium Eco', 'booking_classes': 'W', 'three_comp_code': 'Y'}, {'curtain_version': '', 'compartment': 'Y', 'valid_since': datetime.datetime(2017, 1, 1, 0, 0, tzinfo=datetime.timezone.utc), 'valid_until': datetime.datetime(2035, 12, 31, 0, 0, tzinfo=datetime.timezone.utc), 'seats': 0, 'code': 'Y', 'priority': 4, 'name': 'Economy', 'booking_classes': 'SYBHKLMNQTVXGUE', 'three_comp_code': 'Y'}], 'aircraft_restriction': [], 'aircraft_owner': '5Y', 'aircraft_subtype': '74Y', 'fleet_print_code': '74Y', 'rotational_flag': 'Y', 'standard_configuration': 'STN', 'payload': 0, 'mtom': 0, 'max_range': 0, 'noise_category': None, 'average_speed': 0, 'aircraft_group': '74F', 'icao_code': '74Y', 'name': 'Boeing 747-400 Freighter', 'body_type': 'W', 'category': 'J', 'is_active_aircraft': True}, 'url': 'https://errors.pydantic.dev/2.12/v/missing'}\n",
      "      - {'type': 'string_type', 'loc': ('valid_since',), 'msg': 'Input should be a valid string', 'input': datetime.datetime(2017, 6, 1, 0, 0, tzinfo=datetime.timezone.utc), 'url': 'https://errors.pydantic.dev/2.12/v/string_type'}\n",
      "      - {'type': 'string_type', 'loc': ('valid_until',), 'msg': 'Input should be a valid string', 'input': datetime.datetime(2035, 12, 31, 0, 0, tzinfo=datetime.timezone.utc), 'url': 'https://errors.pydantic.dev/2.12/v/string_type'}\n",
      "\n",
      "  Alternative: Validate using metadata store...\n",
      "    Store-based validation: 0/5 valid\n",
      "\n",
      "✓ Runtime validation completed\n",
      "  Note: Validated data is ready for quality assurance checks\n"
     ]
    }
   ],
   "source": [
    "# Reconstruct the complete contract from stored components\n",
    "reconstructed_contract = build_contract_from_store(\n",
    "    store=store,\n",
    "    schema_id=schema_id,\n",
    "    version=None,  # Use latest version\n",
    "    include_metadata=True,\n",
    "    include_ownership=True,\n",
    "    include_governance=True\n",
    ")\n",
    "\n",
    "print(\"=\" * 70)\n",
    "print(\"Contract Reconstruction\")\n",
    "print(\"=\" * 70)\n",
    "print(f\"✓ Reconstructed contract from metadata store\")\n",
    "print(f\"\\n  Contract Components:\")\n",
    "print(f\"    Schema: {'✓' if 'schema' in reconstructed_contract else '✗'}\")\n",
    "print(f\"    Metadata: {'✓' if 'metadata' in reconstructed_contract else '✗'}\")\n",
    "print(f\"    Ownership: {'✓' if 'ownership' in reconstructed_contract else '✗'}\")\n",
    "print(f\"    Governance Rules: {'✓' if 'governance_rules' in reconstructed_contract else '✗'}\")\n",
    "\n",
    "if \"schema\" in reconstructed_contract:\n",
    "    print(f\"\\n  Schema Details:\")\n",
    "    print(f\"    Title: {reconstructed_contract['schema'].get('title', 'N/A')}\")\n",
    "    print(f\"    Version: {reconstructed_contract['schema'].get('version', 'N/A')}\")\n",
    "    print(f\"    Properties: {len(reconstructed_contract['schema'].get('properties', {}))} fields\")\n",
    "\n",
    "print(f\"\\n✓ Contract successfully reconstructed from store\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "======================================================================\n",
      "Step 7: Quality Assurance - Check Data Quality\n",
      "======================================================================\n",
      "✓ Initialized QualityCheck with metadata store and database session\n",
      "✓ Prepared 20 records for quality check\n",
      "\n",
      "Running quality check...\n",
      "\n",
      "======================================================================\n",
      "Quality Check Results\n",
      "======================================================================\n",
      "  Records Checked: 20\n",
      "  Valid Records: 0\n",
      "  Invalid Records: 20\n",
      "  Total Violations: 20\n",
      "\n",
      "  Quality Metrics:\n",
      "    Overall Score: 0.00/100\n",
      "    Accuracy: 0.00%\n",
      "    Completeness: 0.00%\n",
      "    Violation Rate: 100.00%\n",
      "\n",
      "  ⚠ Threshold Breaches:\n",
      "    - Overall score 0.00 below threshold 80.00\n",
      "    - Violation rate 100.00% exceeds threshold 5.00%\n",
      "    - Completeness 0.00% below threshold 7000.00%\n",
      "    - Accuracy 0.00% below threshold 7500.00%\n",
      "\n",
      "  Field-Level Metrics (Top 5 with violations):\n",
      "    https:\n",
      "      Completeness: 100.00%\n",
      "      Violation Rate: 330.00%\n",
      "      Valid Count: -46/20\n",
      "    required:\n",
      "      Completeness: 100.00%\n",
      "      Violation Rate: 100.00%\n",
      "      Valid Count: 0/20\n",
      "\n",
      "  Data Profiling Summary:\n",
      "    Total Fields: N/A\n",
      "    Numeric Fields: N/A\n",
      "    String Fields: N/A\n",
      "\n",
      "  Quality Check Status: ✗ FAILED\n",
      "  Check Timestamp: 2025-12-18 09:09:09.883847\n",
      "\n",
      "======================================================================\n",
      "Violations in Database\n",
      "======================================================================\n",
      "  Found 4 violation records (showing first 10):\n",
      "\n",
      "  Violation 1:\n",
      "    Field: unknown\n",
      "    Error: {'loc': ('unknown',), 'type': 'type_error', 'msg': \"{'type': 'string_type', 'loc': ('valid_until',),...\n",
      "    Error Type: type_error\n",
      "    Severity: warning\n",
      "    Record ID: 309080\n",
      "    Timestamp: 2025-12-18 09:02:03.936216+00:00\n",
      "\n",
      "  Violation 2:\n",
      "    Field: unknown\n",
      "    Error: {'loc': ('unknown',), 'type': 'type_error', 'msg': \"{'type': 'string_type', 'loc': ('last_update',),...\n",
      "    Error Type: type_error\n",
      "    Severity: warning\n",
      "    Record ID: 309080\n",
      "    Timestamp: 2025-12-18 09:02:03.936216+00:00\n",
      "\n",
      "  Violation 3:\n",
      "    Field: unknown\n",
      "    Error: {'loc': ('unknown',), 'type': 'missing_field', 'msg': \"{'type': 'missing', 'loc': ('metadata',), 'ms...\n",
      "    Error Type: missing_field\n",
      "    Severity: warning\n",
      "    Record ID: 310651\n",
      "    Timestamp: 2025-12-18 09:09:09.779593+00:00\n",
      "\n",
      "  Violation 4:\n",
      "    Field: unknown\n",
      "    Error: {'loc': ('unknown',), 'type': 'type_error', 'msg': \"{'type': 'string_type', 'loc': ('valid_until',),...\n",
      "    Error Type: type_error\n",
      "    Severity: warning\n",
      "    Record ID: 310651\n",
      "    Timestamp: 2025-12-18 09:09:09.779593+00:00\n",
      "\n",
      "  Violation 5:\n",
      "    Field: unknown\n",
      "    Error: {'loc': ('unknown',), 'type': 'type_error', 'msg': '{\\'type\\': \\'value_error\\', \\'loc\\': (\\'acars\\',...\n",
      "    Error Type: type_error\n",
      "    Severity: warning\n",
      "    Record ID: 310651\n",
      "    Timestamp: 2025-12-18 09:09:09.779593+00:00\n",
      "\n",
      "  Violation 6:\n",
      "    Field: unknown\n",
      "    Error: {'loc': ('unknown',), 'type': 'type_error', 'msg': \"{'type': 'string_type', 'loc': ('last_update',),...\n",
      "    Error Type: type_error\n",
      "    Severity: warning\n",
      "    Record ID: 310651\n",
      "    Timestamp: 2025-12-18 09:09:09.779593+00:00\n",
      "\n",
      "  Violation 7:\n",
      "    Field: unknown\n",
      "    Error: {'loc': ('unknown',), 'type': 'missing_field', 'msg': \"{'type': 'missing', 'loc': ('metadata',), 'ms...\n",
      "    Error Type: missing_field\n",
      "    Severity: warning\n",
      "    Record ID: 310652\n",
      "    Timestamp: 2025-12-18 09:09:09.789286+00:00\n",
      "\n",
      "  Violation 8:\n",
      "    Field: unknown\n",
      "    Error: {'loc': ('unknown',), 'type': 'type_error', 'msg': \"{'type': 'string_type', 'loc': ('valid_until',),...\n",
      "    Error Type: type_error\n",
      "    Severity: warning\n",
      "    Record ID: 310652\n",
      "    Timestamp: 2025-12-18 09:09:09.789286+00:00\n",
      "\n",
      "  Violation 9:\n",
      "    Field: unknown\n",
      "    Error: {'loc': ('unknown',), 'type': 'type_error', 'msg': '{\\'type\\': \\'value_error\\', \\'loc\\': (\\'acars\\',...\n",
      "    Error Type: type_error\n",
      "    Severity: warning\n",
      "    Record ID: 310652\n",
      "    Timestamp: 2025-12-18 09:09:09.789286+00:00\n",
      "\n",
      "  Violation 10:\n",
      "    Field: unknown\n",
      "    Error: {'loc': ('unknown',), 'type': 'missing_field', 'msg': \"{'type': 'missing', 'loc': ('metadata',), 'ms...\n",
      "    Error Type: missing_field\n",
      "    Severity: warning\n",
      "    Record ID: 25410\n",
      "    Timestamp: 2025-12-18 09:09:09.875536+00:00\n",
      "\n",
      "✓ Quality check completed and metrics persisted to database\n",
      "\n",
      "✓ Disconnected from metadata store\n"
     ]
    }
   ],
   "source": [
    "## Step 5: Generate Pydantic Model\n",
    "\n",
    "Generate a comprehensive Pydantic model from the reconstructed contract. This model includes all validation rules, coercion rules, and business constraints.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generate Pydantic model from the reconstructed contract\n",
    "AircraftConfigurationModel = get_model_from_contract(\n",
    "    contract=reconstructed_contract,\n",
    "    model_name=\"AircraftConfiguration\"\n",
    ")\n",
    "\n",
    "print(\"=\" * 70)\n",
    "print(\"Step 5: Generate Pydantic Model\")\n",
    "print(\"=\" * 70)\n",
    "print(f\"✓ Generated Pydantic model: {AircraftConfigurationModel.__name__}\")\n",
    "print(f\"\\n  Model Details:\")\n",
    "print(f\"    Model Name: {AircraftConfigurationModel.__name__}\")\n",
    "schema_props = AircraftConfigurationModel.model_json_schema().get('properties', {})\n",
    "print(f\"    Properties: {len(schema_props)}\")\n",
    "required = AircraftConfigurationModel.model_json_schema().get('required', [])\n",
    "print(f\"    Required Fields: {len(required)}\")\n",
    "\n",
    "# Test with a sample record (if available)\n",
    "if sample_aircraft_data:\n",
    "    print(f\"\\n  Testing model with sample record:\")\n",
    "    try:\n",
    "        test_record = sample_aircraft_data[0]\n",
    "        result = AircraftConfigurationModel.model_validate(test_record)\n",
    "        print(f\"    ✓ Sample record validated successfully\")\n",
    "        print(f\"      Registration: {result.registration}\")\n",
    "    except Exception as e:\n",
    "        print(f\"    ⚠ Validation issues found in sample record:\")\n",
    "        print(f\"      - {str(e)[:200]}\")\n",
    "\n",
    "print(f\"\\n✓ Model generation completed - ready for runtime validation\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'$defs': {'AircraftConfigurationAircraft_compartmentItem': {'properties': {},\n",
       "   'title': 'AircraftConfigurationAircraft_compartmentItem',\n",
       "   'type': 'object'},\n",
       "  'AircraftConfigurationAircraft_restrictionItem': {'properties': {},\n",
       "   'title': 'AircraftConfigurationAircraft_restrictionItem',\n",
       "   'type': 'object'},\n",
       "  'AircraftConfigurationMetadata': {'properties': {},\n",
       "   'title': 'AircraftConfigurationMetadata',\n",
       "   'type': 'object'}},\n",
       " 'properties': {'metadata': {'$ref': '#/$defs/AircraftConfigurationMetadata'},\n",
       "  'registration': {'description': 'Registration of the Aircraft',\n",
       "   'examples': ['BLBK', 'BLXN'],\n",
       "   'maxLength': 10,\n",
       "   'title': 'Registration',\n",
       "   'type': 'string'},\n",
       "  'valid_since': {'description': 'Start of validity',\n",
       "   'format': 'date-time',\n",
       "   'title': 'Valid Since',\n",
       "   'type': 'string'},\n",
       "  'valid_until': {'description': 'End of validity',\n",
       "   'format': 'date-time',\n",
       "   'title': 'Valid Until',\n",
       "   'type': 'string'},\n",
       "  'ac_operator': {'description': 'Aircraft Operator',\n",
       "   'examples': ['CX', 'KA'],\n",
       "   'maxLength': 3,\n",
       "   'title': 'Ac Operator',\n",
       "   'type': 'string'},\n",
       "  'ac_owner': {'description': 'Owner of the aircraft',\n",
       "   'examples': ['CX', 'KA'],\n",
       "   'maxLength': 3,\n",
       "   'title': 'Ac Owner',\n",
       "   'type': 'string'},\n",
       "  'ac_subtype': {'description': 'Subtype of the aircraft',\n",
       "   'maxLength': 3,\n",
       "   'title': 'Ac Subtype',\n",
       "   'type': 'string'},\n",
       "  'ac_logical_no': {'description': 'Logical number of the Aircraft',\n",
       "   'title': 'Ac Logical No',\n",
       "   'type': 'integer'},\n",
       "  'ac_state': {'description': \"Status of Aircraft('R'=real and 'O'=overflow)\",\n",
       "   'enum': ['R', 'O'],\n",
       "   'maxLength': 1,\n",
       "   'title': 'Ac State',\n",
       "   'type': 'string'},\n",
       "  'dry_operating_wgt': {'description': 'Dry Operating Weight (kg)',\n",
       "   'title': 'Dry Operating Wgt',\n",
       "   'type': 'integer'},\n",
       "  'max_takeoff_wgt': {'description': 'Max Takeoff Weight (kg)',\n",
       "   'title': 'Max Takeoff Wgt',\n",
       "   'type': 'integer'},\n",
       "  'cargo_capacity': {'description': 'Cargo Capacity (kg)',\n",
       "   'title': 'Cargo Capacity',\n",
       "   'type': 'integer'},\n",
       "  'fuel_capacity': {'description': 'Fuel Capacity (kg)',\n",
       "   'title': 'Fuel Capacity',\n",
       "   'type': 'integer'},\n",
       "  'avg_fuel_consump': {'description': 'Average Fuel Consumption (kg per hour)',\n",
       "   'title': 'Avg Fuel Consump',\n",
       "   'type': 'integer'},\n",
       "  'ac_index': {'description': 'Aircraft index',\n",
       "   'title': 'Ac Index',\n",
       "   'type': 'integer'},\n",
       "  'crewsize_cockpit': {'description': 'Number of Cockpit Crewmembers',\n",
       "   'title': 'Crewsize Cockpit',\n",
       "   'type': 'integer'},\n",
       "  'crewsize_cabin': {'description': 'Number of Cabin Crewmembers',\n",
       "   'title': 'Crewsize Cabin',\n",
       "   'type': 'integer'},\n",
       "  'std_version': {'description': 'Aircraft Standard Version',\n",
       "   'maxLength': 20,\n",
       "   'title': 'Std Version',\n",
       "   'type': 'string'},\n",
       "  'ap_restriction': {'description': \"Airport Restriction - e.g. Political Reason, Performance('Y' or 'N')\",\n",
       "   'examples': ['N', 'Y', 'R'],\n",
       "   'maxLength': 1,\n",
       "   'title': 'Ap Restriction',\n",
       "   'type': 'string'},\n",
       "  'ac_owner_name': {'description': 'Full Name of Owner',\n",
       "   'maxLength': 30,\n",
       "   'title': 'Ac Owner Name',\n",
       "   'type': 'string'},\n",
       "  'ac_subtype_name': {'description': 'Full Name of Subtype',\n",
       "   'maxLength': 30,\n",
       "   'title': 'Ac Subtype Name',\n",
       "   'type': 'string'},\n",
       "  'ac_category': {'description': 'SSIM-Category of Aircraft',\n",
       "   'maxLength': 1,\n",
       "   'title': 'Ac Category',\n",
       "   'type': 'string'},\n",
       "  'remark': {'anyOf': [{'type': 'string'}, {'type': 'null'}],\n",
       "   'default': None,\n",
       "   'description': 'Additional remarks',\n",
       "   'title': 'Remark'},\n",
       "  'fuel_measure_unit': {'description': 'Unit for fuel',\n",
       "   'maxLength': 3,\n",
       "   'title': 'Fuel Measure Unit',\n",
       "   'type': 'string'},\n",
       "  'ac_callsign': {'anyOf': [{'type': 'string'}, {'type': 'null'}],\n",
       "   'default': None,\n",
       "   'description': 'Callsign of aircraft',\n",
       "   'title': 'Ac Callsign'},\n",
       "  'radio': {'anyOf': [{'type': 'string'}, {'type': 'null'}],\n",
       "   'default': None,\n",
       "   'description': 'Radio type',\n",
       "   'title': 'Radio'},\n",
       "  'noise': {'anyOf': [{'type': 'string'}, {'type': 'null'}],\n",
       "   'default': None,\n",
       "   'description': 'ICAO noise chapter',\n",
       "   'title': 'Noise'},\n",
       "  'phone': {'anyOf': [{'type': 'string'}, {'type': 'null'}],\n",
       "   'default': None,\n",
       "   'description': 'Onboard mobile phone number',\n",
       "   'title': 'Phone'},\n",
       "  'special_equipment': {'anyOf': [{'type': 'string'}, {'type': 'null'}],\n",
       "   'default': None,\n",
       "   'description': 'Special equipment',\n",
       "   'title': 'Special Equipment'},\n",
       "  'acars': {'anyOf': [{'enum': ['V', 'H', 'S'], 'type': 'string'},\n",
       "    {'type': 'null'}],\n",
       "   'default': None,\n",
       "   'description': 'ACARS type',\n",
       "   'enum': ['V', 'H', 'S'],\n",
       "   'title': 'Acars'},\n",
       "  'alt_reg': {'anyOf': [{'type': 'string'}, {'type': 'null'}],\n",
       "   'default': None,\n",
       "   'description': 'Alternate Registration',\n",
       "   'title': 'Alt Reg'},\n",
       "  'record_id': {'anyOf': [{'type': 'integer'}, {'type': 'null'}],\n",
       "   'default': 0,\n",
       "   'description': 'Record ID',\n",
       "   'title': 'Record Id'},\n",
       "  'last_update': {'anyOf': [{'type': 'string'}, {'type': 'null'}],\n",
       "   'default': None,\n",
       "   'description': 'Timestamp of last update',\n",
       "   'title': 'Last Update'},\n",
       "  'last_update_user_id': {'anyOf': [{'type': 'string'}, {'type': 'null'}],\n",
       "   'default': None,\n",
       "   'description': 'User ID of last update',\n",
       "   'title': 'Last Update User Id'},\n",
       "  'std_version_alt_1': {'anyOf': [{'type': 'string'}, {'type': 'null'}],\n",
       "   'default': None,\n",
       "   'description': 'Alternative Aircraft Version 1',\n",
       "   'title': 'Std Version Alt 1'},\n",
       "  'std_version_alt_2': {'anyOf': [{'type': 'string'}, {'type': 'null'}],\n",
       "   'default': None,\n",
       "   'description': 'Alternative Aircraft Version 2',\n",
       "   'title': 'Std Version Alt 2'},\n",
       "  'std_version_alt_3': {'anyOf': [{'type': 'string'}, {'type': 'null'}],\n",
       "   'default': None,\n",
       "   'description': 'Alternative Aircraft Version 3',\n",
       "   'title': 'Std Version Alt 3'},\n",
       "  'std_version_alt_4': {'anyOf': [{'type': 'string'}, {'type': 'null'}],\n",
       "   'default': None,\n",
       "   'description': 'Alternative Aircraft Version 4',\n",
       "   'title': 'Std Version Alt 4'},\n",
       "  'ils_equipment': {'anyOf': [{'enum': ['I', 'II', 'IIIa', 'IIIb'],\n",
       "     'maxLength': 4,\n",
       "     'type': 'string'},\n",
       "    {'type': 'null'}],\n",
       "   'default': 'I',\n",
       "   'description': 'aircraft equipment supporting ILS CAT approaches(I, II, IIIa, IIIb)',\n",
       "   'enum': ['I', 'II', 'IIIa', 'IIIb'],\n",
       "   'title': 'Ils Equipment'},\n",
       "  'autoland': {'anyOf': [{'enum': ['Y', 'N'],\n",
       "     'maxLength': 1,\n",
       "     'type': 'string'},\n",
       "    {'type': 'null'}],\n",
       "   'default': 'N',\n",
       "   'description': 'Autoland capability indicator',\n",
       "   'enum': ['Y', 'N'],\n",
       "   'title': 'Autoland'},\n",
       "  'homebase': {'anyOf': [{'type': 'string'}, {'type': 'null'}],\n",
       "   'default': None,\n",
       "   'description': 'Homebase of the aircraft (IATA CODE)',\n",
       "   'title': 'Homebase'},\n",
       "  'aircraft_compartment': {'anyOf': [{'items': {'$ref': '#/$defs/AircraftConfigurationAircraft_compartmentItem'},\n",
       "     'type': 'array'},\n",
       "    {'type': 'null'}],\n",
       "   'default': [],\n",
       "   'description': 'List of aircraft compartments with seat configurations',\n",
       "   'title': 'Aircraft Compartment'},\n",
       "  'aircraft_restriction': {'anyOf': [{'items': {'$ref': '#/$defs/AircraftConfigurationAircraft_restrictionItem'},\n",
       "     'type': 'array'},\n",
       "    {'type': 'null'}],\n",
       "   'default': [],\n",
       "   'description': 'List of aircraft restrictions with constraint codes',\n",
       "   'title': 'Aircraft Restriction'}},\n",
       " 'required': ['metadata',\n",
       "  'registration',\n",
       "  'valid_since',\n",
       "  'valid_until',\n",
       "  'ac_operator',\n",
       "  'ac_owner',\n",
       "  'ac_subtype',\n",
       "  'ac_logical_no',\n",
       "  'ac_state',\n",
       "  'dry_operating_wgt',\n",
       "  'max_takeoff_wgt',\n",
       "  'cargo_capacity',\n",
       "  'fuel_capacity',\n",
       "  'avg_fuel_consump',\n",
       "  'ac_index',\n",
       "  'crewsize_cockpit',\n",
       "  'crewsize_cabin',\n",
       "  'std_version',\n",
       "  'ap_restriction',\n",
       "  'ac_owner_name',\n",
       "  'ac_subtype_name',\n",
       "  'ac_category',\n",
       "  'fuel_measure_unit'],\n",
       " 'title': 'AircraftConfiguration',\n",
       " 'type': 'object'}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View the generated model schema\n",
    "AircraftConfigurationModel.model_json_schema()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 6: Runtime Validation\n",
    "# Validate all data records against the generated model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Validate all records using the generated model\n",
    "print(\"=\" * 70)\n",
    "print(\"Step 6: Runtime Validation - Validate All Data Records\")\n",
    "print(\"=\" * 70)\n",
    "print(f\"✓ Prepared {len(sample_aircraft_data)} records for validation\")\n",
    "print()\n",
    "\n",
    "# Method 1: Validate using generated Pydantic model\n",
    "print(\"Validating data using generated Pydantic model...\")\n",
    "print()\n",
    "\n",
    "validation_results = validate_batch(AircraftConfigurationModel, sample_aircraft_data)\n",
    "\n",
    "valid_count = sum(1 for r in validation_results if r.is_valid)\n",
    "invalid_count = sum(1 for r in validation_results if not r.is_valid)\n",
    "\n",
    "print(f\"  Validation Results:\")\n",
    "print(f\"    Total Records: {len(sample_aircraft_data)}\")\n",
    "print(f\"    Valid Records: {valid_count}\")\n",
    "print(f\"    Invalid Records: {invalid_count}\")\n",
    "print(f\"    Success Rate: {valid_count/len(sample_aircraft_data):.2%}\")\n",
    "print()\n",
    "\n",
    "# Show example invalid records\n",
    "if invalid_count > 0:\n",
    "    print(f\"  Example Invalid Records (showing first 3):\")\n",
    "    print()\n",
    "    invalid_results = [r for r in validation_results if not r.is_valid][:3]\n",
    "    for i, result in enumerate(invalid_results, 1):\n",
    "        print(f\"    Record {i}:\")\n",
    "        for error in result.errors[:2]:  # Show first 2 errors\n",
    "            error_msg = str(error)[:150]\n",
    "            print(f\"      - {error_msg}\")\n",
    "\n",
    "# Method 2: Validate using metadata store (alternative approach)\n",
    "print()\n",
    "print(\"  Alternative: Validate using metadata store...\")\n",
    "store_results = []\n",
    "for record in sample_aircraft_data[:5]:  # Test with first 5 records\n",
    "    try:\n",
    "        result = validate_with_store(\n",
    "            store=store,\n",
    "            schema_id=schema_id,\n",
    "            data=record,\n",
    "            strict=False\n",
    "        )\n",
    "        store_results.append(result)\n",
    "    except Exception as e:\n",
    "        print(f\"    Error validating record: {str(e)[:100]}\")\n",
    "        store_results.append(None)\n",
    "\n",
    "store_valid = sum(1 for r in store_results if r and r.is_valid)\n",
    "print(f\"    Store-based validation: {store_valid}/{len(store_results)} valid\")\n",
    "\n",
    "print()\n",
    "print(\"✓ Runtime validation completed\")\n",
    "print(\"  Note: Validated data is ready for quality assurance checks\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get database session for persisting quality metrics and violations\n",
    "db_session = get_session(connection_string)\n",
    "\n",
    "# Create quality check instance\n",
    "quality_check = QualityCheck(store=store, db_session=db_session)\n",
    "\n",
    "print(\"=\" * 70)\n",
    "print(\"Quality Assurance Check\")\n",
    "print(\"=\" * 70)\n",
    "print(f\"✓ Initialized QualityCheck\")\n",
    "print(f\"✓ Prepared {len(sample_aircraft_data)} records for quality check\")\n",
    "\n",
    "# Run quality check with comprehensive options\n",
    "quality_report = quality_check.run(\n",
    "    schema_id=schema_id,\n",
    "    data=sample_aircraft_data,\n",
    "    options=QualityCheckOptions(\n",
    "        calculate_metrics=True,\n",
    "        record_violations=True,\n",
    "        include_field_metrics=True,\n",
    "        include_profiling=True,\n",
    "        check_thresholds=True,\n",
    "        thresholds=QualityThresholds(\n",
    "            min_overall_score=70.0,\n",
    "            min_accuracy=65.0,\n",
    "            min_completeness=60.0,\n",
    "        ),\n",
    "        data_version=\"1.0.0\",\n",
    "        data_source=\"sample_aircraft_data\",\n",
    "        deduplicate_violations=True,\n",
    "    ),\n",
    ")\n",
    "\n",
    "# Display results\n",
    "print(f\"\\n{'='*70}\")\n",
    "print(\"Quality Check Results\")\n",
    "print(f\"{'='*70}\")\n",
    "print(f\"  Records Checked: {quality_report.record_count}\")\n",
    "print(f\"  Valid Records: {quality_report.valid_count}\")\n",
    "print(f\"  Invalid Records: {quality_report.invalid_count}\")\n",
    "print(f\"  Total Violations: {quality_report.violation_count}\")\n",
    "\n",
    "if quality_report.quality_score:\n",
    "    print(f\"\\n  Quality Metrics:\")\n",
    "    print(f\"    Overall Score: {quality_report.quality_score.overall_score:.2f}/100\")\n",
    "    print(f\"    Accuracy: {quality_report.quality_score.accuracy:.2%}\")\n",
    "    print(f\"    Completeness: {quality_report.quality_score.completeness:.2%}\")\n",
    "    print(f\"    Violation Rate: {quality_report.quality_score.violation_rate:.2%}\")\n",
    "    \n",
    "    if quality_report.threshold_breaches:\n",
    "        print(f\"\\n  ⚠ Threshold Breaches:\")\n",
    "        for breach in quality_report.threshold_breaches[:3]:\n",
    "            print(f\"    - {breach}\")\n",
    "    else:\n",
    "        print(f\"\\n  ✓ All quality thresholds met\")\n",
    "\n",
    "# Show field-level metrics\n",
    "if quality_report.field_metrics:\n",
    "    print(f\"\\n  Field-Level Metrics (Top 3 with issues):\")\n",
    "    sorted_fields = sorted(\n",
    "        quality_report.field_metrics.items(),\n",
    "        key=lambda x: x[1].violation_rate,\n",
    "        reverse=True\n",
    "    )[:3]\n",
    "    for field_name, metrics in sorted_fields:\n",
    "        if metrics.violation_rate > 0:\n",
    "            valid_count = metrics.total_count - metrics.violation_count\n",
    "            print(f\"    {field_name}:\")\n",
    "            print(f\"      Valid: {valid_count}/{metrics.total_count}\")\n",
    "            print(f\"      Violation Rate: {metrics.violation_rate:.2%}\")\n",
    "\n",
    "print(f\"\\n  Status: {'✓ PASSED' if quality_report.passed else '✗ FAILED'}\")\n",
    "print(f\"  Timestamp: {quality_report.check_timestamp}\")\n",
    "\n",
    "# Query violations from database\n",
    "print(f\"\\n{'='*70}\")\n",
    "print(\"Violations in Database\")\n",
    "print(f\"{'='*70}\")\n",
    "from pycharter.quality import ViolationTracker\n",
    "violation_tracker = ViolationTracker(store=store, db_session=db_session)\n",
    "violations = violation_tracker.get_violations(\n",
    "    schema_id=schema_id,\n",
    "    status=\"open\",\n",
    "    limit=5\n",
    ")\n",
    "\n",
    "if violations:\n",
    "    print(f\"  Found {len(violations)} violation records:\")\n",
    "    for i, violation in enumerate(violations[:3], 1):\n",
    "        print(f\"\\n  Violation {i}:\")\n",
    "        print(f\"    Record ID: {violation.record_id}\")\n",
    "        print(f\"    Severity: {violation.severity}\")\n",
    "        if violation.field_violations:\n",
    "            fv = violation.field_violations[0]\n",
    "            print(f\"    Field: {fv.get('field', 'unknown')}\")\n",
    "            error_msg = fv.get('error_message', '')[:80]\n",
    "            print(f\"    Error: {error_msg}...\")\n",
    "else:\n",
    "    print(\"  ✓ No open violations found\")\n",
    "\n",
    "# Cleanup\n",
    "db_session.close()\n",
    "store.disconnect()\n",
    "\n",
    "print(f\"\\n✓ Quality check completed and metrics persisted to database\")\n",
    "print(f\"✓ Disconnected from metadata store\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "This notebook demonstrated the complete PyCharter data journey:\n",
    "\n",
    "1. ✅ **Contract Parsing** - Decomposed contract into components\n",
    "2. ✅ **Metadata Storage** - Stored all components in PostgreSQL with versioning\n",
    "3. ✅ **Contract Reconstruction** - Rebuilt complete contract from stored components\n",
    "4. ✅ **Model Generation** - Created comprehensive Pydantic model with all rules\n",
    "5. ✅ **Runtime Validation** - Validated data records against the contract\n",
    "6. ✅ **Quality Assurance** - Checked data quality, calculated metrics, and tracked violations\n",
    "\n",
    "### Key Takeaways\n",
    "\n",
    "- **Single Source of Truth**: Contracts are stored centrally and can be reconstructed\n",
    "- **Versioning**: All components are versioned and tracked\n",
    "- **Comprehensive Validation**: Data is validated against schema, coercion, and validation rules\n",
    "- **Quality Monitoring**: Quality metrics and violations are tracked in the database\n",
    "- **Production Ready**: All services work together to ensure data quality throughout the pipeline\n",
    "\n",
    "### Next Steps\n",
    "\n",
    "- Explore the API endpoints for programmatic access\n",
    "- Set up automated quality checks in your data pipeline\n",
    "- Create additional contracts for other data domains\n",
    "- Integrate PyCharter into your existing data infrastructure\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (pycharter-dev)",
   "language": "python",
   "name": "pycharter-dev"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
