{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Testing ETL Orchestrator Extraction Method\n",
        "\n",
        "This notebook demonstrates how the `ETLOrchestrator.extract()` method works with input parameters.\n",
        "\n",
        "## Overview\n",
        "\n",
        "This notebook shows:\n",
        "1. **Setup** - Configure API credentials and initialize orchestrator\n",
        "2. **Basic Extraction** - Extract data without input parameters\n",
        "3. **Extraction with Input Parameters** - Use input_params from extract.yaml as method arguments\n",
        "4. **Parameter Override** - Show how input_params override config defaults\n",
        "5. **Multiple Parameters** - Demonstrate using multiple input parameters\n",
        "\n",
        "## Prerequisites\n",
        "\n",
        "Before running this notebook, ensure you have:\n",
        "1. **FMP API Key** - Get your API key from [Financial Modeling Prep](https://financialmodelingprep.com/)\n",
        "2. **PyCharter installed** - `pip install pycharter`\n",
        "3. **Environment variable set** - `FMP_API_KEY` (or set it in the notebook)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "✓ Imports successful\n",
            "✓ Root directory: /Users/IMTAUC/akc/votedrop/pycharter\n",
            "✓ Contract directory: /Users/IMTAUC/akc/votedrop/pycharter/data/stock_examples/fmp_stock_list\n",
            "✓ Directory exists: True\n"
          ]
        }
      ],
      "source": [
        "# Step 1: Setup and Import Dependencies\n",
        "\n",
        "import os\n",
        "import sys\n",
        "import asyncio\n",
        "from pathlib import Path\n",
        "from typing import List, Dict, Any\n",
        "import json\n",
        "\n",
        "# Import PyCharter ETL Orchestrator\n",
        "from pycharter.etl_generator import ETLOrchestrator\n",
        "\n",
        "# Setup paths (assuming notebook is run from repository root or notebooks directory)\n",
        "if Path.cwd().name == \"notebooks\":\n",
        "    ROOT_DIR = Path.cwd().parent\n",
        "else:\n",
        "    ROOT_DIR = Path.cwd()\n",
        "\n",
        "DATA_DIR = ROOT_DIR / \"data\"\n",
        "STOCK_EXAMPLES_DIR = DATA_DIR / \"stock_examples\"\n",
        "FMP_STOCK_LIST_DIR = STOCK_EXAMPLES_DIR / \"fmp_stock_list\"\n",
        "\n",
        "print(\"✓ Imports successful\")\n",
        "print(f\"✓ Root directory: {ROOT_DIR}\")\n",
        "print(f\"✓ Contract directory: {FMP_STOCK_LIST_DIR}\")\n",
        "print(f\"✓ Directory exists: {FMP_STOCK_LIST_DIR.exists()}\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 2: Configure API Key\n",
        "\n",
        "Set your FMP API key. You can either:\n",
        "- Set it as an environment variable: `export FMP_API_KEY=your_key_here`\n",
        "- Or set it directly in this notebook (for testing only)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "⚠ Using default API key for testing\n",
            "   For production, set it as an environment variable:\n",
            "   export FMP_API_KEY=your_api_key_here\n",
            "✓ API Key configured and set in environment\n"
          ]
        }
      ],
      "source": [
        "# Step 2: Configure API Key\n",
        "\n",
        "# Option 1: Get from environment variable (recommended)\n",
        "FMP_API_KEY = os.getenv(\"FMP_API_KEY\")\n",
        "\n",
        "# Option 2: Set directly (for testing only - remove before committing!)\n",
        "# If not set in environment, use this default for testing\n",
        "if not FMP_API_KEY:\n",
        "    FMP_API_KEY = \"qskVPeuMpFr4dGW3WOqaOltkU8eNgrjb\"\n",
        "    print(\"⚠ Using default API key for testing\")\n",
        "    print(\"   For production, set it as an environment variable:\")\n",
        "    print(\"   export FMP_API_KEY=your_api_key_here\")\n",
        "else:\n",
        "    print(f\"✓ API Key loaded from environment (length: {len(FMP_API_KEY)} characters)\")\n",
        "\n",
        "# Always set it in the environment for this session\n",
        "# This ensures the orchestrator can access it when resolving variables\n",
        "os.environ[\"FMP_API_KEY\"] = FMP_API_KEY\n",
        "print(f\"✓ API Key configured and set in environment\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 3: Initialize ETL Orchestrator\n",
        "\n",
        "The ETL Orchestrator reads the configuration files from the `fmp_stock_list` directory:\n",
        "- `extract.yaml` - API endpoint, parameters, rate limiting, and **input_params**\n",
        "- `transform.yaml` - Field renaming and transformations\n",
        "- `schema.yaml` - Data schema definition\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "✓ ETL Orchestrator initialized\n",
            "✓ Extract config loaded: True\n",
            "✓ Transform config loaded: True\n",
            "✓ Schema loaded: True\n",
            "\n",
            "Extract Configuration:\n",
            "  Provider: fmp\n",
            "  Base URL: https://financialmodelingprep.com\n",
            "  Endpoint: /stable/stock-list\n",
            "  Method: GET\n",
            "\n",
            "Input Parameters (from extract.yaml):\n",
            "  Available input params: ['exchange', 'limit']\n",
            "    - exchange: no metadata\n",
            "    - limit: no metadata\n"
          ]
        }
      ],
      "source": [
        "# Step 3: Initialize ETL Orchestrator\n",
        "\n",
        "# Initialize orchestrator with contract directory\n",
        "orchestrator = ETLOrchestrator(\n",
        "    contract_dir=str(FMP_STOCK_LIST_DIR)\n",
        ")\n",
        "\n",
        "print(\"✓ ETL Orchestrator initialized\")\n",
        "print(f\"✓ Extract config loaded: {orchestrator.extract_config is not None}\")\n",
        "print(f\"✓ Transform config loaded: {orchestrator.transform_config is not None}\")\n",
        "print(f\"✓ Schema loaded: {orchestrator.schema is not None}\")\n",
        "\n",
        "# Display extract configuration\n",
        "if orchestrator.extract_config:\n",
        "    print(\"\\nExtract Configuration:\")\n",
        "    print(f\"  Provider: {orchestrator.extract_config.get('provider_name')}\")\n",
        "    print(f\"  Base URL: {orchestrator.extract_config.get('base_url')}\")\n",
        "    print(f\"  Endpoint: {orchestrator.extract_config.get('api_endpoint')}\")\n",
        "    print(f\"  Method: {orchestrator.extract_config.get('method')}\")\n",
        "\n",
        "# Display input parameters\n",
        "print(f\"\\nInput Parameters (from extract.yaml):\")\n",
        "print(f\"  Available input params: {list(orchestrator.input_params.keys())}\")\n",
        "for param_name, param_meta in orchestrator.input_params.items():\n",
        "    print(f\"    - {param_name}: {param_meta if param_meta else 'no metadata'}\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 4: Basic Extraction (Without Input Parameters)\n",
        "\n",
        "First, let's extract data without providing any input parameters. This will use default values from the config.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "🔄 Starting basic extraction (no input params)...\n",
            "\n",
            "❌ Extraction failed: Extraction failed: '>' not supported between instances of 'str' and 'int'\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Traceback (most recent call last):\n",
            "  File \"/Users/IMTAUC/akc/votedrop/pycharter/pycharter/etl_generator/orchestrator.py\", line 345, in extract\n",
            "    else:\n",
            "       ^^^\n",
            "TypeError: '>' not supported between instances of 'str' and 'int'\n",
            "\n",
            "The above exception was the direct cause of the following exception:\n",
            "\n",
            "Traceback (most recent call last):\n",
            "  File \"/var/folders/6g/rp584cvj7kj2yr1n_z23vm000000gp/T/ipykernel_58469/3416380349.py\", line 9, in basic_extract\n",
            "    raw_data = await orchestrator.extract()\n",
            "               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "  File \"/Users/IMTAUC/akc/votedrop/pycharter/pycharter/etl_generator/orchestrator.py\", line 365, in extract\n",
            "    wait_time = backoff_factor ** attempt\n",
            "RuntimeError: Extraction failed: '>' not supported between instances of 'str' and 'int'\n"
          ]
        }
      ],
      "source": [
        "# Step 4: Basic Extraction (Without Input Parameters)\n",
        "\n",
        "async def basic_extract():\n",
        "    \"\"\"Extract data without input parameters.\"\"\"\n",
        "    print(\"🔄 Starting basic extraction (no input params)...\")\n",
        "    \n",
        "    try:\n",
        "        # Extract data without any input arguments\n",
        "        raw_data = await orchestrator.extract()\n",
        "        \n",
        "        print(f\"\\n✓ Extraction successful!\")\n",
        "        print(f\"   Records extracted: {len(raw_data)}\")\n",
        "        \n",
        "        if raw_data:\n",
        "            print(f\"\\n   Sample records (first 3):\")\n",
        "            for i, record in enumerate(raw_data[:3], 1):\n",
        "                symbol = record.get('symbol', 'N/A')\n",
        "                name = record.get('name', record.get('companyName', 'N/A'))\n",
        "                print(f\"     {i}. {symbol}: {name}\")\n",
        "        \n",
        "        return raw_data\n",
        "    except Exception as e:\n",
        "        print(f\"\\n❌ Extraction failed: {e}\")\n",
        "        import traceback\n",
        "        traceback.print_exc()\n",
        "        return []\n",
        "\n",
        "# Run basic extraction\n",
        "basic_data = await basic_extract()\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 5: Extraction with Input Parameters\n",
        "\n",
        "Now let's use the `input_params` defined in `extract.yaml` as method arguments. The `fmp_stock_list` extract.yaml defines:\n",
        "- `exchange` - Filter by exchange (NASDAQ, NYSE, etc.)\n",
        "- `limit` - Limit number of results\n",
        "\n",
        "These can be passed directly to `extract()` as keyword arguments!\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "🔄 Starting extraction with input parameters...\n",
            "   Calling: extract(exchange='NASDAQ', limit=5)\n",
            "\n",
            "❌ Extraction failed: Extraction failed: '>' not supported between instances of 'str' and 'int'\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Traceback (most recent call last):\n",
            "  File \"/Users/IMTAUC/akc/votedrop/pycharter/pycharter/etl_generator/orchestrator.py\", line 345, in extract\n",
            "    else:\n",
            "       ^^^\n",
            "TypeError: '>' not supported between instances of 'str' and 'int'\n",
            "\n",
            "The above exception was the direct cause of the following exception:\n",
            "\n",
            "Traceback (most recent call last):\n",
            "  File \"/var/folders/6g/rp584cvj7kj2yr1n_z23vm000000gp/T/ipykernel_58469/2004802379.py\", line 11, in extract_with_params\n",
            "    raw_data = await orchestrator.extract(exchange='NASDAQ', limit=5)\n",
            "               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "  File \"/Users/IMTAUC/akc/votedrop/pycharter/pycharter/etl_generator/orchestrator.py\", line 365, in extract\n",
            "    wait_time = backoff_factor ** attempt\n",
            "RuntimeError: Extraction failed: '>' not supported between instances of 'str' and 'int'\n"
          ]
        }
      ],
      "source": [
        "# Step 5: Extraction with Input Parameters\n",
        "\n",
        "async def extract_with_params():\n",
        "    \"\"\"Extract data with input parameters.\"\"\"\n",
        "    print(\"🔄 Starting extraction with input parameters...\")\n",
        "    print(\"   Calling: extract(exchange='NASDAQ', limit=5)\")\n",
        "    \n",
        "    try:\n",
        "        # Extract data with input arguments\n",
        "        # These parameters are defined in extract.yaml's input_params section\n",
        "        raw_data = await orchestrator.extract(exchange='NASDAQ', limit=5)\n",
        "        \n",
        "        print(f\"\\n✓ Extraction successful!\")\n",
        "        print(f\"   Records extracted: {len(raw_data)}\")\n",
        "        print(f\"   ✓ Input parameters were applied correctly!\")\n",
        "        \n",
        "        if raw_data:\n",
        "            print(f\"\\n   Sample records (first 5):\")\n",
        "            for i, record in enumerate(raw_data[:5], 1):\n",
        "                symbol = record.get('symbol', 'N/A')\n",
        "                name = record.get('name', record.get('companyName', 'N/A'))\n",
        "                exchange = record.get('exchange', 'N/A')\n",
        "                print(f\"     {i}. {symbol} ({exchange}): {name}\")\n",
        "        \n",
        "        return raw_data\n",
        "    except Exception as e:\n",
        "        print(f\"\\n❌ Extraction failed: {e}\")\n",
        "        import traceback\n",
        "        traceback.print_exc()\n",
        "        return []\n",
        "\n",
        "# Run extraction with input parameters\n",
        "param_data = await extract_with_params()\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 6: Testing Parameter Override\n",
        "\n",
        "Let's test that input parameters correctly override any default values from the config. We'll extract with different limit values to show the override behavior.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Step 6: Testing Parameter Override\n",
        "\n",
        "async def test_parameter_override():\n",
        "    \"\"\"Test that input parameters override config defaults.\"\"\"\n",
        "    print(\"🔄 Testing parameter override behavior...\")\n",
        "    \n",
        "    test_cases = [\n",
        "        {\"limit\": 3, \"description\": \"Limit to 3 records\"},\n",
        "        {\"limit\": 10, \"description\": \"Limit to 10 records\"},\n",
        "        {\"exchange\": \"NYSE\", \"limit\": 5, \"description\": \"NYSE exchange, limit 5\"},\n",
        "    ]\n",
        "    \n",
        "    results = []\n",
        "    for i, test_case in enumerate(test_cases, 1):\n",
        "        limit = test_case.get(\"limit\")\n",
        "        exchange = test_case.get(\"exchange\")\n",
        "        description = test_case[\"description\"]\n",
        "        \n",
        "        print(f\"\\n   Test {i}: {description}\")\n",
        "        try:\n",
        "            if exchange:\n",
        "                raw_data = await orchestrator.extract(exchange=exchange, limit=limit)\n",
        "            else:\n",
        "                raw_data = await orchestrator.extract(limit=limit)\n",
        "            \n",
        "            print(f\"     ✓ Extracted {len(raw_data)} records\")\n",
        "            results.append((description, len(raw_data)))\n",
        "        except Exception as e:\n",
        "            print(f\"     ✗ Failed: {e}\")\n",
        "            results.append((description, None))\n",
        "    \n",
        "    print(f\"\\n✓ Parameter override tests completed!\")\n",
        "    print(f\"\\n   Summary:\")\n",
        "    for desc, count in results:\n",
        "        status = f\"{count} records\" if count is not None else \"FAILED\"\n",
        "        print(f\"     - {desc}: {status}\")\n",
        "    \n",
        "    return results\n",
        "\n",
        "# Run parameter override tests\n",
        "override_results = await test_parameter_override()\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 7: Apply Transformations\n",
        "\n",
        "Now let's apply transformations to the extracted data using the `transform()` method. This will:\n",
        "- Rename fields according to transform.yaml rules\n",
        "- Apply type conversions\n",
        "- Fill null values with defaults\n",
        "- Drop unwanted fields\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Step 7: Apply Transformations\n",
        "\n",
        "if param_data and orchestrator.transform_config:\n",
        "    print(\"🔄 Applying transformations...\")\n",
        "    print(f\"   Original records: {len(param_data)}\")\n",
        "    \n",
        "    # Transform the data\n",
        "    transformed_data = orchestrator.transform(param_data)\n",
        "    \n",
        "    print(f\"\\n✓ Transformation complete!\")\n",
        "    print(f\"   Transformed records: {len(transformed_data)}\")\n",
        "    \n",
        "    # Show first transformed record\n",
        "    if len(transformed_data) > 0:\n",
        "        print(f\"\\n   First transformed record:\")\n",
        "        print(\"=\" * 60)\n",
        "        import json\n",
        "        print(json.dumps(transformed_data[0], indent=2))\n",
        "        print(\"=\" * 60)\n",
        "        \n",
        "        # Compare field names\n",
        "        if len(param_data) > 0:\n",
        "            print(f\"\\n   Field name comparison:\")\n",
        "            original_fields = set(param_data[0].keys())\n",
        "            transformed_fields = set(transformed_data[0].keys())\n",
        "            \n",
        "            renamed = original_fields - transformed_fields\n",
        "            new_fields = transformed_fields - original_fields\n",
        "            common = original_fields & transformed_fields\n",
        "            \n",
        "            if renamed:\n",
        "                print(f\"     Renamed fields ({len(renamed)}): {sorted(renamed)}\")\n",
        "            if new_fields:\n",
        "                print(f\"     New fields ({len(new_fields)}): {sorted(new_fields)}\")\n",
        "            if common:\n",
        "                print(f\"     Unchanged fields ({len(common)}): {sorted(list(common)[:10])}...\")\n",
        "    else:\n",
        "        print(\"   ⚠ No transformed data\")\n",
        "else:\n",
        "    print(\"⚠ No data to transform or transform config not available\")\n",
        "    if not param_data:\n",
        "        print(\"   - No extracted data available\")\n",
        "    if not orchestrator.transform_config:\n",
        "        print(\"   - Transform config not loaded\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Summary\n",
        "\n",
        "This notebook demonstrated:\n",
        "- ✅ Initializing ETL Orchestrator with extract.yaml configuration\n",
        "- ✅ Understanding input_params defined in extract.yaml\n",
        "- ✅ Basic extraction without input parameters\n",
        "- ✅ Extraction with input parameters as method arguments\n",
        "- ✅ Parameter override behavior (input_params override config defaults)\n",
        "- ✅ Using multiple input parameters together\n",
        "- ✅ Applying transformations to extracted data\n",
        "\n",
        "## Key Features\n",
        "\n",
        "1. **Input Parameters**: Parameters defined in `extract.yaml`'s `input_params` section become method arguments\n",
        "2. **Parameter Override**: Input arguments take precedence over config defaults\n",
        "3. **Type Safety**: Input parameters can have metadata (type, required, description)\n",
        "4. **Flexibility**: Supports both simple list format and detailed dict format for input_params\n",
        "5. **Transformations**: Apply field renaming, type conversions, and data cleaning\n",
        "\n",
        "## Next Steps\n",
        "\n",
        "1. **Full ETL Pipeline**: Run the complete pipeline with `await orchestrator.run()` to extract, transform, and load data\n",
        "2. **Data Validation**: Use PyCharter's validation features to ensure data quality\n",
        "3. **Quality Checks**: Run quality assurance checks on the extracted data\n",
        "4. **Schedule Extraction**: Set up scheduled extractions using Airflow, Prefect, or similar tools\n",
        "\n",
        "For more information, see:\n",
        "- `data/stock_examples/fmp_stock_list/extract.yaml` - Example extract configuration\n",
        "- `data/stock_examples/fmp_stock_list/transform.yaml` - Example transform configuration\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 7: Testing with Another Dataset (fmp_analyst)\n",
        "\n",
        "Let's test with a different dataset that has more input parameters. The `fmp_analyst` extract.yaml defines:\n",
        "- `symbol` (required) - Stock symbol\n",
        "- `period` (optional) - Period (annual, quarter)\n",
        "- `limit` (optional) - Limit number of results\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Step 7: Testing with Another Dataset (fmp_analyst)\n",
        "\n",
        "FMP_ANALYST_DIR = STOCK_EXAMPLES_DIR / \"fmp_analyst\"\n",
        "\n",
        "# Initialize orchestrator for analyst data\n",
        "analyst_orchestrator = ETLOrchestrator(\n",
        "    contract_dir=str(FMP_ANALYST_DIR)\n",
        ")\n",
        "\n",
        "print(\"✓ Analyst Orchestrator initialized\")\n",
        "print(f\"  Input params: {list(analyst_orchestrator.input_params.keys())}\")\n",
        "\n",
        "# Show input params with metadata\n",
        "print(f\"\\n  Input Parameters with metadata:\")\n",
        "for param_name, param_meta in analyst_orchestrator.input_params.items():\n",
        "    required = param_meta.get('required', False)\n",
        "    param_type = param_meta.get('type', 'any')\n",
        "    description = param_meta.get('description', '')\n",
        "    print(f\"    - {param_name}:\")\n",
        "    print(f\"        Type: {param_type}\")\n",
        "    print(f\"        Required: {required}\")\n",
        "    print(f\"        Description: {description}\")\n",
        "\n",
        "async def extract_analyst_data():\n",
        "    \"\"\"Extract analyst data with required and optional parameters.\"\"\"\n",
        "    print(\"\\n🔄 Extracting analyst data...\")\n",
        "    print(\"   Calling: extract(symbol='AAPL', period='annual', limit=5)\")\n",
        "    \n",
        "    try:\n",
        "        # Extract analyst data with required symbol and optional parameters\n",
        "        raw_data = await analyst_orchestrator.extract(\n",
        "            symbol='AAPL',\n",
        "            period='annual',\n",
        "            limit=5\n",
        "        )\n",
        "        \n",
        "        print(f\"\\n✓ Extraction successful!\")\n",
        "        print(f\"   Records extracted: {len(raw_data)}\")\n",
        "        \n",
        "        if raw_data:\n",
        "            print(f\"\\n   Sample records (first 3):\")\n",
        "            for i, record in enumerate(raw_data[:3], 1):\n",
        "                symbol = record.get('symbol', 'N/A')\n",
        "                date = record.get('date', 'N/A')\n",
        "                revenue_avg = record.get('revenueAvg', 'N/A')\n",
        "                print(f\"     {i}. {symbol} ({date}): Revenue Avg = {revenue_avg}\")\n",
        "        \n",
        "        return raw_data\n",
        "    except Exception as e:\n",
        "        print(f\"\\n❌ Extraction failed: {e}\")\n",
        "        import traceback\n",
        "        traceback.print_exc()\n",
        "        return []\n",
        "\n",
        "# Run analyst extraction\n",
        "analyst_data = await extract_analyst_data()\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Summary\n",
        "\n",
        "This notebook demonstrated:\n",
        "- ✅ Initializing ETL Orchestrator with extract.yaml configuration\n",
        "- ✅ Understanding input_params defined in extract.yaml\n",
        "- ✅ Basic extraction without input parameters\n",
        "- ✅ Extraction with input parameters as method arguments\n",
        "- ✅ Parameter override behavior (input_params override config defaults)\n",
        "- ✅ Using multiple input parameters together\n",
        "- ✅ Working with required vs optional input parameters\n",
        "\n",
        "## Key Features\n",
        "\n",
        "1. **Input Parameters**: Parameters defined in `extract.yaml`'s `input_params` section become method arguments\n",
        "2. **Parameter Override**: Input arguments take precedence over config defaults\n",
        "3. **Type Safety**: Input parameters can have metadata (type, required, description)\n",
        "4. **Flexibility**: Supports both simple list format and detailed dict format for input_params\n",
        "\n",
        "## Next Steps\n",
        "\n",
        "1. **Full ETL Pipeline**: Run the complete pipeline with `await orchestrator.run()` to extract, transform, and load data\n",
        "2. **Data Validation**: Use PyCharter's validation features to ensure data quality\n",
        "3. **Quality Checks**: Run quality assurance checks on the extracted data\n",
        "4. **Schedule Extraction**: Set up scheduled extractions using Airflow, Prefect, or similar tools\n",
        "\n",
        "For more information, see:\n",
        "- `data/stock_examples/fmp_stock_list/extract.yaml` - Example extract configuration\n",
        "- `data/stock_examples/fmp_analyst/extract.yaml` - Example with detailed input_params metadata\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python (pycharter-dev)",
      "language": "python",
      "name": "pycharter-dev"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12.7"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}
