{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# FMP Stock List Data Extraction\n",
        "\n",
        "This notebook demonstrates how to extract stock list data from the Financial Modeling Prep (FMP) API using PyCharter's ETL Orchestrator.\n",
        "\n",
        "## Overview\n",
        "\n",
        "This notebook shows:\n",
        "1. **Setup** - Configure API credentials and import dependencies\n",
        "2. **Data Extraction** - Extract stock list data from FMP API\n",
        "3. **Data Inspection** - View and analyze extracted data\n",
        "4. **Optional: Transformation** - Apply transformations to the extracted data\n",
        "\n",
        "## Prerequisites\n",
        "\n",
        "Before running this notebook, ensure you have:\n",
        "1. **FMP API Key** - Get your API key from [Financial Modeling Prep](https://financialmodelingprep.com/)\n",
        "2. **PyCharter installed** - `pip install pycharter`\n",
        "3. **Environment variable set** - `FMP_API_KEY` (or set it in the notebook)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Step 1: Setup and Import Dependencies\n",
        "\n",
        "import os\n",
        "import sys\n",
        "import asyncio\n",
        "from pathlib import Path\n",
        "from typing import List, Dict, Any\n",
        "import json\n",
        "\n",
        "# Import PyCharter ETL Orchestrator\n",
        "from pycharter.etl_generator import ETLOrchestrator\n",
        "\n",
        "# Setup paths (assuming notebook is run from repository root or notebooks directory)\n",
        "# If running from notebooks/, go up one level to get to repository root\n",
        "if Path.cwd().name == \"notebooks\":\n",
        "    ROOT_DIR = Path.cwd().parent\n",
        "else:\n",
        "    ROOT_DIR = Path.cwd()\n",
        "\n",
        "DATA_DIR = ROOT_DIR / \"data\"\n",
        "STOCK_EXAMPLES_DIR = DATA_DIR / \"stock_examples\"\n",
        "FMP_STOCK_LIST_DIR = STOCK_EXAMPLES_DIR / \"fmp_stock_list\"\n",
        "\n",
        "print(\"✓ Imports successful\")\n",
        "print(f\"✓ Root directory: {ROOT_DIR}\")\n",
        "print(f\"✓ Contract directory: {FMP_STOCK_LIST_DIR}\")\n",
        "print(f\"✓ Directory exists: {FMP_STOCK_LIST_DIR.exists()}\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 2: Configure API Key\n",
        "\n",
        "Set your FMP API key. You can either:\n",
        "- Set it as an environment variable: `export FMP_API_KEY=your_key_here`\n",
        "- Or set it directly in this notebook (not recommended for production)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Step 2: Configure API Key\n",
        "\n",
        "# Option 1: Get from environment variable (recommended)\n",
        "FMP_API_KEY = os.getenv(\"FMP_API_KEY\")\n",
        "\n",
        "# Option 2: Set directly (for testing only - remove before committing!)\n",
        "# FMP_API_KEY = \"your_api_key_here\"\n",
        "\n",
        "if not FMP_API_KEY:\n",
        "    print(\"⚠ Warning: FMP_API_KEY not set!\")\n",
        "    print(\"Please set it as an environment variable:\")\n",
        "    print(\"  export FMP_API_KEY=your_key_here\")\n",
        "    print(\"\\nOr set it directly in the cell above (for testing only)\")\n",
        "else:\n",
        "    print(f\"✓ API Key configured (length: {len(FMP_API_KEY)} characters)\")\n",
        "    # Set it in the environment for this session\n",
        "    os.environ[\"FMP_API_KEY\"] = FMP_API_KEY\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 3: Initialize ETL Orchestrator\n",
        "\n",
        "The ETL Orchestrator reads the configuration files from the `fmp_stock_list` directory:\n",
        "- `extract.yaml` - API endpoint, parameters, rate limiting\n",
        "- `transform.yaml` - Field renaming and transformations\n",
        "- `schema.yaml` - Data schema definition\n",
        "- `load.yaml` - Database loading configuration (optional for extraction-only)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Step 3: Initialize ETL Orchestrator\n",
        "\n",
        "# Initialize orchestrator with contract directory\n",
        "orchestrator = ETLOrchestrator(\n",
        "    contract_dir=str(FMP_STOCK_LIST_DIR)\n",
        ")\n",
        "\n",
        "print(\"✓ ETL Orchestrator initialized\")\n",
        "print(f\"✓ Extract config loaded: {orchestrator.extract_config is not None}\")\n",
        "print(f\"✓ Transform config loaded: {orchestrator.transform_config is not None}\")\n",
        "print(f\"✓ Schema loaded: {orchestrator.schema is not None}\")\n",
        "\n",
        "# Display extract configuration\n",
        "if orchestrator.extract_config:\n",
        "    print(\"\\nExtract Configuration:\")\n",
        "    print(f\"  Provider: {orchestrator.extract_config.get('provider_name')}\")\n",
        "    print(f\"  Endpoint: {orchestrator.extract_config.get('api_endpoint')}\")\n",
        "    print(f\"  Method: {orchestrator.extract_config.get('method')}\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 4: Extract Data from FMP API\n",
        "\n",
        "Now we'll extract stock list data from the FMP API. The orchestrator will:\n",
        "1. Make HTTP request to the configured endpoint\n",
        "2. Handle rate limiting and retries\n",
        "3. Parse the JSON response\n",
        "4. Return the raw data\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Step 4: Extract Data from FMP API\n",
        "\n",
        "async def extract_data():\n",
        "    \"\"\"Extract data from FMP API.\"\"\"\n",
        "    print(\"🔄 Starting data extraction...\")\n",
        "    print(f\"   Endpoint: {orchestrator.extract_config.get('api_endpoint')}\")\n",
        "    print(f\"   Method: {orchestrator.extract_config.get('method')}\")\n",
        "    \n",
        "    try:\n",
        "        # Extract data using the orchestrator\n",
        "        raw_data = await orchestrator.extract()\n",
        "        \n",
        "        print(f\"\\n✓ Extraction successful!\")\n",
        "        print(f\"   Records extracted: {len(raw_data)}\")\n",
        "        \n",
        "        return raw_data\n",
        "    except Exception as e:\n",
        "        print(f\"\\n❌ Extraction failed: {e}\")\n",
        "        import traceback\n",
        "        traceback.print_exc()\n",
        "        return []\n",
        "\n",
        "# Run extraction\n",
        "raw_data = await extract_data()\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 5: Inspect Extracted Data\n",
        "\n",
        "Let's examine the structure and content of the extracted data.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Step 5: Inspect Extracted Data\n",
        "\n",
        "if raw_data:\n",
        "    print(f\"Total records: {len(raw_data)}\\n\")\n",
        "    \n",
        "    # Show first record structure\n",
        "    if len(raw_data) > 0:\n",
        "        print(\"First record structure:\")\n",
        "        print(\"=\" * 60)\n",
        "        first_record = raw_data[0]\n",
        "        print(json.dumps(first_record, indent=2))\n",
        "        print(\"=\" * 60)\n",
        "        \n",
        "        # Show all field names\n",
        "        print(f\"\\nField names ({len(first_record)} fields):\")\n",
        "        for field in sorted(first_record.keys()):\n",
        "            value = first_record[field]\n",
        "            value_type = type(value).__name__\n",
        "            value_preview = str(value)[:50] if value is not None else \"None\"\n",
        "            print(f\"  - {field}: {value_type} = {value_preview}\")\n",
        "        \n",
        "        # Show sample records\n",
        "        print(f\"\\n\\nSample records (first 5):\")\n",
        "        print(\"=\" * 60)\n",
        "        for i, record in enumerate(raw_data[:5], 1):\n",
        "            symbol = record.get('symbol', 'N/A')\n",
        "            name = record.get('name', record.get('companyName', 'N/A'))\n",
        "            print(f\"{i}. {symbol}: {name}\")\n",
        "else:\n",
        "    print(\"⚠ No data extracted. Please check:\")\n",
        "    print(\"  1. API key is correctly set\")\n",
        "    print(\"  2. Network connection is available\")\n",
        "    print(\"  3. FMP API endpoint is accessible\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 6: Apply Transformations (Optional)\n",
        "\n",
        "If you want to transform the data according to the `transform.yaml` configuration, you can use the orchestrator's transform method. This will:\n",
        "- Rename fields (e.g., `companyName` → `company_name`)\n",
        "- Apply type conversions\n",
        "- Fill null values with defaults\n",
        "- Drop unwanted fields\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Step 6: Apply Transformations (Optional)\n",
        "\n",
        "if raw_data and orchestrator.transform_config:\n",
        "    print(\"🔄 Applying transformations...\")\n",
        "    \n",
        "    # Transform the data\n",
        "    transformed_data = orchestrator.transform(raw_data)\n",
        "    \n",
        "    print(f\"✓ Transformation complete!\")\n",
        "    print(f\"   Original records: {len(raw_data)}\")\n",
        "    print(f\"   Transformed records: {len(transformed_data)}\")\n",
        "    \n",
        "    # Show first transformed record\n",
        "    if len(transformed_data) > 0:\n",
        "        print(\"\\nFirst transformed record:\")\n",
        "        print(\"=\" * 60)\n",
        "        print(json.dumps(transformed_data[0], indent=2))\n",
        "        print(\"=\" * 60)\n",
        "        \n",
        "        # Compare field names\n",
        "        if len(raw_data) > 0:\n",
        "            print(\"\\nField name changes:\")\n",
        "            original_fields = set(raw_data[0].keys())\n",
        "            transformed_fields = set(transformed_data[0].keys())\n",
        "            \n",
        "            renamed = original_fields - transformed_fields\n",
        "            new_fields = transformed_fields - original_fields\n",
        "            common = original_fields & transformed_fields\n",
        "            \n",
        "            if renamed:\n",
        "                print(f\"  Renamed fields ({len(renamed)}): {sorted(renamed)}\")\n",
        "            if new_fields:\n",
        "                print(f\"  New fields ({len(new_fields)}): {sorted(new_fields)}\")\n",
        "            if common:\n",
        "                print(f\"  Unchanged fields ({len(common)}): {sorted(common)}\")\n",
        "else:\n",
        "    print(\"⚠ No data to transform or transform config not available\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 7: Save Extracted Data (Optional)\n",
        "\n",
        "You can save the extracted data to a JSON file for later use or analysis.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Step 7: Save Extracted Data (Optional)\n",
        "\n",
        "if raw_data:\n",
        "    output_file = ROOT_DIR / \"notebooks\" / \"fmp_stock_list_data.json\"\n",
        "    \n",
        "    with open(output_file, 'w', encoding='utf-8') as f:\n",
        "        json.dump(raw_data, f, indent=2, ensure_ascii=False)\n",
        "    \n",
        "    print(f\"✓ Data saved to: {output_file}\")\n",
        "    print(f\"   File size: {output_file.stat().st_size / 1024:.2f} KB\")\n",
        "    print(f\"   Records: {len(raw_data)}\")\n",
        "else:\n",
        "    print(\"⚠ No data to save\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Summary\n",
        "\n",
        "This notebook demonstrated:\n",
        "- ✅ Setting up the ETL Orchestrator with FMP Stock List configuration\n",
        "- ✅ Extracting data from the FMP API\n",
        "- ✅ Inspecting and analyzing the extracted data\n",
        "- ✅ Applying transformations (optional)\n",
        "- ✅ Saving data to JSON file (optional)\n",
        "\n",
        "## Next Steps\n",
        "\n",
        "1. **Full ETL Pipeline**: Run the complete pipeline with `await orchestrator.run()` to extract, transform, and load data\n",
        "2. **Data Validation**: Use PyCharter's validation features to ensure data quality\n",
        "3. **Quality Checks**: Run quality assurance checks on the extracted data\n",
        "4. **Schedule Extraction**: Set up scheduled extractions using Airflow, Prefect, or similar tools\n",
        "\n",
        "For more information, see:\n",
        "- `data/stock_examples/fmp_stock_list/README.md` - Configuration details\n",
        "- `data/stock_examples/fmp_stock_list/USAGE.md` - Usage examples\n"
      ]
    }
  ],
  "metadata": {
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}
