{
  "cells": [
    {
      "id": "intro-md",
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Transform raw data into MATER-ready data\n",
        "\n",
        "**Source :** `{{ dataset_path }}`\n",
        "\n",
        "> **💡 Tip:** Select the `.venv` Python environment in the kernel selector (top right). This ensures you're using the project's virtual environment with all required dependencies."
      ]
    },
    {
      "id": "title-md",
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Imports"
      ]
    },
    {
      "id": "imports-code",
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import json\n",
        "import sys\n",
        "from pathlib import Path\n",
        "import pandas as pd\n",
        "\n",
        "from src.settings import MaterConfig\n",
        "from src.lib.validation import validate_mater_ready_data\n",
        "\n",
        "# Add project root to path\n",
        "current = Path.cwd()\n",
        "project_root = current\n",
        "while project_root != project_root.parent:\n",
        "    if (project_root / \"pyproject.toml\").exists():\n",
        "        break\n",
        "    project_root = project_root.parent\n",
        "\n",
        "if str(project_root) not in sys.path:\n",
        "    sys.path.insert(0, str(project_root))"
      ]
    },
    {
      "id": "load-md",
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Load raw data\n",
        "\n",
        "*Adjust as needed.*\n"
      ]
    },
    {
      "id": "load-code",
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "def load_data():\n",
        "    \"\"\"Load data from source file\"\"\"\n",
        "# PLACEHOLDER_LOADING_CODE\n",
        "    return df\n",
        "\n",
        "\n",
        "df = load_data()\n",
        "display(df)"
      ]
    },
    {
      "id": "transform-md",
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Transform to MATER format\n",
        "\n",
        "Each record must contain:\n",
        "\n",
        "- **time** (int): Temporal interval (e.g., year)\n",
        "- **value** (float): Quantitative data point\n",
        "- **scenario** (str): Scenario identifier (e.g., \"historical\")\n",
        "- **unit** (str): Unit of measurement\n",
        "- **variable** (str): MATER variable name (see `uv run mater-cli simulation list-variables`)\n",
        "- **dimensions_values** (dict): Dimension key-value pairs specific to the variable\n",
        "\n",
        "**Example:**\n",
        "\n",
        "```json\n",
        "{\n",
        "  \"time\": 1900,\n",
        "  \"value\": 1000,\n",
        "  \"scenario\": \"historical\",\n",
        "  \"unit\": \"number of objects\",\n",
        "  \"variable\": \"exogenous_stock\",\n",
        "  \"dimensions_values\": {\n",
        "    \"location\": \"Spain\",\n",
        "    \"object\": \"light-duty vehicule\"\n",
        "  }\n",
        "}\n",
        "```\n",
        "\n",
        "\n",
        "*Adjust the transformation logic to match your data structure.*"
      ]
    },
    {
      "id": "transform-code",
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "def transform_to_mater(df: pd.DataFrame) -> list[dict]:\n",
        "    \"\"\"Transform raw data to MATER-ready data\"\"\"\n",
        "    return [\n",
        "        {\n",
        "            \"time\": ...,\n",
        "            \"value\": ...,\n",
        "            \"scenario\": ...,\n",
        "            \"unit\": ...,\n",
        "            \"variable\": ...,\n",
        "            \"dimensions_values\": {...},\n",
        "        }\n",
        "        for row in df.to_dict(\"records\")\n",
        "    ]\n",
        "\n",
        "\n",
        "input_data = transform_to_mater(df)\n",
        "display(pd.DataFrame(input_data))\n"
      ]
    },
    {
      "id": "assemble-md",
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Assembling\n",
        "\n",
        "Complete structure with provider and metadata and validate it."
      ]
    },
    {
      "id": "assemble-code",
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "final_data = {\n",
        "    \"input_data\": input_data,\n",
        "    \"provider\": {\n",
        "        \"first_name\": \"{{ provider.first_name }}\",\n",
        "        \"last_name\": \"{{ provider.last_name }}\",\n",
        "        \"email_address\": \"{{ provider.email_address }}\",\n",
        "    },\n",
        "    \"metadata\": {\n",
        "        \"source\": \"{{ metadata.source }}\",\n",
        "        \"link\": \"{{ metadata.link }}\",\n",
        "        \"project\": \"{{ metadata.project }}\",\n",
        "    },\n",
        "}\n",
        "\n",
        "validate_mater_ready_data(final_data)"
      ]
    },
    {
      "id": "save-md",
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Save MATER-ready data\n",
        "\n",
        "Once you are satisfied, save data to JSON.\n"
      ]
    },
    {
      "id": "save-code",
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "config = MaterConfig()\n",
        "dataset_name = Path(\"{{ dataset_path }}\").stem\n",
        "output_path = project_root / config.paths.input_path / f\"{dataset_name}.json\"\n",
        "\n",
        "with open(output_path, \"w\", encoding=\"utf-8\") as f:\n",
        "    json.dump(final_data, f, indent=2, ensure_ascii=False)\n",
        "\n",
        "print(f\"Saved {len(final_data['input_data'])} records to {output_path}\")"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}