{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "13159a42-6261-42c4-bd6d-4d92d765a230",
   "metadata": {},
   "source": [
    "# Export and Validate Metadata\n",
    "\n",
    "This tutorial shows how to **export and validate metadata** using the `paidiverpy` package.\n",
    "\n",
    "## What You'll Learn\n",
    "\n",
    "- 📂 Load metadata from JSON and CSV formats\n",
    "- ✅ Validate metadata structure and contents\n",
    "- 💾 Export metadata in standardized formats (iFDO)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fd8e9d9-324e-41ee-b6ba-53b4761baddf",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "To use this package effectively, you need a metadata file that describes the images being processed.\n",
    "This can be either a `.json` file (following the iFDO standard) or a `.csv` file.\n",
    "\n",
    "Metadata provides essential context such as filenames, timestamps, and geospatial coordinates, which are critical for accurate image analysis.\n",
    "\n",
    "---\n",
    "\n",
    "### iFDO Standard JSON File\n",
    "\n",
    "The **iFDO** (image FAIR Digital Object) format is a standardized way to structure metadata for image datasets. It supports a rich set of attributes including:\n",
    "\n",
    "- **filename**: Name of the image file\n",
    "- **datetime**: Timestamp of image capture\n",
    "- **lat / lon**: Geospatial coordinates of the image location\n",
    "\n",
    "To ensure iFDO compliance, structure the JSON file according to the standard.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49e7b59c-ff31-4f94-8e0a-4e48dba469f2",
   "metadata": {},
   "source": [
    "## Validate iFDO Metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e1536ff-a482-4b93-8bce-503478db374c",
   "metadata": {},
   "source": [
    "You can validate your metadata using the `validate_ifdo` function included in the `paidiverpy` package:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a5734e4-55f0-42ad-8ec3-081785d51b8c",
   "metadata": {},
   "source": [
    "### Import dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "908dd731-e15e-4b30-a4bf-ea1b9b3158ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "from paidiverpy.metadata_parser.ifdo_tools import validate_ifdo"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54313372-bf5a-451a-a2b9-f62e1d33ae0b",
   "metadata": {},
   "source": [
    "### Opening the metadata\n",
    "\n",
    "The validation function accepts a path to a json file or a json file loaded in python as a dictionary.\n",
    "\n",
    "The output of the validation function is a list of validation errors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b4213eb9-cc48-4bc3-a1c8-125c7bd928f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# metadata file path\n",
    "metadata_file = \"../metadata/metadata_ifdo_hf.json\"\n",
    "errors = validate_ifdo(metadata_file)\n",
    "\n",
    "# You can also validate the metadata as a dict object on python metadadata file path\n",
    "# with open(metadata_file, \"r\") as file:\n",
    "#     metadata_data = json.dum\n",
    "\n",
    "# validate_ifdo(metadata_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7486d2a4-e79b-4cca-8600-be0f0e223f46",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'path': ['image-set-header'],\n",
       "  'message': \"'image-set-handle' is a required property\"},\n",
       " {'path': ['image-set-header'],\n",
       "  'message': \"'image-altitude-meters' is a required property\"},\n",
       " {'path': ['image-set-header'],\n",
       "  'message': \"'image-coordinate-uncertainty-meters' is a required property\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744807147.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744811443.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744853552.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744857927.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744862146.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744866443.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744870740.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744875021.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744879318.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744883630.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744887927.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744892224.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744901661.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744905958.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744910271.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744918864.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744923161.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744927442.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744934333.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744938630.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744942926.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744947223.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744951692.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744962270.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744966129.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744970426.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744974707.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744979020.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744989332.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987744994488.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745006504.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745010816.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745015191.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745019395.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745023691.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745027988.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745035754.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745040035.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745044332.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745048629.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745052926.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745064097.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745068379.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745072691.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745076972.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745088941.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745092456.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745103628.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745107925.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745112222.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745116519.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745121003.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745125097.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745138003.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745142300.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745146581.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745150878.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745155190.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745159487.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745163768.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745168081.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745172362.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745176674.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745180955.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745185268.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745189565.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745194721.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745199002.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745203643.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745210189.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745219642.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745223939.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745231673.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745235955.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745240267.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745244564.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745249720.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745254017.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745258361.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745262595.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745267282.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745271204.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745275501.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745287688.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745291829.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745296126.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745300422.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745304704.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745311579.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745315891.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745320188.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745324485.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745328781.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745333078.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745337359.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745342531.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745346828.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745351406.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745355422.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"},\n",
       " {'path': ['image-set-items', 'M58_10441297_12987745365734.jpg'],\n",
       "  'message': \"Missing fields: ['image-hash-sha256', 'image-handle']. Additional fields (this will not make the metadata invalid): ['image-area-square-meter', 'image-media-type'].\"}]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "errors"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8860a546-ec67-422e-a4c5-175820eed7fc",
   "metadata": {},
   "source": [
    "As you can see above, there are several validation errors with the iFDO metadata. The package will work with those validation problems. However, some fields are mandatory per the iFDO specification, while others are required by `paidiverpy` for full pipeline functionality. These are the mandatory fields:\n",
    "\n",
    "- `image-set-header` >> `image-set-ifdo-version`: Version of the iFDO schema used.\n",
    "- `image-set-items` >> `image-filename`: Name of the image file.\n",
    "\n",
    "It is also important to mention the this validation will also run automatically when you run a pipeline or when you instantiate a `MetadataParser` class with a iFDO metadata."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22895ac4-7558-4979-910a-7d081f079ace",
   "metadata": {},
   "source": [
    "## Export Metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c03b0a8-ece1-4384-8ae7-c2271f473bd0",
   "metadata": {},
   "source": [
    "You can export metadata to several formats using the `export_metadata` function inside the `MetadataParser` class.\n",
    "This is useful for interoperability, archiving, or sharing."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d0bee72-5f89-4f08-ac50-8e6cbec41e47",
   "metadata": {},
   "source": [
    "### Import dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "256be5d6-274d-40b8-9ce2-6d15100022e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from paidiverpy.pipeline import Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fafc2437-4198-4593-9984-1931d9aa0187",
   "metadata": {},
   "source": [
    "### Parsing the existing metadata\n",
    "\n",
    "You can parse and open the metadata using the Pipeline class or the MetadataParser class.\n",
    "\n",
    "The only difference is that, if you use the Pipeline class, after you run the pipeline, metadata can be added to the output metadata based on the outputs of the pipeline steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1b7bfff6-35e2-448b-9177-0a81966678bf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m☁ paidiverpy ☁  |       INFO | 2025-09-11 16:44:58 | Processing images using 8 cores\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:paidiverpy:Processing images using 8 cores\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m☁ paidiverpy ☁  |       INFO | 2025-09-11 16:44:58 | Running step 0: raw - OpenLayer\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:paidiverpy:Running step 0: raw - OpenLayer\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[########################################] | 100% Completed | 219.05 ms\n",
      "\u001b[92m☁ paidiverpy ☁  |       INFO | 2025-09-11 16:44:58 | Step 0 completed\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:paidiverpy:Step 0 completed\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m☁ paidiverpy ☁  |       INFO | 2025-09-11 16:44:58 | Running step 1: colour_correction - ColourLayer\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:paidiverpy:Running step 1: colour_correction - ColourLayer\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m☁ paidiverpy ☁  |       INFO | 2025-09-11 16:44:59 | Step 1 completed\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:paidiverpy:Step 1 completed\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m☁ paidiverpy ☁  |       INFO | 2025-09-11 16:44:59 | Running step 2: datetime - SamplingLayer\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:paidiverpy:Running step 2: datetime - SamplingLayer\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m☁ paidiverpy ☁  |       INFO | 2025-09-11 16:44:59 | Number of images before the sampling step: 20. Total number of images after: 14\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:paidiverpy:Number of images before the sampling step: 20. Total number of images after: 14\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m☁ paidiverpy ☁  |       INFO | 2025-09-11 16:44:59 | Step 2 completed\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:paidiverpy:Step 2 completed\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m☁ paidiverpy ☁  |       INFO | 2025-09-11 16:44:59 | Running step 3: overlapping - SamplingLayer\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:paidiverpy:Running step 3: overlapping - SamplingLayer\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m☁ paidiverpy ☁  |       INFO | 2025-09-11 16:44:59 | Number of images before the sampling step: 14. Total number of images after: 9\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:paidiverpy:Number of images before the sampling step: 14. Total number of images after: 9\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m☁ paidiverpy ☁  |       INFO | 2025-09-11 16:44:59 | Step 3 completed\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:paidiverpy:Step 3 completed\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m☁ paidiverpy ☁  |       INFO | 2025-09-11 16:44:59 | Running step 4: colour_correction - ColourLayer\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:paidiverpy:Running step 4: colour_correction - ColourLayer\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m☁ paidiverpy ☁  |       INFO | 2025-09-11 16:44:59 | Step 4 completed\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:paidiverpy:Step 4 completed\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m☁ paidiverpy ☁  |       INFO | 2025-09-11 16:45:00 | Running step 5: sharpen - ColourLayer\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:paidiverpy:Running step 5: sharpen - ColourLayer\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m☁ paidiverpy ☁  |       INFO | 2025-09-11 16:45:00 | Step 5 completed\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:paidiverpy:Step 5 completed\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m☁ paidiverpy ☁  |       INFO | 2025-09-11 16:45:00 | Running step 6: contrast - ColourLayer\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:paidiverpy:Running step 6: contrast - ColourLayer\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m☁ paidiverpy ☁  |       INFO | 2025-09-11 16:45:00 | Step 6 completed\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:paidiverpy:Step 6 completed\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[########################################] | 100% Completed | 1.01 sms\n"
     ]
    }
   ],
   "source": [
    "# Using the Pipeline class and get the metadata after the run the pipeline\n",
    "pipeline = Pipeline(config_file_path=\"../config_files/config_benthic.yml\")\n",
    "pipeline.run()\n",
    "\n",
    "# Using the MetadataParser class\n",
    "# config = Config(config_file_path=\"/path/to/your/config.yaml\")\n",
    "# metadata_parser = MetadataParser(config=config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cffbcdd5-e1f0-451c-9cba-b1aa2359f833",
   "metadata": {},
   "source": [
    "### Exporting the metadata\n",
    "\n",
    "\n",
    "**1. CSV**\n",
    "\n",
    "- Useful for compatibility with spreadsheets or data analysis tools.\n",
    "- Dataset metadata is added as columns.\n",
    "- Column names follow iFDO naming conventions where possible.\n",
    "\n",
    "**2. JSON**\n",
    "\n",
    "- A flexible, widely supported format.\n",
    "- Each image’s metadata includes dataset-level metadata as additional keys.\n",
    "\n",
    "**3. iFDO**\n",
    "\n",
    "- Exports metadata in the native iFDO JSON format.\n",
    "- `image-set-header`: Built from `dataset_metadata`\n",
    "- `image-set-items`: Built from the image-level metadata\n",
    "- Automatically converts EXIF and pipeline-derived metadata to iFDO fields\n",
    "- Fills missing required fields with default descriptions from the iFDO schema for easier review and editing\n",
    "\n",
    "After exporting to iFDO, the file is validated and any schema errors will be printed to the console.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4ab80606-7fad-45c1-a90a-1f08d083c376",
   "metadata": {},
   "outputs": [],
   "source": [
    "# export to csv\n",
    "pipeline.metadata.export_metadata(output_format=\"csv\", metadata=pipeline.get_metadata(flag=\"all\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4b9e5202-5c27-438b-8228-d6373b404c46",
   "metadata": {},
   "outputs": [],
   "source": [
    "# export to json\n",
    "pipeline.metadata.export_metadata(\"json\", metadata=pipeline.get_metadata(flag=\"all\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "41cdfc2e-7cc8-4146-868e-8da377ee0e51",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Missing required fields in iFDO header or items\n",
      "WARNING:root:You need to set then in the metadata or dataset_metadata arguments\n",
      "WARNING:root:The missing fields will be set to the description value on the iFDO schema file\n",
      "WARNING:root:Missing fields in iFDO header: ['image-set-name', 'image-set-ifdo-version', 'image-altitude-meters', 'image-coordinate-reference-system', 'image-coordinate-uncertainty-meters', 'image-context', 'image-context:name', 'image-project', 'image-project:name', 'image-event', 'image-event:name', 'image-platform', 'image-platform:name', 'image-sensor', 'image-sensor:name', 'image-pi', 'image-pi:name', 'image-creators', 'image-license', 'image-license:name', 'image-copyright', 'image-abstract']\n",
      "WARNING:root:Missing fields in iFDO items: [['image-hash-sha256', 'image-handle'], ['image-hash-sha256', 'image-handle'], ['image-hash-sha256', 'image-handle'], ['image-hash-sha256', 'image-handle'], ['image-hash-sha256', 'image-handle'], ['image-hash-sha256', 'image-handle'], ['image-hash-sha256', 'image-handle'], ['image-hash-sha256', 'image-handle'], ['image-hash-sha256', 'image-handle']]\n"
     ]
    }
   ],
   "source": [
    "# export to iFDO Standard metadata\n",
    "pipeline.metadata.export_metadata(\"ifdo\", output_path=\"metadata_ifdo\", metadata=pipeline.get_metadata(flag=\"all\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfa0679f-b6ba-4f6c-ad3a-a0e1f13dfcd9",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "As you can see, there are several validation errors in the **iFDO metadata**. Please read them carefully to see if you need to update any information in your `dataset_metadata` and `metadata` as input arguments to the export_metadata` function.\n",
    "It is important to note that the output metadata will fill in missing required fields with default descriptions from the iFDO schema for easy review and editing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40a5c1f1-327b-475c-babb-5232a4f9a3f2",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
