{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Roboflow Dataset Analysis with Filoma\n",
    "\n",
    "This tutorial demonstrates how to use **Filoma** to analyze a computer vision dataset downloaded from Roboflow. We will cover:\n",
    "1. Downloading a dataset using the Roboflow API.\n",
    "2. Probing the dataset structure and metadata using Filoma.\n",
    "3. Using Polars for efficient data manipulation.\n",
    "4. Visualizing file size distributions across dataset splits."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Setup and Installation\n",
    "\n",
    "First, we'll install the necessary dependencies and set up the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reset -f\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "!uv pip install roboflow seaborn matplotlib polars python-dotenv --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from pathlib import Path\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import polars as pl\n",
    "import seaborn as sns\n",
    "from dotenv import load_dotenv\n",
    "from roboflow import Roboflow\n",
    "\n",
    "import filoma as flm\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Download Dataset from Roboflow\n",
    "\n",
    "To use this notebook, you should have a Roboflow API key. \n",
    "\n",
    "**Recommendation:** Create a `.env` file in the root of your project and add your key there:\n",
    "```text\n",
    "ROBOFLOW_API_KEY=your_actual_api_key_here\n",
    "```\n",
    "The code below will automatically load it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loading Roboflow workspace...\n",
      "loading Roboflow project...\n",
      "Dataset downloaded to: /home/kalfasy/repos/filoma/notebooks/Weeds-3\n"
     ]
    }
   ],
   "source": [
    "# Load environment variables from .env file\n",
    "load_dotenv()\n",
    "\n",
    "# Access your Roboflow API Key\n",
    "api_key = os.getenv(\"ROBOFLOW_API_KEY\", \"YOUR_API_KEY_HERE\")\n",
    "\n",
    "rf = Roboflow(api_key=api_key)\n",
    "project = rf.workspace(\"augmented-startups\").project(\"weeds-nxe1w\")\n",
    "version = project.version(3)\n",
    "dataset = version.download(\"multiclass\")\n",
    "\n",
    "# Define dataset path\n",
    "dataset_path = Path(\"./Weeds-3/\").resolve()\n",
    "print(f\"Dataset downloaded to: {dataset_path}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Profiling the Dataset with Filoma\n",
    "\n",
    "Filoma makes it incredibly easy to scan a directory and create a Polars DataFrame containing file, file-path, and image metadata information. We can make use of this to analyze our Roboflow dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m2025-12-30 22:21:20.462\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36mfiloma.directories.directory_profiler\u001b[0m:\u001b[36m__init__\u001b[0m:\u001b[36m381\u001b[0m - \u001b[34m\u001b[1mInteractive environment detected, disabling progress bars to avoid conflicts\u001b[0m\n",
      "\u001b[32m2025-12-30 22:21:20.462\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mfiloma.directories.directory_profiler\u001b[0m:\u001b[36mprobe\u001b[0m:\u001b[36m474\u001b[0m - \u001b[1mStarting directory analysis of '/home/kalfasy/repos/filoma/notebooks/Weeds-3' using 🦀 Rust (Parallel) implementation\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m2025-12-30 22:21:20.532\u001b[0m | \u001b[32m\u001b[1mSUCCESS \u001b[0m | \u001b[36mfiloma.directories.directory_profiler\u001b[0m:\u001b[36mprobe\u001b[0m:\u001b[36m490\u001b[0m - \u001b[32m\u001b[1mDirectory analysis completed in 0.07s - Found 4,212 items (4,208 files, 4 folders) using 🦀 Rust (Parallel)\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of images in each split:\n",
      " - Train: 3664\n",
      " - Validation: 359\n",
      " - Test: 180\n"
     ]
    }
   ],
   "source": [
    "# Use filoma's probe_to_df to create a DataFrame with file metadata\n",
    "# and simply filter for .jpg images\n",
    "df = flm.probe_to_df(dataset_path.as_posix()).filter_by_extension(\".jpg\")\n",
    "\n",
    "# Extract the dataset split (train/valid/test) from the \"parent\" directory name\n",
    "# since all images are stored in respective subdirectories\n",
    "df = df.with_columns(split=pl.col(\"parent\").str.split(\"/\").list.last())\n",
    "\n",
    "# And we can now separate the DataFrame into different splits\n",
    "df_train = df.filter(pl.col(\"split\") == \"train\")\n",
    "df_val = df.filter(pl.col(\"split\") == \"valid\")\n",
    "df_test = df.filter(pl.col(\"split\") == \"test\")\n",
    "\n",
    "print(\"Number of images in each split:\")\n",
    "print(f\" - Train: {df_train.shape[0]}\")\n",
    "print(f\" - Validation: {df_val.shape[0]}\")\n",
    "print(f\" - Test: {df_test.shape[0]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Analyzing File Sizes\n",
    "\n",
    "We can easily explore data sizes using the `size_bytes` column provided by Filoma."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training total size: 258.30 MB\n",
      "Validation total size: 20.01 MB\n",
      "Test total size: 9.91 MB\n"
     ]
    }
   ],
   "source": [
    "# Let's compute the total size of each split\n",
    "def get_stats(df, label):\n",
    "    \"\"\"\n",
    "    Compute and print the total size of files in the given DataFrame.\n",
    "    \"\"\"\n",
    "    total_bytes = df.select(pl.col(\"size_bytes\").sum()).item()\n",
    "    total_mb = total_bytes / (1024 * 1024)\n",
    "    print(f\"{label} total size: {total_mb:.2f} MB\")\n",
    "    return total_mb\n",
    "\n",
    "\n",
    "# Get stats for each split\n",
    "get_stats(df_train, \"Training\")\n",
    "get_stats(df_val, \"Validation\")\n",
    "get_stats(df_test, \"Test\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average file size (MB):\n",
      " - Train: 0.07 MB\n",
      " - Validation: 0.06 MB\n",
      " - Test: 0.06 MB\n"
     ]
    }
   ],
   "source": [
    "# Let's get the average file size per split\n",
    "avg_train_size = df_train.select(pl.col(\"size_bytes\").mean()).item() / (1024 * 1024)\n",
    "avg_val_size = df_val.select(pl.col(\"size_bytes\").mean()).item() / (1024 * 1024)\n",
    "avg_test_size = df_test.select(pl.col(\"size_bytes\").mean()).item() / (1024 * 1024)\n",
    "print(\"Average file size (MB):\")\n",
    "print(f\" - Train: {avg_train_size:.2f} MB\")\n",
    "print(f\" - Validation: {avg_val_size:.2f} MB\")\n",
    "print(f\" - Test: {avg_test_size:.2f} MB\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks a bit suspicious already, let's plot the density distributions of file sizes across the different splits to visualize this better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Visualizing the Distribution\n",
    "\n",
    "Finally, let's visualize the density of file sizes across our splits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 89644 characters]",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Convert size_bytes to MB for plotting\n",
    "df = df.with_columns((pl.col(\"size_bytes\") / (1024 * 1024)).alias(\"size_MB\"))\n",
    "train_mb = df.filter(pl.col(\"split\") == \"train\").select(\"size_MB\").to_series().to_list()\n",
    "val_mb = df.filter(pl.col(\"split\") == \"valid\").select(\"size_MB\").to_series().to_list()\n",
    "test_mb = df.filter(pl.col(\"split\") == \"test\").select(\"size_MB\").to_series().to_list()\n",
    "\n",
    "plt.figure(figsize=(12, 6))\n",
    "\n",
    "sns.kdeplot(train_mb, label=\"Train\", fill=True, common_norm=False, alpha=0.5)\n",
    "sns.kdeplot(val_mb, label=\"Validation\", fill=True, common_norm=False, alpha=0.5)\n",
    "sns.kdeplot(test_mb, label=\"Test\", fill=True, common_norm=False, alpha=0.5)\n",
    "\n",
    "plt.xlabel(\"File Size (MB)\")\n",
    "plt.ylabel(\"Density\")\n",
    "plt.title(\"Density of File Sizes (MB) by Split\")\n",
    "plt.xlim(left=0)\n",
    "plt.legend()\n",
    "plt.grid(True, linestyle=\"--\", alpha=0.7)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the training set has larger files on average compared to validation and test sets. This insight can help in understanding the dataset's characteristics and planning for model training and evaluation. Let's plot some random images from each split to get a better sense of the data and why those size differences might exist."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Plotting random images from train/val/test splits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 791080 characters]",
      "text/plain": [
       "<Figure size 1500x500 with 5 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 767384 characters]",
      "text/plain": [
       "<Figure size 1500x500 with 5 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 755816 characters]",
      "text/plain": [
       "<Figure size 1500x500 with 5 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_random_images(df, split, num_images=5):\n",
    "    sampled_df = df.filter(pl.col(\"split\") == split).sample(n=num_images)\n",
    "    fig, axes = plt.subplots(1, num_images, figsize=(15, 5))\n",
    "    for ax, row in zip(axes, sampled_df.iter_rows()):\n",
    "        img_path = row[df.get_column_index(\"path\")]\n",
    "        img = plt.imread(img_path)\n",
    "        ax.imshow(img)\n",
    "        ax.axis(\"off\")\n",
    "        ax.set_title(f\"{split.capitalize()} Sample\")\n",
    "    plt.suptitle(f\"Random {num_images} Images from {split.capitalize()} Set\")\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "plot_random_images(df, \"train\", num_images=5)\n",
    "plot_random_images(df, \"valid\", num_images=5)\n",
    "plot_random_images(df, \"test\", num_images=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training images seem to have augmentations applied, which could explain their larger file sizes.."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
