{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9ad5f259",
   "metadata": {},
   "source": [
    "# Providing Data - Level 1\n",
    "## How to tell Hyrax what data to give to a model.\n",
    "\n",
    "Every model needs data. It learns from data during training, \n",
    "and it makes predictions from data during inference.\n",
    "In Hyrax, that flow of information happens through two main pieces: \n",
    "a ``HyraxDataset`` and a ``DataProvider``.\n",
    "A ``HyraxDataset`` is the code that knows how to read specific data from disk.\n",
    "A ``DataProvider`` is the part we actually ask for data — it calls on one or more datasets,\n",
    "retrieves the fields we need, and hands everything back as a clean, well-structured Python dictionary.\n",
    "\n",
    "In this guide, we’ll take our very first steps with data in Hyrax. Here’s what we’ll do:\n",
    "- Learn how to use a DataProvider to tell Hyrax what data a model should see.\n",
    "- Look inside the DataProvider to understand what the data looks like once it’s ready.\n",
    "\n",
    "To keep things simple, we’ll practice with a built-in Dataset called ``HyraxRandomDataset``.\n",
    "Think of it as “practice data” that stands in for the real thing.\n",
    "\n",
    "As always, our first move will be to create an instance of the Hyrax class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e20e33fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "from hyrax import Hyrax\n",
    "\n",
    "h = Hyrax()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72cfea63",
   "metadata": {},
   "source": [
    "Next we'll try to tell Hyrax that we want to use the ``HyraxRandomDataset`` as the source for our data provider."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "129366f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_inputs_definition = {\n",
    "    \"train\": {\n",
    "        \"data\": {\"dataset_class\": \"HyraxRandomDataset\"},\n",
    "    }\n",
    "}\n",
    "h.set_config(\"model_inputs\", model_inputs_definition)\n",
    "\n",
    "# Prepare \"model_inputs\"\n",
    "d = h.prepare()\n",
    "\n",
    "# Print a sample of the data\n",
    "d[\"train\"].sample_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a16736fe",
   "metadata": {},
   "source": [
    "Hooray — success! 🎉 We just told Hyrax which Dataset to use, called ``h.prepare()`` to set it up,\n",
    "and printed out a sample of the data.\n",
    "The configuration we used is called ``model_inputs``, because it defines what will be sent into the model.\n",
    "(And yes, the name is plural — models can take more than one input, but we’ll save that for the next notebook.)\n",
    "\n",
    "What we’ve created here is the simplest possible setup: it tells Hyrax to use ``HyraxRandomDataset``\n",
    "as the source of data for training. That’s all we need to get started!\n",
    "\n",
    "Before we move on, there are two details worth highlighting:\n",
    "- The dictionary key (\"data\") is up to you. You can name it whatever you like, as long as each key is unique.\n",
    "- The value of \"dataset_class\" is the Dataset you want Hyrax to use. Here we picked ``HyraxRandomDataset``, but you could swap in any class."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d44bef84",
   "metadata": {},
   "source": [
    "## Examine some of the data\n",
    "Now that our ``DataProvider`` is set up to return only the fields we care about,\n",
    "let’s take a closer look at the data itself.\n",
    "\n",
    "We’ve already seen the ``DataProvider.sample_data()`` function, which returns the first sample it can find.\n",
    "Because Hyrax retrieves data by index, it’s also easy to explore different parts\n",
    "of the dataset by sampling at random indices.\n",
    "This helps you get a feel for the form and structure of the data before feeding it into a model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba66ba7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "d[\"train\"][12]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40690a17",
   "metadata": {},
   "source": [
    "No big surprise — the output here looks very similar to what we saw with ``d.sample_data()``.\n",
    "Remember, the data is returned as a nested dictionary, with the top-level key matching the friendly name, \"data\".\n",
    "\n",
    "You can use any integer index up to the size of the dataset, but how do we figure out that size?\n",
    "Simple: just use Python’s ``len(...)`` function!\n",
    "This lets us see exactly how many samples are available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1ffb5f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(d[\"train\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d9515ce",
   "metadata": {},
   "source": [
    "## The primary id field\n",
    "One more important option is `primary_id_field` — it tells Hyrax which value to include with each sample so you can trace inference outputs back to the original input.\n",
    "\n",
    "Each dataset can provide a unique identifier per sample (a name, an index, or any value). In this dataset the identifier is returned by `get_object_id`, so request it by setting \"primary_id_field\" to \"object_id\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13b741ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_inputs_definition = {\n",
    "    \"train\": {\n",
    "        \"data\": {\n",
    "            \"dataset_class\": \"HyraxRandomDataset\",\n",
    "            \"primary_id_field\": \"object_id\",\n",
    "        },\n",
    "    },\n",
    "}\n",
    "\n",
    "h.set_config(\"model_inputs\", model_inputs_definition)\n",
    "\n",
    "# Prepare \"model_inputs\"\n",
    "d = h.prepare()\n",
    "\n",
    "# Print a sample of the data\n",
    "d[\"train\"].sample_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4491a13",
   "metadata": {},
   "source": [
    "With that change the model input is fully specified; training or inference can now be run using the built-in Hyrax models (for instance `h.train()` or `h.infer()`). We'll leave that as an exercise — training and inference are covered in more detail in a later notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eced5315",
   "metadata": {},
   "source": [
    "## Persisting the configuration\n",
    "\n",
    "In Hyrax, all configurations — including ``model_inputs`` — are saved to the configuration .toml file,\n",
    "along with any results. This makes it easy to reuse or share your setup later.\n",
    "\n",
    "For our example, the saved configuration would look like this:\n",
    "\n",
    "    [model_inputs]\n",
    "    [model_inputs.train]\n",
    "    [model_inputs.train.data]\n",
    "    dataset_class = 'HyraxRandomDataset'\n",
    "    primary_id_field = 'object_id'\n",
    "\n",
    "This ensures that Hyrax remembers exactly which Dataset and fields you want for future runs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48c94e3c",
   "metadata": {},
   "source": [
    "## Recap\n",
    "\n",
    "Great job! We covered a lot of ground in this notebook and learned the basics of providing data to models in Hyrax. \n",
    "Here’s a quick summary of what you accomplished:\n",
    "\n",
    "- Learned how to use `DataProvider` to supply data for your models.\n",
    "- Configured which dataset to use by updating the \"model_inputs\" configuration.\n",
    "- Previewed sample data and checked the dataset size to understand your data better.\n",
    "- Specified a primary ID field for traceability in your results.\n",
    "- Saw how Hyrax saves your configuration for easy reuse and sharing.\n",
    "\n",
    "You’re now ready to set up data for training or inference with Hyrax!\n",
    "If you want more control of the datasets used for training and inference,\n",
    "\"model_inputs\" can accept more configurations.\n",
    "Checkout out Providing Data - Level 2 for more."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "hyrax",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
