{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`firefly/ntbks/reader_tutorial.ipynb`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "from IPython.display import YouTubeVideo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A recording of this jupyter notebook in action is available at:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/jpeg": "[encoded data, 18716 characters]\n",
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"400\"\n",
       "            height=\"300\"\n",
       "            src=\"https://www.youtube.com/embed/lYPGa6DibOk\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.YouTubeVideo at 0x10411aca0>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "YouTubeVideo(\"lYPGa6DibOk\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "don't have phil's colormaps\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "import os\n",
    "import numpy as np\n",
    "## ignore this line, you do not need to add this if Firefly is pip installed into your PYTHONPATH\n",
    "sys.path.insert(0,\"/Users/agurvich/research/repos/firefly/src/\")\n",
    "from firefly.data_reader import Reader,ArrayReader,ParticleGroup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial notebook: Using the `Reader` class\n",
    "One of the main purposes of Firefly is to enable users to interactively explore their *own* data (or else interactively explore someone else's respective own data). While it is possible to format one's data manually using a text editor we have provided a python API for producing the `.json` files that are necessary to run an iteration of firefly.\n",
    "\n",
    "The central piece of that API is the `firefly.data_reader.Reader` class, which acts to collect the different parts of the API together to produce consistently formatted and linked `.json` files that the Firefly webapp can interpret. The `Reader` class is <a href=\"https://alexbgurvi.ch/Firefly/docs/build/html/data_reader/reader.html\">documented here</a> but here we only provide a brief example of how to use the API. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Reader instance\n",
    "To begin, we'll start by initializing a `Reader` object. Users are encouraged to familiarize themselves with the different keyword arguments through the documentation linked above. \n",
    "\n",
    "Perhaps the most important keyword argument is the `JSONdir`, which tells your `Reader` object where it should collect the different `.json` files it will produce. The `.json` files have to be readable from the `firefly/static/data` directory of the iteration of Firefly that's trying to open the data. The `Reader` class will automatically create a shortcut to the directory if you don't choose a path that lives in `firefly/static/data`. If you enter a relative path it will assume you mean relative to your `${HOME}` directory. If no `JSONdir` is provided then it will default to `${HOME}/<JSONprefix>` (which itself defaults to `Data` if nothing is passed). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "## initialize a Reader object, cwd will be firefly/ntbks\n",
    "JSONdir = os.path.abspath(os.path.join(os.getcwd(),'..','static','data','tutorial'))\n",
    "my_reader = Reader(JSONdir=JSONdir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "## let's create some sample data, a grid of points in a 3d cube\n",
    "my_coords = np.linspace(-10,10,20)\n",
    "xs,ys,zs = np.meshgrid(my_coords,my_coords,my_coords)\n",
    "xs,ys,zs = xs.flatten(),ys.flatten(),zs.flatten()\n",
    "coords = np.array([xs,ys,zs]).T\n",
    "\n",
    "## we'll pick some random field values to demonstrate filtering/colormapping\n",
    "fields = np.random.random(size=xs.size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Store the coordinates in a ParticleGroup\n",
    "Particle data is validated and organized in `firefly.data_reader.ParticleGroup` objects. In general users should not sub-class the `ParticleGroup` class but if you're an enterprising user with a specific use case I'm a tutorial not a cop! For details about how the `ParticleGroup` class works, check the <a href=\"https://alexbgurvi.ch/Firefly/docs/build/html/data_reader/particle_group.html\">particle group documentation</a>.\n",
    "\n",
    "For our purposes, we'll take advantage of the fact that any keyword arguments passed here go directly to the `particleGroup.settings_default` dictionary which controls which elements appear in the particle panes in the UI, see the <a href=\"https://alexbgurvi.ch/Firefly/docs/build/html/data_reader/settings.html\">settings documentation</a> or see `settings_tutorial.ipynb` for an example.\n",
    "\n",
    "**Note:**\n",
    "Sometimes data is too large to load directly into Firefly, we encourage users who are trying a new dataset for the first time to use the `decimation_factor` keyword argument to reduce the dataset size by the factor specified (the implementation is just a `shuffle(coords)[::decimation_factor]`). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Make sure each field_array (1) has a field_filter_flag (0), assuming True.\n",
      "Make sure each field_array (1) has a field_colormap_flag (0), assuming True.\n",
      "Make sure each field_array (1) has a field_radius_flag (0), assuming False.\n",
      "Make sure each field_array (1) has a field_filter_flag (0), assuming True.\n",
      "Make sure each field_array (1) has a field_colormap_flag (0), assuming True.\n",
      "Make sure each field_array (1) has a field_radius_flag (0), assuming False.\n"
     ]
    }
   ],
   "source": [
    "## create a particle group that contains our test coordinates\n",
    "my_particleGroup = ParticleGroup(\n",
    "    'partname',\n",
    "    coords,\n",
    "    sizeMult=5, ## increase the size of the particles to make the particles a bit easier to see since there's so few of them\n",
    "    color = [0,0,1,1], ## make them blue, colors should be RGBA list,\n",
    "    field_arrays=[fields], ## track the dummy field to demonstrate how to pass field data\n",
    "    field_names=['testfield']) ## name the dummy field\n",
    "\n",
    "## sometimes data is too large to load directly into Firefly\n",
    "my_decimated_particleGroup = ParticleGroup(\n",
    "    'decimated',\n",
    "    coords,\n",
    "    sizeMult=5, ## increase the size of the particles to make the particles a bit easier to see since there's so few of them\n",
    "    color = [0,0,1,1], ## make them blue, colors should be RGBA list,\n",
    "    field_arrays=[fields], ## track the dummy field to demonstrate how to pass field data\n",
    "    field_names=['testfield'], ## name the dummy field\n",
    "    decimation_factor=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All that's left is to connect the `ParticleGroup` object to the `Reader` object using the `.addParticleGroup` method. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reader with 2 particle groups\n",
      "[partname - 8000/8000 particles - 1 tracked fields\n",
      " decimated - 800/8000 particles - 1 tracked fields]\n"
     ]
    }
   ],
   "source": [
    "## instructs my_reader to keep track of my_particleGroup\n",
    "my_reader.addParticleGroup(my_particleGroup)\n",
    "my_reader.addParticleGroup(my_decimated_particleGroup)\n",
    "print(my_reader)\n",
    "print(my_reader.particleGroups)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the decimation factor is represented by the fraction 800/8000 in the second particle group \"decimated\". "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Outputting to `.ffly`\n",
    "At this point we're ready to output our data to `.ffly` format in order to load in with Firefly. The `Reader` object will automatically dump all of the necessary files associated with each of the `ParticleGroup` objects and `Settings` objects we've attached to it as described in the <a href=\"https://alexbgurvi.ch/Firefly/docs/build/html/data_reader/reader.html\">reader documentation</a>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Outputting: partname - 8000/8000 particles - 1 tracked fields\n",
      "Outputting: decimated - 800/8000 particles - 1 tracked fields\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "''"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## have the reader dump all its data to the different JSON files\n",
    "my_reader.writeToDisk(loud=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that `.writeToDisk` returned an empty string, this is because the `.ffly` and configuration `.json` files were written to disk. Another option is instead to produce a single `.json` formatted string with all the data that would've been written to disk. This is useful for transmitting data through Flask, which is the [subject of another tutorial](http://www.alexbgurvi.ch/Firefly/docs/build/html/data_reader/flask_tutorial.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Outputting: partname - 8000/8000 particles - 1 tracked fields\n",
      "Outputting: decimated - 800/8000 particles - 1 tracked fields\n",
      "big_JSON has 451720 characters\n"
     ]
    }
   ],
   "source": [
    "## have the reader dump all its data to a single big string\n",
    "big_JSON = my_reader.writeToDisk(loud=True,write_to_disk=False,extension='.json')\n",
    "\n",
    "print(\"big_JSON has %d characters\"%len(big_JSON))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using an `ArrayReader` sub-class\n",
    "\n",
    "The procedure outlined above is a common use case, and so we've provided a sub-class to `firefly.data_reader.Reader`, `firefly.data_reader.ArrayReader` which wraps the `ParticleGroup` and `.addParticleGroup` so the user can get a `Reader` containing their data with a single initialization. It will automatically name particle groups and fields unless they are specified directly (see <a href=\"https://alexbgurvi.ch/Firefly/docs/build/html/data_reader/reader.html\">reader documentation</a>)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Make sure each field_array (1) has a field_filter_flag (0), assuming True.\n",
      "Make sure each field_array (1) has a field_colormap_flag (0), assuming True.\n",
      "Make sure each field_array (1) has a field_radius_flag (0), assuming False.\n",
      "Outputting: PGroup_0 - 8000/8000 particles - 1 tracked fields\n"
     ]
    }
   ],
   "source": [
    "my_arrayReader = ArrayReader(\n",
    "    coords,\n",
    "    fields=fields,\n",
    "    JSONdir=JSONdir,\n",
    "    write_to_disk=True)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
