{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Writing Containers to a tabular format\n",
    "\n",
    "The `TableWriter`/`TableReader` sub-classes allow you to write a `ctapipe.core.Container` class and its meta-data to an output table. They treat the `Field`s in the `Container` as columns in the output, and automatically generate a schema.  Here we will go through an example of writing out data and reading it back with *Pandas*, *PyTables*, and a `ctapipe.io.TableReader`:\n",
    "\n",
    "In this example, we will use the `HDF5TableWriter`, which writes to HDF5 datasets using *PyTables*. Currently this is the only implemented TableWriter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Caveats to think about:\n",
    "* vector columns in Containers *can* be written, but some lilbraries like Pandas can not read those (so you must use pytables or astropy to read outputs that have vector columns)\n",
    "* units are stored in the table metadata, but some libraries like Pandas ignore them and all other metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create some example Containers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ctapipe.io import HDF5TableWriter\n",
    "from ctapipe.core import Container, Field\n",
    "from astropy import units as u\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class VariousTypesContainer(Container):\n",
    "    \n",
    "    a_int = Field(int, 'some int value')\n",
    "    a_float = Field(float, 'some float value with a unit', unit=u.m)\n",
    "    a_bool = Field(bool, 'some bool value')\n",
    "    a_np_int = Field(np.int, 'a numpy int')\n",
    "    a_np_float = Field(np.float, 'a numpy float')\n",
    "    a_np_bool = Field(np.bool, 'np.bool')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "let's also make a dummy stream (generator) that will create a series of these containers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_stream(n_event):\n",
    "    \n",
    "    data = VariousTypesContainer()\n",
    "    for i in range(n_event):\n",
    "        \n",
    "        data.a_int = int(i)\n",
    "        data.a_float = float(i) * u.cm # note unit conversion will happen\n",
    "        data.a_bool = (i % 2) == 0\n",
    "        data.a_np_int = np.int(i)\n",
    "        data.a_np_float = np.float(i)\n",
    "        data.a_np_bool = np.bool((i % 2) == 0)\n",
    "        \n",
    "        yield data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for data in create_stream(2):\n",
    "        \n",
    "    for key, val in data.items():\n",
    "        \n",
    "        print('{}: {}, type : {}'.format(key, val, type(val)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Writing the Data (and good practices)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How not to do it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "h5_table = HDF5TableWriter('container.h5', group_name='data')\n",
    "\n",
    "for data in create_stream(10):\n",
    "    \n",
    "    h5_table.write('table', data)\n",
    "\n",
    "h5_table.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In that case the file is not garenteed to close properly for instance if one does a mistake in the for loop. Let's just add a stupid mistake and see what happens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    h5_table = HDF5TableWriter('container.h5', group_name='data')\n",
    "\n",
    "    for data in create_stream(10):\n",
    "\n",
    "        h5_table.write('table', data)\n",
    "        0/0  # cause an error\n",
    "        \n",
    "    h5_table.close()\n",
    "except Exception as err:\n",
    "    print(\"FAILED!\", err)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the file did not close properly. So let's try to correct the mistake and execute the code again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    h5_table = HDF5TableWriter('container.h5', group_name='data')\n",
    "\n",
    "    for data in create_stream(10):\n",
    "\n",
    "        h5_table.write('table', data)\n",
    "        0/0  # cause an error\n",
    "    h5_table.close()\n",
    "except Exception as err:\n",
    "    print(\"FAILED!\", err)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ah it seems that the file did not close! Now I am stuck. Maybe I should restart the kernel? ahh no I don't want to loose everything. Can I just close it ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "h5_table.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It worked!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Better to use context management!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    with HDF5TableWriter('container.h5', group_name='data') as h5_table:\n",
    "\n",
    "        for data in create_stream(10):\n",
    "\n",
    "            h5_table.write('table', data)\n",
    "            0/0\n",
    "except Exception as err:\n",
    "    print(\"FAILED:\", err)\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls container.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Appending new Containers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To append some new containers we need to set the writing in append mode by using: 'mode=a'. But let's now first look at what happens if we don't."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(2):\n",
    "\n",
    "    with HDF5TableWriter('container.h5', mode='w', group_name='data_{}'.format(i)) as h5_table:\n",
    "    \n",
    "        for data in create_stream(10):\n",
    "    \n",
    "            h5_table.write('table', data)\n",
    "\n",
    "        print(h5_table._h5file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -f container.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok so the writer destroyed the content of the file each time it opens the file. Now let's try to append some data group to it! (using mode='a')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(2):\n",
    "\n",
    "    with HDF5TableWriter('container.h5', mode='a', group_name='data_{}'.format(i)) as h5_table:\n",
    "    \n",
    "        for data in create_stream(10):\n",
    "    \n",
    "            h5_table.write('table', data)\n",
    "\n",
    "        print(h5_table._h5file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we can append some data groups. As long as the data group_name does not already exists. Let's try to overwrite the data group : data_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    with HDF5TableWriter('container.h5', mode='a', group_name='data_1') as h5_table:\n",
    "        for data in create_stream(10):\n",
    "            h5_table.write('table', data)\n",
    "except Exception as err:\n",
    "    print(\"Failed as expected:\", err)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Good ! I cannot overwrite my data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(bool(h5_table._h5file.isopen))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading the whole table at once:\n",
    "\n",
    "For this, you have several choices.  Since we used the HDF5TableWriter in this example, we have at least these options avilable:\n",
    "\n",
    "* Pandas\n",
    "* PyTables\n",
    "* Astropy Table\n",
    "\n",
    "For other TableWriter implementations, others may be possible (depending on format)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Reading with Pandas:\n",
    "\n",
    "Pandas is a convenient way to read the output.  **HOWEVER BE WARNED** that so far Pandas does not support reading the table *meta-data* or *units* for colums, so that information is lost! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "data = pd.read_hdf('container.h5', key='/data_0/table')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Reading with PyTables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tables\n",
    "h5  = tables.open_file('container.h5')\n",
    "table = h5.root['data_0']['table']\n",
    "table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "note that here we can still access the metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "table.attrs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading one-row-at-a-time:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rather than using the full-table methods, if you want to read it row-by-row (e.g. to maintain compatibility with an existing event loop), you can use a `TableReader` instance.\n",
    "\n",
    "The advantage here is that units and other metadata are retained and re-applied"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ctapipe.io import HDF5TableReader\n",
    "\n",
    "def read(mode):\n",
    "    \n",
    "    print('reading mode {}'.format(mode))\n",
    "\n",
    "    with HDF5TableReader('container.h5', mode=mode) as h5_table:\n",
    "\n",
    "        for group_name in ['data_0/', 'data_1/']:\n",
    "\n",
    "            group_name = '/{}table'.format(group_name)\n",
    "            print(group_name)\n",
    "\n",
    "            for data in h5_table.read(group_name, VariousTypesContainer()):\n",
    "\n",
    "                print(data.as_dict())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "read('r')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "read('r+')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "read('a')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "read('w')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
