{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mpes import fprocessing as fp\n",
    "# from imp import reload\n",
    "# reload(fp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "fpath = r'../data/data_20180605_131.h5'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Loading HDF5 files\n",
    "HDF5 files can be read using a few different classes operating on different levels. The hierarchy meaningful to the end user is in the following (from low to high),\n",
    "* **mpes.fprocessing.File()** -- local import of h5py.File(), a low-level Python HDF5 parser (wrapped over even lower C code).\n",
    "* **mpes.fprocessing.hdf5Reader()** -- built on the File() class, with the inclusion of several file structure parsing, file component readout and format conversion functions.\n",
    "* **mpes.fprocessing.hdf5Splitter()** -- built on the hdf5Reader() class, used for splitting large hdf5 files.\n",
    "* **mpes.fprocessing.hdf5Processor()** -- built on the hdf5Reader() class, with the inclusion of binning operations and io.\n",
    "\n",
    "The hierarchy goes **File $\\in$ hdf5Reader $\\in$ (hdf5Splitter, hdf5Processor)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<HDF5 file \"data_20180605_131.h5\" (mode r+)>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hdff = fp.File(fpath)\n",
    "hdff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<HDF5 file \"data_20180605_131.h5\" (mode r+)>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hdfr = fp.hdf5Reader(fpath)\n",
    "hdfr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**New attributes and methods in the hdf5Reader() class**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['groupAliases', 'readGroup', 'nameLookupDict', 'CHUNK_SIZE', 'attributeNames', 'convert', '_assembleGroups', 'summarize', 'name2alias', 'faddress', 'readAttribute', 'getAttributeNames', 'getGroupNames', 'ncores', 'nEvents', 'groupNames']\n"
     ]
    }
   ],
   "source": [
    "print( list(set(dir(hdfr)) - set(dir(hdff))) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<HDF5 file \"data_20180605_131.h5\" (mode r+)>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hdfp = fp.hdf5Processor(fpath)\n",
    "hdfp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**New attributes and methods in the hdf5Processer() class**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['saveHistogram', 'toSplitter', 'toBandStructure', 'histdict', 'localBinning', '_addBinners', 'viewEventHistogram', 'loadMapping', 'hdfdict', 'distributedBinning', 'saveParameters', '_delayedBinning', 'ua', 'axesdict', 'updateHistogram', 'distributedProcessBinning']\n"
     ]
    }
   ],
   "source": [
    "print( list(set(dir(hdfp)) - set(dir(hdfr))) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Retrieving components from HDF5 files\n",
    "Reading components can also be done at different levels, the level of hdf5Reader() or above is recommended."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** HDF5 file info ***\n",
      " File address = /scratch/metis_storage/data_20180605_131.h5\n",
      "\n",
      "\n",
      ">>> Attributes <<<\n",
      "\n",
      "CAClientMajorVersion = 0\n",
      "CAClientMinorVersion = 1\n",
      "CompileTimeStamp = Wed Jun 13 15:31:04 2018\n",
      "KTOF:Lens:A:VSet = 514.63\n",
      "KTOF:Lens:B:VSet = 2199.8\n",
      "KTOF:Lens:C:VSet = 76.402\n",
      "KTOF:Lens:D:VSet = 261.24\n",
      "KTOF:Lens:E:VSet = 558.98\n",
      "KTOF:Lens:Extr:VSet = 6000.0\n",
      "KTOF:Lens:F:VSet = 48.904\n",
      "KTOF:Lens:Foc:VSet = 167.0\n",
      "KTOF:Lens:G:VSet = 20.1\n",
      "KTOF:Lens:H:VSet = 35.0\n",
      "KTOF:Lens:I:VSet = 42.25\n",
      "KTOF:Lens:MCPback:VSet = 1825.0\n",
      "KTOF:Lens:MCPfront:VSet = 20.0\n",
      "KTOF:Lens:TOF:VSet = 20.0\n",
      "KTOF:Lens:UCA:VSet = 1200.0\n",
      "KTOF:Lens:UFA:VSet = 600.0\n",
      "KTOF:Lens:Z1:VSet = 2452.9\n",
      "KTOF:Lens:Z2:VSet = 1489.9\n",
      "\n",
      ">>> Groups <<<\n",
      "\n",
      "EventFormat, Shape = (64,), Alias = None\n",
      "Stream_0, Shape = (27296214,), Alias = X\n",
      "Stream_1, Shape = (27296214,), Alias = Y\n",
      "Stream_2, Shape = (27296214,), Alias = t\n",
      "Stream_3, Shape = (27296214,), Alias = MasterRstCtr\n",
      "Stream_4, Shape = (27296214,), Alias = ADC\n",
      "Stream_5, Shape = (27296214,), Alias = State Input\n"
     ]
    }
   ],
   "source": [
    "hdfp.summarize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 64, 64, 0, 64, 0, 64, 0, 64, 0, 53, 11, 42, 11, 16, 26, 15, 1, 15, 0, 0, 15, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n"
     ]
    }
   ],
   "source": [
    "print(list(hdfr.readGroup(hdfr, 'EventFormat')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Converting HDF5 files\n",
    "Conversion of hdf5 to Matlab (mat) format (no data processing)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "hdfr.convert('mat', save_addr='../data/data_131')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conversion to parquet format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hdfr.convert('parquet', save_addr='../data/data_131_parquet', pq_append=False, chunksz=1e7, \\\n",
    "             compression='gzip')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 Splitting HDF5 files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hdfs = fp.hdf5Splitter(fpath)\n",
    "hdfs.split(nsplit=50, save_addr=r'../data/data_114_parts/data_114_', pbar=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.5 Retrieve binned data from stored HDF5 file\n",
    "Read binned data over 3 axes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fpath_binned = r'../data/binres_114.h5'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bindict = fp.readBinnedhdf5(fpath_binned, combined=True)\n",
    "bindict.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read binned data over 4 axes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "fpath_binned = r'../data/data_114_4axis_binned.h5'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['ADC', 'X', 'Y', 't', 'V'])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bindict = fp.readBinnedhdf5(fpath_binned, combined=True)\n",
    "bindict.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['ADC', 'X', 'Y', 't', 'V0', 'V1', 'V10', 'V11', 'V12', 'V13', 'V14', 'V15', 'V16', 'V17', 'V18', 'V19', 'V2', 'V20', 'V21', 'V22', 'V23', 'V24', 'V25', 'V26', 'V27', 'V28', 'V29', 'V3', 'V30', 'V31', 'V32', 'V33', 'V34', 'V35', 'V36', 'V37', 'V38', 'V39', 'V4', 'V40', 'V41', 'V42', 'V43', 'V44', 'V45', 'V46', 'V47', 'V48', 'V49', 'V5', 'V6', 'V7', 'V8', 'V9'])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bindict = fp.readBinnedhdf5(fpath_binned, combined=False)\n",
    "bindict.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
