{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "from py4xs.hdf import h5xs,h5exp,lsh5\n",
    "from py4xs.data2d import Axes2dPlot\n",
    "import pylab as plt\n",
    "import numpy as np\n",
    "import warnings,json\n",
    "\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detector configuration assignment\n",
    "\n",
    "Before data processing can start, the detector configurations need to be assoicated with the h5 file. If the detector information has been previously saved, it can be recalled automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "dt = h5xs(\"data/AgBH-2020Aug16.h5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The h5 file itself can be accessed using the fh5 attribute of the h5xs object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## h5 data organization \n",
    "\n",
    "The h5 file packaged at the LiX beamline is a selected copy of information from the NSLS-II databroker, \n",
    "organized by sample, stream, then data groups. This data structure can be visualized using a viewer like\n",
    "hdfview, or using the lsh5() function. For instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " AgBH_2020Aug16\n",
      "= primary\n",
      "== data\n",
      "=== em1_sum_all_mean_value\n",
      "=== em2_current1_mean_value\n",
      "=== em2_current2_mean_value\n",
      "=== em2_sum_all_mean_value\n",
      "=== pil1M_image\n",
      "=== pilW2_image\n",
      "== time\n",
      "== timestamps\n",
      "=== em1_sum_all_mean_value\n",
      "=== em2_current1_mean_value\n",
      "=== em2_current2_mean_value\n",
      "=== em2_sum_all_mean_value\n",
      "=== pil1M_image\n",
      "=== pilW2_image\n",
      "= processed\n",
      "== _SAXS\n",
      "== _WAXS2\n",
      "== merged\n"
     ]
    }
   ],
   "source": [
    "lsh5(dt.fh5, print_attrs=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: in fly scans, the monitor data (transmitted beam intensity) are recorded in separate streams from *primary*. These data typically do not have the same time stamps. The fast axis also moves in the snake pattern to save time. Thefore the position of the fast-axis motor and the corresponding monitor counts may not be accessed by simple indexing.   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generation of intensity maps/profiles \n",
    "\n",
    "Currently py4xs provides batch processing to perform aximuthal average on all data contained in the h5 file, including averaging on each detector, then merging the data from multiple detectors. This will create new data under the *processed* group. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data received: sn=AgBH_2020Aug16, fr1=0\n"
     ]
    }
   ],
   "source": [
    "dt.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['AgBH_2020Aug16']"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(dt.d1s.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['merged', '_SAXS', '_WAXS2']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(dt.d1s[dt.samples[0]].keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 20824 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "dt.d1s[dt.samples[0]]['merged'][0].plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Batch processing for 2D data conversion will eventually become avialable, once decisions are made on what values will be used for the $q$ and $\\phi$ grid, and utilities are put in place to save and load the converted data. For now this kind of conversion can be performs frame by frame, as shown in the previous example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "d2s = dt.get_d2(det_ext=\"_SAXS\", frn=0)\n",
    "d2s.conv_Iqphi(100, 60, mask=dt.detectors[0].exp_para.mask)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 14104 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(6,4))\n",
    "ax = plt.gca()\n",
    "paxr = Axes2dPlot(ax, d2s.qphi_data)\n",
    "paxr.plot(aspect=\"auto\", log=True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "## Background subtraction\n",
    "\n",
    "Background subtraction is essential, especially for samples that do not produce prominet diffraction peaks. During the experiments, be sure that empty cell scattering is measured. This may be position-dependent if the sample is measured in air. The scattering intensity must be scaled based on the transmitted beam intensity before subtraction. This value is recorded in the <i>em2_sum_all_mean_value</i> datagroup, which can be specified when the h5xs object is initilized. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "dt = h5xs(\"data/AgBH-2020Aug16.h5\", transField=\"em2_sum_all_mean_value\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This value is then passed on to the *Data1d* objects in `load_data()`. Similar feature will need to be implemented for 2D data, so that this information can be preserved in the data derived from the 2D maps. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature extraction\n",
    "Extracting features from the processed 1D or 2D data should be straightforward. This will mainly rely on the code provided by the user, and very specific to the type of structures only the user may be kownledgable of. Some generic functions are trivial to implement. For instance, *numpy* can be used to integrate intensity in a given $q$-range or calculate the center-of-mass of a angular distribution. Others are mmore complex to implement and could be included in py4xs, e.g. peak finding (assuming that there are only a handful). Requests may be entertained.   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parallelization\n",
    "\n",
    "Currently py4xs makes use of multi-processing when batch processing the data (in `load_data()`). The number of processes used\n",
    "can be specified. Note that the gain in procesing speed is not linearly proportional\n",
    "to the number of processes used. And sometimes a single process needs to be used to avoid crashes.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
