

[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.4651325.svg)](https://doi.org/10.5281/zenodo.4651325) [![Documentation Status](https://readthedocs.org/projects/hextof-processor/badge/?version=latest)](https://hextof-processor.readthedocs.io/en/latest/?badge=latest)



# hextof-processor
This code is used to analyze data measured at FLASH using the HEXTOF (high 
energy X-ray time of flight) instrument. The HEXTOF uses a delay line detector 
(DLD) to measure the position and arrival time of single electron events.

The analysis of the data is based on "clean tables" of single events as dask dataframes. 
There are two dataframes generated in the data readout process. 
The main dataframe `dd` contains all detected electrons and can be binned according to the needs of the experiment. 
The second dataframe `ddMicrobunches` contains the FEL pulses and is commonly used for normalization.

The class `DldProcessor` contains the dask dataframes as well as the methods to perform binning in a parallelized fashion.

The `DldFlashDataframeCreatorExpress` class subclasses `DldProcessor` and is used for creating the dataframes from the hdf5 files generated by the DAQ system. 


# Installation

In this section we will walk you through all you need to get up and running with the hextof-processor.

For using this package with the old FLASH data structure, please refer to [README_DEPR.md](https://github.com/momentoscope/hextof-processor/blob/master/README_DEPR.md).

### 1. Python
If you don't have python on your local machine yet we suggest to start with `anaconda` or `miniconda`. 
Details about how to install can be found [here](https://docs.anaconda.com/anaconda/install/).

### 2. Install hextof-processor
Download the package by cloning to a local folder.

```bash
$ git clone https://github.com/momentoscope/hextof-processor.git
```
#### 2.1 Virtual environment
Create a clean new environment (We *strongly* suggest you to always do so!)

If you are using conda:
```bash
$ conda env create -f environment.yml
```
now, to activate your new environment (windows):
```bash
$ conda activate hextof-express
```
if you are using linux:
```bash
$ source activate hextof-express
```
#### 2.2 Virtual environment in Jupyter Notebooks

To add the newly created environment to the **Jupyter Notebooks** kernel list,
and install your new kernel:
```bash
(hextof-express)$ python -m ipykernel install --user --name=hextof-express
```
### 3. Local Setup
#### 3.1 Initialize settings

Finally, you need to initialize your local settings. This can be done by running 
`InitializeSettings.py`, in the same repository folder

```bash
(hextof-env)$ python InitializeSettings.py
```

This will create a file called **SETTINGS.ini** in the local repository folder. This is used to store the local settings as well as calibration values (will change in future..) and other options. 

#### 3.2 Setting up local paths
In order to make sure your folders are in the right place, open this file and modify the paths in the `[path]` section.
 * **data_raw_dir** - location where the raw h5 files from FLASH are stored
 * **data_h5_dir** - storage of binned hdf5 files
 * **data_parquet_dir** where the apache parquet data files from the generated single event tables are stored (we suggest using an SSD for this folder, since would greatly improve the binning performance.)
 * **data_results_dir** folder where to save results (figures and binned arrays)

if you are installing on Maxwell, we suggest setting the following paths:
```ini
[paths]
data_raw_dir =     /asap3/flash/gpfs/pg2/YYYY/data/xxxxxxxx/raw/
data_h5_dir =      /asap3/flash/gpfs/pg2/YYYY/data/xxxxxxxx/processed/
data_parquet_dir = /asap3/flash/gpfs/pg2/YYYY/data/xxxxxxxx/processed/parquet/
data_results_dir = /asap3/flash/gpfs/pg2/YYYY/data/xxxxxxxx/processed/*USER_NAME*/binned/
```
Where `YYYY` is the current year and `xxxxxxxx` is the beamtime number.

#### 3.3 Calculating sector_correction list
If you like, in the settings, you can add the sector_correction list, which will shift any misalignment of the sectors.
At the very least, this should include the "bit stealing hack" correction, where the last bits of the dldTime
are set so they encode dldSectorId. This can be achieved by using the calibration.gen_sector_correction function
which will generate the list for you, given the energy shifts you want.

#### 3.4 Installing Doniach-Sunjic gaussian broadened
Please refer to [XPSdoniachs/README.md](https://github.com/momentoscope/hextof-processor/blob/master/XPSdoniachs/README.md) for compilation instructions.

### 4. Test your installation
In order to test your local installation, we have provided a series of tutorial Jupyter Notebooks. 
You can find all the relevant material in the [tutorial folder](https://github.com/momentoscope/hextof-processor/tree/master/tutorial) in the main repository.
We suggest setting testing [Express data readout.ipynb](https://github.com/momentoscope/hextof-processor/blob/master/tutorial/Express%20data%20readout.ipynb).


# Documentation

The documentation of the package can be found [here](https://momentoscope.github.io/hextof-processor/).

Examples are available as [Jupyter Notebooks](https://github.com/momentoscope/hextof-processor/tree/master/tutorial).
Some example data is provided together with the examples. More compatible data is being collected and will soon be added to online open-access repositories.

# Citation and and acknowledgments 

If you use this software, please consider citing these two papers:
 * [D. Kutnyakhov *et al.* Review of Scientific Instruments 91, 013109 (2020);](https://aip.scitation.org/doi/full/10.1063/1.5118777)
 * [R. P. Xian *et al.* Scientific Data 7, 442 (2020)](https://www.nature.com/articles/s41597-020-00769-8).
