{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading ASCII files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How files are read and interpreted\n",
    "A common problem is to read ASCII files as the format of data is often not\n",
    "very intuitive designed. \n",
    "Often there is additional metadata before or after a matrix like block or in the filename.\n",
    "Or information is given as column headers in the first line (e.g csv data) \n",
    "\n",
    "Jscatter uses a simple concept to classify lines :\n",
    "\n",
    "* 2 numbers at the beginning of a line are data (matrix like data block).\n",
    "* a name followed by a number (and more) is an attribute with name and content.\n",
    "* everything else is a comment (but can later be converted to an attribute).\n",
    "* The filename is **always** stored in attribute ``.name``\n",
    "\n",
    "A new dataArray is created if, while reading a file, a data block with a parameter block\n",
    "(preceded or appended) is found or a keyword indicates the new dataset (see [dataArray](https://jscatter.readthedocs.io/en/latest/dataArray.html)).\n",
    "\n",
    "Often it is just necessary to replace some characters to fit into this concept.\n",
    "This can be done during reading using some simple options in *dataArray/dataList* creation:\n",
    "\n",
    "* replace={‘old’:’new’,’,’:’.’}     ==>  replace char and strings\n",
    "* skip complete bad lines by\n",
    "   ``skiplines=lambda words: any(w in words for w in [‘’,’ ‘,’NAN’,’‘*])``\n",
    "* takeline='ATOM'   ==> select specific lines\n",
    "* ignore ='#'       ==> skip lines starting with this character\n",
    "* usecols =[1,2,5]  ==> select specific columns\n",
    "* lines2parameter=[2,3,4]  ==> use these data lines as comment and not as data.\n",
    "\n",
    "See [dataArray](https://jscatter.readthedocs.io/en/latest/dataArray.html) for all options and how to use them.\n",
    "\n",
    "If there is more information in comments or filename this can be extracted by using the comment lines.\n",
    " * data.getfromcomment('nameatfirstcolumn') ==> extract a list of words in this line\n",
    " * data.name  ==> filename, see below examples.\n",
    " \n",
    "#### Prerequisite"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install jscatter\n",
    "%matplotlib notebook\n",
    "\n",
    "import jscatter as js\n",
    "import numpy as np\n",
    "js.usempl(True)   # force matplotlib, not needed in Linux"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### dataArray (js.dA) reads **one** dataset from a file    \n",
    "Read one dataset. If multiple then the first is choosen or as given by index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dat=js.dA(js.examples.datapath+'/1hho.Dq')                  # take dataset in file \n",
    "dat=js.dA(js.examples.datapath+'/iqt_1hho.dat',index=3)     # take 4th dataset in file with multiple datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### dataList  (js.dL) reads **all** datasets from one/multiple files (may differ in shape) \n",
    ".append(...) shares the same options as dL(...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data=js.dL(js.examples.datapath+'/iqt_1hho.dat')                          # read all dataset in file\n",
    "data=js.dL(js.examples.datapath+'/iqt_1hho.dat',index=2)                  # select only the second dataset in the file\n",
    "data=js.dL(js.examples.datapath+'/iqt_1hho.dat',index=[2,6,-1])           # select given in list\n",
    "data=js.dL(js.examples.datapath+'/iqt_1hho.dat',index=slice(None,None,2)) # select from slice, here each second\n",
    "data.append(dat)                                                          # append single dataArray read from above\n",
    "data.append(js.examples.datapath+'/iqt_1hho.dat',index=[2,6,-1])          # append with same possibilities as js.dL(...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Read all, filter later\n",
    "Create dataList from multiple files\n",
    "\n",
    "Uses [glob](https://docs.python.org/2/library/glob.html#module-glob) patterns  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data=js.dL(js.examples.datapath+'/*.dat')              # reads all '.dat' files starting with 'latest' in data folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Filter according to attributes or name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data2=data.filter(lambda a:a.name.find('/polymer')>-1)   # by name\n",
    "data3=data2.filter(lambda a:a.detDist==1205)             # e.g. detector distance\n",
    "data4=data[1:-1:3]                                       # drop first and last and use only each 3rd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same with glob (using name patterns as ?,*) treating"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "data=js.dL()  # empty dataList\n",
    "for filename in glob.glob(js.examples.datapath+'/*.dat'):\n",
    "   data.append(filename)                       # you may add options"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add attributes e.g. from filename or comments\n",
    "Attribut pressure was detected with following unit as list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to extract information from comment lines or names\n",
    "Lines as 'pressure 1013 14' are used automatically to set an as .pressure\n",
    "\n",
    "If data.name is a string like 'mydata_T_273_bla.dat' extract Temp as below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data=js.dL(js.examples.datapath+'/mydata_T*.dat') \n",
    "for dat in data:\n",
    "   dat.Temp=float(dat.name.split('_')[2])     # extract attribute from name (which is always the filename read from)\n",
    "   dat.mw=dat.comment[0].split()[2]     # if second comment line contains 'pressure 1000 mbar'\n",
    "\n",
    "data.showattr()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data=js.dA(js.examples.datapath+'/mydata_T_273_bla.dat')\n",
    "# lines as 'pressure 1013 14' are used automatically to set an attribute as .pressure\n",
    "# if data.name is a string as 'adh_Temp273_conc02.dat' extract Temp and conc like\n",
    "temp=data.name.split('_')\n",
    "data.Temp=float(temp[2])\n",
    "#\n",
    "# if same line is in comment use\n",
    "data.Temp=data.comment[0].split('_')\n",
    "# or use data.getfromComment(...)\n",
    "\n",
    "data.showattr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Some explicit examples and how to read them"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### File with matrix like data and name-value combinations \n",
    "filename: data1_273K_10mM.dat (e.g. Instrument JNSE@MLZ, Garching) ::"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datacontent='''\n",
    "this is just a comment or description of the data\n",
    "temp     293\n",
    "pressure 1013 14\n",
    "detectorsetting up\n",
    "name     temp1bsa\n",
    "0.854979E-01  0.178301E+03  0.383044E+02\n",
    "0.882382E-01  0.156139E+03  0.135279E+02\n",
    "0.909785E-01  0.150313E+03  0.110681E+02\n",
    "0.937188E-01  0.147430E+03  0.954762E+01\n",
    "0.964591E-01  0.141615E+03  0.846613E+01\n",
    "0.991995E-01  0.141024E+03  0.750891E+01\n",
    "0.101940E+00  0.135792E+03  0.685011E+01\n",
    "0.104680E+00  0.140996E+03  0.607993E+01\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read by"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data=js.dA(js.examples.datapath+'/data1_273K_10mM.dat')\n",
    "data.getfromcomment('detectorsetting')           # creates attribute detectorsetting with string value 'up' found in comments\n",
    "data.Temp=float(data.name.split('_')[1][:-1])    # extracts the temperature from filename\n",
    "data.conc=float(data.name.split('_')[2][:2])    # same for concentration\n",
    "\n",
    "data.showattr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NSE measurement from IN15 at ILL Grenoble \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datacontent=\"\"\"\n",
    "ftime         E_SUM       EERR_SUM    EQ_0.0596   EERRQ_0.0596  EQ_0.0662   EERRQ_0.0662   EQ_0.0728   EERRQ_0.0728   EQ_0.0793   EERRQ_0.0793   EQ_0.0859   EERRQ_0.0859\n",
    "Amplitude    -1.0000e+00  0.0000e+00  3.3149e+00  1.9984e-03  3.4203e+00  2.0375e-03  3.2560e+00  1.9803e-03  2.7188e+00  1.8161e-03  1.8634e+00  1.5032e-03\n",
    "Polarisation -1.0000e+00  0.0000e+00  2.3719e+00  4.4403e-03  2.3723e+00  4.6673e-03  2.1675e+00  4.6726e-03  1.7156e+00  4.4392e-03  1.1127e+00  3.7890e-03\n",
    "0.0000e+00 1.0000e+00  1.0318e-03  1.0000e+00  1.9261e-03  1.0000e+00  2.0252e-03  1.0000e+00  2.2186e-03  1.0000e+00  2.6615e-03  1.0000e+00  3.4992e-03\n",
    "2.2428e-01  9.7447e-01  3.4201e-03  9.7363e-01  6.3708e-03  9.7026e-01  6.6990e-03  9.8392e-01  7.3605e-03  9.8819e-01  8.8623e-03  9.5632e-01  1.1831e-02\n",
    "2.9474e-01  9.8425e-01  3.3694e-03  9.9020e-01  6.1962e-03  9.7785e-01  6.5809e-03  9.9125e-01  7.2723e-03  9.8005e-01  8.8698e-03  9.9022e-01  1.1909e-02\n",
    "3.6520e-01  9.7910e-01  3.3071e-03  9.8269e-01  6.0875e-03  9.8190e-01  6.4363e-03  9.7275e-01  7.1155e-03  9.8566e-01  8.7117e-03  9.7766e-01  1.1829e-02\n",
    "5.0612e-01  9.7927e-01  3.2226e-03  9.7898e-01  5.9112e-03  9.7517e-01  6.2379e-03  9.8108e-01  6.9563e-03  9.8669e-01  8.5569e-03  9.8611e-01  1.1557e-02\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read by "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    " # column 1,2 are averages over following columns. First line contains q values\n",
    " data=js.dL()   # empty dataList\n",
    " temp=js.dA(js.examples.datapath+'/017112345.txt')         # read all then sort later\n",
    " for i in [3,5,7,9]:\n",
    "    data.append(temp[[0,i,i+1]])\n",
    "    data[-1].Amplitude=temp.Amplitude[i-1:i+1]\n",
    "    data[-1].Polarisation=temp.Polarisation[i-1:i+1]\n",
    "    data[-1].q=float(temp.comment[0].split()[i].split('_')[1])\n",
    "    \n",
    "data.showattr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A complex file with atomic coordinates file\n",
    "aspirin.pdb: Atomic coordinates for aspirin [AIN](http://ligand-expo.rcsb.org/reports/A/AIN/AIN_ideal.pdb)\n",
    "from [Protein Data Bank, PDB](http://www.rcsb.org/ligand/AIN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "content='''\n",
    "Header\n",
    "Remarks blabla\n",
    "Remarks in pdb files are sometimes more than 100 lines\n",
    "ATOM      1  O1  AIN A   1       1.731   0.062  -2.912  1.00 10.00           O\n",
    "ATOM      2  C7  AIN A   1       1.411   0.021  -1.604  1.00 10.00           C\n",
    "ATOM      3  O2  AIN A   1       2.289   0.006  -0.764  1.00 10.00           O\n",
    "ATOM      4  C3  AIN A   1      -0.003  -0.006  -1.191  1.00 10.00           C\n",
    "ATOM      5  C4  AIN A   1      -1.016   0.010  -2.153  1.00 10.00           C\n",
    "ATOM      6  C5  AIN A   1      -2.337  -0.015  -1.761  1.00 10.00           C\n",
    "ATOM      7  C6  AIN A   1      -2.666  -0.063  -0.416  1.00 10.00           C\n",
    "ATOM      8  C1  AIN A   1      -1.675  -0.085   0.544  1.00 10.00           C\n",
    "ATOM      9  C2  AIN A   1      -0.340  -0.060   0.168  1.00 10.00           C\n",
    "ATOM     10  O3  AIN A   1       0.634  -0.083   1.111  1.00 10.00           O\n",
    "ATOM     11  C8  AIN A   1       0.314   0.035   2.410  1.00 10.00           C\n",
    "ATOM     12  O4  AIN A   1      -0.824   0.277   2.732  1.00 10.00           O\n",
    "ATOM     13  C9  AIN A   1       1.376  -0.134   3.466  1.00 10.00           C\n",
    "ATOM     14  HO1 AIN A   1       2.659   0.080  -3.183  1.00 10.00           H\n",
    "ATOM     15  H4  AIN A   1      -0.765   0.047  -3.203  1.00 10.00           H\n",
    "ATOM     16  H5  AIN A   1      -3.119   0.001  -2.505  1.00 10.00           H\n",
    "ATOM     17  H6  AIN A   1      -3.704  -0.082  -0.117  1.00 10.00           H\n",
    "ATOM     18  H1  AIN A   1      -1.939  -0.123   1.591  1.00 10.00           H\n",
    "ATOM     19  H91 AIN A   1       0.931  -0.004   4.453  1.00 10.00           H\n",
    "ATOM     20  H92 AIN A   1       1.807  -1.133   3.391  1.00 10.00           H\n",
    "ATOM     21  H93 AIN A   1       2.158   0.610   3.318  1.00 10.00           H\n",
    "CONECT    1    2   14 may appear at the end\n",
    "HETATOM lines may appear at the end\n",
    "END'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Read by (several methods)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take 'ATOM' lines, but only column 6-8 as x,y,z coordinates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "js.dA(js.examples.datapath+'/AIN_ideal.pdb',takeline='ATOM',replace={'ATOM':'0'},usecols=[6,7,8])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Replace 'ATOM' string by number and set XYZ for convenience"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "js.dA(js.examples.datapath+'/AIN_ideal.pdb',replace={'ATOM':'0'},usecols=[6,7,8],XYeYeX=[0,1,None,None,2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Only the Oxygen atoms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "js.dA(js.examples.datapath+'/AIN_ideal.pdb',takeline=lambda w:(w[0]=='ATOM') & (w[2][0]=='O'),replace={'ATOM':'0'},usecols=[6,7,8])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using regular expressions we can decode the atom specifier into a scattering length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re    # regular expression \n",
    "rHO=re.compile('HO\\d') # 14 is HO1\n",
    "rH=re.compile('H\\d+')  # represents something like 'H11' or 'H1' see regular expressions\n",
    "rC=re.compile('C\\d+')\n",
    "rO=re.compile('O\\d+')\n",
    "# replace atom specifier by number and use it as last column\n",
    "js.dA(js.examples.datapath+'/AIN_ideal.pdb',replace={'ATOM':'0',rC:1,rH:5,rO:2,rHO:5},usecols=[6,7,8,2],XYeYeX=[0,1,None,None,2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read only atoms and use it to retrieve atom data from js.formel.Elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "atoms=js.dA(js.examples.datapath+'/AIN_ideal.pdb',replace={'ATOM':'0'},usecols=[2],XYeYeX=[0,1,None,None,2])[0].array\n",
    "al=[js.formel.Elements[a[0].lower()] for a in atoms]\n",
    "al"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data with lots of non non number content in matrix\n",
    "data2.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "content='''\n",
    "# this is just a comment or description of the data\n",
    "# temp     ;    293\n",
    "# pressure ; 1013 14  bar\n",
    "# name     ; temp1bsa\n",
    "&doit\n",
    "0,854979E-01  0,178301E+03  0,383044E+02\n",
    "0,882382E-01  0,156139E+03  0,135279E+02\n",
    "0,909785E-01  *             0,110681E+02\n",
    "0,937188E-01  0,147430E+03  0,954762E+01\n",
    "0,964591E-01  0,141615E+03  0,846613E+01\n",
    "nan           nan           0\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read by\n",
    "- ignore is by default '#', so switch it of\n",
    "- skip lines with non numbers in data\n",
    "- replace some char by others or remove by replacing with empty string ''."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "js.dA(js.examples.datapath+'/data2.txt',replace={'#':'',';':'',',':'.'},skiplines=['*','nan'],ignore='' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### pdh format used in some SAXS instruments (first real data point is line 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "content='''\n",
    "SAXS BOX\n",
    "      2057         0         0         0         0         0         0         0\n",
    "  0.000000E+00   3.053389E+02   0.000000E+00   1.000000E+00   1.541800E-01\n",
    "  0.000000E+00   1.332462E+00   0.000000E+00   0.000000E+00   0.000000E+00\n",
    "-1.069281E-01   2.277691E+03   1.168599E+00\n",
    "-1.037351E-01   2.239132E+03   1.275602E+00\n",
    "-1.005422E-01   2.239534E+03   1.068182E+00\n",
    "-9.734922E-02   2.219594E+03   1.102175E+00\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read by\n",
    "- Use only first two column\n",
    "- Use lines 2,3,4 as comment\n",
    "- As demo: skip first 5  and last 50 and use each second"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this saves the prepended lines in attribute line_2,...\n",
    "empty=js.dA(js.examples.datapath+'/buffer_averaged_corrected_despiked.pdh',usecols=[0,1],lines2parameter=[2,3,4])\n",
    "# next just ignores the first lines (and last 50) and uses every second line,\n",
    "empty2=js.dA(js.examples.datapath+'/buffer_averaged_corrected_despiked.pdh',usecols=[0,1],block=[5,-50,2])\n",
    "\n",
    "empty.showattr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read csv data by (comma separated list) \n",
    "- replace ';' by ' ' (space)\n",
    "- if needed from Windows computers the encoding may be specified. encoding='cp1252' (US),'cp1251' (with German öäüß)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```js.dA('data2.txt',replace={',':' '})\n",
    "If tabs separate the columns\n",
    "js.dA('data2.txt',replace={',':' ','\\t':' '})\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
