{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Jscatter\n",
    "\n",
    "#### The aim of Jscatter is treatment of experimental data and models\n",
    "\n",
    "\n",
    "* Reading and analyzing experimental data with associated attributes as temperature, wavevector, comment, ....\n",
    "* **Multidimensional fitting** taking attributes into account.\n",
    "* Providing useful models for **neutron and X-ray scattering** as form factors, structure factors\n",
    "  and dynamic models (quasi elastic neutron scattering) and other topics.\n",
    "* Simplified plotting with paper ready quality (preferred in xmgrace).\n",
    "* Easy model building for non programmers.\n",
    "* Python scripts to document data evaluation and modelling.\n",
    "\n",
    "For more details see the \n",
    "[Documentation](https://jscatter.readthedocs.io/en/latest/index.html), \n",
    "[Beginners Guide](https://jscatter.readthedocs.io/en/latest/BeginnersGuide.html) or\n",
    "[Examples](https://jscatter.readthedocs.io/en/latest/examples.html)\n",
    "\n",
    "Included models are documented at \n",
    "[formfactor](https://jscatter.readthedocs.io/en/latest/formfactor.html), \n",
    "[structurefactor](https://jscatter.readthedocs.io/en/latest/structurefactor.html) or\n",
    "[dynamic](https://jscatter.readthedocs.io/en/latest/dynamic.html)\n",
    "\n",
    "This Notebook aims to give a short introduction for Beginners with the possibility to play around with some examples to explore the posibilities without the need to do a full installation.\n",
    "\n",
    "A typical example for data analysis is given in [Jscatter_DataAnalysis.ipynb](Jscatter_DataAnalysis.ipynb).\n",
    "\n",
    "### Content\n",
    "- Demonstration for 1D data in **dataArray** with later fitting a parabola\n",
    "- Demonstration of multiple 1D data in **dataList**\n",
    "- Demonstration of multiple 1D data in a **dataList** using attributes from read data\n",
    "- 2D fit data with an X,Z grid data and Y values\n",
    "\n",
    "\n",
    "![Jscatter](https://jscatter.readthedocs.io/en/latest/_images/Jscatter.jpeg \"Jscatter\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Prerequisite \n",
    "Install Jscatter and prerequisites on readonly Jupyter server\n",
    "\n",
    "For installing on Linux, Mac or Windows visit the [installation instructions](https://jscatter.readthedocs.io/en/latest/Installation.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys,site\n",
    "# Install jscatter as user (optional: \"==0.9.3\" forces a specific version)\n",
    "!{sys.executable} -m pip install jscatter --user          \n",
    "sys.path.append(site.USER_BASE)\n",
    "%matplotlib notebook\n",
    "\n",
    "import jscatter as js\n",
    "import numpy as np\n",
    "js.usempl(True)   # force matplotlib, not needed in Linux"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## dataArray with 1D data and later fitting \n",
    "\n",
    "**dataArray** is a container for matrix like data like a spreadsheet (numpy ndarray). Additional to the entries in the spreadsheet we have attributes describing metadata like the temperature of the measurement related to the spreadsheet data accessible as data.temperature.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load example data into dataArray, add attributes and inspect it\n",
    "Additional options allow tunning how datafiles are read. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data=js.dA(js.examples.datapath+'/exampledata0.dat')\n",
    "data.pressure=200\n",
    "data.q=0.96\n",
    "data.comment=['Its cold here as _2 K']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The attribute temperature is found in the ASCII file as line ``temperature -20`` and is automatically added. Attributes are identified from lines with name followed by a number (and more)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.Y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Inspect attributes and extract one from comment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.attr    # get list of attributes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.showattr()    # show attributes with content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data.pressure    # This may be used for model calculations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comment is all from ASCII file what is not identified as data or attribute with name. We can extract new attributes from the comments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.comment    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.Temp=float(data.comment[0].split('_')[1].split()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.Temp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Manipulate data\n",
    "\n",
    "Calculations work with vectorized data, so data.Y is a column in the dataArray.\n",
    "Numpy functions like np.sin or np.exp can deal with this and speedup computations. \n",
    "Also matrix operations are feasible.\n",
    "\n",
    ".Y column can be changed by setColumnIndex(...). \n",
    "Direct indexing is done in usual way. Copy includes also attributes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data0=data.copy()                # copy with attributes\n",
    "data0.Y = data0.Y*data0.q**2     # inplace changes\n",
    "data0[1] = data0[1]* data0.q**2  # same by direct indexing \n",
    "data0[1,::2]=0                   # each second of 1 column set to zero\n",
    "\n",
    "# create new dataArray with additional column\n",
    "data2 = data0.addColumn(n=1,values=data[0]**2*np.sin(data[1]))           "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Slicing, indexing, cut, boolean indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data4 = data[:,[1,2,3,4]]               # take index 1,2,3,4\n",
    "data5 = data[:,3:-3:2]                  # cut first and last 3 and ech second\n",
    "data6 = data[:,data.X>0]                # take only positive .X (Boolean or “mask” index arrays from numpy)\n",
    "data6[:,data6.X>1] = data6[:,data6.X>1]*2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# p=js.grace() # on Linux using Xmgrace\n",
    "p=js.mplot()   # empty plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using dataarray attributes .X .Y .eY  \n",
    "Needed attributes are taken automatically and used for plotting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p.Plot(data,le='test $q 1/nm')   # use .X,.Y,.eY automatically"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explicit given X, Y and eY components. \n",
    "Original matplotlib functionallity is accessible with small letter commands (e.g. p.title(...) ) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p.Plot(data.X ,np.abs(data.Y*data.X),np.ones_like(data.X)*20,le='more')\n",
    "p.Plot(data.X,10*data.X**2,symbol=0,line=1,legend='xxxxx')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pretty Up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p.Yaxis(label='test')\n",
    "p.Title(r'This is a nonsense plot with $\\int_0^\\inf\\xi \\partial\\xi$')\n",
    "p.Legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to fit\n",
    "#### Define model \n",
    "A model gets as input a number of parameters where typically one is an array with X values. The return value should be the model values Y or a **dataArray** with ``.Y``.\n",
    "Here ``q`` gets list of values as numpy array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def parabola(q,a,b,c,d):\n",
    "   return d*(q-a)**2+b*q+c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fit it defining the model, free parameters, fixed parameters and map model names to data names\n",
    "#### Use free *d* to improve fit\n",
    "('X' is always x axis) We get some output thats documents fit progress and the final result if successful.\n",
    "\n",
    "Additional options define conditions, fit methods and more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.fit(parabola ,freepar={'a':2,'b':4,'d':1}, fixpar={'c':-20}, mapNames={'q':'X'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Show an errorPlot with the result. \n",
    "Repeating the .fit command will always plot to the same errPlot. The errplot can be created (empty) prior to the fit by .makeErrPlot to observe the progress graphically (only in xmgrace on Linux as matplotlib is too slow.)\n",
    "\n",
    "We may simulate a changed parameter by setting it like``data.showlastErrPlot(a=6)``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "data.showlastErrPlot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The fit result is accessible in the .lastfit attribute as dataArray and should be saved as the fit result.\n",
    "\n",
    "1-$\\sigma$ errors are accessible as par_err for free parameters. Additional the $\\chi^2$ and covariance matrix are included as attributes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.lastfit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.a_err"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## dataList of multiple 1D data\n",
    "\n",
    "**dataList** is a container for a list of dataArrays with variable size.\n",
    "\n",
    "#### Create some data as sinusoidal with changing amplitude an phase"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import jscatter as js\n",
    "import numpy as np\n",
    "\n",
    "x = np.r_[0:10:0.1]\n",
    "data = js.dL()\n",
    "ef = 0.1  # increase this to increase error bars of final result\n",
    "for ff in [0.001, 0.4, 0.8, 1.2, 1.6]:\n",
    "    data.append(np.c_[x, (1.234 + ff) * np.sin(x + ff) + ef * ff * np.random.randn(len(x)), x * 0 + ef * ff].T)\n",
    "    data[-1].B = 0.2 * ff / 2  # add attributes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sinusoidal model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sinus(x, A, a, B, p):\n",
    "    return A * np.sin(a * x + p) + B"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fit with a common parameter *p* for all data (``float`` start value), obviously wrong result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data.fit(sinus, {'a': 1.2, 'p': 0, 'A': 1.2}, {}, {'x': 'X'})\n",
    "data.showlastErrPlot()  # show fit\n",
    "data.errPlotTitle('Fit Sine with attribut and common fit parameter')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Improve fit using independent `p`, `A` and `B` \n",
    "Independent fit parameters are indicated using a list of start parameters ('B' = [0,0.1]). \n",
    "Mixing is allowed. The `[]` indicate the list, the last start parameter in this list is repeated to have a value for all missing.\n",
    "\n",
    "The same is used for different fixed parameter values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data.fit(sinus, {'a': 1.2, 'p': [0], 'B': [0, 0.1], 'A': [1]}, {}, {'x': 'X'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Change title."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data.errPlotTitle('Fit Sine with attribut and non common fit parameter')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## dataList of multiple 1D data using attributes from read data\n",
    "\n",
    "#### Here intermediate scattering function for 16 wavevectors \n",
    "The data corresponds to the measurement of a protein with translational and rotational diffusion as measured by Neutron Spinecho Spectroscopy.\n",
    "\n",
    "Each dataArray has a parameter q for the wavevector that is automatically used in the diffusion model.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i5=js.dL(js.examples.datapath+'/iqt_1hho.dat')\n",
    "print(i5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Look at attribute q as wavevector "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(i5.q)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Model for fitting\n",
    "Use numpy functions (here np.exp) as these deal with numpy arrays !!!\n",
    "\n",
    "Also lineal algebra as matrix multilication is done in a fast way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def diffusion(A,D,t,elastic,wavevector=0):\n",
    "    return A*np.exp(-wavevector**2*D*t)+elastic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### makeErrPlot: see progress of intermediate steps with residuals \n",
    "(updated all 2 seconds if used on your personal Linux computer with xmgrace, matplotlib is too slow)\n",
    "\n",
    "Change lin/log scale by pressing k,l for x,y axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i5.makeErrPlot(title='diffusion model residual plot')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fit it\n",
    "We need only to define that the 'wavevector' is named 'q' in th data. The values are used as found in the data as attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i5.fit(model=diffusion,                                 # the fit function\n",
    "       freepar={'D':[0.2],'A':1},                       # start values; [..] indicate independent fit parameter\n",
    "       fixpar={'elastic':0.0},                          # fixed parameters, single values indicates common parameter\n",
    "       mapNames= {'t':'X','wavevector':'q'},            # map names of the model to names of data attributes\n",
    "       condition=lambda a: (a.X>0.01) & (a.Y>0.01)  )   # a condition to include only specific values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspect resulting parameters and 1-$\\sigma$ errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "print(i5.D)\n",
    "print(i5.D_err)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plot the main result from the fit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p=js.mplot()\n",
    "p.Plot(i5.q,i5.D,i5.D_err,symbol=[1,1,3],le='trans. + rot. diffusion')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "scrolled": false
   },
   "source": [
    "Pretty Up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "p.Yaxis(label=r'$D\\; /\\; nm^2/ns$')\n",
    "p.Xaxis(label=r'$Q\\; /\\; nm^{-1}$')\n",
    "p.Plot([i5.q.min,i5.q.max],[i5.D[:4].mean()]*2 ,li=[2,3,1],sy=0, le='trans. diffusion')\n",
    "p.Legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Last fit result in .lastfit  with errors and covariance matrix\n",
    "Save it with data.lastfit.save('mydata_fitdiffusionmodel.dat')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i5.lastfit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Simulate with changed parameters\n",
    "We can also simulate in the above errPlot by ``data.showlastErrPlot(D=0.1)``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "i5.modelValues(D=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2D fit data with an X,Z grid data and Y values \n",
    "\n",
    "As we use for 1D data independent variables .X with dependent variable .Y (measurement or output of a model) we use for 2D and 3D data dependent variable .X,.Z,.W to stay at the dependent .Y variable name. \n",
    "\n",
    "For fitting we need data with X,Z,W,Y columns indicated .\n",
    "\n",
    "In this example we create synthetic data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### We create synthetic 2D data with X,Z axes and Y values as Y=f(X,Z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is ONE way to make a grid. For fitting it can be unordered, non-gridded X,Z data\n",
    "x, z = np.mgrid[-5:5:0.25, -5:5:0.25]\n",
    "xyz = js.dA(np.c_[x.flatten(), z.flatten(), 0.3 * np.sin(x * z / np.pi).flatten() + 0.01 * np.random.randn(\n",
    "    len(x.flatten())), 0.01 * np.ones_like(x).flatten()].T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Set columns where to find X,Z coordinates and Y values and eY errors )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xyz.setColumnIndex(ix=0, iz=1, iy=2, iey=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define model and fit it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ff(x, z, a, b):\n",
    "    return a * np.sin(b * x * z)\n",
    "\n",
    "\n",
    "xyz.fit(model=ff, freepar={'a': 1, 'b': 1 / 3.}, fixpar={}, mapNames={'x': 'X', 'z': 'Z'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Show in 2D plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "from matplotlib import cm\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111, projection='3d')\n",
    "ax.set_title('2D Sinusoidal fit')\n",
    "# plot data as points\n",
    "ax.scatter(xyz.X, xyz.Z, xyz.Y)\n",
    "# plot fit as contour lines\n",
    "ax.tricontour(xyz.lastfit.X, xyz.lastfit.Z, xyz.lastfit.Y, cmap=cm.coolwarm, antialiased=False)\n",
    "plt.show(block=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## For more examples please visit the Jscatter documentation."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
