{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analysing experimental data\n",
    "\n",
    "We need:\n",
    "\n",
    "- **Data** need to be read and prepared.\n",
    " - Data may be a single dataset (usually in a *dataArray*) or several of these (in a *dataList*) like\n",
    "   multiple measurements with same or changing parameters (e.g. wavevectors).\n",
    "   Coordinates are in *.X* and values in *.Y*\n",
    " - 2D data (e.g. a detector image with 2 dimensions) need to be transformed to coordinates\n",
    "   *.X*,*.Z* with values in *.Y*. This also gives pixels coordinates in an image a physical\n",
    "   interpretation as e.g. wavevectors.\n",
    "   See examples ['2D fitting'](https://jscatter.readthedocs.io/en/latest/examples.html#d-fitting) and [`Fitting the 2D scattering of a lattice`](https://jscatter.readthedocs.io/en/latest/examples.html#fitting-the-2d-scattering-of-a-lattice)\n",
    " - Attributes need to be extracted from read data (from Comments or filename or from a list..)\n",
    "   In the below example the Temperature is stored in the data as attribute.\n",
    "\n",
    "- A **model**, which can be defined in different ways.\n",
    "\n",
    " - See below or in the notebook `Jscatter_BuildModels` for different ways.\n",
    "\n",
    "  - Please avoid using lists as parameters as list are used to discriminate\n",
    "    between common parameters (a single float) and individual fit parameters\n",
    "    (a list of float for each) in dataList.\n",
    "\n",
    "- **Fit algorithm**\n",
    "\n",
    "  We use methods from the scipy.optimize module that are incorporated in the *.fit*\n",
    "  method of *dataArray/dataList*. *.fit* supports different fit algorithms\n",
    "  (see dataList.fit Examples how to choose and about speed differences):\n",
    " - method='leastsq' (default) is a wrapper around MINPACK’s lmdif and lmder, which is a modification\n",
    "   of the [Levenberg-Marquardt](https://en.wikipedia.org/wiki/Levenberg-Marquardt_algorithm) algorithm.\n",
    "   This is what you usually expect by \"fitting\" including error bars (and a covariance matrix for experts....).\n",
    "   Errors are *1-sigma* errors as they are calculated from the covariance matrix and not directly depend\n",
    "   on the errors of the *.eY*. Still the relative weight of values according to *.eY* is relevant.\n",
    " - method='BFGS' Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm.  Not as fast as 'leastsq' but gives also error bars.\n",
    " - method='differential_evolution' is a global optimization method using iterative improving candidate solutions.\n",
    "   In general it needs a large number of function calls but may find a global minimum.\n",
    " - method=‘Nelder-Mead’,'CG', ..... other optimization methods.\n",
    "   These are slower converging than 'leastsq' and give no error bars.\n",
    "   Some require a gradient function ore more.\n",
    "   They are more for advanced users if someone really knows why using it in special cases.\n",
    "\n",
    "### Warning:: **Save results !!!**\n",
    "\n",
    "- The resulting parameters with errors are in *.lastfit*. **Save the fit result !!!**\n",
    " \n",
    "- I regularly have to ask PhD students \"What are the errors ?\" and they repeat all their work again.\n",
    " \n",
    "- Fit results without Errors are meaningless.\n",
    "\n",
    "\n",
    "**If the fit finishes it tells if there was success or if it failed. For error messages the final parameters are no valid fit results!!! Please always check this.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A typical example of several datasets from a SAXS measurement (polymer in solution)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install jscatter \n",
    "%matplotlib notebook\n",
    "\n",
    "import jscatter as js\n",
    "import numpy as np\n",
    "js.usempl(True)   # force matplotlib, not needed in Linux"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Read data\n",
    "\n",
    "Inspect attributes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data=js.dL(js.examples.datapath+'/polymer.dat')\n",
    "data.showattr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### Merge equal Temperatures each measured with two detector distances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.mergeAttribut('Temp',limit=0.01,isort='X')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### define model\n",
    "q will get the X values from your data as numpy ndarray."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gCpower(q,I0,Rg,A,beta,bgr):\n",
    "    \"\"\"Model Gaussian chain  + power law and background\"\"\"\n",
    "    gc=js.ff.gaussianChain(q=q,Rg=Rg)\n",
    "    # add power law and background\n",
    "    gc.Y=I0*gc.Y+A*q**beta+bgr\n",
    "    # add attributes for later documentation, these are additional content of lastfit (see below)\n",
    "    gc.A=A\n",
    "    gc.I0=I0\n",
    "    gc.bgr=bgr\n",
    "    gc.beta=beta\n",
    "    gc.comment=['gaussianChain with power law and bgr','a second comment']\n",
    "    return gc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Make errplot and set limits\n",
    "For log scale klick on the plot and use k,l keys to switch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.makeErrPlot()           # additional errorplot with intermediate output\n",
    "data.setlimit(bgr=[0,1])     # upper and lower soft limit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Here we use individual parameter ([]) for all except a common beta ( no [] )\n",
    "- Please try removing the [] and play with it :-)\n",
    "- mapNames tells that q is *.X* (maps model names to data names )\n",
    "- condition limits the range to fit (may also contain something like (a.Y>0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.fit(model=gCpower,\n",
    "         freepar={'I0':[0.1],'Rg':[3],'A':[1],'bgr':[0.01],'beta':-3},\n",
    "         fixpar={},\n",
    "         mapNames={'q':'X'},\n",
    "         condition =lambda a:(a.X>0.05) & (a.X<4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To fix a parameter move it to fixpar dict (bgr is automatically extended)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.fit(model=gCpower,\n",
    "         freepar={'I0':[0.1],'Rg':[3],'A':[1]},\n",
    "         fixpar={'bgr':[0.001, 0.0008, 0.0009],'beta':-4},\n",
    "         mapNames={'q':'X'},\n",
    "         condition =lambda a:(a.X>0.05) & (a.X<4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Result parameter and error (example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.lastfit.Rg\n",
    "data.lastfit.Rg_err\n",
    "# as result dataArray\n",
    "result=js.dA(np.c_[data.Tempmean,data.lastfit.Rg,data.lastfit.Rg_err].T)\n",
    "# plot it\n",
    "p=js.mplot()\n",
    "p.Plot(result,sy=[1,0.5,4],li=[2,2,1])\n",
    "p.Xaxis(label='Temperature /  Celsius')\n",
    "p.Yaxis(label='$R_{gyration}\\; /\\;  nm$')\n",
    "# save the fit result including parameters, errors and covariance matrix\n",
    "# and your model description\n",
    "data.lastfit.save('polymer_fitDebye.dat')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### You may want to fit dataArray in a dataList individually.\n",
    "Do it in a loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from the above\n",
    "for dat in data:\n",
    "   dat.fit(model=gCpower,\n",
    "        freepar={'I0':0.1,'Rg':3,'A':1,},\n",
    "        fixpar={'bgr':0.001,'beta':-3},\n",
    "        mapNames={'q':'X'},\n",
    "        condition =lambda a:(a.X>0.05) & (a.X<4))\n",
    "\n",
    "# each dataArray has its own .lastfit, .errPlot and so on\n",
    "data[0].showlastErrPlot()\n",
    "# for saving with individual names\n",
    "for dat in data:\n",
    "   dat.lastfit.save('fitresult_Temp%2g.dat' %(dat.Tempmean))\n",
    "\n",
    "# to get a list of the lastfit\n",
    "fitresult=js.dL([dat.lastfit for dat in data])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### To get a list of the lastfit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fitresult=js.dL([dat.lastfit for dat in data])\n",
    "fitresult"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
