{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# User Guide\n",
    "## Examples on how to store and retrieve data using StorageGroup "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "import json\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import requests\n",
    "import xarray as xr\n",
    "from tonik import StorageGroup "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can define any directory structure within a  storage group. We here use the hierarchy that is common for seismometers which is network, site, sensor, and channel. Data can be stored under each of them. We will start by generating fake spectrogram data and then store that under a channel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 29148 characters]",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dates = pd.date_range(\"2024-01-02\", freq='10min', periods=288)\n",
    "data = np.abs(np.cumsum(np.random.normal(0, 8., len(dates))))\n",
    "data = np.tile(data, (10, 1))\n",
    "freqs = np.arange(10)\n",
    "xrd = xr.Dataset({'spectrogram': xr.DataArray(data, coords=[freqs, dates],\n",
    "                                              dims=['frequency', 'datetime'])})\n",
    "fig = xrd['spectrogram'].plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will store the data under two different sites of the same experiment using the same sensor and channel names for both."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Group: experiment\n",
       "/tmp/experiment\n",
       "  /tmp/experiment/NET1\n",
       "    /tmp/experiment/NET1/MDR1\n",
       "      /tmp/experiment/NET1/MDR1/00\n",
       "        /tmp/experiment/NET1/MDR1/00/HHZ\n",
       "    /tmp/experiment/NET1/MDR2\n",
       "      /tmp/experiment/NET1/MDR2/00\n",
       "        /tmp/experiment/NET1/MDR2/00/HHZ"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "g = StorageGroup('experiment', rootdir='/tmp')\n",
    "st1 = g.get_store('NET1', 'MDR1', '00', 'HHZ')\n",
    "st2 = g.get_store('NET1', 'MDR2', '00', 'HHZ')\n",
    "st1.save(xrd)\n",
    "st2.save(xrd)\n",
    "g"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we want to retrieve the data we just saved. Before retrieving data we have to set the timespan over which we want to retrieve data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.QuadMesh at 0x7f6bfcacaec0>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 25924 characters]",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "g.starttime = datetime(2024, 1, 2, 18, 0, 0)\n",
    "g.endtime = datetime(2024, 1, 3, 6, 0, 0)\n",
    "st = g.get_store('NET1', 'MDR2', '00', 'HHZ')\n",
    "st('spectrogram').plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now start the API server by running the following command in a terminal\n",
    "# tonik_api --rootdir /tmp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first have a look what is available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'experiment': [{'NET1': [{'MDR1': [{'00': [{'HHZ': ['spectrogram']}]}]},\n",
       "    {'MDR2': [{'00': [{'HHZ': ['spectrogram']}]}]},\n",
       "    {'MDR3': ['rsam']},\n",
       "    {'MDR3name=rsam': []}]}]}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url = f\"http://localhost:8003/inventory?group={g.name}\"\n",
    "requests.get(url).json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's request the data we just stored."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.QuadMesh at 0x7f6bfca34040>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 23344 characters]",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "url = \"http://localhost:8003/feature?\"\n",
    "url += f\"group={g.name}&subdir=NET1&subdir=MDR2&subdir=00&subdir=HHZ&name=spectrogram\"\n",
    "url += f\"&starttime={g.starttime.isoformat()}&endtime={g.endtime.isoformat()}\" \n",
    "spec = pd.read_csv(url, parse_dates=True, index_col=[0, 1], date_format='ISO8601').to_xarray()\n",
    "spec.transpose('freqs', 'dates')['feature'].plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we repeated the `subdir` parameter multiple times. It is important that you pass the subdirs in the order of your directory structure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using pandas instead of xarray"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we'll create some fake data again. Note that we name the feature in the DataFrame and also that we name the index `datetime`. The `save` function uses this information when creating the files on disk. If the dataframe contains multiple features they will be stored in separate files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: xlabel='datetime'>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 55964 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dates = pd.date_range(\"2024-01-02\", freq='10min', periods=288)\n",
    "data = np.abs(np.cumsum(np.random.normal(0, 8., len(dates))))\n",
    "df = pd.DataFrame({'rsam': data}, index=dates)\n",
    "df.index.name = 'datetime'\n",
    "df.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For illustration, this time we store the data under the site."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "st3 = g.get_store('NET1', 'MDR3')\n",
    "st3.save(df.to_xarray())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: xlabel='dates'>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 51812 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "url = \"http://localhost:8003/feature?\"\n",
    "url += f\"group={g.name}&subdir=NET1&subdir=MDR3&name=rsam\"\n",
    "url += f\"&starttime=2024-01-02T00:00:00&endtime=2024-01-04T00:00:00\" \n",
    "rsam = pd.read_csv(url, parse_dates=True, index_col=0, date_format='ISO8601')\n",
    "rsam.plot()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
