{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This notebook shows step by step how water leaks of different severity can be detected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import pytz\n",
    "import inspect\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import datetime as dt\n",
    "import matplotlib.pyplot as plt\n",
    "import tmpo\n",
    "\n",
    "from opengrid import config\n",
    "from opengrid.library import plotting\n",
    "from opengrid.library import houseprint\n",
    "\n",
    "c=config.Config()\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = 16,8\n",
    "\n",
    "# path to data\n",
    "path_to_data = c.get('data', 'folder')\n",
    "if not os.path.exists(path_to_data):\n",
    "    raise IOError(\"Provide your path to the data in your config.ini file.  This is a folder containing a 'zip' and 'csv' subfolder.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "hp = houseprint.Houseprint()\n",
    "\n",
    "#set start and end date\n",
    "end = pd.Timestamp('2015/1/4')\n",
    "start = end - dt.timedelta(days=60)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "hp.init_tmpo()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def dif_interp(ts, freq='min', start=None, end=None):\n",
    "    \"\"\"\n",
    "    Return a fixed frequency discrete difference time series from an unevenly spaced cumulative time series\n",
    "    \"\"\"\n",
    "    if ts.empty and (start is None or end is None):\n",
    "        return ts\n",
    "    start = start or ts.index[0]\n",
    "    start = start.replace(tzinfo=pytz.utc)\n",
    "    end = end or max(start, ts.index[-1])\n",
    "    end = end.replace(tzinfo=pytz.utc)\n",
    "    start = min(start, end)\n",
    "    newindex = pd.DataFrame([0, 0], index=[start, end]).resample(freq).index\n",
    "    if ts.dropna().empty:\n",
    "        tsmin = ts.reindex(newindex)\n",
    "    else:\n",
    "        tsmin = ts.reindex(ts.index + newindex)\n",
    "        tsmin = tsmin.interpolate(method='time')\n",
    "        tsmin = tsmin.reindex(newindex)\n",
    "    return tsmin.diff()*3600/60"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df = hp.get_data(sensortype='water', head=start, tail=end).diff()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "water_sensors = [sensor for sensor in hp.get_sensors('water') if sensor.key in df.columns]\n",
    "print \"{} water sensors\".format(len(water_sensors))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The purpose is to automatically detect leaks, undesired high consumption, etc.. so we can warn the user\n",
    "\n",
    "Let's first have a look at the carpet plots in order to see whether we have such leaks, etc... in our database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "for sensor in water_sensors:\n",
    "    ts = df[sensor.key]\n",
    "    if not ts.dropna().empty:\n",
    "        plotting.carpet(ts, title=sensor.device.key, zlabel=r'Flow [l/hour]')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yes, we do! The most obvious is FL03001579 with a more or less constant leak in the first month and later on some very large leaks during several hours. FL03001556 has a moderate leak once and seems to have similar, but less severe leaks later again. Also in FL03001561 there was once a strange (but rather short) issue and later on small, stubborn and irregularly deteriorating leaks of a different kind.\n",
    "\n",
    "So, out of 6 water consumption profiles, there are 3 with possible leaks of different types and severities! This looks a very promising case to detect real issues and show the value of opengrid.\n",
    "\n",
    "So, we would like to detect the following issues:\n",
    "* FL03001579: constant leak in first month and big water leak during several hours on some days (toilet leaking?)\n",
    "* FL03001556: moderate leak once and small water leak during several hours on some days (toilet leaking?)\n",
    "* FL03001561: rather short leak and small, irregularly deteriorating water leak?\n",
    "\n",
    "How could we detect these? Let's look first at the daily load curves of each sensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for sensor in water_sensors:\n",
    "    ts = df[sensor.key]\n",
    "    if not ts.dropna().empty:\n",
    "        tsday = ts.resample('D', how='sum')\n",
    "        tsday.plot(label=sensor.device.key)\n",
    "        (tsday*0.+1000.).plot(style='--', lw=3, label='_nolegend_')\n",
    "        plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, the big water leaks of FL03001579 is relatively easy to detect, e.g. by raising an alarm as soon as the daily consumption exceeds 1500 l. However, by that time a lot of water has been wasted already.  One could lower the threshold a bit, but below 1000l a false alarm would be raised for FL03001525 on one day. Moreover, the other issues are not detected by such an alarm.\n",
    "\n",
    "Let's try it in a different way. First have a look at the load duration curve, maybe there we could find something usefull for the alarm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "for sensor in water_sensors:\n",
    "    ts = df[sensor.key]\n",
    "    if not ts.dropna().empty:\n",
    "        plt.figure()\n",
    "        for day in pd.date_range(start, end):\n",
    "            try:\n",
    "                tsday = ts[day.strftime('%Y/%m/%d')].order(ascending=False) * 60.\n",
    "                plt.plot(tsday.values/60.)\n",
    "                x = np.arange(len(tsday.values)) + 10.\n",
    "                plt.plot(x + 100., 500./x**1.5, 'k--')\n",
    "                plt.gca().set_yscale('log')\n",
    "                plt.ylim(ymin=1/60.)\n",
    "                plt.title(sensor.device.key)\n",
    "            except:\n",
    "                pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This way, most of the issues could be detected, but some marginally. For small leaks it may take a full day before the alarm is raised.\n",
    "\n",
    "Maybe we can improve this. A more reliable way may be to look for consecutive minutes with high load. So, let's have a look at the 60 minutes rolling minimum of the load."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for sensor in water_sensors:\n",
    "    ts = df[sensor.key] * 60.\n",
    "    if not ts.dropna().empty:\n",
    "        tsday = pd.rolling_min(ts, 60)\n",
    "        ax = tsday.plot(label=sensor.device.key)\n",
    "(tsday*0.+20.).plot(style='--', lw=3, label='_nolegend_')\n",
    "plt.gca().set_yscale('log')\n",
    "ax.set_ylim(ymin=1)\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The large leaks are very pronounced and easily detected (remark that this is a logarithmic scale!) one hour after the leak started. But the smaller leaks are still not visible.\n",
    "\n",
    "Remark that a typical characteristic of a leak is that it is more or less constant, and thus the mean is probably pretty close to the minimum. An expected high load typically varies a lot more over one hour and thus its mean is probably a lot higher than its minimum. So, we could exploit this characteristic of leaks by subtracting some fraction of the rolling mean from the rolling minimum. Leaks should then stand out compared to normal loads."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for sensor in water_sensors:\n",
    "    ts = df[sensor.key] * 60.\n",
    "    if not ts.dropna().empty:\n",
    "        tsday = pd.rolling_min(ts, 60) - 0.4*pd.rolling_mean(ts, 60)\n",
    "        ax = tsday.plot(label=sensor.device.key)\n",
    "(tsday*0.+1.).plot(style='--', lw=3, label='_nolegend_')\n",
    "ax.set_yscale('log')\n",
    "ax.set_ylim(ymin=0.1)\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now this works! The large leaks of FL03001579 stand out by two orders of magnitude, but also the small leaks of FL03001556 and FL03001561 are detected one hour after they started."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
