{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conduit Examples\n",
    "\n",
    "This notebook demonstrates various examples of computation using Conduit using streams selected from BTrDB. Note that these particular examples are using data fetched directly from Dominion; but this notebook should be refactored to select other data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import btrdb\n",
    "import conduit\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'majorVersion': 5, 'build': '5.6.3', 'proxy': {'proxyEndpoints': []}}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db = btrdb.connect(profile=\"dominion.predictivegrid.com\")\n",
    "db.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Selection\n",
    "\n",
    "Right now conduits are constructed by instantiating a `Conduit` with a collection of `btrdb.Stream` objects. In my opinion this should be the starting place for most computation, though there are several design elements that must be considered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def label_prefix(prefix):\n",
    "    def label_prefix_filter(stream):\n",
    "        meta, _ = stream.annotations()\n",
    "        return meta.get(\"label\", \"\").startswith(prefix)\n",
    "    return label_prefix_filter\n",
    "\n",
    "streams = conduit.Conduit(*db.streams_in_collection(\"dfr/DFR!PMU-NOR2_237\"))\n",
    "streams = streams.subset(func=label_prefix(\"Northeast Carver 3\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(streams.describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stream Filtering\n",
    "\n",
    "Right now conduit uses filter masks to select streams to apply various computations. For example, to select all voltage phases, you would do the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "voltages = streams.subset(streams.unit_mask(\"VPHM\", \"VPHA\"))\n",
    "print(voltages.describe())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "streams.unit_mask(\"VPHM\", \"VPHA\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get all of the phase A streams you would filter as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "phase_a = streams.subset(streams.phase_mask(\"A\"))\n",
    "print(phase_a.describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Masks can combined with binary operators, e.g. `&` and `|` to put together more complex queries. For example to get the Phase A phasor pair (voltage and current phasors) you would combine the two masks with the `&` operator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "phasor_mask = streams.unit_mask(\"VPHM\", \"VPHA\", \"IPHM\", \"IPHA\") & streams.phase_mask(\"A\")\n",
    "phasor_pair = streams.subset(phasor_mask)\n",
    "print(phasor_pair.describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `subset` method can take any boolean mask and the `unit_mask` and `phase_mask` helper methods create filter masks based on the metadata of the stream. Note that you can create more complex streams filters with functions as shown in the first stream selection example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Phasor Compositions\n",
    "\n",
    "At the moment, you cannot compose `Phasor`, `PhasorPair`, `PhasorGroup` or `PhasorPairGroup` objects directly from a `Conduit` object since these objects extend a `Conduit` and the current code structure would then have a circular dependency. However, you can use the filter masks from above to quickly construct these objects and call their computations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vmask = streams.unit_mask(\"VPHM\", \"VPHA\")\n",
    "imask = streams.unit_mask(\"IPHM\", \"IPHA\")\n",
    "\n",
    "phaseVA = conduit.Phasor(*streams.subset(vmask & streams.phase_mask(\"A\")))\n",
    "phaseIB = conduit.Phasor(*streams.subset(imask & streams.phase_mask(\"B\")))\n",
    "\n",
    "print(phaseVA)\n",
    "print(phaseIB)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vpg = conduit.PhasorGroup(*streams.subset(vmask))\n",
    "\n",
    "for phasor in vpg.phasors():\n",
    "    print(phasor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Phasor` objects validate their inputs and raise an exception if they are composed incorrectly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    conduit.Phasor(*streams.subset(vmask))\n",
    "except Exception as e:\n",
    "    print(\"{}: {}\".format(type(e), str(e)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    conduit.Phasor(streams[0], streams[1])\n",
    "except Exception as e:\n",
    "    print(\"{}: {}\".format(type(e), str(e)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fetch Data\n",
    "\n",
    "The `Conduit` and `Phasor` compositions are all intended to store meta data about streams, so that data can be fetched easily on demand and various standard computations applied. Data is returned as a `pd.DataFrame` from the `values`, `windows`, and `aligned_windows` methods as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "phase_a.values(\"2019-08-22 18:27:00.000\", \"2019-08-22 18:27:15.000\").head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the `DataFrame` will ensure that the time index is joined properly, e.g. if there are any streams that do not have a data point for the time index, they will be filled in with `np.nan`.\n",
    "\n",
    "You can also use the `windows` and `aligned_windows` queries to get `DataFrames` with aggregate values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "phase_a.aligned_windows(\"2019-08-22 18:00:00.000\", \"2019-08-22 20:00:00.000\", 39, agg=\"mean\").head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It may be possible to have more interesting data frame headers from different phase groups and phase group metadata:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vpg.values_poc(\"2019-08-22 18:27:00.000\", \"2019-08-22 18:27:15.000\").head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applying Transformations\n",
    "\n",
    "The first method of applying computations is to apply a transformation function with an optional mask that determines which streams will have the transformation applied on them. The transformation function should be as follows:\n",
    "\n",
    "```python\n",
    "def transformer(series, tags={}, annotations={}):\n",
    "    return series\n",
    "```\n",
    "\n",
    "The caveat is that right now the transformer must only accept a single stream and return only a single stream. \n",
    "\n",
    "Note that if the transformer filters rows, these will simply become `np.nan` in the final `DataFrame`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mags = streams.subset(streams.unit_mask(\"VPHM\", \"IPHM\"))\n",
    "mags.values(\"2019-08-22 18:27:00.000\", \"2019-08-22 18:27:15.000\").head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def per_unit(units, base_kv=None):\n",
    "    if units == \"IPHM\":\n",
    "        def per_unit_current_transformer(series, tags=None, annotations=None):\n",
    "            bkv = base_kv or annotations.get(\"base_kv\", None)\n",
    "            if bkv is None:\n",
    "                raise ValueError(\"could not find required base kV for current to per-unit\")\n",
    "            tags[\"unit\"] = \"per-unit\"\n",
    "            return series / (100000.0 / (bkv *np.sqrt(3)))\n",
    "        return per_unit_current_transformer\n",
    "    \n",
    "    if units == \"VPHM\":\n",
    "        def per_unit_voltage_transformer(series, tags=None, annotations=None):\n",
    "            bkv = base_kv or annotations.get(\"base_kv\", None)\n",
    "            if bkv is None:\n",
    "                raise ValueError(\"could not find required base kV for voltage to per-unit\")\n",
    "            return series * np.sqrt(3) / (bkv * 1000)\n",
    "        return per_unit_voltage_transformer\n",
    "    \n",
    "    raise ValueError(\"unknown per-unit transformation unis: {}\".format(units))\n",
    "\n",
    "    \n",
    "vphm_mask = streams.unit_mask(\"VPHM\")\n",
    "iphm_mask = streams.unit_mask(\"IPHM\")\n",
    "\n",
    "mags.apply(per_unit(\"VPHM\", base_kv=140), vphm_mask)\n",
    "mags.apply(per_unit(\"IPHM\", base_kv=140), iphm_mask)\n",
    "\n",
    "\n",
    "mags.values(\"2019-08-22 18:27:00.000\", \"2019-08-22 18:27:15.000\").head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compositional Computation\n",
    "\n",
    "This is a proof of concept about how the specific `Phasor` objects could be used to perform compositional style computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vpg.complex_poc(\"2019-08-22 18:27:00.000\", \"2019-08-22 18:27:15.000\").head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vpg.symmetrical_components_poc(\"2019-08-22 18:27:00.000\", \"2019-08-22 18:27:15.000\").head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Phasor Pair Power"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "pair = conduit.PhasorPair(*streams.subset(streams.phase_mask('A')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2019-08-22 18:27:00.033333300    (16895121.864918888+3777910.7101272354j)\n",
       "2019-08-22 18:27:00.066666600    (16682646.532123093+3802666.5067336573j)\n",
       "2019-08-22 18:27:00.099999700    (16863629.584425475+3872083.8382258536j)\n",
       "2019-08-22 18:27:00.133333000     (16836907.401072208+3982758.949077471j)\n",
       "2019-08-22 18:27:00.166666800     (16767446.797394488+3936075.092472428j)\n",
       "Name: complex power, dtype: complex128"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pair.complex_power_poc(\"2019-08-22 18:27:00.000\", \"2019-08-22 18:27:15.000\").head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2019-08-22 18:27:00.033333300    1.689512e+07\n",
       "2019-08-22 18:27:00.066666600    1.668265e+07\n",
       "2019-08-22 18:27:00.099999700    1.686363e+07\n",
       "2019-08-22 18:27:00.133333000    1.683691e+07\n",
       "2019-08-22 18:27:00.166666800    1.676745e+07\n",
       "Name: real power, dtype: float64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pair.real_power_poc(\"2019-08-22 18:27:00.000\", \"2019-08-22 18:27:15.000\").head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2019-08-22 18:27:00.033333300    3.777911e+06\n",
       "2019-08-22 18:27:00.066666600    3.802667e+06\n",
       "2019-08-22 18:27:00.099999700    3.872084e+06\n",
       "2019-08-22 18:27:00.133333000    3.982759e+06\n",
       "2019-08-22 18:27:00.166666800    3.936075e+06\n",
       "Name: reactive power, dtype: float64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pair.reactive_power_poc(\"2019-08-22 18:27:00.000\", \"2019-08-22 18:27:15.000\").head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2019-08-22 18:27:00.033333300    1.731236e+07\n",
       "2019-08-22 18:27:00.066666600    1.711055e+07\n",
       "2019-08-22 18:27:00.099999700    1.730246e+07\n",
       "2019-08-22 18:27:00.133333000    1.730156e+07\n",
       "2019-08-22 18:27:00.166666800    1.722324e+07\n",
       "Name: apparent power, dtype: float64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pair.apparent_power_poc(\"2019-08-22 18:27:00.000\", \"2019-08-22 18:27:15.000\").head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2019-08-22 18:27:00.033333300    0.975899\n",
       "2019-08-22 18:27:00.066666600    0.974992\n",
       "2019-08-22 18:27:00.099999700    0.974638\n",
       "2019-08-22 18:27:00.133333000    0.973144\n",
       "2019-08-22 18:27:00.166666800    0.973536\n",
       "Name: power factor, dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pair.power_factor_poc(\"2019-08-22 18:27:00.000\", \"2019-08-22 18:27:15.000\").head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Phasor Pair Group Power\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "pg = conduit.PhasorPairGroup(*streams.subset(streams.phase_mask('A', 'B', 'C')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2019-08-22 18:27:00.033333300     (48859665.16463483+14009554.774793874j)\n",
       "2019-08-22 18:27:00.066666600      (48582604.93651591+13929174.85814242j)\n",
       "2019-08-22 18:27:00.099999700    (48910205.167097494+14186188.961366167j)\n",
       "2019-08-22 18:27:00.133333000    (48951617.038899146+14176898.834545221j)\n",
       "2019-08-22 18:27:00.166666800    (48720651.197375774+14191429.478326395j)\n",
       "Name: complex power, dtype: complex128"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pg.complex_power_poc(\"2019-08-22 18:27:00.000\", \"2019-08-22 18:27:15.000\").head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2019-08-22 18:27:00.033333300    4.885967e+07\n",
       "2019-08-22 18:27:00.066666600    4.858260e+07\n",
       "2019-08-22 18:27:00.099999700    4.891021e+07\n",
       "2019-08-22 18:27:00.133333000    4.895162e+07\n",
       "2019-08-22 18:27:00.166666800    4.872065e+07\n",
       "Name: real power, dtype: float64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pg.real_power_poc(\"2019-08-22 18:27:00.000\", \"2019-08-22 18:27:15.000\").head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2019-08-22 18:27:00.033333300    1.400955e+07\n",
       "2019-08-22 18:27:00.066666600    1.392917e+07\n",
       "2019-08-22 18:27:00.099999700    1.418619e+07\n",
       "2019-08-22 18:27:00.133333000    1.417690e+07\n",
       "2019-08-22 18:27:00.166666800    1.419143e+07\n",
       "Name: reactive power, dtype: float64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pg.reactive_power_poc(\"2019-08-22 18:27:00.000\", \"2019-08-22 18:27:15.000\").head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2019-08-22 18:27:00.033333300    5.091303e+07\n",
       "2019-08-22 18:27:00.066666600    5.062708e+07\n",
       "2019-08-22 18:27:00.099999700    5.100956e+07\n",
       "2019-08-22 18:27:00.133333000    5.103601e+07\n",
       "2019-08-22 18:27:00.166666800    5.082061e+07\n",
       "Name: apparent power, dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pg.apparent_power_poc(\"2019-08-22 18:27:00.000\", \"2019-08-22 18:27:15.000\").head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2019-08-22 18:27:00.033333300    2.879308\n",
       "2019-08-22 18:27:00.066666600    2.879450\n",
       "2019-08-22 18:27:00.099999700    2.876954\n",
       "2019-08-22 18:27:00.133333000    2.878037\n",
       "2019-08-22 18:27:00.166666800    2.876576\n",
       "Name: power factor, dtype: float64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pg.power_factor_poc(\"2019-08-22 18:27:00.000\", \"2019-08-22 18:27:15.000\").head()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
