Metering Introduction

# Metering Package Indroduction¶

The 'metering' package is intended to automate many of the routine processes encountered when handling data collected from meters. It allows for further analysis to be preformed in Python, or for data to be written to a .csv file and analyzed in a different program, like Excel, Matlab, or R. If you haven't installed Python, you can download it here by selecting the 64-bit version of Python 2.7.

The code included below can be run from the console window or copied into a script. The 'In [1]' indicates the input at line 1, and if an output is returned, it will appear next to 'Out [1]'.

### Installing metering¶

Use pip to install the metering package from pypi.org.

In [1]:

    !pip install metering

    Requirement already satisfied: metering in c:\users\ari.jackson\appdata\local\continuum\anaconda2\lib\site-packages

If metering is already installed, you can use pip to upgrade the version.

In [2]:

    !pip install metering -U

    Collecting metering
      Downloading metering-0.0.20.tar.gz
    Building wheels for collected packages: metering
      Running setup.py bdist_wheel for metering: started
      Running setup.py bdist_wheel for metering: finished with status 'done'
      Stored in directory: C:\Users\ari.jackson\AppData\Local\pip\Cache\wheels\d8\f5\09\ccb67b1ef57496b4db5423357132b9bbe9bfbb72cadbc385b9
    Successfully built metering
    Installing collected packages: metering
      Found existing installation: metering 0.0.19
        Uninstalling metering-0.0.19:
          Successfully uninstalled metering-0.0.19
    Successfully installed metering-0.0.20

### Importing metering¶

Now that metering is installed, modules can be imported from python.

In [2]:

    import metering
    metering

Out[2]:

    <module 'metering' from 'C:\Users\ari.jackson\AppData\Local\Continuum\Anaconda2\lib\site-packages\metering\__init__.pyc'>

Inside of metering are several modules that can be accessed using dot notation. We'll be working with the 'exports' module.

In [3]:

    import metering.exports
    metering.exports

The exports module contains classes for reading data files that have been exported from various applications. Here we'll look at .csv files exported from HOBOWare. Accessing these classes similarly uses dot notation. The top level class in the exports module is called 'Export,' which contains attirbutes and methods that apply generically, and below it are classes specific to a certain application, including the 'HOBOWare' class.

In [4]:

    from metering.exports import Export
    Export

Out[4]:

    <class metering.exports.Export at 0x000000000992E528>

In [5]:

    import sys

    path = str()
    for s in sys.path:
        if s.split('\\')[-1] == 'site-packages':
            path = s
    path

Out[5]:

    'C:\\Users\\ari.jackson\\AppData\\Local\\Continuum\\Anaconda2\\lib\\site-packages'

### Directory Setup¶

When a file is read by the Export, it is essential for the module to know what application that file came from because this will determine how it is processed. Currently (and this can change) I use the directory the data is stored in to indicate to Export the type of file that is being passed. This process requires directories have specific names and that the data contained in them all comes from the same application. For files exported form HOBOWare, the directory name needs to be 'HOBOWare.'

In the Windows Explorer address bar, you can see the path to the five files we'll be working with, where the open directory is 'HOBOWare.' These five files include data from U30s, plug load meters, state loggers, and Temp/RH loggers.

In [6]:

    from PIL import Image
    Image.open(path + '\\metering\\data\\Images\\Directory.png')

Out[6]:

### Processing Data¶

To process these five export files, we'll pass their path to the method Export.make() with a second argument to specify the sampling frequency we'd like to use for later analysis. The file path is a string the indicates the location of the file and is make up of the directories the file is in and the file name. Here each path will begin with 'F:\metering demo\Exports\HOBOWare\' and end with the name of the file, including the '.csv' at the end. This path depends on where you've stored your data.

A quick way to compile these paths is to use python's os.listdir method, which takes a directory path as an argument and returnes the files it contains. Appending these file names to the directory path, we can create a list of file paths that can be passed to Export.make().

In [7]:

    import os
    dir_path = path + '\\metering\\data\\Exports\\HOBOWare'
    l = list()
    for i in os.listdir(dir_path):
        l.append(dir_path + '\\' + i)
    l

Out[7]:

    ['C:\\Users\\ari.jackson\\AppData\\Local\\Continuum\\Anaconda2\\lib\\site-packages\\metering\\data\\Exports\\HOBOWare\\M0003-Worcester-10064511-20150417-091745.csv',
     'C:\\Users\\ari.jackson\\AppData\\Local\\Continuum\\Anaconda2\\lib\\site-packages\\metering\\data\\Exports\\HOBOWare\\M0100-West-10181391 (1).csv',
     'C:\\Users\\ari.jackson\\AppData\\Local\\Continuum\\Anaconda2\\lib\\site-packages\\metering\\data\\Exports\\HOBOWare\\M0100-West-10181391 (2).csv',
     'C:\\Users\\ari.jackson\\AppData\\Local\\Continuum\\Anaconda2\\lib\\site-packages\\metering\\data\\Exports\\HOBOWare\\M0100-West-10365288-20150413-125200.csv',
     'C:\\Users\\ari.jackson\\AppData\\Local\\Continuum\\Anaconda2\\lib\\site-packages\\metering\\data\\Exports\\HOBOWare\\MG074-Manhattan-10580517-20160922-222222.csv']

A note on file names: The files here all follow a convention Cadmus has used in the past, which is to seperate by a hyphen a site identifier, a last name, a meter serial number, and in some cases the date and time. This convention has been useful, but won't be necessary for every project. Currently, the HOBOWare export class does require at least a site identifier followed by a hyphen.

We can now call the Export.make() method with one of the file paths above and the additional argument '1Min' (to indicate we'd like the data stored in 1 minute intervals), and this call will return a HOBOWareExport object that we'll store as the variable 'e1.'

In [8]:

    file_path = l[0]
    w = '1Min'
    e1 = Export.make_Export(file_path, w)
    e1

Out[8]:

    <metering.exports.HOBOWare instance at 0x0000000009A0B808>

What's going on behind the Export.make() method are all the processes required for further analysis, including reading in the data, creating and localizing a timestamp index, clean out unused columns or rows, resampling, interpolating, and storing identifiers so the data can later be mapped to some piece of equipiment or quantity. We can look at some of the data attributes stored in an export object.

In [9]:

    # Identifier associated with an export
    e1.idy

Out[9]:

    'M0003_10064511'

In [10]:

    # A portion of the data from the export file
    e1.data.head()

Out[10]:

| Motor (LGR S/N: 10064511, SEN S/N: 10064511, LBL: OilBurner)
--- | ---
Date Time, GMT-04:00 |
2014-10-27 10:55:03-04:00 | 1.0
2014-10-27 11:04:13-04:00 | 0.0
2014-10-27 11:04:27-04:00 | 1.0
2014-10-27 11:04:40-04:00 | 0.0
2014-10-27 11:04:58-04:00 | 1.0

In [11]:

    # The application the data was exported form
    e1.app

Out[11]:

    'HOBOWare'

Another attribute of any export object is a list of 'Measurements.' Measurement is a class used to represent specific measurement like temperature or power, as opposed to Export, which represents a file exported from an applicaiton. Since the export file we read in before has four columns, the measurement list attribute of the export object will have four elements.

In [12]:

    e1.meas_list

Out[12]:

    [<metering.exports.HOBOWareMeasurement instance at 0x0000000009D41C88>]

The important attributes of Measurement objects are the data and the measurement descriptions. Measurement description is meant to be a unique identifier so that when multiple exports are used to meter the same quantity (say you have two file each measuring the same temperature during two different periods of time) so the they can later be combine into a single time series. Below we'll assign the first Measurement object in the exports measurement list and access it's attributes.

In [13]:

    m = e1.meas_list[0]
    m.meas_des

Out[13]:

    'M0003_10064511_10064511 Motor'

In [14]:

    m.data.head()

Out[14]:

    Date Time, GMT-04:00
    2014-10-27 10:55:00-04:00    1.0
    2014-10-27 10:56:00-04:00    1.0
    2014-10-27 10:57:00-04:00    1.0
    2014-10-27 10:58:00-04:00    1.0
    2014-10-27 10:59:00-04:00    1.0
    Freq: T, Name: M0003_10064511_10064511 Motor, dtype: float64

We can process the rest of export files from before (takes a few seconds).

In [15]:

    e2 = Export.make_Export(l[1], w)
    e3 = Export.make_Export(l[2], w)
    e4 = Export.make_Export(l[3], w)
    e5 = Export.make_Export(l[4], w)

If we wanted to group the data from these exports together, say because they all came from the same site, we can use the Location class. The function this class serves is to combine measurements across multiple exports and compile everything into a single data frame that can be written to a .csv file. This process is accomplished by calling the Location.make_Location() method with the arguments of the exports you'd like to combine as a list. Here we'll input exports e1, e2, and e3 because they were all collected at the same site.

In [16]:

    from metering.exports import Location
    export_list = [e1, e2, e3]
    site_id = 'Test Site'
    loc = Location.make_Location(export_list, site_id)

If we look at the measurment lists of the individual exports inputted into the MeasurementGroup.make() method and those of the MeasurementGroup object mg, we see that four of the Measurement objects were combine across exports, this was because the represented the same measurements taken at a site but that occured during different periods of time.

In [17]:

    e1.meas_list

Out[17]:

    [<metering.exports.HOBOWareMeasurement instance at 0x0000000009D41C88>]

In [18]:

    e2.meas_list

Out[18]:

    [<metering.exports.HOBOWareMeasurement instance at 0x0000000009D39D88>,
     <metering.exports.HOBOWareMeasurement instance at 0x0000000009A0BE08>,
     <metering.exports.HOBOWareMeasurement instance at 0x0000000009A0BE48>,
     <metering.exports.HOBOWareMeasurement instance at 0x0000000009D13288>]

In [19]:

    e3.meas_list

Out[19]:

    [<metering.exports.HOBOWareMeasurement instance at 0x000000000A1DDAC8>,
     <metering.exports.HOBOWareMeasurement instance at 0x000000000A1DDFC8>,
     <metering.exports.HOBOWareMeasurement instance at 0x0000000009FE0608>,
     <metering.exports.HOBOWareMeasurement instance at 0x00000000099FA8C8>]

In [20]:

    loc.meas_list

Out[20]:

    [<metering.exports.HOBOWareMeasurement instance at 0x0000000009D41C88>,
     <metering.exports.HOBOWareMeasurement instance at 0x0000000009D39D88>,
     <metering.exports.HOBOWareMeasurement instance at 0x0000000009A0BE08>,
     <metering.exports.HOBOWareMeasurement instance at 0x0000000009A0BE48>,
     <metering.exports.HOBOWareMeasurement instance at 0x0000000009D13288>]

### Plotting¶

Using the matplotlib package we can plot some of this processed data.

In [21]:

    import matplotlib.pyplot as plt
    import matplotlib.dates as mdates
    m = loc.meas_list[2]
    fig = plt.figure(figsize=(10, 8))
    plt.plot(m.data.index, m.data)
    plt.ylim(bottom=0, top=100)
    plt.grid()
    plt.title(m.meas_des)
    fig.autofmt_xdate()
    plt.show()
    plt.close('all')

In [ ]: