
pywws.DataStore
***************


Introduction
============

This module is at the core of my weather station software. It stores
data on disc, but without the overhead of a full scale database
system. I have designed it to run on a small memory machine such as my
Asus router. To minimise memory usage it only loads one day's worth of
data at a time into memory.

From a "user" point of view, the data is accessed as a cross between a
list and a dictionary. Each data record is indexed by a "datetime"
object (dictionary behaviour), but records are stored in order and can
be accessed as slices (list behaviour).

For example, to access the hourly data for Christmas day 2009, one
might do the following:

   from datetime import datetime
   import DataStore
   hourly = DataStore.hourly_store('weather_data')
   for data in hourly[datetime(2009, 12, 25):datetime(2009, 12, 26)]:
       print data['idx'], data['temp_out']

The module provides five classes to store different data. "data_store"
takes "raw" data from the weather station; "calib_store",
"hourly_store", "daily_store" and "monthly_store" store processed data
(see *pywws.Process*). All three are derived from the same
"core_store" class, they only differ in the keys and types of data
stored in each record.


Detailed API
============

DataStore.py - stores readings in easy to access files

A separate file is used for each day's data, to keep memory load
reasonable. One day at a time is held in memory, and saved to disc
when another day needs to be accessed.

Data is accessed in a cross between dictionary and list behaviour. The
following are all valid: # get value nearest 9:30 on Christmas day
data[data.nearest(datetime(2008, 12, 25, 9, 30))] # get entire array,
equivalent to data[:] or just data data[datetime.min:datetime.max] #
get last 12 hours of data data[datetime.utcnow() -
timedelta(hours=12):]

-[ Functions ]-

+------------+--------------------------------------------------------------------------------------------+
+------------+--------------------------------------------------------------------------------------------+

-[ Classes ]-

+------------+--------------------------------------------------------------------------------------------+
| "data_stor | Stores raw weather station data.                                                           |
+------------+--------------------------------------------------------------------------------------------+
| "calib_sto | Stores 'calibrated' weather station data.                                                  |
+------------+--------------------------------------------------------------------------------------------+
| "hourly_st | Stores hourly summary weather station data.                                                |
+------------+--------------------------------------------------------------------------------------------+
| "daily_sto | Stores daily summary weather station data.                                                 |
+------------+--------------------------------------------------------------------------------------------+
| "monthly_s | Stores monthly summary weather station data.                                               |
+------------+--------------------------------------------------------------------------------------------+
+------------+--------------------------------------------------------------------------------------------+
| "params"(r | Parameters are stored in a file "weather.ini" in root_dir.                                 |
+------------+--------------------------------------------------------------------------------------------+

pywws.DataStore.safestrptime(date_string, format=None)

class class pywws.DataStore.params(root_dir)

   Parameters are stored in a file "weather.ini" in root_dir.

   flush()

   get(section, option, default=None)

      Get a parameter value and return a string.

      If default is specified and section or option are not defined in
      the weather.ini file, they are created and set to default, which
      is then the return value.

   get_datetime(section, option, default=None)

   set(section, option, value)

      Set option in section to string value.

class class pywws.DataStore.core_store(root_dir)

   before(idx)

      Return datetime of newest existing data record whose datetime is
      < idx.

      Might not even be in the same year! If no such record exists,
      return None.

   after(idx)

      Return datetime of oldest existing data record whose datetime is
      >= idx.

      Might not even be in the same year! If no such record exists,
      return None.

   nearest(idx)

      Return datetime of record whose datetime is nearest idx.

   flush()

class class pywws.DataStore.data_store(root_dir)

   Stores raw weather station data.

   key_list = ['idx', 'delay', 'hum_in', 'temp_in', 'hum_out', 'temp_out', 'abs_pressure', 'wind_ave', 'wind_gust', 'wind_dir', 'rain', 'status', 'illuminance', 'uv']

   conv = {'status': <type 'int'>, 'wind_ave': <type 'float'>, 'rain': <type 'float'>, 'hum_in': <type 'int'>, 'temp_out': <type 'float'>, 'wind_dir': <type 'int'>, 'hum_out': <type 'int'>, 'illuminance': <type 'float'>, 'wind_gust': <type 'float'>, 'idx': <function safestrptime at 0x1ced938>, 'uv': <type 'int'>, 'temp_in': <type 'float'>, 'delay': <type 'int'>, 'abs_pressure': <type 'float'>}

class class pywws.DataStore.calib_store(root_dir)

   Stores 'calibrated' weather station data.

   key_list = ['idx', 'delay', 'hum_in', 'temp_in', 'hum_out', 'temp_out', 'abs_pressure', 'rel_pressure', 'wind_ave', 'wind_gust', 'wind_dir', 'rain', 'status', 'illuminance', 'uv']

   conv = {'status': <type 'int'>, 'wind_ave': <type 'float'>, 'rain': <type 'float'>, 'rel_pressure': <type 'float'>, 'hum_in': <type 'int'>, 'temp_out': <type 'float'>, 'wind_dir': <type 'int'>, 'hum_out': <type 'int'>, 'illuminance': <type 'float'>, 'wind_gust': <type 'float'>, 'idx': <function safestrptime at 0x1ced938>, 'uv': <type 'int'>, 'temp_in': <type 'float'>, 'delay': <type 'int'>, 'abs_pressure': <type 'float'>}

class class pywws.DataStore.hourly_store(root_dir)

   Stores hourly summary weather station data.

   key_list = ['idx', 'hum_in', 'temp_in', 'hum_out', 'temp_out', 'abs_pressure', 'rel_pressure', 'pressure_trend', 'wind_ave', 'wind_gust', 'wind_dir', 'rain', 'illuminance', 'uv']

   conv = {'pressure_trend': <type 'float'>, 'wind_ave': <type 'float'>, 'rain': <type 'float'>, 'rel_pressure': <type 'float'>, 'hum_in': <type 'int'>, 'temp_out': <type 'float'>, 'wind_dir': <type 'int'>, 'hum_out': <type 'int'>, 'wind_gust': <type 'float'>, 'idx': <function safestrptime at 0x1ced938>, 'uv': <type 'int'>, 'temp_in': <type 'float'>, 'illuminance': <type 'float'>, 'abs_pressure': <type 'float'>}

class class pywws.DataStore.daily_store(root_dir)

   Stores daily summary weather station data.

   key_list = ['idx', 'start', 'hum_out_ave', 'hum_out_min', 'hum_out_min_t', 'hum_out_max', 'hum_out_max_t', 'temp_out_ave', 'temp_out_min', 'temp_out_min_t', 'temp_out_max', 'temp_out_max_t', 'hum_in_ave', 'hum_in_min', 'hum_in_min_t', 'hum_in_max', 'hum_in_max_t', 'temp_in_ave', 'temp_in_min', 'temp_in_min_t', 'temp_in_max', 'temp_in_max_t', 'abs_pressure_ave', 'abs_pressure_min', 'abs_pressure_min_t', 'abs_pressure_max', 'abs_pressure_max_t', 'rel_pressure_ave', 'rel_pressure_min', 'rel_pressure_min_t', 'rel_pressure_max', 'rel_pressure_max_t', 'wind_ave', 'wind_gust', 'wind_gust_t', 'wind_dir', 'rain', 'illuminance_ave', 'illuminance_max', 'illuminance_max_t', 'uv_ave', 'uv_max', 'uv_max_t']

   conv = {'temp_in_min': <type 'float'>, 'temp_in_max': <type 'float'>, 'uv_ave': <type 'float'>, 'temp_out_max': <type 'float'>, 'temp_out_min': <type 'float'>, 'abs_pressure_max': <type 'float'>, 'start': <function safestrptime at 0x1ced938>, 'hum_in_max_t': <function safestrptime at 0x1ced938>, 'uv_max_t': <function safestrptime at 0x1ced938>, 'temp_in_max_t': <function safestrptime at 0x1ced938>, 'abs_pressure_min_t': <function safestrptime at 0x1ced938>, 'illuminance_ave': <type 'float'>, 'rel_pressure_ave': <type 'float'>, 'rain': <type 'float'>, 'rel_pressure_min_t': <function safestrptime at 0x1ced938>, 'wind_gust': <type 'float'>, 'temp_out_min_t': <function safestrptime at 0x1ced938>, 'wind_gust_t': <function safestrptime at 0x1ced938>, 'hum_out_ave': <type 'float'>, 'rel_pressure_min': <type 'float'>, 'hum_out_min_t': <function safestrptime at 0x1ced938>, 'temp_out_max_t': <function safestrptime at 0x1ced938>, 'uv_max': <type 'int'>, 'illuminance_max_t': <function safestrptime at 0x1ced938>, 'rel_pressure_max': <type 'float'>, 'temp_in_ave': <type 'float'>, 'hum_in_min_t': <function safestrptime at 0x1ced938>, 'hum_out_max': <type 'int'>, 'illuminance_max': <type 'float'>, 'wind_ave': <type 'float'>, 'abs_pressure_max_t': <function safestrptime at 0x1ced938>, 'temp_in_min_t': <function safestrptime at 0x1ced938>, 'hum_out_max_t': <function safestrptime at 0x1ced938>, 'hum_out_min': <type 'int'>, 'rel_pressure_max_t': <function safestrptime at 0x1ced938>, 'wind_dir': <type 'int'>, 'temp_out_ave': <type 'float'>, 'abs_pressure_ave': <type 'float'>, 'idx': <function safestrptime at 0x1ced938>, 'hum_in_max': <type 'int'>, 'abs_pressure_min': <type 'float'>, 'hum_in_ave': <type 'float'>, 'hum_in_min': <type 'int'>}

class class pywws.DataStore.monthly_store(root_dir)

   Stores monthly summary weather station data.

   key_list = ['idx', 'start', 'hum_out_ave', 'hum_out_min', 'hum_out_min_t', 'hum_out_max', 'hum_out_max_t', 'temp_out_ave', 'temp_out_min_lo', 'temp_out_min_lo_t', 'temp_out_min_hi', 'temp_out_min_hi_t', 'temp_out_min_ave', 'temp_out_max_lo', 'temp_out_max_lo_t', 'temp_out_max_hi', 'temp_out_max_hi_t', 'temp_out_max_ave', 'hum_in_ave', 'hum_in_min', 'hum_in_min_t', 'hum_in_max', 'hum_in_max_t', 'temp_in_ave', 'temp_in_min_lo', 'temp_in_min_lo_t', 'temp_in_min_hi', 'temp_in_min_hi_t', 'temp_in_min_ave', 'temp_in_max_lo', 'temp_in_max_lo_t', 'temp_in_max_hi', 'temp_in_max_hi_t', 'temp_in_max_ave', 'abs_pressure_ave', 'abs_pressure_min', 'abs_pressure_min_t', 'abs_pressure_max', 'abs_pressure_max_t', 'rel_pressure_ave', 'rel_pressure_min', 'rel_pressure_min_t', 'rel_pressure_max', 'rel_pressure_max_t', 'wind_ave', 'wind_gust', 'wind_gust_t', 'wind_dir', 'rain', 'illuminance_ave', 'illuminance_max_lo', 'illuminance_max_lo_t', 'illuminance_max_hi', 'illuminance_max_hi_t', 'illuminance_max_ave', 'uv_ave', 'uv_max_lo', 'uv_max_lo_t', 'uv_max_hi', 'uv_max_hi_t', 'uv_max_ave']

   conv = {'uv_ave': <type 'float'>, 'illuminance_max_hi_t': <function safestrptime at 0x1ced938>, 'uv_max_lo_t': <function safestrptime at 0x1ced938>, 'temp_out_max_hi_t': <function safestrptime at 0x1ced938>, 'temp_out_max_lo': <type 'float'>, 'abs_pressure_max': <type 'float'>, 'rel_pressure_min_t': <function safestrptime at 0x1ced938>, 'temp_in_min_hi': <type 'float'>, 'temp_in_min_lo': <type 'float'>, 'start': <function safestrptime at 0x1ced938>, 'hum_in_max_t': <function safestrptime at 0x1ced938>, 'uv_max_hi_t': <function safestrptime at 0x1ced938>, 'abs_pressure_min_t': <function safestrptime at 0x1ced938>, 'temp_out_max_hi': <type 'float'>, 'illuminance_ave': <type 'float'>, 'temp_out_min_lo': <type 'float'>, 'illuminance_max_ave': <type 'float'>, 'uv_max_hi': <type 'int'>, 'rel_pressure_ave': <type 'float'>, 'rain': <type 'float'>, 'temp_out_min_hi': <type 'float'>, 'wind_gust': <type 'float'>, 'temp_in_max_hi_t': <function safestrptime at 0x1ced938>, 'temp_in_max_lo': <type 'float'>, 'temp_in_min_ave': <type 'float'>, 'wind_gust_t': <function safestrptime at 0x1ced938>, 'hum_out_ave': <type 'float'>, 'rel_pressure_min': <type 'float'>, 'hum_out_min_t': <function safestrptime at 0x1ced938>, 'illuminance_max_lo': <type 'float'>, 'temp_in_min_lo_t': <function safestrptime at 0x1ced938>, 'illuminance_max_hi': <type 'float'>, 'temp_out_max_lo_t': <function safestrptime at 0x1ced938>, 'temp_in_max_hi': <type 'float'>, 'rel_pressure_max': <type 'float'>, 'temp_in_ave': <type 'float'>, 'uv_max_ave': <type 'float'>, 'temp_in_max_ave': <type 'float'>, 'hum_in_min_t': <function safestrptime at 0x1ced938>, 'hum_out_max': <type 'int'>, 'temp_in_min_hi_t': <function safestrptime at 0x1ced938>, 'temp_out_min_ave': <type 'float'>, 'temp_out_min_hi_t': <function safestrptime at 0x1ced938>, 'wind_ave': <type 'float'>, 'abs_pressure_max_t': <function safestrptime at 0x1ced938>, 'hum_out_max_t': <function safestrptime at 0x1ced938>, 'hum_out_min': <type 'int'>, 'rel_pressure_max_t': <function safestrptime at 0x1ced938>, 'uv_max_lo': <type 'int'>, 'wind_dir': <type 'int'>, 'temp_out_ave': <type 'float'>, 'abs_pressure_ave': <type 'float'>, 'idx': <function safestrptime at 0x1ced938>, 'temp_out_min_lo_t': <function safestrptime at 0x1ced938>, 'illuminance_max_lo_t': <function safestrptime at 0x1ced938>, 'hum_in_max': <type 'int'>, 'abs_pressure_min': <type 'float'>, 'temp_in_max_lo_t': <function safestrptime at 0x1ced938>, 'hum_in_ave': <type 'float'>, 'temp_out_max_ave': <type 'float'>, 'hum_in_min': <type 'int'>}
