pywws.toservice — pywws 13.06_r1023 documentation

### Navigation

- index
- modules |
- pywws 13.06_r1023 documentation »
- Code du module »

# Code source de pywws.toservice

    #!/usr/bin/env python

    # pywws - Python software for USB Wireless Weather Stations
    # http://github.com/jim-easterbrook/pywws
    # Copyright (C) 2008-13  Jim Easterbrook  jim@jim-easterbrook.me.uk

    # This program is free software; you can redistribute it and/or
    # modify it under the terms of the GNU General Public License
    # as published by the Free Software Foundation; either version 2
    # of the License, or (at your option) any later version.

    # This program is distributed in the hope that it will be useful,
    # but WITHOUT ANY WARRANTY; without even the implied warranty of
    # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    # GNU General Public License for more details.

    # You should have received a copy of the GNU General Public License
    # along with this program; if not, write to the Free Software
    # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.

    """Post weather update to services such as Weather Underground
    ::

    %s

    Introduction
    ------------

    Several organisations allow weather stations to upload data using a
    simple HTTP 'POST' or 'GET' request, with the data encoded as a
    sequence of ``key=value`` pairs separated by ``&`` characters.

    This module enables pywws to upload readings to these organisations.
    It is highly customisable using configuration files. Each 'service'
    requires a configuration file and two templates in ``pywws/services``
    (that should not need to be edited by the user) and a section in
    ``weather.ini`` containing user specific data such as your site ID and
    password.

    There are currently six services for which configuration files have
    been written.

    +-----------------------------------------------------------------------+-----------------------+-------------------------------------------------------+
    | organisation                                                          | service name          | config file                                           |
    +=======================================================================+=======================+=======================================================+
    | `UK Met Office <http://wow.metoffice.gov.uk/>`_                       | ``metoffice``         | :download:`../../pywws/services/metoffice.ini`        |
    +-----------------------------------------------------------------------+-----------------------+-------------------------------------------------------+
    | `Open Weather Map <http://openweathermap.org/>`_                      | ``openweathermap``    | :download:`../../pywws/services/openweathermap.ini`   |
    +-----------------------------------------------------------------------+-----------------------+-------------------------------------------------------+
    | `PWS Weather <www.pwsweather.com>`_                                   | ``pwsweather``        | :download:`../../pywws/services/pwsweather.ini`       |
    +-----------------------------------------------------------------------+-----------------------+-------------------------------------------------------+
    | `Stacja Pogody <http://stacjapogody.waw.pl/index.php?id=mapastacji>`_ | ``stacjapogodywawpl`` | :download:`../../pywws/services/stacjapogodywawpl.ini`|
    +-----------------------------------------------------------------------+-----------------------+-------------------------------------------------------+
    | `temperatur.nu <http://www.temperatur.nu/>`_                          | ``temperaturnu``      | :download:`../../pywws/services/temperaturnu.ini`     |
    +-----------------------------------------------------------------------+-----------------------+-------------------------------------------------------+
    | `Weather Underground <http://www.wunderground.com/>`_                 | ``underground``       | :download:`../../pywws/services/underground.ini`      |
    +-----------------------------------------------------------------------+-----------------------+-------------------------------------------------------+
    | `wetter.com <http://www.wetter.com/community/>`_                      | ``wetterarchivde``    | :download:`../../pywws/services/wetterarchivde.ini`   |
    +-----------------------------------------------------------------------+-----------------------+-------------------------------------------------------+

    Configuration
    -------------

    If you haven't already done so, visit the organisation's web site and
    create an account for your weather station. Make a note of any site ID
    and password details you are given.

    Stop any pywws software that is running and then run ``toservice`` to
    create a section in ``weather.ini``::

        python -m pywws.toservice data_dir service_name

    ``service_name`` is a single word service name, such as ``metoffice``,
    ``data_dir`` is your weather data directory, as usual.

    Edit ``weather.ini`` and find the section corresponding to the service
    name, e.g. ``[underground]``. Copy your site details into this
    section, for example::

        [underground]
        password = secret
        station = ABCDEFG1A

    Now you can test your configuration::

        python -m pywws.toservice -vvv data_dir service_name

    This should show you the data string that is uploaded. Any failure
    should generate an error message.

    Upload old data
    ---------------

    Now you can upload your last 7 days' data, if the service supports it.
    Edit your ``status.ini`` file and remove the appropriate line from the
    ``last update`` section, then run ``toservice`` with the catchup
    option::

        python -m pywws.toservice -cvv data_dir service_name

    This may take 20 minutes or more, depending on how much data you have.

    Add service(s) upload to regular tasks
    --------------------------------------

    Edit your ``weather.ini`` again, and add a list of services to the
    ``[live]``, ``[logged]``, ``[hourly]``, ``[12 hourly]`` or ``[daily]``
    section, depending on how often you want to send data. For example::

        [live]
        twitter = []
        plot = []
        text = []
        services = ['underground_rf']

        [logged]
        twitter = []
        plot = []
        text = []
        services = ['metoffice', 'stacjapogodywawpl']

        [hourly]
        twitter = []
        plot = []
        text = []
        services = ['underground']

    Note that the ``[live]`` section is only used when running
    :py:mod:`~pywws.LiveLog`. It is a good idea to repeat any service
    selected in ``[live]`` in the ``[logged]`` or ``[hourly]`` section in
    case you switch to running :py:mod:`~pywws.Hourly`.

    Restart your regular pywws program (:py:mod:`~pywws.Hourly` or
    :py:mod:`~pywws.LiveLog`) and visit the appropriate web site to see
    regular updates from your weather station.

    Notes on the services
    ---------------------

    UK Met Office
    =============

    * Create account: https://register.metoffice.gov.uk/WaveRegistrationClient/public/register.do?service=weatherobservations
    * API: http://wow.metoffice.gov.uk/support?category=dataformats#automatic
    * Example ``weather.ini`` section::

        [metoffice]
        site id = 12345678
        aws pin = 987654

    Open Weather Map
    ================

    * Create account: http://openweathermap.org/login
    * API: http://openweathermap.org/API
    * Example ``weather.ini`` section::

        [openweathermap]
        lat = 51.501
        long = -0.142
        alt = 10
        user = Elizabeth Windsor
        password = corgi
        id = Buck House

    The default behaviour is to use your user name to identify the weather
    station. However, it's possible for a user to have more than one
    weather station, so there is an undocumented ``name`` parameter in the
    API that can be used to identify the station. This appears as ``id``
    in ``weather.ini``. Make sure you don't choose a name that is already
    in use.

    PWS Weather
    ===================

    * Create account: http://www.pwsweather.com/register.php
    * API based on WU protocol: `<http://wiki.wunderground.com/index.php/PWS_-_Upload_Protocol>`_
    * Example ``weather.ini`` section::

        [pwsweather]
        station = ABCDEFGH1
        password = xxxxxxx

    Weather Underground
    ===================

    * Create account: http://www.wunderground.com/members/signup.asp
    * API: `<http://wiki.wunderground.com/index.php/PWS_-_Upload_Protocol>`_
    * Example ``weather.ini`` section::

        [underground]
        station = ABCDEFGH1
        password = xxxxxxx

    API
    ---

    """

    __docformat__ = "restructuredtext en"
    __usage__ = """
     usage: python -m pywws.toservice [options] data_dir service_name
     options are:
      -h or --help     display this help
      -c or --catchup  upload all data since last upload
      -v or --verbose  increase amount of reassuring messages
     data_dir is the root directory of the weather data
     service_name is the service to upload to, e.g. underground
    """
    __doc__ %= __usage__
    __usage__ = __doc__.split('\n')[0] + __usage__

    from ConfigParser import SafeConfigParser
    import getopt
    import logging
    import os
    import re
    import socket
    import sys
    import urllib
    import urllib2
    from datetime import datetime, timedelta

    from pywws import DataStore
    from pywws.Logger import ApplicationLogger
    from pywws import Template

    FIVE_MINS = timedelta(minutes=5)

    class ToService(object):
        """Upload weather data to weather services such as Weather
        Underground.

        """
        def __init__(self, params, status, calib_data, service_name):
            """

            :param params: pywws configuration.

            :type params: :class:`pywws.DataStore.params`

            :param status: pywws status store.

            :type status: :class:`pywws.DataStore.status`

            :param calib_data: 'calibrated' data.

            :type calib_data: :class:`pywws.DataStore.calib_store`

            :param service_name: name of service to upload to.

            :type service_name: string

            """
            self.logger = logging.getLogger('pywws.ToService(%s)' % service_name)
            self.params = params
            self.status = status
            self.data = calib_data
            self.service_name = service_name
            # 'derived' services such as 'underground_rf' share their
            # parent's config and templates
            config_section = self.service_name.split('_')[0]
            self.old_response = None
            self.old_ex = None
            # set default socket timeout, so urlopen calls don't hang forever
            socket.setdefaulttimeout(30)
            # open params file
            service_params = SafeConfigParser()
            service_params.optionxform = str
            service_params.readfp(open(os.path.join(
                os.path.dirname(__file__), 'services', '%s.ini' % (self.service_name))))
            # get URL
            self.server = service_params.get('config', 'url')
            # get fixed part of upload data
            self.fixed_data = dict()
            for name, value in service_params.items('fixed'):
                if value[0] == '*':
                    value = self.params.get(config_section, value[1:], 'unknown')
                self.fixed_data[name] = value
            # create templater
            self.templater = Template.Template(
                self.params, self.status, self.data, self.data, None, None,
                use_locale=False)
            self.template_file = os.path.join(
                os.path.dirname(__file__), 'services', '%s_template_%s.txt' % (
                    config_section, self.params.get('config', 'ws type')))
            # get other parameters
            self.catchup = eval(service_params.get('config', 'catchup'))
            self.use_get = eval(service_params.get('config', 'use get'))
            self.expected_result = eval(service_params.get('config', 'result'))

        def translate_data(self, current, fixed_data):
            """Convert a weather data record to upload format.

            The :obj:`current` parameter contains the data to be uploaded.
            It should be a 'calibrated' data record, as stored in
            :class:`pywws.DataStore.calib_store`.

            The :obj:`fixed_data` parameter contains unvarying data that
            is site dependent, for example an ID code and authentication
            data.

            :param current: the weather data record.

            :type current: dict

            :param fixed_data: unvarying upload data.

            :type fixed_data: dict

            :return: converted data, or :obj:`None` if invalid data.

            :rtype: dict(string)

            """
            # check we have enough data
            if current['temp_out'] is None or current['hum_out'] is None:
                return None
            # convert data
            result = dict(fixed_data)
            template_data = self.templater.make_text(self.template_file, current)
            result.update(eval(template_data))
            return result

        def send_data(self, data, server, fixed_data):
            """Upload a weather data record.

            The :obj:`data` parameter contains the data to be uploaded.
            It should be a 'calibrated' data record, as stored in
            :class:`pywws.DataStore.calib_store`.

            The :obj:`fixed_data` parameter contains unvarying data that
            is site dependent, for example an ID code and authentication
            data.

            :param data: the weather data record.

            :type data: dict

            :param server: web address to upload to.

            :type server: string

            :param fixed_data: unvarying upload data.

            :type fixed_data: dict

            :return: success status

            :rtype: bool

            """
            coded_data = self.translate_data(data, fixed_data)
            if not coded_data:
                return True
            self.logger.debug(coded_data)
            coded_data = urllib.urlencode(coded_data)
            # have three tries before giving up
            for n in range(3):
                try:
                    if sys.hexversion <= 0x020406ff:
                        wudata = urllib.urlopen('%s?%s' % (server, coded_data))
                    else:
                        try:
                            if self.use_get:
                                wudata = urllib2.urlopen(
                                    '%s?%s' % (server, coded_data))
                            else:
                                wudata = urllib2.urlopen(server, coded_data)
                        except urllib2.HTTPError, ex:
                            if ex.code != 400:
                                raise
                            wudata = ex
                    response = wudata.readlines()
                    wudata.close()
                    if len(response) == len(self.expected_result):
                        for actual, expected in zip(response, self.expected_result):
                            if not re.match(expected, actual):
                                break
                        else:
                            self.old_response = response
                            return True
                    if response != self.old_response:
                        for line in response:
                            self.logger.error(line.strip())
                    self.old_response = response
                except Exception, ex:
                    e = str(ex)
                    if e != self.old_ex:
                        self.logger.error(e)
                        self.old_ex = e
            return False

        def Upload(self, catchup):
            """Upload one or more weather data records.

            This method uploads either the most recent weather data
            record, or all records since the last upload (up to 7 days),
            according to the value of :obj:`catchup`.

            It sets the ``last update`` configuration value to the time
            stamp of the most recent record successfully uploaded.

            :param catchup: upload all data since last upload.

            :type catchup: bool

            :return: success status

            :rtype: bool

            """
            if catchup and self.catchup > 0:
                start = datetime.utcnow() - timedelta(days=self.catchup)
                last_update = self.params.get_datetime(
                    self.service_name, 'last update')
                if last_update:
                    self.params.unset(self.service_name, 'last update')
                    self.status.set('last update', self.service_name,
                                    last_update.isoformat(' '))
                last_update = self.status.get_datetime(
                    'last update', self.service_name)
                if last_update:
                    start = max(start, last_update + timedelta(minutes=1))
                count = 0
                for data in self.data[start:]:
                    if not self.send_data(data, self.server, self.fixed_data):
                        return False
                    self.status.set('last update', self.service_name,
                                    data['idx'].isoformat(' '))
                    count += 1
                if count:
                    self.logger.info('%d records sent', count)
            else:
                # upload most recent data
                last_update = self.data.before(datetime.max)
                if not self.send_data(
                        self.data[last_update], self.server, self.fixed_data):
                    return False
                self.status.set(
                    'last update', self.service_name, last_update.isoformat(' '))
            return True

    def main(argv=None):
        if argv is None:
            argv = sys.argv
        try:
            opts, args = getopt.getopt(
                argv[1:], "hcv", ['help', 'catchup', 'verbose'])
        except getopt.error, msg:
            print >>sys.stderr, 'Error: %s\n' % msg
            print >>sys.stderr, __usage__.strip()
            return 1
        # process options
        catchup = False
        verbose = 0
        for o, a in opts:
            if o == '-h' or o == '--help':
                print __usage__.strip()
                return 0
            elif o == '-c' or o == '--catchup':
                catchup = True
            elif o == '-v' or o == '--verbose':
                verbose += 1
        # check arguments
        if len(args) != 2:
            print >>sys.stderr, "Error: 2 arguments required"
            print >>sys.stderr, __usage__.strip()
            return 2
        logger = ApplicationLogger(verbose)
        return ToService(
            DataStore.params(args[0]), DataStore.status(args[0]),
            DataStore.calib_store(args[0]), args[1]).Upload(catchup)

    if __name__ == "__main__":
        sys.exit(main())

### Navigation

- index
- modules |
- pywws 13.06_r1023 documentation »
- Code du module »