pyncei
======

This module provides tools to request data from the `Climate Data Online
webservices <http://www.ncdc.noaa.gov/cdo-web/webservices/v2#gettingStarted>`__
provided by NOAA's National Centers for Environmental information
(formerly the National Center for Climate Data). Install with:

::

    pip install pyncei

Documentation for this project is available on `Read The
Docs <http://pyncei.readthedocs.org/en/latest/pyncei.html>`__.

Getting started
---------------

To use the NCEI webservices, you'll need a token. The token is a
32-character string provided by NCEI; users can request one
`here <https://www.ncdc.noaa.gov/cdo-web/token>`__. Pass the token to
`pyncei.reader.NCEIReader() <http://pyncei.readthedocs.org/en/latest/pyncei.html#pyncei.reader.NCEIReader>`__
to get started:

.. code:: python

    from pyncei.reader import NCEIReader

    token = 'AnExampleTokenFromTheNCEIWebsite'
    ncei = NCEIReader(token)

The NCEIReader class includes methods corresponding to each of the
endpoints described on the CDO website. Query parameters specified by
CDO can be passed as arguments:

.. code:: python

    ncei.get_stations(location='FIPS:11')
    ncei.get_data(dataset='GHCND',
                  station=['COOP:010957'],
                  datatype=['TMIN','TMAX'],
                  startdate='2015-03-01',
                  enddate='2016-03-01')

The table below provides a quick overview of the various endpoints and
available parameters:

+---------------+--------------+-------------------+--------------+-----------------+
| CDO Endpoint  | CDO Query    | NCEIReader Method | NCEIReader   | Values          |
|               | Parameter    |                   | Argument     |                 |
+===============+==============+===================+==============+=================+
| `datasets <ht | datasetid    | `get\_datasets()  | dataset      | `datasets.csv < |
| tp://www.ncdc |              | <http://pyncei.re |              | https://github. |
| .noaa.gov/cdo |              | adthedocs.org/en/ |              | com/adamancer/p |
| -web/webservi |              | latest/pyncei.htm |              | yncei/tree/mast |
| ces/v2#datase |              | l#pyncei.reader.N |              | er/pyncei/files |
| ts>`__        |              | CEIReader.get_dat |              | /datasets.csv>` |
|               |              | asets>`__         |              | __              |
+---------------+--------------+-------------------+--------------+-----------------+
| `datacategori | datacategory | `get\_data\_categ | datatype     | `datatypes.csv  |
| es <http://ww | id           | ories() <http://p |              | <https://github |
| w.ncdc.noaa.g |              | yncei.readthedocs |              | .com/adamancer/ |
| ov/cdo-web/we |              | .org/en/latest/py |              | pyncei/tree/mas |
| bservices/v2# |              | ncei.html#pyncei. |              | ter/pyncei/file |
| dataCategorie |              | reader.NCEIReader |              | s/datatypes.csv |
| s>`__         |              | .get_data_categor |              | >`__            |
|               |              | ies>`__           |              |                 |
+---------------+--------------+-------------------+--------------+-----------------+
| `datatypes <h | datatypeid   | `get\_data\_types | datacategory | `datacategories |
| ttp://www.ncd |              | () <http://pyncei |              | .csv <https://g |
| c.noaa.gov/cd |              | .readthedocs.org/ |              | ithub.com/adama |
| o-web/webserv |              | en/latest/pyncei. |              | ncer/pyncei/tre |
| ices/v2#dataT |              | html#pyncei.reade |              | e/master/pyncei |
| ypes>`__      |              | r.NCEIReader.get_ |              | /files/datacate |
|               |              | data_types>`__    |              | gories.csv>`__  |
+---------------+--------------+-------------------+--------------+-----------------+
| `locationcate | locationcate | `get\_location\_c | locationcate | `locationcatego |
| gories <http: | goryid       | ategories() <http | gory         | ries.csv <https |
| //www.ncdc.no |              | ://pyncei.readthe |              | ://github.com/a |
| aa.gov/cdo-we |              | docs.org/en/lates |              | damancer/pyncei |
| b/webservices |              | t/pyncei.html#pyn |              | /tree/master/py |
| /v2#locationC |              | cei.reader.NCEIRe |              | ncei/files/loca |
| ategories>`__ |              | ader.get_location |              | tioncategories. |
|               |              | _categories>`__   |              | csv>`__         |
+---------------+--------------+-------------------+--------------+-----------------+
| `locations <h | locationid   | `get\_locations() | location     | `locations.csv  |
| ttp://www.ncd |              |  <http://pyncei.r |              | <https://github |
| c.noaa.gov/cd |              | eadthedocs.org/en |              | .com/adamancer/ |
| o-web/webserv |              | /latest/pyncei.ht |              | pyncei/tree/mas |
| ices/v2#locat |              | ml#pyncei.reader. |              | ter/pyncei/file |
| ions>`__      |              | NCEIReader.get_lo |              | s/locations.csv |
|               |              | cations>`__       |              | >`__            |
+---------------+--------------+-------------------+--------------+-----------------+
| `stations <ht | stationid    | `get\_stations()  | station      | --              |
| tp://www.ncdc |              | <http://pyncei.re |              |                 |
| .noaa.gov/cdo |              | adthedocs.org/en/ |              |                 |
| -web/webservi |              | latest/pyncei.htm |              |                 |
| ces/v2#statio |              | l#pyncei.reader.N |              |                 |
| ns>`__        |              | CEIReader.get_sta |              |                 |
|               |              | tions>`__         |              |                 |
+---------------+--------------+-------------------+--------------+-----------------+
| `data <http:/ | --           | `get\_data() <htt | --           | --              |
| /www.ncdc.noa |              | p://pyncei.readth |              |                 |
| a.gov/cdo-web |              | edocs.org/en/late |              |                 |
| /webservices/ |              | st/pyncei.html#py |              |                 |
| v2#data>`__   |              | ncei.reader.NCEIR |              |                 |
|               |              | eader.get_data>`_ |              |                 |
|               |              | _                 |              |                 |
+---------------+--------------+-------------------+--------------+-----------------+

Each NCEIReader method accepts either a single positional argument (used
to return data for a single id) or a series of keyword arguments.

Note that id fields used by CDO have been renamed. For example,
datasetid has been renamed dataset, and locationid has been renamed
location. Unlike CDO, which accepts only ids, NCEIReader will accept
either ids or name strings. If names are used, NCEIReader attempts to
map the name strings to valid id using
`NCEIReader.map\_name() <http://pyncei.readthedocs.org/en/latest/pyncei.html#pyncei.reader.NCEIReader.map_name>`__,
called manually here:

.. code:: python

    ncei.map_name('District of Columbia', 'locations')
    # returns ('FIPS:11', True)

When the mapping function fails to find an exact match, it throws an
exception containing a list of similar values that can be used to refine
the original query.

Finding the right term
----------------------

If you have no idea what data is available or where to look, you have a
few options. You can search the terms available for each endpoint using
`NCEIReader.find\_in\_endpoint() <http://pyncei.readthedocs.org/en/latest/pyncei.html#pyncei.reader.NCEIReader.find_in_endpoint>`__:

.. code:: python

    ncei.find_in_endpoint('District of Columbia', 'locations')
    # returns ['FIPS:11 => District of Columbia',
    #          'FIPS:11001 => District of Columbia County, DC']

If the search term is None, this function will return ALL available ids
for the given endpoint. You can search across all endpoints using
`NCEIReader.find\_all() <http://pyncei.readthedocs.org/en/latest/pyncei.html#pyncei.reader.NCEIReader.find_all>`__:

.. code:: python

    ncei.find_all('temperature')
    # returns [('datacategories', 'ANNTEMP', 'Annual Temperature'),
    #          ('datacategories', 'AUTEMP', 'Autumn Temperature'),...]

You can also browse the source files in the Values column of the table
above. The data in these files shouldn't change much, but they can be
updated using
`NCEIReader.refresh\_lookups() <http://pyncei.readthedocs.org/en/latest/pyncei.html#pyncei.reader.NCEIReader.refresh_lookups>`__
if necessary:

.. code:: python

    ncei.refresh_lookups()

Queries are cached for one day by default. Users can change this
behavior using the cache parameter when initializing an NCEIReader
object. This parameter specifies the number of seconds pages should
persist in the cache; a value of zero disables the cache entirely.

Example: Find and return data from a station
--------------------------------------------

.. code:: python

    import csv
    from datetime import date

    from pyncei import NCEIReader


    # Initialize NCEIReader object using your token string
    ncei = NCEIReader('AnExampleTokenFromTheNCEIWebsite')
    ncei.debug = True  # this flag produces verbose output

    # Set the parameters you're looking for. You can use ncei.find_all() or
    # ncei_find_in_endpoint() to search the available parameters if you don't
    # know what to use.
    mindate = '1966-01-01'  # either yyyy-mm-dd or a datetime object
    maxdate = '2015-12-31'
    datatypes = ['TMIN', 'TMAX']
    dataset = 'GHCND'

    # You can manually verify parameters if you're so inclined
    for datatype in datatypes:
        ncei.map_name(datatype, 'datatypes')

    # Get all DC stations operating between mindate and maxdate. The date
    # parameters in station queries are a little odd. According to the docs,
    # queries will return stations with data from on/before the enddate and
    # on/after the startdate. If both parameters are included, the result set
    # seems to include all stations that EITHER have data from on/before the
    # startdate OR have data on/after the enddate.
    stations = ncei.get_stations(location='District of Columbia',
                                 dataset=dataset,
                                 datatype=datatypes,
                                 enddate=mindate)

    # Filter out stations no longer operating using maxdate
    stations = [station for station in stations
                if station['maxdate'] >= maxdate]

    # Find the station with the best data coverage in the result set
    stations.sort(key=lambda s:s['datacoverage'], reverse=True)
    station = stations[0]
    minyear = int(station['mindate'][:4])

    # Get temperature data for the the lifetime of the station. Note that for the
    # data endpoint, you can't request more than one year's worth of data at a
    # time.
    year = date.today().year - 1
    results = []
    while year >= minyear:
        results.extend(ncei.get_data(dataset=dataset,
                                     station=station['id'],
                                     datatype=datatypes,
                                     startdate=date(year, 1, 1),
                                     enddate=date(year, 12, 31)))
        year -= 1

    # Write results to csv
    fn = station['id'].replace(':', '') + '.csv'
    with open(fn, 'wb') as f:
        writer = csv.writer(f, delimiter=',', quotechar='"')
        keys = ('date', 'datatype', 'value')
        writer.writerow(keys)
        for row in results:
            row['date'] = row['date'].split('T')[0].replace('-', '')
            writer.writerow([row[key] for key in keys])

