{"body": "
\n

# How to use the storage backends\u00b6

\n

By default, Collectors uses a simple python `list` for each series. You\ncan use other storage backends to handle very large amounts of data or to get a\nsimple MS Excel export. You can also add your own storage classes very easily.

\n

All storage classes can be found in submodules of `collectors.storage`\n(e.g. `collectors.storage.pytables.PyTables`) but you can also import\n`PyTables` and\n`Excel` directly from\n`collectors.storage`.

\n

You must pass an instance of the storage as keyword argument `backend` to a\nnew Collector. Each storage instance should only be used with one Collector\ninstance.

\n

    from collectors import Collector\nfrom collectors.storage import MyStorage\nc = Collector(..., backend=MyStorage())\n

\n
\n
\n

## PyTables/HDF5\u00b6

\n

PyTables is not bundled with this package.\nInstructions follow:

\n

Mac OS X (10.6 Snow Leopard)

\n

You should not use the precompiled version of HDF5 because it\u2019s linked against\nszip, which is not bundled with HDF5 and available under a license you might\nnot want. So you need to compile it yourself:

\n

\n
1. Download the source from ftp://ftp.hdfgroup.org/HDF5/current/src/
\n
2. Build and install (PyTables will auto detect it if you install it under\n`/usr/local`):
\n

\n

    $ ./configure --prefix=/usr/local\n$ make\n$ sudo make install\n

\n
\n

\n
1. Finally install PyTables:
\n

\n

    $ sudo pip install tables\n

\n
\n

Ubuntu (9.10 Karmic Koala)

\n

Ubuntu\u2019s package for PyTables is somehow broken, so you need to build your own.\nIf gcc is already installed, you just need to add the development files for\npython and HDF5 before you can build and install PyTables from PyPI:

\n

    $ sudo aptitude install python-dev libhdf5-serial-dev\n$ sudo pip install tables\n

\n
\n

Windows

\n

Download the installer from here\nand execute it. Further information can be found in the PyTables manual.

\n
\n

### Example\u00b6

\n

    >>> import tables\n>>> from collectors import Collector, get, storage\n>>>\n>>> class Spam(object):\n...     a = 1\n...     b = 2\n...\n>>> spam = Spam()\n>>> h5file = tables.openFile('/tmp/example.h5', mode='w')\n>>>\n>>> collector = Collector(get(spam, 'a', 'b'),\n...         backend=storage.PyTables(h5file, 'spamgroup', ('int', 'int'))\n... )\n>>>\n>>> for values in zip(range(10), reversed(range(10))):\n...     spam.a, spam.b = values\n...     collector()\n...\n>>> print collector.a.read(), collector.b.read()\n[0 1 2 3 4 5 6 7 8 9] [9 8 7 6 5 4 3 2 1 0]\n>>> print collector.a.read().mean(), collector.b.read().max()\n4.5 9\n>>> h5file.close()\n

\n
\n

The `PyTables` storage stores the results\nfor multiple Collector instances in one file. For each Collector, a new group\nwill be create and each observed variable will have its own array within that\ngroup, so the group name musst be unique among all collectors that use the same\nHDF5 file.

\n

You also need to specifiy the data types of the observed variables. They must be\npassed as list of strings (like e.g. `'int'`, `'float'` or `'bool'`, see\nthe NumPy Docs for more details).

\n

The series are now EArrays instead of\nsimple lists. The `read` function returns the complete series for that\nvariable as a NumPy array so you can do further calculations on them very fast.

\n
\n
\n
\n

## MS Excel\u00b6

\n

The `Excel` storage allows you to store your\ndata directly in an Excel file.

\n

To use this storage backend, you need to install `xlwt` (like \u201cExcel\nWrite\u201d); `xlrd` (\u201cExcel Read\u201d) can be used to read from an Excel file:

\n

    $ sudo pip install xlwt\n

\n
\n
\n

### Example\u00b6

\n

    >>> from xlwt import Workbook\n>>> from collectors import Collector, get, storage\n>>>\n>>> w = Workbook()\n>>>\n>>> class ObserveMe(object):\n...     pass\n...\n>>> obj = ObserveMe()\n>>> c = Collector(get(obj, 'value_a', 'value_b'),\n...         backend=storage.Excel(w, 'my collected data'))\n>>>\n>>> for a, b in zip(range(10), reversed(range(10))):\n...     obj.value_a, obj.value_b = a, b\n...     c()\n...\n>>> w.save('/tmp/example.xls')\n

\n
\n

Using the the Excel storage is quite easy. Just create a new `Workbook` and\npass it with a name for the sheet to `Excel`\nconstructor. Alternatively, you can create a sheet with\n`workbook.add_sheet('name')` and pass the sheet instance instead of a string\nwith the sheet\u2019s name.

\n

When you are done collecting data, save the Workbook to a file by calling its\n`save()` method.

\n
\n
\n
\n

## What\u2019s next?\u00b6

\n

Collectors was originally developed to use it with SimPy, so we\u2019ll show how both packages can be used\ntogether in the next section.

\n
\n
\n", "display_toc": true, "title": "How to use the storage backends", "sourcename": "storages.txt", "current_page_name": "storages", "next": {"link": "../simpy/", "title": "How to use Collectors with SimPy"}, "rellinks": [["genindex", "General Index", "I", "index"], ["modindex", "Global Module Index", "M", "modules"], ["simpy", "How to use Collectors with SimPy", "N", "next"], ["collector", "The Collector in-depth", "P", "previous"]], "meta": {}, "parents": [], "toc": "

\n
- How to use the storage backends
  \n
  - PyTables/HDF5
    \n
    - Example
    \n
    \n
  \n
  - MS Excel
    \n
    - Example
    \n
    \n
  \n
  - What\u2019s next?
  \n
  \n
\n

\n", "prev": {"link": "../collector/", "title": "The Collector in-depth"}, "metatags": ""}