Storage APIs — ZODB documentation

# Storage APIs

Contents

- Storage APIs
  - Storage interfaces
    - IStorage
    - IStorageIteration
    - IStorageUndoable
    - IStorageCurrentRecordIteration
    - IBlobStorage
    - IStorageRecordInformation
    - IStorageTransactionInformation
  - Included storages
    - FileStorage
      - FileStorage text configuration
    - MappingStorage
    - MappingStorage text configuration
    - DemoStorage
    - DemoStorage text configuration
  - Noteworthy non-included storages

## Storage interfaces

There are various storage implementations that implement standard storage interfaces. They differ primarily in their constructors.

Application code rarely calls storage methods, and those it calls are generally called indirectly through databases. There are interface-defined methods that are called internally by ZODB. These aren’t shown below.

### IStorage

interface `ZODB.interfaces.` `IStorage`

  A storage is responsible for storing and retrieving data of objects.

  Consistency and locking

  When transactions are committed, a storage assigns monotonically increasing transaction identifiers (tids) to the transactions and to the object versions written by the transactions. ZODB relies on this to decide if data in object caches are up to date and to implement multi-version concurrency control.

  There are methods in IStorage and in derived interfaces that provide information about the current revisions (tids) for objects or for the database as a whole. It is critical for the proper working of ZODB that the resulting tids are increasing with respect to the object identifier given or to the databases. That is, if there are 2 results for an object or for the database, R1 and R2, such that R1 is returned before R2, then the tid returned by R2 must be greater than or equal to the tid returned by R1. (When thinking about results for the database, think of these as results for all objects in the database.)

  This implies some sort of locking strategy. The key method is tcp_finish, which causes new tids to be generated and also, through the callback passed to it, returns new current tids for the objects stored in a transaction and for the database as a whole.

  The IStorage methods affected are lastTransaction, load, store, and tpc_finish. Derived interfaces may introduce additional methods.

  `__len__`()

    The approximate number of objects in the storage

    This is used soley for informational purposes.

  `close`()

    Close the storage.

    Finalize the storage, releasing any external resources. The storage should not be used after this method is called.

    Note that databses close their storages when they’re closed, so this method isn’t generally called from application code.

  `getName`()

    The name of the storage

    The format and interpretation of this name is storage dependent. It could be a file name, a database name, etc..

    This is used soley for informational purposes.

  `getSize`()

    An approximate size of the database, in bytes.

    This is used soley for informational purposes.

  `history`(oid, size=1)

    Return a sequence of history information dictionaries.

    Up to size objects (including no objects) may be returned.

    The information provides a log of the changes made to the object. Data are reported in reverse chronological order.

    Each dictionary has the following keys:

    time
      UTC seconds since the epoch (as in time.time) that the object revision was committed.
    tid
      The transaction identifier of the transaction that committed the version.
    serial
      An alias for tid, which expected by older clients.
    user_name
      The user identifier, if any (or an empty string) of the user on whos behalf the revision was committed.
    description
      The transaction description for the transaction that committed the revision.
    size
      The size of the revision data record.

    If the transaction had extension items, then these items are also included if they don’t conflict with the keys above.

  `isReadOnly`()

    Test whether a storage allows committing new transactions

    For a given storage instance, this method always returns the same value. Read-only-ness is a static property of a storage.

  `lastTransaction`()

    Return the id of the last committed transaction.

    If no transactions have been committed, return a string of 8 null (0) characters.

  `pack`(pack_time, referencesf)

    Pack the storage

    It is up to the storage to interpret this call, however, the general idea is that the storage free space by:

    - discarding object revisions that were old and not current as of the given pack time.
    - garbage collecting objects that aren’t reachable from the root object via revisions remaining after discarding revisions that were not current as of the pack time.

    The pack time is given as a UTC time in seconds since the epoch.

    The second argument is a function that should be used to extract object references from database records. This is needed to determine which objects are referenced from object revisions.

  `sortKey`()

    Sort key used to order distributed transactions

    When a transaction involved multiple storages, 2-phase commit operations are applied in sort-key order. This must be unique among storages used in a transaction. Obviously, the storage can’t assure this, but it should construct the sort key so it has a reasonable chance of being unique.

    The result must be a string.

### IStorageIteration

interface `ZODB.interfaces.` `IStorageIteration`

  API for iterating over the contents of a storage.

  `iterator`(start=None, stop=None)

    Return an IStorageTransactionInformation iterator.

    If the start argument is not None, then iteration will start with the first transaction whose identifier is greater than or equal to start.

    If the stop argument is not None, then iteration will end with the last transaction whose identifier is less than or equal to stop.

    The iterator provides access to the data as available at the time when the iterator was retrieved.

### IStorageUndoable

interface `ZODB.interfaces.` `IStorageUndoable`

  A storage supporting transactional undo.

  `undoInfo`(first=0, last=-20, specification=None)

    Return a sequence of descriptions for undoable transactions.

    This is like undoLog(), except for the specification argument. If given, specification is a dictionary, and undoInfo() synthesizes a filter function f for undoLog() such that f(desc) returns true for a transaction description mapping desc if and only if desc maps each key in specification to the same value specification maps that key to. In other words, only extensions (or supersets) of specification match.

    ZEO note: undoInfo() passes the specification argument from a ZEO client to its ZEO server (while a ZEO client ignores any filter argument passed to undoLog()).

  `undoLog`(first, last, filter=None)

    Return a sequence of descriptions for undoable transactions.

    Application code should call undoLog() on a DB instance instead of on the storage directly.

    A transaction description is a mapping with at least these keys:

    > “time”: The time, as float seconds since the epoch, when
    >   the transaction committed.
    > “user_name”: The value of the .user attribute on that
    >   transaction.
    > “description”: The value of the .description attribute on
    >   that transaction.
    > “id`” A string uniquely identifying the transaction to the
    >   storage. If it’s desired to undo this transaction, this is the transaction_id to pass to undo().

    In addition, if any name+value pairs were added to the transaction by setExtendedInfo(), those may be added to the transaction description mapping too (for example, FileStorage’s undoLog() does this).

    filter is a callable, taking one argument. A transaction description mapping is passed to filter for each potentially undoable transaction. The sequence returned by undoLog() excludes descriptions for which filter returns a false value. By default, filter always returns a true value.

    ZEO note: Arbitrary callables cannot be passed from a ZEO client to a ZEO server, and a ZEO client’s implementation of undoLog() ignores any filter argument that may be passed. ZEO clients should use the related undoInfo() method instead (if they want to do filtering).

    Now picture a list containing descriptions of all undoable transactions that pass the filter, most recent transaction first (at index 0). The first and last arguments specify the slice of this (conceptual) list to be returned:

    > first: This is the index of the first transaction description
    >   in the slice. It must be >= 0.
    > last: If >= 0, first:last acts like a Python slice, selecting
    >   the descriptions at indices first, first+1, ..., up to but not including index last. At most last-first descriptions are in the slice, and last should be at least as large as first in this case. If last is less than 0, then abs(last) is taken to be the maximum number of descriptions in the slice (which still begins at index first). When last < 0, the same effect could be gotten by passing the positive first-last for last instead.

### IStorageCurrentRecordIteration

interface `ZODB.interfaces.` `IStorageCurrentRecordIteration`
  `record_iternext`(next=None)

    Iterate over the records in a storage

    Use like this:

        >>> next = None
        >>> while 1:
        ...     oid, tid, data, next = storage.record_iternext(next)
        ...     # do things with oid, tid, and data
        ...     if next is None:
        ...         break

### IBlobStorage

interface `ZODB.interfaces.` `IBlobStorage`

  A storage supporting BLOBs.

  `temporaryDirectory`()

    Return a directory that should be used for uncommitted blob data.

    If Blobs use this, then commits can be performed with a simple rename.

### IStorageRecordInformation

interface `ZODB.interfaces.` `IStorageRecordInformation`

  Provide information about a single storage record

  `data` = <zope.interface.interface.Attribute object>

    The data record

  `data_txn` = <zope.interface.interface.Attribute object>

    The previous transaction id

  `oid` = <zope.interface.interface.Attribute object>

    The object id

  `tid` = <zope.interface.interface.Attribute object>

    The transaction id

  `version` = <zope.interface.interface.Attribute object>

    The version id

### IStorageTransactionInformation

interface `ZODB.interfaces.` `IStorageTransactionInformation`

  Provide information about a storage transaction.

  Can be iterated over to retrieve the records modified in the transaction.

  `__iter__`()

    Iterate over the transaction’s records given as IStorageRecordInformation objects.

  `description` = <zope.interface.interface.Attribute object>

    Transaction Description

  `extension` = <zope.interface.interface.Attribute object>

    A dictionary carrying the transaction’s extension data

  `status` = <zope.interface.interface.Attribute object>

    Transaction Status

  `tid` = <zope.interface.interface.Attribute object>

    Transaction id

  `user` = <zope.interface.interface.Attribute object>

    Transaction user

## Included storages

### FileStorage

class `ZODB.FileStorage.FileStorage.` `FileStorage`(file_name, create=False, read_only=False, stop=None, quota=None, pack_gc=True, pack_keep_old=True, packer=None, blob_dir=None)

  Storage that saves data in a file

  `__init__`(file_name, create=False, read_only=False, stop=None, quota=None, pack_gc=True, pack_keep_old=True, packer=None, blob_dir=None)

    Create a file storage

    Parameters: | file_name (str) – Path to store data file create (bool) – Flag indicating whether a file should be created even if it already exists. read_only (bool) – Flag indicating whether the file is read only. Only one process is able to open the file non-read-only. stop (bytes) – Time-travel transaction id When the file is opened, data will be read up to the given transaction id. Transaction ids correspond to times and you can compute transaction ids for a given time using `TimeStamp`. quota (int) – File-size quota pack_gc (bool) – Flag indicating whether garbage collection should be performed when packing. pack_keep_old (bool) – flag indicating whether old data files should be retained after packing as a `.old` file. packer (callable) – An alternative `packer`. blob_dir (str) – A blob-directory path name. Blobs will be supported if this option is provided.

    A file storage stores data in a single file that behaves like a traditional transaction log. New data records are appended to the end of the file. Periodically, the file is packed to free up space. When this is done, current records as of the pack time or later are copied to a new file, which replaces the old file.

    FileStorages keep in-memory indexes mapping object oids to the location of their current records in the file. Back pointers to previous records allow access to non-current records from the current records.

    In addition to the data file, some ancillary files are created. These can be lost without affecting data integrity, however losing the index file may cause extremely slow startup. Each has a name that’s a concatenation of the original file and a suffix. The files are listed below by suffix:

    .index
      Snapshot of the in-memory index. This are created on shutdown, packing, and after rebuilding an index when one was not found. For large databases, creating a file-storage object without an index file can take very long because it’s necessary to scan the data file to build the index.
    .lock
      A lock file preventing multiple processes from opening a file storage on non-read-only mode.
    .tmp
      A file used to store data being committed in the first phase of 2-phase commit
    .index_tmp
      A temporary file used when saving the in-memory index to avoid overwriting an existing index until a new index has been fully saved.
    .pack
      A temporary file written while packing containing current records as of and after the pack time.
    .old
      The previous database file after a pack.

    When the database is packed, current records as of the pack time and later are written to the `.pack` file. At the end of packing, the `.old` file is removed, if it exists, and the data file is renamed to the `.old` file and finally the `.pack` file is rewritten to the data file.

interface `ZODB.FileStorage.interfaces.` `IFileStoragePacker`
  `__call__`(storage, referencesf, stop, gc)

    Pack the file storage into a new file

    Parameters: | storage (FileStorage) – The storage object to be packed referencesf (callable) – A function that extracts object references from a pickle bytes string. This is usually `ZODB.serialize.referencesf`. stop (bytes) – A transaction id representing the time at which to stop packing. gc (bool) – A flag indicating whether garbage collection should be performed.

    The new file will have the same name as the old file with `.pack` appended. (The packer can get the old file name via storage._file.name.) If blobs are supported, if the storages blob_dir attribute is not None or empty, then a .removed file must be created in the blob directory. This file contains records of the form:

        (oid+serial).encode('hex')+'\n'

    or, of the form:

        oid.encode('hex')+'\n'

    If packing is unnecessary, or would not change the file, then no pack or removed files are created None is returned, otherwise a tuple is returned with:

    - the size of the packed file, and
    - the packed index

    If and only if packing was necessary (non-None) and there was no error, then the commit lock must be acquired. In addition, it is up to FileStorage to:

    - Rename the .pack file, and
    - process the blob_dir/.removed file by removing the blobs corresponding to the file records.

#### FileStorage text configuration

File storages are configured using the `filestorage` section:

    <filestorage>
      path Data.fs
    </filestorage>

which accepts the following options:

`blob-dir` (existing-dirpath)
  If supplied, the file storage will provide blob support and this is the name of a directory to hold blob data. The directory will be created if it doesn’t exist. If no value (or an empty value) is provided, then no blob support will be provided. (You can still use a BlobStorage to provide blob support.)
`create` (boolean)
  Flag that indicates whether the storage should be truncated if it already exists.
`pack-gc` (boolean, default: true)
  If false, then no garbage collection will be performed when packing. This can make packing go much faster and can avoid problems when objects are referenced only from other databases.
`pack-keep-old` (boolean, default: true)
  If true, a copy of the database before packing is kept in a ”.old” file.
`packer` (string)
  The dotted name (dotted module name and object name) of a packer object. This is used to provide an alternative pack implementation.
`path` (existing-dirpath, required)
  Path name to the main storage file. The names for supplemental files, including index and lock files, will be computed from this.
`quota` (byte-size)
  Maximum allowed size of the storage file. Operations which would cause the size of the storage to exceed the quota will result in a ZODB.FileStorage.FileStorageQuotaError being raised.
`read-only` (boolean)
  If true, only reads may be executed against the storage. Note that the “pack” operation is not considered a write operation and is still allowed on a read-only filestorage.

### MappingStorage

class `ZODB.MappingStorage.` `MappingStorage`(name='MappingStorage')

  In-memory storage implementation

  Note that this implementation is somewhat naive and inefficient with regard to locking. Its implementation is primarily meant to be a simple illustration of storage implementation. It’s also useful for testing and exploration where scalability and efficiency are unimportant.

  `__init__`(name='MappingStorage')

    Create a mapping storage

    The name parameter is used by the `getName()` and `sortKey()` methods.

### MappingStorage text configuration

File storages are configured using the `mappingstorage` section:

    <mappingstorage>
    </mappingstorage>

Options:

`name` (string, default: Mapping Storage)
  The storage name, used by the `getName()` and `sortKey()` methods.

### DemoStorage

class `ZODB.DemoStorage.` `DemoStorage`(name=None, base=None, changes=None, close_base_on_close=None, close_changes_on_close=None)

  A storage that stores changes against a read-only base database

  This storage was originally meant to support distribution of application demonstrations with populated read-only databases (on CDROM) and writable in-memory databases.

  Demo storages are extemely convenient for testing where setup of a base database can be shared by many tests.

  Demo storages are also handy for staging appplications where a read-only snapshot of a production database (often accomplished using a beforestorage) is combined with a changes database implemented with a `FileStorage`.

  `__init__`(name=None, base=None, changes=None, close_base_on_close=None, close_changes_on_close=None)

    Create a demo storage

    Parameters: | name (str) – The storage name used by the `getName()` and `sortKey()` methods. base (object) – base storage changes (object) – changes storage close_base_on_close (bool) – A Flag indicating whether the base database should be closed when the demo storage is closed. close_changes_on_close (bool) – A Flag indicating whether the changes database should be closed when the demo storage is closed.

    If a base database isn’t provided, a `MappingStorage` will be constructed and used.

    If `close_base_on_close` isn’t specified, it will be `True` if a base database was provided and `False` otherwise.

    If a changes database isn’t provided, a `MappingStorage` will be constructed and used and blob support will be provided using a temporary blob directory.

    If `close_changes_on_close` isn’t specified, it will be `True` if a changes database was provided and `False` otherwise.

  `pop`()

    Close the changes database and return the base.

  `push`(changes=None)

    Create a new demo storage using the storage as a base.

    The given changes are used as the changes for the returned storage and `False` is passed as `close_base_on_close`.

### DemoStorage text configuration

Demo storages are configured using the `demostorage` section:

    <demostorage>
      <filestorage base>
        path base.fs
      </filestorage>
      <mappingstorage changes>
        name Changes
      </mappingstorage>
    </demostorage>

`demostorage` sections can contain up to 2 storage subsections, named `base` and `changes`, specifying the demo storage’s base and changes storages. See `ZODB.DemoStorage.DemoStorage.__init__()` for more on the base anc changes storages.

Options:

`name` (string)
  The storage name, used by the `getName()` and `sortKey()` methods.

## Noteworthy non-included storages

A number of important ZODB storages are distributed separately, including:

RelStorage

  RelStorage stores data in relational databases. This is especially useful when you have requirements or existing infrastructure for storing data in relational databases. Unlike the included storages, multiple processes can share the same database.

  For more information, see http://relstorage.readthedocs.io/en/latest/.

ZEO

  ZEO is a client-server database implementation for ZODB. To use ZEO, you run a ZEO server, and use ZEO clients in your application. Unlike the included storages, multiple processes can share the same database.

  For more information, see https://github.com/zopefoundation/ZEO.

ZRS

  ZRS provides replication from one database to another. It’s most commonly used with ZEO. With ZRS, you create a ZRS primary database around a `FileStorage` and in a separate process, you creatre a ZRS secondary storage around any `storage`. As transactions are committed on the primary, they’re copied asynchronously to secondaries.

  For more information, see https://github.com/zc/zrs.

zlibstorage

  zlibstorage compresses database records using the compression algorithm used by gzip.

  For more information, see https://pypi.python.org/pypi/zc.zlibstorage.

beforestorage

  beforestorage provides a point-in-time view of a database that might be changing. This can be useful to provide a non-changing view of a production database for use with a `DemoStorage`.

  For more information, see https://pypi.python.org/pypi/zc.beforestorage.

cipher.encryptingstorage

  cipher.encryptingstorage provided compression and encryption of database records.

  For more information, see https://pypi.python.org/pypi/cipher.encryptingstorage/.