API — dogpile.cache 0.2.1 documentation

### Navigation

- index
- modules |
- previous |
- dogpile.cache 0.2.1 documentation »

# API

## Region

class `dogpile.cache.region.` `CacheRegion`(name=None, function_key_generator=<function function_key_generator at 0x10173bf50>, key_mangler=None)

  A front end to a particular cache backend.

  - Parameters: | name – Optional, a string name for the region. This isn’t used internally but can be accessed via the `.name` parameter, helpful for configuring a region from a config file. function_key_generator – Optional. A function that will produce a “cache key” given a data creation function and arguments, when using the `CacheRegion.cache_on_arguments()` method. The structure of this function should be two levels: given the data creation function, return a new function that generates the key based on the given arguments. Such as:
        def my_key_generator(namespace, fn):
            fname = fn.__name__
            def generate_key(*arg):
                return namespace + "_" + fname + "_".join(str(s) for s in arg)
            return generate_key

        region = make_region(
            function_key_generator = my_key_generator
        ).configure(
            "dogpile.cache.dbm",
            expiration_time=300,
            arguments={
                "filename":"file.dbm"
            }
        )
    The `namespace` is that passed to `CacheRegion.cache_on_arguments()`. It’s not consulted outside this function, so in fact can be of any form. For example, it can be passed as a tuple, used to specify arguments to pluck from **kw:
        def my_key_generator(namespace, fn):
            def generate_key(*arg, **kw):
                return ":".join(
                        [kw[k] for k in namespace] +
                        [str(x) for x in arg]
                    )
    Where the decorator might be used as:
        @my_region.cache_on_arguments(namespace=('x', 'y'))
        def my_function(a, b, **kw):
            return my_data()
  key_mangler – Function which will be used on all incoming keys before passing to the backend. Defaults to `None`, in which case the key mangling function recommended by the cache backend will be used. A typical mangler is the SHA1 mangler found at `sha1_mangle_key()` which coerces keys into a SHA1 hash, so that the string length is fixed. To disable all key mangling, set to `False`.

  `cache_on_arguments`(namespace=None, expiration_time=None)

    A function decorator that will cache the return value of the function using a key derived from the function itself and its arguments.

    E.g.:

        @someregion.cache_on_arguments()
        def generate_something(x, y):
            return somedatabase.query(x, y)

    The decorated function can then be called normally, where data will be pulled from the cache region unless a new value is needed:

        result = generate_something(5, 6)

    The function is also given an attribute `invalidate`, which provides for invalidation of the value. Pass to `invalidate()` the same arguments you’d pass to the function itself to represent a particular value:

        generate_something.invalidate(5, 6)

    The default key generation will use the name of the function, the module name for the function, the arguments passed, as well as an optional “namespace” parameter in order to generate a cache key.

    Given a function `one` inside the module `myapp.tools`:

        @region.cache_on_arguments(namespace="foo")
        def one(a, b):
            return a + b

    Above, calling `one(3, 4)` will produce a cache key as follows:

        myapp.tools:one|foo|3, 4

    The key generator will ignore an initial argument of `self` or `cls`, making the decorator suitable (with caveats) for use with instance or class methods. Given the example:

        class MyClass(object):
            @region.cache_on_arguments(namespace="foo")
            def one(self, a, b):
                return a + b

    The cache key above for `MyClass().one(3, 4)` will again produce the same cache key of `myapp.tools:one|foo|3, 4` - the name `self` is skipped.

    The `namespace` parameter is optional, and is used normally to disambiguate two functions of the same name within the same module, as can occur when decorating instance or class methods as below:

        class MyClass(object):
            @region.cache_on_arguments(namespace='MC')
            def somemethod(self, x, y):
                ""

        class MyOtherClass(object):
            @region.cache_on_arguments(namespace='MOC')
            def somemethod(self, x, y):
                ""

    Above, the `namespace` parameter disambiguates between `somemethod` on `MyClass` and `MyOtherClass`. Python class declaration mechanics otherwise prevent the decorator from having awareness of the `MyClass` and `MyOtherClass` names, as the function is received by the decorator before it becomes an instance method.

    The function key generation can be entirely replaced on a per-region basis using the `function_key_generator` argument present on `make_region()` and `CacheRegion`. If defaults to `function_key_generator()`.

    Parameters: | namespace – optional string argument which will be established as part of the cache key. This may be needed to disambiguate functions of the same name within the same source file, such as those associated with classes - note that the decorator itself can’t see the parent class on a function as the class is being declared. expiration_time – if not None, will override the normal expiration time.

  `configure`(backend, expiration_time=None, arguments=None, _config_argument_dict=None, _config_prefix=None)

    Configure a `CacheRegion`.

    The `CacheRegion` itself is returned.

    Parameters: | backend – Required. This is the name of the `CacheBackend` to use, and is resolved by loading the class from the `dogpile.cache` entrypoint. expiration_time – Optional. The expiration time passed to the dogpile system. The `CacheRegion.get_or_create()` method as well as the `CacheRegion.cache_on_arguments()` decorator (though note: not the `CacheRegion.get()` method) will call upon the value creation function after this time period has passed since the last generation. arguments – Optional. The structure here is passed directly to the constructor of the `CacheBackend` in use, though is typically a dictionary.

  `configure_from_config`(config_dict, prefix)

    Configure from a configuration dictionary and a prefix.

    Example:

        local_region = make_region()
        memcached_region = make_region()

        # regions are ready to use for function
        # decorators, but not yet for actual caching

        # later, when config is available
        myconfig = {
            "cache.local.backend":"dogpile.cache.dbm",
            "cache.local.arguments.filename":"/path/to/dbmfile.dbm",
            "cache.memcached.backend":"dogpile.cache.pylibmc",
            "cache.memcached.arguments.url":"127.0.0.1, 10.0.0.1",
        }
        local_region.configure_from_config(myconfig, "cache.local.")
        memcached_region.configure_from_config(myconfig,
                                            "cache.memcached.")

  `delete`(key)

    Remove a value from the cache.

    This operation is idempotent (can be called multiple times, or on a non-existent key, safely)

  `get`(key)

    Return a value from the cache, based on the given key.

    While it’s typical the key is a string, it’s passed through to the underlying backend so can be of any type recognized by the backend. If the value is not present, returns the token `NO_VALUE`. `NO_VALUE` evaluates to False, but is separate from `None` to distinguish between a cached value of `None`. Note that the `expiration_time` argument is not used here - this method is a direct line to the backend’s behavior.

  `get_or_create`(key, creator, expiration_time=None)

    Similar to `get`, will use the given “creation” function to create a new value if the value does not exist.

    This will use the underlying dogpile/ expiration mechanism to determine when/how the creation function is called.

    Parameters: | key – Key to retrieve creator – function which creates a new value. expiration_time – optional expiration time which will overide the expiration time already configured on this `CacheRegion` if not None. To set no expiration, use the value -1.

  `set`(key, value)

    Place a new value in the cache under the given key.

`dogpile.cache.region.` `make_region`(*arg, **kw)

  Instantiate a new `CacheRegion`.

  Currently, `make_region()` is a passthrough to `CacheRegion`. See that class for constructor arguments.

`dogpile.cache.region.` `value_version` = 1

  An integer placed in the `CachedValue` so that new versions of dogpile.cache can detect cached values from a previous, backwards-incompatible version.

`dogpile.cache.util.` `function_key_generator`(namespace, fn)

  Return a function that generates a string key, based on a given function as well as arguments to the returned function itself.

  This is used by `CacheRegion.cache_on_arguments()` to generate a cache key from a decorated function.

  It can be replaced using the `function_key_generator` argument passed to `make_region()`.

## Backend API

See the section Creating Backends for details on how to register new backends.

class `dogpile.cache.api.` `CacheBackend`(arguments)

  Base class for backend implementations.

  `delete`(key)

    Delete a value from the cache.

    The key will be whatever was passed to the registry, processed by the “key mangling” function, if any.

    The behavior here should be idempotent, that is, can be called any number of times regardless of whether or not the key exists.

  `get`(key)

    Retrieve a value from the cache.

    The returned value should be an instance of `CachedValue`, or `NO_VALUE` if not present.

  `get_mutex`(key)

    Return an optional mutexing object for the given key.

    This object need only provide an `acquire()` and `release()` method.

    May return `None`, in which case the dogpile lock will use a regular `threading.Lock` object to mutex concurrent threads for value creation. The default implementation returns `None`.

    Different backends may want to provide various kinds of “mutex” objects, such as those which link to lock files, distributed mutexes, memcached semaphores, etc. Whatever kind of system is best suited for the scope and behavior of the caching backend.

    A mutex that takes the key into account will allow multiple regenerate operations across keys to proceed simultaneously, while a mutex that does not will serialize regenerate operations to just one at a time across all keys in the region. The latter approach, or a variant that involves a modulus of the given key’s hash value, can be used as a means of throttling the total number of value recreation operations that may proceed at one time.

  `key_mangler` = None

    Key mangling function.

    May be None, or otherwise declared as an ordinary instance method.

  `set`(key, value)

    Set a value in the cache.

    The key will be whatever was passed to the registry, processed by the “key mangling” function, if any. The value will always be an instance of `CachedValue`.

class `dogpile.cache.api.` `CachedValue`

  Represent a value stored in the cache.

  `CachedValue` is a two-tuple of `(payload, metadata)`, where `metadata` is dogpile.cache’s tracking information ( currently the creation time). The metadata and tuple structure is pickleable, if the backend requires serialization.

  `metadata`

    Named accessor for the dogpile.cache metadata dictionary.

  `payload`

    Named accessor for the payload.

`dogpile.cache.api.` `NO_VALUE` = <dogpile.cache.api.NoValue object at 0x10173a210>

  Value returned from `get()` that describes a key not present.

class `dogpile.cache.api.` `NoValue`

  Describe a missing cache value.

  The `NO_VALUE` module global should be used.

## Backends

### Memory Backend

Provides a simple dictionary-based backend.

class `dogpile.cache.backends.memory.` `MemoryBackend`(arguments)

  A backend that uses a plain dictionary.

  There is no size management, and values which are placed into the dictionary will remain until explicitly removed. Note that Dogpile’s expiration of items is based on timestamps and does not remove them from the cache.

  E.g.:

      from dogpile.cache import make_region

      region = make_region().configure(
          'dogpile.cache.memory'
      )

  To use a Python dictionary of your choosing, it can be passed in with the `cache_dict` argument:

      my_dictionary = {}
      region = make_region().configure(
          'dogpile.cache.memory',
          arguments={
              "cache_dict":my_dictionary
          }
      )

### Memcached Backends

Provides backends for talking to memcached.

class `dogpile.cache.backends.memcached.` `GenericMemcachedBackend`(arguments)

  Base class for memcached backends.

  This base class accepts a number of paramters common to all backends.

  Parameters: | url – the string URL to connect to. Can be a single string or a list of strings. This is the only argument that’s required. distributed_lock – boolean, when True, will use a memcached-lock as the dogpile lock (see `MemcachedLock`). Use this when multiple processes will be talking to the same memcached instance. When left at False, dogpile will coordinate on a regular threading mutex. memcached_expire_time – integer, when present will be passed as the `time` parameter to `pylibmc.Client.set`. This is used to set the memcached expiry time for a value. Note This parameter is different from Dogpile’s own `expiration_time`, which is the number of seconds after which Dogpile will consider the value to be expired. When Dogpile considers a value to be expired, it continues to use the value until generation of a new value is complete, when using `CacheRegion.get_or_create()`. Therefore, if you are setting `memcached_expire_time`, you’ll want to make sure it is greater than `expiration_time` by at least enough seconds for new values to be generated, else the value won’t be available during a regeneration, forcing all threads to wait for a regeneration each time a value expires.

  The `GenericMemachedBackend` uses a `threading.local()` object to store individual client objects per thread, as most modern memcached clients do not appear to be inherently threadsafe.

  In particular, `threading.local()` has the advantage over pylibmc’s built-in thread pool in that it automatically discards objects associated with a particular thread when that thread ends.

  `client`

    Return the memcached client.

    This uses a threading.local by default as it appears most modern memcached libs aren’t inherently threadsafe.

  `set_arguments` = {}

    Additional arguments which will be passed to the `set()` method.

class `dogpile.cache.backends.memcached.` `MemcachedBackend`(arguments)

  A backend using the standard Python-memcached library.

  Example:

      from dogpile.cache import make_region

      region = make_region().configure(
          'dogpile.cache.memcached',
          expiration_time = 3600,
          arguments = {
              'url':"127.0.0.1:11211"
          }
      )

class `dogpile.cache.backends.memcached.` `PylibmcBackend`(arguments)

  A backend for the pylibmc memcached client.

  A configuration illustrating several of the optional arguments described in the pylibmc documentation:

      from dogpile.cache import make_region

      region = make_region().configure(
          'dogpile.cache.pylibmc',
          expiration_time = 3600,
          arguments = {
              'url':["127.0.0.1"],
              'binary':True,
              'behaviors':{"tcp_nodelay": True,"ketama":True}
          }
      )

  Arguments accepted here include those of `GenericMemcachedBackend`, as well as those below.

  Parameters: | binary – sets the `binary` flag understood by `pylibmc.Client`. behaviors – a dictionary which will be passed to `pylibmc.Client` as the `behaviors` parameter. min_compres_len – Integer, will be passed as the `min_compress_len` parameter to the `pylibmc.Client.set` method.

class `dogpile.cache.backends.memcached.` `BMemcachedBackend`(arguments)

  A backend for the python-binary-memcached memcached client.

  This is a pure Python memcached client which includes the ability to authenticate with a memcached server using SASL.

  A typical configuration using username/password:

      from dogpile.cache import make_region

      region = make_region().configure(
          'dogpile.cache.bmemcached',
          expiration_time = 3600,
          arguments = {
              'url':["127.0.0.1"],
              'username':'scott',
              'password':'tiger'
          }
      )

  Arguments which can be passed to the `arguments` dictionary include:

  Parameters: | username – optional username, will be used for SASL authentication. password – optional password, will be used for SASL authentication.

class `dogpile.cache.backends.memcached.` `MemcachedLock`(client_fn, key)

  Simple distributed lock using memcached.

  This is an adaptation of the lock featured at http://amix.dk/blog/post/19386

### File Backends

Provides backends that deal with local filesystem access.

class `dogpile.cache.backends.file.` `DBMBackend`(arguments)

  A file-backend using a dbm file to store keys.

  Basic usage:

      from dogpile.cache import make_region

      region = make_region().configure(
          'dogpile.cache.dbm',
          expiration_time = 3600,
          arguments = {
              "filename":"/path/to/cachefile.dbm"
          }
      )

  DBM access is provided using the Python `anydbm` module, which selects a platform-specific dbm module to use. This may be made to be more configurable in a future release.

  Note that different dbm modules have different behaviors. Some dbm implementations handle their own locking, while others don’t. The `DBMBackend` uses a read/write lockfile by default, which is compatible even with those DBM implementations for which this is unnecessary, though the behavior can be disabled.

  The DBM backend by default makes use of two lockfiles. One is in order to protect the DBM file itself from concurrent writes, the other is to coordinate value creation (i.e. the dogpile lock). By default, these lockfiles use the `flock()` system call for locking; this is only available on Unix platforms.

  Currently, the dogpile lock is against the entire DBM file, not per key. This means there can only be one “creator” job running at a time per dbm file.

  A future improvement might be to have the dogpile lock using a filename that’s based on a modulus of the key. Locking on a filename that uniquely corresponds to the key is problematic, since it’s not generally safe to delete lockfiles as the application runs, implying an unlimited number of key-based files would need to be created and never deleted.

  Parameters to the `arguments` dictionary are below.

  Parameters: | filename – path of the filename in which to create the DBM file. Note that some dbm backends will change this name to have additional suffixes. rw_lockfile – the name of the file to use for read/write locking. If omitted, a default name is used by appending the suffix ”.rw.lock” to the DBM filename. If False, then no lock is used. dogpile_lockfile – the name of the file to use for value creation, i.e. the dogpile lock. If omitted, a default name is used by appending the suffix ”.dogpile.lock” to the DBM filename. If False, then dogpile.cache uses the default dogpile lock, a plain thread-based mutex.

class `dogpile.cache.backends.file.` `FileLock`(filename)

  Use lockfiles to coordinate read/write access to a file.

## Plugins

### Mako Integration

dogpile.cache includes a Mako plugin that replaces Beaker as the cache backend. Setup a Mako template lookup using the “dogpile.cache” cache implementation and a region dictionary:

    from dogpile.cache import make_region
    from mako.lookup import TemplateLookup

    my_regions = {
        "local":make_region(
                    "dogpile.cache.dbm",
                    expiration_time=360,
                    arguments={"filename":"file.dbm"}
                ),
        "memcached":make_region(
                    "dogpile.cache.pylibmc",
                    expiration_time=3600,
                    arguments={"url":["127.0.0.1"]}
                )
    }

    mako_lookup = TemplateLookup(
        directories=["/myapp/templates"],
        cache_impl="dogpile.cache",
        cache_args={
            'regions':my_regions
        }
    )

To use the above configuration in a template, use the `cached=True` argument on any Mako tag which accepts it, in conjunction with the name of the desired region as the `cache_region` argument:

    <%def name="mysection()" cached=True cache_region="memcached">
        some content that's cached
    </%def>

class `dogpile.cache.plugins.mako_cache.` `MakoPlugin`(cache)

  A Mako `CacheImpl` which talks to dogpile.cache.

## Utilities

`dogpile.cache.util.` `function_key_generator`(namespace, fn)

  Return a function that generates a string key, based on a given function as well as arguments to the returned function itself.

  This is used by `CacheRegion.cache_on_arguments()` to generate a cache key from a decorated function.

  It can be replaced using the `function_key_generator` argument passed to `make_region()`.

`dogpile.cache.util.` `sha1_mangle_key`(key)

  a SHA1 key mangler.

`dogpile.cache.util.` `length_conditional_mangler`(length, mangler)

  a key mangler that mangles if the length of the key is past a certain threshold.

### Navigation

- index
- modules |
- previous |
- dogpile.cache 0.2.1 documentation »