django-redis documentation
==========================
Andrey Antukh, <niwi@niwi.be>
4.4.2
:toc: left
:numbered:
:source-highlighter: pygments
:pygments-style: friendly


Introduction
------------

_django-redis_ is a xref:license[BSD Licensed], full featured Redis cache/session backend for Django.


Why use django-redis?
~~~~~~~~~~~~~~~~~~~~~

Because:

- In active development.
- Uses native redis-py url notation connection strings.
- Pluggable clients.
- Pluggable parsers.
- Pluggable serializers.
- Master-Slave support in the default client.
- Complete battery of tests.
- Used in production in several projects as cache and session storage.
- Supports infinite timeouts.
- Facilities for raw access to Redis client/connection pool.
- Highly configurable (can emulate memcached exception behavior, for example).
- Unix sockets supported by default.
- With support for python 2.7, 3.3, 3.4 and 3.5



Supported django-redis versions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Supported stable version: *4.4.0*
- Supported stable version: *3.8.4*


How version number is handled
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Versions like _3.6_, _3.7_, ... are considered major releases
and can contain some backward incompatibilities. For more information
is very recommended see the changelog before update.

Versions like _3.7.0_, _3.7.1_, ... are considered minor or bug
fix releases and are should contain only bug fixes. No new features.


Requirements
~~~~~~~~~~~~

Django version support
^^^^^^^^^^^^^^^^^^^^^^

- *django-redis 3.8.x* will maintain support for django 1.4, 1.5, 1.6, 1.7 (and maybe 1.8)
- *django-redis 4.4.0* will maintain support for django 1.6, 1.7, 1.8 and 1.9


Redis Server Support
^^^^^^^^^^^^^^^^^^^^

- *django-redis 3.x.y* will maintain support for redis-server 2.6.x and upper.
- *django-redis 4.x.y* will maintain support for redis-server 2.8.x and upper.


Other requirements
^^^^^^^^^^^^^^^^^^

All supported versions of *django-redis* depends on `redis-py >= 2.10.0`.


User guide
----------

Installation
~~~~~~~~~~~~

The simplest way to use *django-redis* in your project is to install it with *pip*:

[source,text]
----
pip install django-redis
----

Configure as cache backend
~~~~~~~~~~~~~~~~~~~~~~~~~~

To start using *django-redis*, you should change your Django cache settings to something like this:

[source, python]
----
CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://127.0.0.1:6379/1",
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
        }
    }
}
----

django-redis, since 3.8.0, it starts using redis-py native url notation for connection strings,
that allows better interoperability and have a connection string in more "standard" way.

.This is a examples of url format
----
redis://[:password]@localhost:6379/0
rediss://[:password]@localhost:6379/0
unix://[:password]@/path/to/socket.sock?db=0
----

Three URL schemes are supported:

- `redis://`: creates a normal TCP socket connection
- `rediss://`: creates a SSL wrapped TCP socket connection
- `unix://` creates a Unix Domain Socket connection

There are several ways to specify a database number:

- A `db` querystring option, e.g. redis://localhost?db=0
- If using the redis:// scheme, the path argument of the url, e.g. `redis://localhost/0`


NOTE: if you are coming fron django-redis < 3.8.x, you are probably using redis_cache. Since
django-redis 3.8.x, `redis_cache` module is deprecated in favor to `django_redis`. The
`redis_cache` module will be removed in 3.9.x versions.



Configure as session backend
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Django can by default use any cache backend as session backend and you benefit from that by using
*django-redis* as backend for session storage without installing any additional backends:

[source, python]
----
SESSION_ENGINE = "django.contrib.sessions.backends.cache"
SESSION_CACHE_ALIAS = "default"
----


Testing with django-redis
~~~~~~~~~~~~~~~~~~~~~~~~~

`django-redis` supports customizing the underlying Redis client (see <<_pluggable_redis_client>>).
This can be used for testing purposes, e.g., by replacing the default client with `fakeredis` 
(https://github.com/jamesls/fakeredis) or `mockredis` (https://github.com/locationlabs/mockredis).
Doing so allows you to run your integration tests without depending on a real Redis server.

.Example setup using `fakeredis`: 
[source, python]
----
import fakeredis
CACHES = {
    "default": {
        "OPTIONS": {
            "REDIS_CLIENT_CLASS": "fakeredis.FakeStrictRedis",
        }
    }
}
----

In case you want to flush all data from the cache after a test, add the
following lines to your `TestCase`:
[source, python]
----
def tearDown(self):
    from django_redis import get_redis_connection
    get_redis_connection("default").flushall()
----


Advanced usage
--------------

Pickle version
~~~~~~~~~~~~~~

For almost all values, *django-redis* uses pickle to serialize objects.

The latest available version of pickle is used by default. If you want set a concrete version, you
can do it, using `PICKLE_VERSION` option:

[source, python]
----
CACHES = {
    "default": {
        # ...
        "OPTIONS": {
            "PICKLE_VERSION": -1  # Use the latest protocol version
        }
    }
}
----


Socket timeout
~~~~~~~~~~~~~~

Socket timeout can be set using `SOCKET_TIMEOUT` and `SOCKET_CONNECT_TIMEOUT`
options:

[source, python]
----
CACHES = {
    "default": {
        # ...
        "OPTIONS": {
            "SOCKET_CONNECT_TIMEOUT": 5,  # in seconds
            "SOCKET_TIMEOUT": 5,  # in seconds
        }
    }
}
----

`SOCKET_CONNECT_TIMEOUT` is the timeout for the connection to be established and
`SOCKET_TIMEOUT` is the timeout for read and write operations after the connection
is established.


Compression support
~~~~~~~~~~~~~~~~~~~

_django_redis_ comes with compression support out of the box, but is deactivated by default. 
You can activate it setting up a concrete backend:


[source, python]
----
CACHES = {
    "default": {
        # ...
        "OPTIONS": {
            "COMPRESSOR": "django_redis.compressor.zlib.ZlibCompressor",
        }
    }
}
----

Let see an example, of how make it work with *lzma* compression format:

[source, python]
----
import lzma

CACHES = {
    "default": {
        # ...
        "OPTIONS": {
            "COMPRESSOR": "django_redis.compressor.lzma.LzmaCompressor",
        }
    }
}
----


Memcached exceptions behavior
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In some situations, when Redis is only used for cache, you do not want exceptions when Redis is down.
This is default behavior in the memcached backend and it can be emulated in *django-redis*.

For setup memcached like behaviour (ignore connection exceptions), you should
set `IGNORE_EXCEPTIONS` settings on your cache configuration:

[source, python]
----
CACHES = {
    "default": {
        # ...
        "OPTIONS": {
            "IGNORE_EXCEPTIONS": True,
        }
    }
}
----

Also, you can apply the same settings to all configured caches, you can set the global flag in
your settings:

[source, python]
----
DJANGO_REDIS_IGNORE_EXCEPTIONS = True
----


Log Ignored Exceptions
~~~~~~~~~~~~~~~~~~~~~~

When ignoring exceptions with `IGNORE_EXCEPTIONS` or `DJANGO_REDIS_IGNORE_EXCEPTIONS`,
you may optionally log exceptions using the global variable `DJANGO_REDIS_LOG_IGNORED_EXCEPTIONS`
in your settings file.

[source, python]
----
DJANGO_REDIS_LOG_IGNORED_EXCEPTIONS = True
----

If you wish to specify the logger in which the exceptions are output, simply set the global
variable `DJANGO_REDIS_LOGGER` to the string name and/or path of the desired logger. This will
default to `__name__` if no logger is specified and `DJANGO_REDIS_LOG_IGNORED_EXCEPTIONS` is `True`

[source, python]
----
DJANGO_REDIS_LOGGER = 'some.specified.logger'
----


Infinite timeout
~~~~~~~~~~~~~~~~

*django-redis* comes with infinite timeouts support out of the box. And it behaves in same way
as django backend contract specifies:

- `timeout=0` expires the value immediately.
- `timeout=None` infinite timeout

[source, python]
----
cache.set("key", "value", timeout=None)
----


Get ttl (time-to-live) from key
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

With redis, you can access to ttl of any stored key, for it, django-redis exposes `ttl` function.

It returns:

- ttl value for any volatile key (any key that has expiration)
- 0 for expired and not existent keys
- None for keys that does not have expiration

.Simple search keys by pattern
[source,pycon]
----
>>> from django.core.cache import cache
>>> cache.set("foo", "value", timeout=25)
>>> cache.ttl("foo")
25
>>> cache.ttl("not-existent")
0
----


Expire & Persist
~~~~~~~~~~~~~~~~

Additionally to the simple ttl query, you can send persist a concrete key or specify
a new expiration timeout using the `persist` and `expire` methods:

.Example using `persist` method
[source, pycon]
-----
>>> cache.set("foo", "bar", timeout=22)
>>> cache.ttl("foo")
22
>>> cache.persist("foo")
>>> cache.ttl("foo")
None
-----

.Example using expire method
[source,pycon]
----
>>> cache.set("foo", "bar", timeout=22)
>>> cache.expire("foo", timeout=5)
>>> cache.ttl("foo")
5
----


Locks
~~~~~

It also supports the redis ability to create redis distributed named locks. The Lock
interface is identical to the `threading.Lock` so you can use it as replacement.

.Example allocating a lock using python context managers facilities.
[source, python]
----
with cache.lock("somekey"):
    do_some_thing()
----


Scan & Delete keys in bulk
~~~~~~~~~~~~~~~~~~~~~~~~~~

*django-redis* comes with some additional methods that help with searching or deleting keys
using glob patterns.

.Simple search keys by pattern
[source,pycon]
----
>>> from django.core.cache import cache
>>> cache.keys("foo_*")
["foo_1", "foo_2"]
----

A simple search like this will return all matched values. In databases with a large number of keys
this isn't suitable method. Instead, you can use the `iter_keys` function that works like the `keys`
function but uses redis>=2.8 server side cursors. Calling `iter_keys` will return a generator that
you can then iterate over efficiently.

.Search using server side cursors
[source,pycon]
----
>>> from django.core.cache import cache
>>> cache.iter_keys("foo_*")
<generator object algo at 0x7ffa9c2713a8>
>>> next(cache.iter_keys("foo_*"))
"foo_1"
----

For deleting keys, you should use `delete_pattern` which has the same glob pattern syntax
as the `keys` function and returns the number of deleted keys.

.Example use of delete_pattern
[source, pycon]
----
>>> from django.core.cache import cache
>>> cache.delete_pattern("foo_*")
----


Redis native commands
~~~~~~~~~~~~~~~~~~~~~

*django-redis* has limited support for some Redis atomic operations, such as the commands `SETNX`
 and `INCR`.

You can use the `SETNX` command through the backend `set()` method with the `nx` parameter:

.Example:
[source, pycon]
----
>>> from django.core.cache import cache
>>> cache.set("key", "value1", nx=True)
True
>>> cache.set("key", "value2", nx=True)
False
>>> cache.get("key")
"value1"
----

Also, `incr` and `decr` methods uses redis atomic operations when value that contains a key is suitable
for it.


Raw client access
~~~~~~~~~~~~~~~~~

In some situations your application requires access to a raw Redis client to use some advanced
features that aren't exposed by the Django cache interface. To avoid storing another setting for
creating a raw connection, *django-redis* exposes functions with which you can obtain a raw client
reusing the cache connection string: `get_redis_connection(alias)`.

[source, pycon]
----
>>> from django_redis import get_redis_connection
>>> con = get_redis_connection("default")
>>> con
<redis.client.StrictRedis object at 0x2dc4510>
----

WARNING: Not all pluggable clients support this feature.


Connection pools
~~~~~~~~~~~~~~~~

Behind the scenes, *django-redis* uses the underlying *redis-py* connection pool implementation,
and exposes a simple way to configure it. Alternatively, you can directly customize a
connection/connection pool creation for a backend.

The default *redis-py* behavior is to not close connections, recycling them when possible.


Configure default connection pool
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The default connection pool is simple. You can only customize the maximum number of connections
in the pool, by setting `CONNECTION_POOL_KWARGS` in the `CACHES` setting:

[source, python]
----
CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        ...
        "OPTIONS": {
            "CONNECTION_POOL_KWARGS": {"max_connections": 100}
        }
    }
}

----

You can verify how many connections the pool has opened with the following snippet:

[source, python]
----
from django.core.cache import get_cache
from django_redis import get_redis_connection

r = get_redis_connection("default")  # Use the name you have defined for Redis in settings.CACHES
connection_pool = r.connection_pool
print("Created connections so far: %d" % connection_pool._created_connections)
----

Use your own connection pool subclass
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Sometimes you want to use your own subclass of the connection pool. This is possible with
*django-redis* using the `CONNECTION_POOL_CLASS` parameter in the backend options.

._myproj/mypool.py_
[source, python]
----
from redis.connection import ConnectionPool

class MyOwnPool(ConnectionPool):
    # Just doing nothing, only for example purpose
    pass
----

._settings.py_
[source, python]
----
# Omitting all backend declaration boilerplate code.

"OPTIONS": {
    "CONNECTION_POOL_CLASS": "myproj.mypool.MyOwnPool",
}
----

Customize connection factory
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

If none of the previous methods satisfies you, you can get in the middle of the
*django-redis* connection factory process and customize or completely rewrite it.

By default, *django-redis* creates connections through the `django_redis.pool.ConnectionFactory`
class that is specified in the global Django setting `DJANGO_REDIS_CONNECTION_FACTORY`.

.Partial interface of `ConnectionFactory` class
[source, python]
----
# Note: Using Python 3 notation for code documentation ;)

class ConnectionFactory(object):
    def get_connection_pool(self, params:dict):
        # Given connection parameters in the `params` argument,
        # return new connection pool.
        # It should be overwritten if you want do something
        # before/after creating the connection pool, or return your
        # own connection pool.
        pass

    def get_connection(self, params:dict):
        # Given connection parameters in the `params` argument,
        # return a new connection.
        # It should be overwritten if you want to do something
        # before/after creating a new connection.
        # The default implementation uses `get_connection_pool`
        # to obtain a pool and create a new connection in the
        # newly obtained pool.
        pass

    def get_or_create_connection_pool(self, params:dict):
        # This is a high layer on top of `get_connection_pool` for
        # implementing a cache of created connection pools.
        # It should be overwritten if you want change the default
        # behavior.
        pass

    def make_connection_params(self, url:str) -> dict:
        # The responsibility of this method is to convert basic connection
        # parameters and other settings to fully connection pool ready
        # connection parameters.
        pass

    def connect(self, url:str):
        # This is really a public API and entry point for this
        # factory class. This encapsulates the main logic of creating
        # the previously mentioned `params` using `make_connection_params`
        # and creating a new connection using the `get_connection` method.
        pass
----


Pluggable parsers
~~~~~~~~~~~~~~~~~

*redis-py* (the Python Redis client used by *django-redis*) comes with a pure Python Redis parser
that works very well for most common task, but if you want some performance boost, you can use
*hiredis*.

*hiredis* is a Redis client written in C and it has its own parser that can be used with *django-redis*.

[source, python]
----
"OPTIONS": {
    "PARSER_CLASS": "redis.connection.HiredisParser",
}
----


Pluggable clients
~~~~~~~~~~~~~~~~~

_django_redis_ is designed for to be very flexible and very configurable. For it, it exposes a
pluggable backends that make easy extend the default behavior, and it comes with few ones
out the box.

Default client
^^^^^^^^^^^^^^

Almost all about the default client is explained, with one exception: the default client comes
with master-slave support.

To connect to master-slave redis setup, you should change the `LOCATION` to something like this:

[source, python]
----
"LOCATION": [
    "redis://127.0.0.1:6379/1",
    "redis://127.0.0.1:6378/1",
]
----

The first connection string represents a master server and the rest to slave servers.

WARNING: Master-Slave setup is not heavily tested in production environments.


Shard client
^^^^^^^^^^^^

This pluggable client implements client-side sharding. It inherits almost all functionality from
the default client. To use it, change your cache settings to something like this:

[source, python]
----
CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": [
            "redis://127.0.0.1:6379/1",
            "redis://127.0.0.1:6379/2",
        ],
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.ShardClient",
        }
    }
}
----

WARNING: Shard client is still experimental, so be careful when using it in production environments.


Herd client
^^^^^^^^^^^

This pluggable client helps dealing with the thundering herd problem. You can read more about it
on link:http://en.wikipedia.org/wiki/Thundering_herd_problem[Wikipedia].

Like previous pluggable clients, it inherits all functionality from the default client, adding some
additional methods for getting/setting keys.

.Example setup
[source, python]
----
 CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://127.0.0.1:6379/1",
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.HerdClient",
        }
    }
}
----

This client exposes additional settings:

- `CACHE_HERD_TIMEOUT`: Set default herd timeout. (Default value: 60s)


Pluggable serializer
~~~~~~~~~~~~~~~~~~~~

The pluggable clients serialize data before sending it to the
server. By default, _django_redis_ serialize the data using Python
`pickle`. This is very flexible and can handle a large range of object
types.

To serialize using JSON instead, the serializer `JSONSerializer` is
also available.

.Example setup
[source, python]
----
 CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://127.0.0.1:6379/1",
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
            "SERIALIZER": "django_redis.serializers.json.JSONSerializer",
        }
    }
}
----


There's also support for serialization using 'MsgPack' 'http://msgpack.org/'
(that requires the msgpack-python library):

.Example setup
[source, python]
----
 CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://127.0.0.1:6379/1",
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
            "SERIALIZER": "django_redis.serializers.msgpack.MSGPackSerializer",
        }
    }
}
----


Pluggable redis client
~~~~~~~~~~~~~~~~~~~~~~

_django_redis_ uses the Redis client `redis.client.StrictClient` by default.
It is possible to use an alternative client, for example `fakeredis` for
testing purpose (see <<_testing_with_django_redis>>).

You can customize the client used by setting `REDIS_CLIENT_CLASS` in the
`CACHES` setting. Optionally, you can provide arguments to this class
by setting `REDIS_CLIENT_KWARGS`.

.Example setup
[source, python]
----
CACHES = {
    "default": {
        "OPTIONS": {
            "REDIS_CLIENT_CLASS": "my.module.ClientClass",
            "REDIS_CLIENT_KWARGS": {"some_setting": True},
        }
    }
}
----


[[license]]
License
-------

[source,text]
----
Copyright (c) 2011-2015 Andrey Antukh <niwi@niwi.nz>
Copyright (c) 2011 Sean Bleier

All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in the
   documentation and/or other materials provided with the distribution.
3. The name of the author may not be used to endorse or promote products
   derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
----
