persizmq
========

persizmq provides persistence to zeromq. Messages are received in a background and stored on disk before further
manipulation.

Currently, we only support the zeromq subscriber. Adding support for other classes can be easily done; we simply have
not had need for them so far.

Usage
=====
Subscriber
----------
The persistent subscriber wraps a zeromq subscriber. We split up the persistence subscription in two components:
a threaded subscriber that listens on the messages in background, and a persistence component that stores the messages
on disk.

Threaded Subscriber
~~~~~~~~~~~~~~~~~~~
The threaded subscriber is implemented as ``persizmq.ThreadedSubscriber``. You need to specify a callback which is
called upon each received message.

You also need to specify on-exception callback in order to handle exceptions raised in the listening thread.

Example:

.. code-block:: python

    import time

    import zmq

    import persizmq

    context = zmq.Context()
    subscriber = context.socket(zmq.SUB)
    subscriber.setsockopt_string(zmq.SUBSCRIBE, "")
    subscriber.connect("ipc:///some-queue.zeromq")

    def callback(msg: bytes)->None:
        print("received a message: {}".format(msg))

    def on_exception(exception: Exception)->None:
        print("an exception was raised in the listening thread: {}".format(exception))

    with persizmq.ThreadedSubscriber(callback=callback, subscriber=subscriber, on_exception=on_exception):
        # do something while we are listening on messages...
        time.sleep(10)



Storage
~~~~~~~
We provide two storage modes for the received messages:

1. ``persizmq.PersistentStorage``: stores messages in a FIFO queue on disk.
2. ``persizmq.PersistentLatestStorage``: solely stores the newest message on disk.

The storage component is passed directly to the threaded subscriber as a callback.

Example:

.. code-block:: python

    import pathlib

    import zmq

    import persizmq

    context = zmq.Context()
    subscriber = context.socket(zmq.SUB)
    subscriber.setsockopt_string(zmq.SUBSCRIBE, "")
    subscriber.connect("ipc:///some-queue.zeromq")

    persistent_dir = pathlib.Path("/some/dir")
    storage = persizmq.PersistentStorage(persistent_dir=persistent_dir)

    def on_exception(exception: Exception)->None:
        print("an exception was raised in the listening thread: {}".format(exception))

    with persizmq.ThreadedSubscriber(callback=storage.add_message, subscriber=subscriber, on_exception=on_exception):
        msg = storage.front()  # non-blocking
        if msg is not None:
            print("Received a persistent message: {}".format(msg))
            storage.pop_front()

Filtering
~~~~~~~~~
We also provide filtering components which can be chained on the threaded subscriber. The filtering chains are
particularly handy if you want to persist only a small amount of messages and ignore the rest.

The filters are implemented in ``persizmq.filter`` module.

Example:

.. code-block:: python

    import pathlib

    import zmq

    import persizmq
    import persizmq.filter

    context = zmq.Context()
    subscriber = context.socket(zmq.SUB)
    subscriber.setsockopt_string(zmq.SUBSCRIBE, "")
    subscriber.connect("ipc:///some-queue.zeromq")

    persistent_dir = pathlib.Path("/some/dir")
    storage = persizmq.PersistentStorage(persistent_dir=persistent_dir)

    def on_exception(exception: Exception)->None:
        print("an exception was raised in the listening thread: {}".format(exception))

    with persizmq.ThreadedSubscriber(
        lambda msg: storage.add_message(persizmq.filter.MaxSize(max_size=1000)(msg)),
        subscriber=subscriber,
        on_exception=on_exception):

        msg = storage.front()  # non-blocking
        if msg is not None:
            print("Received a persistent message: {}".format(msg))
            storage.pop_front()


Installation
============

* Create a virtual environment:

.. code-block:: bash

    python3 -m venv venv3

* Activate it:

.. code-block:: bash

    source venv3/bin/activate

* Install persizmq with pip:

.. code-block:: bash

    pip3 install persizmq

Development
===========

* Check out the repository.

* In the repository root, create the virtual environment:

.. code-block:: bash

    python3 -m venv venv3

* Activate the virtual environment:

.. code-block:: bash

    source venv3/bin/activate

* Install the development dependencies:

.. code-block:: bash

    pip3 install -e .[dev]

* We use tox for testing and packaging the distribution. Assuming that the virtual environment has been activated and
  the development dependencies have been installed, run:

.. code-block:: bash

    tox

* We also provide a set of pre-commit checks that lint and check code for formatting. Run them locally from an activated
  virtual environment with development dependencies:

.. code-block:: bash

    ./precommit.py

* The pre-commit script can also automatically format the code:

.. code-block:: bash

    ./precommit.py  --overwrite

Versioning
==========
We follow `Semantic Versioning <http://semver.org/spec/v1.0.0.html>`_. The version X.Y.Z indicates:

* X is the major version (backward-incompatible),
* Y is the minor version (backward-compatible), and
* Z is the patch version (backward-compatible bug fix).