Metadata-Version: 1.1
Name: gearhorn
Version: 0.0.2
Summary: A gearman worker which enables efficient broadcast communication
Home-page: http://www.openstack.org/
Author: OpenStack
Author-email: openstack-dev@lists.openstack.org
License: UNKNOWN
Description: ===============================
        gearhorn
        ===============================
        
        
        A gearman worker which enables efficient broadcast communication
        
        * Free software: Apache license
        * Documentation: http://docs.openstack.org/developer/gearhorn
        * Source: http://git.openstack.org/cgit/openstack-infra/gearhorn
        * Bugs: http://bugs.launchpad.net/gearhorn
        
        =============
        Fanout Worker
        =============
        
        Gearman has no in-built way to copy jobs to multiple workers. This worker
        will do that to support the common messaging pattern known as "Fanout"
        
        Once running, the fanout worker will subscribe to the queues
        "register_fanout_subscriber" and "fanout".
        
        subscribe_fanout unsubscribe_fanout
        -----------------------------------
        
        These queues will have a JSON payload consisting of a mapping with two
        keys: topic, client_id. The client_id will be kept as a set under the
        topic for use in sending messages to the appropriate subscriber queue. It
        should be unique for every intended recipient queue for every fanout
        request. Usually this will be unique per host.
        
        fanout
        ------
        
        This queue will have a JSON payload consisting of a mapping with two keys:
        topic, payload. The topic will be used to search for a list of subscriber
        ids. For each subscriber_id found, a copy of the payload will be sent
        to the queue named  topic_subscriber_id. So if we had two subscribers to
        "officememos" with ids "bob" and "alice", then a message to fanout with
        this payload::
        
            {"topic": "officememos",
             "payload": "please go home early today."}
        
        Optionally one can add a 'unique' key to make use of Gearman's
        unique/coalescing features. Also the 'background' key can be used to
        set a truthy value, which will tell gearhorn not to wait for receivers
        before moving on to more messages.
        
        Would result in the worker sending a copy of the payload to the queues
        "officememos_bob" and "officememos_alice".
        
        Matchmaking
        ~~~~~~~~~~~
        
        In order to match up topics with subscribers workers must share a list
        of subscribers for each topic. The mapping is maintained in a backend
        data store. Any time new registration events are seen the list is updated
        in that store and a message is sent to the __matchmaking topic. Workers
        automatically add and remove themselves to/from the __matchmaking list
        in order to ensure they're informed and able to clear cache whenever
        the canonical list is updated. Workers that fail violently must be
        manually removed.
        
        ===========
        Failed Idea
        ===========
        
        The following was idea #1, and fails to be any more efficient than just
        runnint Redis as a pub/sub and direct comm backend. It remains here as
        a reminder not to reimplement Redis.
        
        The expected way to use it is to have a gearman client that wants to
        receive broadcasts request the given broadcast function with a unique ID
        that is the last seen sequence ID. When this daemon receives that work,
        it looks for any items with sequence number greater than this ID, and
        if it finds them, reply with a json payload of::
        
            {"sequence": 2,
             "payload": "foobarbaz"}
        
        Clients would consume this, and then submit a new job with the
        new_seq. This allows a stream of sync jobs to become a stream of
        broadcasted payloads with a good chance of a client receiving all
        sequences.
        
        TODO
        ----
        
        * True functional tests
        
        * Make all the things configurable
        
          * Work queue name(s)
        
          * Tuning backlog and flush frequency
        
        * Consider a work queue that workers can use to share the current sequence
          they should be using.
        
          * Would this be too racy or given the "best effort" is it enough to
            just try hard?
        
          * Do we need a UUID of some kind just to allow clients to detect if
            all the workers went away and sequences reset at 0?
        
        * Add a client helper module which implements the sequence fetching to
          make it easier to write a consumer in python
        
        
Platform: UNKNOWN
Classifier: Environment :: OpenStack
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.4
