Metadata-Version: 1.0
Name: logcollect
Version: 0.9.1
Summary: Helper for collecting logs to ELK stack via RabbitMQ
Home-page: http://github.com/rutube/logcollect/
Author: tumbler
Author-email: zimbler@gmail.com
License: Beerware
Description: # logcollect
        
        Python library for centralized log collecting
        
        Provides simple configuration for collecting python logs to ELK stack via
        RabbitMQ.
        
        Supported message flow is following:
        
        ```
        
        python.logging
              ||
              \/
          logcollect
              ||
              \/
           RabbitMQ  
              ||
              \/
           Logstash
              ||
              \/
         ElasticSearch
              ||
              \/
            Kibana
        
        ```
        
        ## Mechanics
        
        ### Native logging
        
        `logcollect.boot.default_config` ensures that root logger has correctly
        configured amqp handler.
        
        ### Django
        `logcollect.boot.django_dict_config` modifies `django.conf.settings.LOGGING`
        to ensure correct amqp handler for root logger.
        It should be called in settings module after LOGGING definition.
        
        ### Celery
        
        `logcollect.boot.celery_config` adds signal handler for `worker_process_init`
        signal, and after that adds amqp handler to `task_logger` base handler.
        If necessary, root logger can be also attached to amqp handler.
        
        
        ## Tips for configuration
        
        ### Logstash
        
        ```ruby
        
        input {
          rabbitmq {
            exchange => "logstash"
            queue => "logstash"
            host => "rabbitmq-host"
            type => "amqp"
            durable => true
            codec => "json"
          }
        }
        output {
          elasticsearch { host => localhost }
          stdout { codec => rubydebug }
        }
        
        ```
        
        ### logcollect
        
        All boot helpers have same parameters:
        
        * broker_uri - celery-style RabbitMQ connection string, i.e.
        `amqp://guest@localhost//vhost`
        * exchange, routing_key - message routing info for RabbitMQ
        * durable - message delivery mode
        * level - handler loglevel
        * activity_identity - dict with "process type info"
        
        ### Activity Identity
        
        Assuming we deployed two projects on same host: "github" and "jenkins".
        Both have web backends and background workers.
        Activity identity helps to identify messages from these workers:
        
        Project |   Worker   | Activity identity
        ------- | ---------- | -----------------
        github  |  backend   | `{"project": "github", "application": "backend"}`
        jenkins | background | `{"project": "jenkins", "application": "background"}`
        
        `loggername` could be used for separating different parts of code within a
        worker. Hostnames and process PIDs are added automatically.
        
        ### Correlation ID
        
        Not supported yet, but idea is marking log messages about same object with ID
        information about this object.
        
        
        ## Examples
        
        ### Native python logging
        
        ```sh
         python test_native/native_logging.py
        
        ```
        
        ### Django
        
        ```sh
         python test_django/manage.py test_log
        
        ```
        
        ### Celery
        
        First, start worker:
        
        ```sh
        celery worker -A test_celery.app.celery
        ```
        
        Then send a task to that worker:
        
        ```sh
        python test_celery/send_task.py
        ```
        
        ## Related works
        
        AMQPHandler and AMQPLogstashFormatter are copied from
        [python-logstash](https://github.com/vklochan/python-logstash).
        
        See also:
        * [RabbitMQ](https://github.com/rabbitmq/rabbitmq-server)
        * [Logstash](https://github.com/elastic/logstash)
        * [Kibana](https://github.com/elastic/kibana)
        * [Django](https://github.com/django/django)
        * [Celery](https://github.com/celery/celery)
        
Platform: UNKNOWN
