Metadata-Version: 1.0
Name: mmstats
Version: 0.6.1
Summary: Stat, metric, and diagnostic publishing and consuming tools
Home-page: https://github.com/schmichael/mmstats
Author: Michael Schurter
Author-email: m@schmichael.com
License: APLv2
Description: =====
        About
        =====
        
        Mmstats is a way to expose and read diagnostic values and metrics for
        applications.
        
        Think of mmstats as /proc for your application and the readers as procps
        utilities.
        
        This project is a Python implementation, but compatible implementations can be
        made in any language (see Goals).
        
        Discuss at https://groups.google.com/group/python-introspection
        
        -----
        Goals
        -----
        
        * Separate publishing/writing from consuming/reading tools
        * Platform/language independent (a Java writer can be read by a Python tool)
        * Predictable performance impact for writers via:
        
          * No locks (1 writer per thread)
          * No syscalls (after instantiation)
          * All in userspace
          * Reading has no impact on writers
        
        * Optional persistent (writer can sync anytime)
        * 1-way (Publish/consume only; mmstats are not management extensions)
        
        =====
        Usage
        =====
        
        ------------
        Requirements
        ------------
        
        CPython 2.6 or 2.7 (Windows is untested)
        
        PyPy (only tested in 1.7, should be faster in 1.8)
        
        -----
        Using
        -----
        
        1. ``python setup.py install``
        2. ``import mmstats``
        3. Create a subclass of mmstats.MmStats like:
        
        ::
        
            class WebStats(mmstats.MmStats):
                status2xx = mmstats.CounterField(label='status.2XX')
                status3xx = mmstats.CounterField(label='status.3XX')
                status4xx = mmstats.CounterField(label='status.4XX')
                status5xx = mmstats.CounterField(label='status.5XX')
                last_hit = mmstats.DoubleField(label='timers.last_hit')
        
        4. Instantiate it once per thread/process:
        
        ::
        
            webstats = WebStats(label_prefix='web.stats.')
        
        5. Record some data:
        
        ::
        
            if response.status_code == 200:
                webstats.status2xx.inc()
        
            webstats.last_hit = time.time()
        
        6. Run ``slurpstats`` to read it
        7. Run ``mmash`` to create a web interface for stats
        8. Run ``pollstats -p web.stats.status 2XX,3XX,4XX,5XX /tmp/mmstats-*`` for a
           vmstat/dstat like view.
        9. Did a process die unexpectedly and leave around a stale mmstat file?
           ``cleanstats /path/to/mmstat/files`` will check to see which files are stale
           and remove them.
        
        -----------
        Development
        -----------
        
        .. image:: https://secure.travis-ci.org/schmichael/mmstats.png?branch=master
           :target: http://travis-ci.org/schmichael/mmstats/
        
        It's easiest to develop mmstats within a virtualenv:
        
        ::
        
            $ git clone git://github.com/schmichael/mmstats.git
            $ cd mmstats
            $ virtualenv .
            $ source bin/activate
            $ python setup.py develop
            $ ./run_flask_example # This starts up a sample web app
            $ curl http://localhost:5001/
            $ curl http://localhost:5001/500
            $ curl http://localhost:5001/status
            $ # If you have ab installed:
            $ ab -n 50 -c 10 http://localhost:5001/
        
        Now to view the stats run the following in a new terminal:
        
        ::
        
            $ # To get a raw view of the data:
            $ slurpstats mmstats-*
            $ # Or start up the web interface:
            $ mmash
            $ # Run pollstats while ab is running:
            $ pollstats -p flask.example. ok,bad,working mmstats-*
        
        To cleanup stray mmstats files: ``rm mmstats-flask-*``
        
        The web interface will automatically reload when you change source files.
        
        Put static files into static/ and template files into templates/
        
        TODO: Factor mmash out into it's own project (with a meaningful name?)
        
        --------
        Testing
        --------
        
        Feel free to use your favorite test runner like `nose
        <http://readthedocs.org/docs/nose/>`_ or `pytest <http://pytest.org/>`_ or just
        run:
        
        ::
        
            $ python setup.py test
        
        ===============
        Data Structures
        ===============
        
        There are two types of data structures so far in mmstats:
        
        #. buffered
        #. unbuffered
        
        Buffered structures use multiple buffers for handling values which cannot be
        written atomically.
        
        Unbuffered structures have ff in the write buffer field.
        
        --------
        Buffered
        --------
        
        +----------------+------------+---------------+------------+------------------+-------------+
        | ``label size`` | ``label``  | ``type size`` | ``type``   | ``write buffer`` | ``buffers`` |
        +================+============+===============+============+==================+=============+
        | ``ushort``     | ``char[]`` | ``ushort``    | ``char[]`` | ``byte``         | ``varies``  |
        +----------------+------------+---------------+------------+------------------+-------------+
        
        The buffers field length = sizeof(type) * buffers.
        
        The current write buffer is referenced by: write_buffer * sizeof(type)
        
        TODO: field for total number of buffers?
        
        ----------
        Unbuffered
        ----------
        
        
        +----------------+------------+---------------+------------+------------------+-------------+
        | ``label size`` | ``label``  | ``type size`` | ``type``   | ``write buffer`` | ``value``   |
        +================+============+===============+============+==================+=============+
        | ``ushort``     | ``char[]`` | ``ushort``    | ``char[]`` | ``ff``           | ``varies``  |
        +----------------+------------+---------------+------------+------------------+-------------+
        
        The value field length = sizeof(type).
        
        
        =======
        History
        =======
        
        Version 0.6.1 "MANIFEST.out" released 2012-03-08
        
        * Fix packaging issue
        
        Version 0.6.0 "PyCon 2012" released 2012-03-08
        
        * [API CHANGE] - MovingAverageField's kwarg changed from window_size => size
        * Refactored __init__.py into fields, models, and default (and imported public
          bits into __init__)
        * Added TimerField (MovingAverageField + context manager)
        * Added docs (don't get too excited, just a start)
        
        Version 0.5.0 "100% More Average" released 2012-02-25
        
        * [API CHANGE] - RunningAverage field is now AverageField
        * Added MovingAverageField with window_size=100 parameter
        * Tests can now be run via "python setup.py test"
        
        Version 0.4.1 "Derpstats" released 2012-01-31
        
        * Fixed pollstats
        * Updated README slightly
        
        Version 0.4.0 "On the Road to Pycon" released 2012-01-17
        
        * Added clean module and cleanstats script to clean stale mmstat files
        * Added path kwarg to MmStats class to allow easy path overriding
        * Added StringField for UTF-8 encoded strings
        * Added StaticFloatField & StaticDoubleField
        * Added created UNIX timestamp (sys.created) to default MmStats class
        * Moved all modules into mmstats package
        * Fixed mmash template packaging
        * Fixed test mmstat file cleanup
        * Refactored reading code into mmstats.reader module
        
        Version 0.3.12 "Meow" released 2011-11-29
        
        * Use ctypes.get_errno() instead of Linux specific wrapper
        
        Version 0.3.11 "Rawr" released 2011-11-29
        
        * Fix libc loading on OSX
        
        Version 0.3.10 "π²" released 2011-11-28
        
        * PyPy support (switched from ctypes._CData.from_buffer to .from_address)
        * Multiple calls to MmStats().remove() no longer error (makes testing easier)
        
        Version 0.3.9 "MLIT" released 1970-01-01
        
        * Mistag of 0.3.8
        
        Version 0.3.8 "Hapiness" released 2011-11-20
        
        * Allow filename templating with %PID% and %TID% placeholders
        * Allow setting filename template via MMSTATS_FILES environment variable
        * Improved docs slightly
        * Fixed Ctrl-Cing run_flask_example script
        * Fixed 64 bit integer fields on 32 bit platforms
        * Fixed StaticInt64Field (was actually a uint64 field before)
        * Fixed slurpstats debug output (always showed first 40 bytes of file)
        * Strip newlines from org.python.version
        
        Version 0.3.7 "Depressive Realism is for Winners" released 2011-11-17
        
        * Add pollstats utility (similar to dstat/vmstat)
        * Cleanup development/testing section of the README
        * Slight improvements to basic flask integration example
        
        Version 0.3.6 "The M is for Mongo" released 2011-11-09
        
        * Allow setting the value of CounterFields
        
        Version 0.3.5 "Ornery Orangutan" released 2011-10-20
        
        * Added a running average field
        * Made mmash more configurable and added a console entry point
        * Updated TODO
        
        
        ====
        TODO
        ====
        
        * Add API to dynamically add fields to MmStat classes
        * Percentiles
        * Time based windows for moving averages (eg last 60 seconds)
        * Multiple exposed fields (average, mean, and percentiles from 1 model field)
        * Add alternative procedural writer API (vs existing declarative models)
        * Test severity of race conditions (especially: byte value indicating write
          buffer)
        * Test performance
        * Vary filename based on class name
        
        ==============
        Scrapped Ideas
        ==============
        
        ---------------------------------------------------------------
        Compounds Fields where 1 Writer Field = Many Mmap/Reader Fields
        ---------------------------------------------------------------
        
        This seemed like a honking great idea at first. Compound fields would look just
        like a mini-MmStat model:
        
        ::
        
            class SamplingCounterField(CompoundField):
                """Records increments per ms every N increments"""
                counter = CounterField()
                per_ms = UInt64Field()
        
                class _Counter(object):
                    """Implement counter/rate-sampling logic here"""
        
                def __get__(self, inst, owner):
                    if inst is None:
                        return self
                    return inst._fields[self.key]._counter_instance
        
        The blocker is that there's no way to atomically update all of the compound
        fields. The only way to accomplish this is for compound fields to appear as a
        single double buffered field with each component field as a type in the type
        signature:
        
        ::
        
            class SamplingCounterField(DoubleBufferedField):
                initial = (
                    CounterField.initial,
                    UInt64Field.initial,
                )
                buffer_type = (
                    CounterField.buffer_type,
                    UInt64Field.buffer_type,
                )
                type_signature = (
                    CounterField.type_signature + UInt64Field.type_signature
                )
        
        Obviously an actual implementation should remove the redundant references to
        the component types.
        
        *Note:* Lack of atomicity is not a blocker for exposing fields such as Mean,
        Median, and Percentiles.
        
        *Solution:* Future versions of the mmstats format should support structs as
        values instead of just scalars so that a single write buffer offset can point
        to multiple values.
        
        ------------------------
        Metadata metaprogramming
        ------------------------
        
        To get around having to dynamically creating the structs due to a variable
        label size, put the labels in a header index with a pointer to the actual
        struct field.
        
        -------------
        Page Flipping
        -------------
        
        Store metadata seperate from values. Then store values in multiple pages and
        flip between pages for read/write buffering.
        
Platform: UNKNOWN
Classifier: License :: OSI Approved :: Apache Software License
