Metadata-Version: 2.1
Name: pipeline-telemetry
Version: 1.0.1
Summary: Measure your data pipelines with easy to use telemetry logic
Home-page: https://github.com/MaartendeRuyter/pipeline-telemetry
Author: Maarten de Ruyter
Author-email: "Maarten de Ruyter" <maarten@geodatagarden.com>
License: GNU
Classifier: License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: error-manager
Requires-Dist: mongoengine
Requires-Dist: jmespath

# Pipeline Telemetry

**Pipeline Telemetry** makes it easy for project to generate store telemetry data from within your data pipelines.

```python

    from counters import INVALID_DATA, PROCESSED_DATA_POINTS


    def my_data_pipeline():
        telemetry = Telemetry(**TELEMETRY_PARAMS)

        from url in data_urls:
            data = get_data_from_url(url)
            telemetry.add_telemetry_counter(telemetry_counter=URL_RETRIEVALS)    # increase counter for retrieved URLS
            if data.invalid:
                telemetry.add_telemetry_counter(telemetry_counter=INVALID_DATA)  # increase counter for failed retrievals
                continue
            
            processed_data_points = process_and_store_data(data)
            telemetry.add_telemetry_counter(                                     # increase counter number of datapoints retrieved
                telemetry_counter=PROCESSED_DATA_POINTS,
                increment=len(processed_data_points))  
        
        telemetry.save_and_close()
```

In this example data is retrieved for a list of urls. Total retrievals, insuccesfull retrievals as well as number of processed datapoints are stored in the telemetry object.
This allows you to closely monitor the quality and behavior of your datapipeline. For example all retrievals might be OK but if the number of processed datapoints suddenly drops there might be an issue with some the endpoints.



[![Supported Versions](https://img.shields.io/pypi/pyversions/pipeline-telemetry.svg)](https://pypi.org/project/pipeline-telemetry)


Changelog
=========
1.0.0 (2024-05-08)
-------------------
* Added python 3.12 support

0.9.4 (2022-11-03)
-------------------
* Added ``PartialToSingleMongoAggregator`` and ``PartialToSingleAggregator``
  classes to allow automated aggregation of multiple partial telemetry objects to a single telemetry object.


0.9.3 (2022-10-26)
-------------------
* Added support for python 3.11


0.9.2 (2022-10-11)
-------------------
* Added automatic deletion of old aggegration telemetry


0.9.1 (2022-10-04)
-------------------
* Fixed defect in MongoStorage.


0.9.0 (2022-10-03)
-------------------
* Added ``DailyMongoAggregator`` and ``DailyAggregator`` classes to allow
  automated aggregation of multiple intraday telemetry objects to daily
  telemetry objects.


0.4.5 (2022-06-09)
-------------------
* Added ``set_telemetry_source_name`` method to ``TelemetryMixin`` class


0.4.4 (2022-06-08)
-------------------
* Added custom __hash__ method to ``TelemetryCounter``


0.4.3 (2022-06-01)
-------------------
* Added ``set_increment`` method to ``TelemetryCounter`` class.


0.4.2 (2022-05-17)
-------------------
* Added ``process_telemetry_counters_from_list`` method to ``TelemetryMixin`` class.


0.4.1 (2022-05-16)
-------------------
* Added ``process_telemetry_counters_from_return_value`` method to ``TelemetryMixin`` class.
* Added documentation on TelemetryCounter


0.4.0 (2022-05-15)
-------------------
* Added ``TelemetryMixin`` class to enable easy including of telemetry update
  methods in any datapipeline class.
  

0.3.1 (2022-05-13)
-------------------
* Added ``process_telemetry_counters_in_return_value`` helper method to easily
  add retrieve and process TelemetryCounter objects from a return value object.
* Added ``process_return_value`` helper method to process both ErrorCode and
  TelemetryCounter objects from a return value object.
* Added more documentation


0.3.0 (2022-05-12)
-------------------
* Added ``add_errors_from_return_value`` helper method to easily add the errors
  from a ``ReturnValueWithStatus`` object from ``error-manager`` package.
* Automatically initialize a missing sub_process when a telemetry counter is
  added to the telemetry instance.


0.2.18 (2022-03-2)
-------------------
* Added ``add_single_usage_telemetry`` decorator to be used for single usage
* telemetry objects


0.2.16 (2022-02-01)
-------------------
* Added ``telemetry_type`` and ``io_time_in_seconds`` attributes to Telemetry
* object including a method to increase ``io_time_in_seconds``.


0.2.12 (2022-01-19)
-------------------
* Added ``traffic_light`` attribute to Telemetry object indicating the success
  state of the datapipeline process that is reporting on
* Some minor refactoring


0.2.11 (2022-01-18)
-------------------
* Added ``created_at`` field in mongo storage class for better date selection
* of the telemetry objects


0.2.10 (2022-01-18)
-------------------
* Added indexes to mongo storage class


0.2.8 (2022-01-13)
------------------
* Fix in storage classes to ensure ``category`` and ``sub_category`` to be
  stored in the toplevel of the telemetry object


0.2.7 (2022-01-13)
------------------
* Added fields ``category`` and ``sub_category`` to the telemetry object to
  allow for better distinction between telemetry sources
* Rename telemetry field 'process_name' to 'soure_name' to be more clear about
  the data source in scope of the telemetry object


0.2.6 (2021-12-08)
------------------
* Added default increment value = 1 to methods ``increase_base_count`` and
  ``increase_fail_count``


0.2.5 (2021-11-11)
------------------
* Implemented ``increase_base_count`` and ``increase_fail_count`` method. They
  can be used to make your code more readable when updating the telemetry


0.2.4 (2021-11-11)
------------------
* Implemented ``add_to`` method in ``TelemetryCounter``. Can be used to  add a
  ``TelemetryCounter`` instance to an object with a telemetry instance attached
  to it. This will make the code more readable.


0.2.3 (2021-11-10)
------------------
* Added attribute ``process_types`` to ``TelemetryCounter`` dataclass next to
  existing ``process_type`` attribute. This allows you to choose between a list
  of process_types or just a single process_type to be in scope of the ``TelemetryCounter``` instance


0.2.2 (2021-11-10)
------------------
* added ``add_mongo_telemetry`` decorator that uses mongo storage class


0.2.1 (2021-11-09)
------------------
* added ``is_telemetry_counter`` method in helper module
* moved ``add_telemetry`` decorator to helper module
* both methods can be directly imported from ``pipeline_telemetry`` module


0.2.0 (2021-11-09)
------------------
* added error field to ``TelemetryCounter`` so that default Errorcodes
  can be used in a ``TelemetryCounter`` instance to keep an error counter
  in the telemetry
* Updated add_telemetry_counter so that it can handle TelemetryCounter instances
  with either an Errorcode of a custom counter. In case of ErrorCode the actual
  error_code will serve as the customer counter. 

  
0.1.0 (2021-11-05)
------------------
* added Telemetry method add_telemetry_counter 
* Added ``TelemetryCounter`` class that can be used to make prefedined objects
  that can be added to the Telemetry instance with the add_telemetry_counter
  method. This will improve readibilty of your code::

    from pipeline_telemetry import TelemetryCounter

    YOUR_PREDFINED_COUNTER = (
        process_type=ProcessTypes.CREATE_DATA_FROM_API,
        sub_process='RETRIEVE_RAW_DATA',
        counter_name='my_custom_counter',
    )

    telemetry.add_telemetry_counter(YOUR_PREDFINED_COUNTER)


0.0.4 (2021-10-31)
------------------
* Seperated the storage module
* Added MongoDb storage class
* Started with documentation


0.0.1 (2021-10-06)
------------------
* First release on PyPI.
