Metadata-Version: 2.1
Name: ftdc-tools
Version: 0.1.0
Summary: Pure python package that provides tools required to decode and process FTDC data.
Home-page: https://github.com/mongodb/ftdc-tools
License: Apache-2.0
Author: Mridul Augustine
Author-email: mridul.augustine@mongodb.com
Requires-Python: >=3.9,<4.0
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.9
Requires-Dist: aiofile (>=3.7.4,<4.0.0)
Requires-Dist: black (>=22.3.0,<23.0.0)
Requires-Dist: flake8 (>=4.0.1,<5.0.0)
Requires-Dist: flake8-annotations (>=2.9.0,<3.0.0)
Requires-Dist: flake8-bandit (>=3.0.0,<4.0.0)
Requires-Dist: flake8-bugbear (>=22.4.25,<23.0.0)
Requires-Dist: flake8-docstrings (>=1.6.0,<2.0.0)
Requires-Dist: flake8-isort (>=4.1.1,<5.0.0)
Requires-Dist: mypy (>=0.960,<0.961)
Requires-Dist: pytest-asyncio (>=0.18.3,<0.19.0)
Requires-Dist: pytest-flake8 (>=1.1.1,<2.0.0)
Requires-Dist: pytest-mypy (>=0.9.1,<0.10.0)
Project-URL: Repository, https://github.com/mongodb/ftdc-tools
Description-Content-Type: text/markdown

# ftdc-tools - Pure Python package for FTDC
This package provides tools required to decode and process FTDC data. So whats unique about this package

* This a pure python package and does not rely on any external binaries for decoding FTDC data.
* This package provides streaming support. With this feature the whole FTDC files is not loaded in memory and is capable of processing large FTDC file.
* Async support for faster processing.

## Getting Started - Users
```
pip install ftdc-tools
```

## Usage

### Decoding FTDC file from URL - Streaming approach
```
from ftdc_tools.ftdc_decoder import FTDC
import requests
url = "https://genny-metrics.s3.amazonaws.com/performance_linux_wt_repl_genny_scale_InsertRemove_patch_b2098c676bdc64e3194734fa632b133c47496646_61f955933066150fca890e4a_22_02_01_15_58_36_0/canary_InsertRemove.ActorFinished"

# Streaming FTDC data
response = requests.get(url, stream=True)
for ftdc_row in FTDC(response.raw):
    print(ftdc_row)
```

### Decoding FTDC file from URL - Non-streaming approach
```
from ftdc_tools.ftdc_decoder import FTDC
import requests
url = "https://genny-metrics.s3.amazonaws.com/performance_linux_wt_repl_genny_scale_InsertRemove_patch_b2098c676bdc64e3194734fa632b133c47496646_61f955933066150fca890e4a_22_02_01_15_58_36_0/canary_InsertRemove.ActorFinished"

response = requests.get(url)
for ftdc_row in FTDC(response.content):
    print(ftdc_row)
```

### Decoding FTDC file from URL - Aysnc streaming approach
```
import asyncio
import aiohttp
from ftdc_tools.ftdc_decoder import FTDC
url = "https://genny-metrics.s3.amazonaws.com/performance_linux_wt_repl_genny_scale_InsertRemove_patch_b2098c676bdc64e3194734fa632b133c47496646_61f955933066150fca890e4a_22_02_01_15_58_36_0/canary_InsertRemove.ActorFinished"

async def decode_ftdc():
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            async for x in FTDC(resp.content.iter_chunked(10000)):
                print(x)


asyncio.run(decode_ftdc())
```

### Decoding FTDC file from URL - Using all the available memory.
One of the disadvantages of using a streaming approach is that it's slow compared to in-memory processing.
To optimize memory usage and achieve the best performance for available memory, users can pass the memory option to the FTDC class.
When combined with an async streaming approach, users should achieve the best possible performance with available memory.
```
import asyncio
import aiohttp
from ftdc_tools.ftdc_decoder import FTDC
url = "https://genny-metrics.s3.amazonaws.com/performance_linux_wt_repl_genny_scale_InsertRemove_patch_b2098c676bdc64e3194734fa632b133c47496646_61f955933066150fca890e4a_22_02_01_15_58_36_0/canary_InsertRemove.ActorFinished"

async def decode_ftdc():
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            async for x in FTDC(resp.content.iter_chunked(10000), memory=10000*1000):
                print(x)


asyncio.run(decode_ftdc())
```

## Getting Started - Developers

Getting the code:
```
$ git clone git@github.com:mongodb/ftdc-tools.git
$ cd ftdc-tools
```

Installation

```
$ pip install poetry
$ poetry install
```

Testing/linting:
```
$ poetry run black ftdc_tools tests
$ poetry run isort ftdc_tools tests
$ poetry run pytest
$ poetry run flake8
```

