Metadata-Version: 2.1
Name: qaviton-io
Version: 2019.11.3.17.39.58.419954
Summary: qaviton io
Home-page: https://github.com/yehonadav/qaviton_io
Author: yehonadav
Author-email: qaviton@gmail.com
License: apache-2.0
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3.7
Description-Content-Type: text/markdown
Requires-Dist: qaviton-processes (>=2019.11.1.13.50.43.917059)

# Qaviton IO  
![logo](https://www.qaviton.com/wp-content/uploads/logo-svg.svg)  
[![version](https://img.shields.io/pypi/v/qaviton_io.svg)](https://pypi.python.org/pypi)
[![license](https://img.shields.io/pypi/l/qaviton_io.svg)](https://pypi.python.org/pypi)
[![open issues](https://img.shields.io/github/issues/yehonadav/qaviton_io)](https://github/issues-raw/yehonadav/qaviton_io)
[![downloads](https://img.shields.io/pypi/dm/qaviton_io.svg)](https://pypi.python.org/pypi)
![code size](https://img.shields.io/github/languages/code-size/yehonadav/qaviton_io)
-------------------------  

Qaviton IO  
is a package with a simple API, making use of python's async & multiprocessing  
to enable fast execution of many asyncable operations.  

## Installation  
```sh  
pip install qaviton-io -U
```  

### Requirements  
- Python 3.6+  

## Features  
* async task manager  
* process task manager  
* task logger  

## Usage  

#### async manager:  

```python
from time import time
from typing import List
from requests import get, Response  # lets make use of requests to make async http calls
from qaviton_io import AsyncManager

# we can save the responses here
rs: List[Response] = []

# let's create an async manager
m = AsyncManager()


# first we make a simple function to make an http call.
# we want to log the result,
# and make sure that in case of an exception
# the manager won't stop
@m.log.task(exceptions=Exception)
def task():
    r = get("https://qaviton.com")
    r.raise_for_status()
    rs.append(r)


# this will run async tasks and measure their duration
def run(tasks):
    t = time()
    m.run(tasks)
    t = time() - t
    print(f'took {round(t, 3)}s')


# let's run our task once and see how long it takes
run([task for _ in range(1)])

# now let's run our task 20 times and see how long it takes
run([task for _ in range(20)])

# let's view the results in the log report
m.log.report()
```  

#### process manager:  

```python
from time import time
from requests import get
from qaviton_io.types import Tasks
from qaviton_io import ProcessManager, Log
from traceback import format_exc

# make sure your tasks are defined at the module level,
# so they can be pickled by multiprocessing

# first we need to make a log registry
log = Log()


# now we make some tasks
# this is a nested task
# we don't want to handle any exceptions
# so in case of failure the parent will not proceed
@log.task()
def task(url):
    r = get(url)
    r.raise_for_status()


# this is the prent task
# we want to handle all exceptions
# so in case of failure the next task will execute
@log.task(exceptions=Exception)
def multi_task():
    for url in [
        "https://qaviton.com",
        "https://qaviton.co.il",  # make sure you enter a valid address
        "https://qaviton.com1",  # make sure you enter a valid address
    ]:
        task(url)


# let's create a function to execute tasks
def execute_tasks(tasks: Tasks, timeout):
    manager = ProcessManager()
    t = time()
    try:
        manager.run_until_complete(tasks, timeout=timeout)
        timed_out = None
    except TimeoutError:
        timed_out = format_exc()
    t = time() - t
    manager.log.report()
    print(f'took {round(t, 3)}s\n')
    manager.log.clear()
    return timed_out


# now all that's left is to run the tasks
if __name__ == "__main__":
    timeouts = [
        execute_tasks([multi_task for _ in range(1)], timeout=3),
        execute_tasks([multi_task for _ in range(20)], timeout=6),
        execute_tasks([multi_task for _ in range(80)], timeout=9),
    ]
    for timeout in timeouts:
        if timeout:
            print(timeout)

```  

## notes:  

* for good performance and easy usage  
you should probably stick with using the AsyncManager  

* The ProcessManager uses async operations as well as multi-processing.  
It distributes tasks across cpus, and those tasks are executed using the AsyncManager  
if you want maximum efficiency you should consider using the ProcessManager  

* The ProcessManager uses the multiprocessing module  
and should be treated with it's restrictions & limitations accordingly  

* The ProcessManager gets stuck easily,  
make sure to use timeouts when using it  


