======================
Asynchronous Requests
======================

Aquests is generating asynchronous requests and fetching data from HTTP2, REST API, XMLRPC, gRPC and several Database engines. It was seperated from Skitai_ on Jan 2017.

Supported requests are:

- HTTP/1.1
- HTTP/2.0 if target server provides
- Websocket
- XML-RPC
- gRPC
- PostgreSQL
- MongoDB
- Redis

.. _Skitai: https://pypi.python.org/pypi/skitai

.. contents:: Table of Contents


Quickstart
=============

For fetching single web page:

.. code-block:: python

  import aquests
  
  aquests.get ("http://127.0.0.1:5000/")
  aquests.fetchall ()

Its result is:

.. code-block:: bash

  user$ REQID 0. HTTP/2.0 200 OK 4210 bytes received
  
Let's add more pages:

.. code-block:: python
  
  for i in range (3):
    aquests.get ("http://127.0.0.1:5000/")     
  aquests.fetchall ()
  
Its result is:

.. code-block:: bash

  REQID 0. HTTP/2.0 200 OK 4210 bytes received
  REQID 1. HTTP/2.0 200 OK 4210 bytes received
  REQID 2. HTTP/2.0 200 OK 4210 bytes received

Now increase fetching workers,

.. code-block:: python

  aquests.configure (3)  # 3 workers
  for i in range (3):
    aquests.get ("http://127.0.0.1:5000/")     
  aquests.fetchall ()

Result is same as above but somewhat faster and REQID is not ordered.

Now increase workers and pages for making load,

.. code-block:: python
  
  aquests.configure (100) # 100 workers
  for i in range (10000):
    aquests.get ("http://127.0.0.1:5000/")      
  aquests.fetchall ()
  
Now result is,

.. code-block:: bash

  REQID 3635. HTTP/2.0 200 OK 4210 bytes received
  REQID 3627. HTTP/2.0 200 OK 4210 bytes received
  REQID 3594. HTTP/2.0 200 OK 4210 bytes received
  REQID 3702. HTTP/2.0 200 OK 4210 bytes received
  REQID 3685. HTTP/2.0 200 OK 4210 bytes received
  REQID 3637. HTTP/2.0 200 OK 4210 bytes received
  REQID 3591. HTTP/2.0 200 OK 4210 bytes received
  REQID 3586. HTTP/2.0 200 OK 4210 bytes received
  (and scrolled fast...)

Installation
===============

.. code-block:: bash

  pip install aquests


Usage
======

Binding Callback
-------------------

.. code-block:: python
  
  def finish_request (response):
    print (response.status_code)
    print (response.content)
  	
  aquests.configure (workers = 10, callback = finish_request)
  for i in range (10):
    aquests.get ("http://127.0.0.1:5000/")    
  aquests.fetchall ()


Making Traffic Load With Generator Style
------------------------------------------

.. code-block:: python
  
  numreq = 0
  limit = 1000000
  workers = 100
  
  def finish_request (response):
    global numreq, limit     
    if numreq < limit:
      aquests.get ("http://127.0.0.1:5000/")
      numreq += 1
  	
  aquests.configure (workers, callback = finish_request)
  for i in range (workers):
    aquests.get ("http://127.0.0.1:5000/")  
    numreq += 1
  aquests.fetchall ()  


Set/Get Request Meta Information
------------------------------------

.. code-block:: python
  
  def finish_request (response):
    print (response.meta ['req_id'])
    print (response.meta ['req_method'])
    print (response.meta ['job_name'])
  	
  aquests.configure (workers = 10, callback = finish_request)
  aquests.get ("http://127.0.0.1:5000/", meta = {'job_name': 'test1'})  
  aquests.get ("http://127.0.0.1:5000/", meta = {'job_name': 'test2'})

Note: meta ['req_id'] and meta ['req_method'] are automatically added by aquests.


Timeout Setting
----------------

.. code-block:: python
  
  aquests.configure (20, timeout = 10) # 10 seconds
  aquests.get ("https://www.google.co.kr/?gfe_rd=cr&ei=3y14WPCTG4XR8gfSjoK4DQ")  
  aquests.fetchall ()

If timeout occured, response status_code will be 702. Also note above 700 codes mostly indicates network related error.


**Caution**

1. You can't specify timout for each task
2. Cause of aquests' single thread coroutine feature, timeout will not work with exactly timeout seconds.


Mixed Requests
----------------

.. code-block:: python

  dbo = aquests.mongodb ("127.0.0.1:27017", "test_database")
  aquests.configure (20)
  for i in range (1000): 
    aquests.get ("http://127.0.0.1:5000/")
    dbo.findone ("posts", {"author": "James Milton"})
  aquests.fetchall ()


Authorization
-----------------

For requesting with basic/digest authorization:

.. code:: python

  s = was.rpc (url, auth = (username, password))
  rs = s.get_prime_number_gt (10000)
  result = rs.getwait (2)

If you provide both (username, password), aquests try basic/digest authorization. But if just (username,) aquests handle username as bearer token like API Key.

Enabling Cookie
------------------

.. code-block:: python
  
  def finish_request (response):
    print (response.cookies)
    
  aquests.configure (20, callback = finish_request, cookie = True)
  aquests.get ("https://www.google.co.kr/?gfe_rd=cr&ei=3y14WPCTG4XR8gfSjoK4DQ")  
  aquests.fetchall ()

**Caution**

This cookie feature shouldn't handle as different sessions per worker. All workers (connections) of aquests share same cookie values per domain. It means a worker sign in a website, so are the others. Imagine lots of FireFox windows on a desktop computer. If you really need session control, use requests_.


Change Logger
--------------

.. code-block:: python
  
  from aquests.lib import logger
  
  aquests.configure (
    workers = 10, 
    logger = logger.file_logger ('/tmp/logs', 'aquests')
  )


Response
---------

I make similar naming with requests_' attribute and method names as possible.

Response has these attributes and method:

- meta: user added meta data including 'req_id' and 'req_method'
- status_code: HTTP status code or DBO query success (200) or failure (500) code
- reason: status text like OK, Not Found...
- content: bytes content or original db result
- data: usally same as content but on RPC, DB query or json response situation, it returns result object.
- logger: logger.log (msg, type ='info'), logger.trace ()
- method: POST, GET, PUT etc for HTTP/RPC and execute, get, set or lrange etc for DBO
- raise_for_status (): raise exception when HTTP status code >= 400 or DBO command excution failure
- reraise (): shortcut for raise_for_status ()

Below things are available only on DBO responses.

- server: database server address
- dbname: database object name
- params: database command parameters

Below things aren't available on DBO responses.

- url: requested url
- version: HTTP protocol version
- headers: Response headers
- text: charset encoded string (unicode)
- raw: file like object for bytes stream has raw.read (), raw.readline (),... methods
- cookies: if configure (cookie = True), returns dictionary
- encoding: extracted from content-type header
- request.headers
- request.payload: request body bytes, not available at upload and grpc
- json (): load JSON data, but if response content-type is application/json, automatically loaded into response.data then you can just use it.
- get_header (key, default = None): returns header value, if not exists return default
- get_header_with_attr (key, default = None): returns header value and attr dict like 'text/html', {'charset': 'utf-8'}
- set_cookie (key, val, domain = None, path = "/")
- get_cookie (key)

.. _requests: https://pypi.python.org/pypi/requests


Configuration Parameters
==========================

.. code-block:: python

  import aquests
  
  aquests.configure (
    workers = 1, 
    logger = None, 
    callback = None, 
    timeout = 10, 
    cookie = False
  )
  
- workers: number of fetching workers, it'not threads
- logger: logger shoukd have 2 method - log (msg, type = 'info') and trace () for exception logging. if not provided, aquests uses aquests.ib.logger.screen_logger
- callback: function has receiving response arg
- timeout: request timeout seconds
- cookie: enable/disable using cookie for request


List of Methods
==================

GET, DELETE and etc.
---------------------

.. code-block:: python

  aquests.get ("http://127.0.0.1:5000/")
  aquests.delete ("http://127.0.0.1:5000/models/ak-40")
  aquests.get ("https://www.google.co.kr/search?q=aquests")

Also aquests.head (), options () and trace () are available.


POST, PUT
---------------

.. code-block:: python

  aquests.post (
    "http://127.0.0.1:5000/", 
    {'author': 'James Milton'}, 
    {'Content-Type': 'application/x-www-form-urlencoded'}
   )
   
  # is equal to:
   
  aquests.postform (
    "http://127.0.0.1:5000/", 
    {'author': 'James Milton'}    
  )

Put example,

.. code-block:: python
  
  aquest.put (
    "http://127.0.0.1:5000/users/jamesmilton",
    {'fullnamer': 'James Milton'},
    {'Content-Type': 'application/json'}
    )
  )
  
  # is equal to:
   
  aquests.putjson (
    "http://127.0.0.1:5000/users/jamesmilton",
    {'fullnamer': 'James Milton'}
  )
  
There're some shorter ways ratehr than specifing content type:

- postform: application/x-www-form-urlencoded, data value should be dictionary
- postjson: application/json, data value should be json dumpable
- postxml: text/xml, data value should be xml string or utf-8 encoded bytes
- postnvp: text/namevalue, data value should be dictionary 

And putform (), putjson ()... is also available.

  
File Upload
------------

.. code-block:: python

  aquests.upload (
    "http://127.0.0.1:5000/", 
    {
      'author': 'James Milton',
      'file': open ('/tmp/mycar.jpg', 'rb')
    }
  )

You should open file with 'rb' mode.

Websocket
-----------

.. code-block:: python

  aquests.ws ("ws://127.0.0.1:5000/websocket/echo", "Hello World")
  # secure websocket channel, use wss
  aquests.wss ("wss://127.0.0.1:5000/websocket/echo", "Hello World")
  aquests.fetchall ()

If you want to specify message type.

.. code-block:: python
  
  from aquests.protocols.ws import OPCODE_TEXT, OPCODE_BINARY
    
  aquests.ws ("ws://127.0.0.1:5000/websocket/echo", (OPCODE_BINARY, b"Hello World"))
  aquests.fetchall ()
  

XML-RPC
----------

.. code-block:: python

  stub = aquests.rpc ("https://pypi.python.org/pypi")
  stub.package_releases('roundup')
  stub.prelease_urls('roundup', '1.4.10')
  aquests.fetchall ()

Returns,

.. code-block:: bash

  ['1.5.1']
  <class 'xmlrpc.client.Fault'> <Fault 1:...>


gRPC
----------

.. code-block:: python
  
  import route_guide_pb2
  
  stub = aquests.grpc ("http://127.0.0.1:5000/routeguide.RouteGuide")
  point = route_guide_pb2.Point (latitude=409146138, longitude=-746188906)
  for i in range (3):
    stub.GetFeature (point)
  aquests.fetchall ()


Returns,

.. code-block:: python

  name: "Berkshire Valley Management Area Trail, Jefferson, NJ, USA"
  location {
    latitude: 409146138
    longitude: -746188906
  }

For more about gRPC and route_guide_pb2, go to `gRPC Basics - Python`_.

.. _`gRPC Basics - Python`: http://www.grpc.io/docs/tutorials/basic/python.html


PostgreSQL
-------------

.. code-block:: python
  
  def finish_request (response):
    print (response.data)  	
  
  aquests.configure (3, callback = finish_request)	
  dbo = aquests.postgresql ("127.0.0.1:5432", "mydb", ("test", "1111"))
  for i in range (10):
    dbo.execute ("SELECT city, prcp, temp_hi, temp_low FROM weather;")

Returns,

.. code-block:: bash

  [
    {'prcp': 0.25, 'temp_hi': 50, 'city': 'San  Francisco', 'temp_lo': 46}, 
    {'prcp': 0.0, 'temp_hi': 54, 'city': 'Hayward', 'temp_lo': 37}
  ]  

MongoDB
---------

.. code-block:: python

  dbo = aquests.mongodb ("127.0.0.1:27017", "test_database")
  for i in range (3):
    dbo.findone ("posts", {"author": "Steve Newman"})  
    dbo.findall ("posts", {"author": "Hans Roh"})
  aquests.fetchall ()

Returns,

.. code-block:: bash

  {
    'starting_from': 0, 
    'number_returned': 1, 
    'cursor_id': 0, 
    'data': [
      {
        '_id': ObjectId('586a11f80d23915c7ec76f01'), 
        'author': 'Steve Newman', 
        'title': 'How to swim'
      }
    ]
  }
  

**Available Functions**

- find (colname, spec, offset = 0, limit = 1)
- findone (colname, spec): equivalent with find (colname, spec, 0, 1)
- findall (colname, spec): equivalent with find (colname, spec, 0, -1)
- insert (colname, docs, continue_on_error = 0)
- update (colname, spec, doc)
- updateone (colname, spec, doc)
- upsert (colname, spec, doc)
- upsertone (colname, spec, doc)
- delete (colname, spec, flag = 0)
- findkc (colname, spec, offset = 0, limit = 1): after finidhing search, it keeps cursor alive. then you can use 'get_more()'
- get_more (colname, cursor_id, num_to_return): cursor_id can be got from (findkc()'s result).data ["cursor_id"]
- kill_cursors (cursor_ids): if you use findkc() and stop fetching documents, you should mannually call this.

Note: User authorization is not supported yet.


Redis
---------

.. code-block:: python

  dbo = aquests.redis ("127.0.0.1:6379")
  dbo.get ("session-5ae675bc")
  dbo.lrange ("user-saved-docs", 0, 3)
  aquests.fetchall ()

Returns,

.. code-block:: bash
  
  response-of-session-5ae675bc
  
  [32534, 3453, 6786]

Possibly you can use all `Redis commands`_.

.. _`Redis commands`: https://redis.io/commands


Note: User authorization is not supported yet.


SQLite3 For Fast App Prototyping
---------------------------------

Usage is almost same with PostgreSQL. This service IS NOT asynchronous BUT just emulating.

.. code:: python
  
  dbo = aquests.sqlite3 ("sqlite3.db")
  dbo.execute ("""
    drop table if exists people;
    create table people (name_last, age);
    insert into people values ('Cho', 42);
  """)
  aquests.fetchall ()
  
  
Requests Parameters
========================

For get, post*, put*, upload, delete, options, trace parameters are the same.

.. code-block:: python

  aquests.get (url, params = None, headers = None, auth = None, meta = {})
  
- url: request url string
- params: None or dictionary, if it provide with get method, it will be attached on tail of url with '?'
- headers: None or dictionary
- auth: None or tuple (username, password)
- meta: dictionary

For rpc, grpc stub creation:

.. code-block:: python

  stub = aquests.rpc (url, headers = None, auth = None, meta = {})
  stub = aquests.grpc (url, headers = None, auth = None, meta = {})
  
- url: request url string
- headers: None or dictionary
- auth: None or tuple (username, password)
- meta: dictionary

Note: stub's methods and parameters are defined by RPC service providers

For postgresql, mongodb, redis dbo creation:

.. code-block:: python

  dbo = aquests.postgresql (server, dbname = None, auth = None, meta = {})
  dbo = aquests.mongodb (server, dbname = None, auth = None, meta = {})  
  dbo = aquests.redis (server, dbname = None, auth = None, meta = {})
  
- server: address:port formated string
- dbname: None or string
- auth: None or tuple (username, password)
- meta: dictionary

Note: stub's methods and parameters are defined by database engines. Please read above related chapters But SQL based postgresql has only 1 method and parameters - execute(sql) or do(sql) just for your convinience.


History
=========

- 0.4.9: changed response properties - request.method -> method, request.server -> server, request.dbname -> dbname and request.params -> params
- 0.4.4: add lib.athreads
- 0.4.2: fix http2 large content download
- 0.4.1: add a few examples
- 0.4: add timeout feature
- 0.3.10: fix http2 frame length validation, add cookie feature
- 0.3.8: fix dbo request shutdown behavior
- 0.3.1: add HEAD, OPTIONS, TRACE
- 0.3: fix installation error
- 0.2.13: change default display callback
- 0.2.10: fix xmlrpc

