API Reference — Dugong 3.7.2 documentation

# API Reference

## Classes

class `HTTPConnection`(hostname, port=None, ssl_context=None, proxy=None)

  This class encapsulates a HTTP connection.

  Methods whose name begin with `co_` return coroutines. Instead of blocking, a coroutines will yield a `PollNeeded` instance that encapsulates information about the IO operation that would block. The coroutine should be resumed once the operation can be performed without blocking.

  `HTTPConnection` instances can be used as context managers. The `disconnect` method will be called on exit from the managed block.

  `co_discard`()

    Read and discard current response body

    This method returns a coroutine. `discard` is a regular method implementing the same functionality.

  `co_read`(len_=None)

    Read up to len_ bytes of response body data

    This method may return less than len_ bytes, but will return `b''` only if the response body has been read completely.

    If len_ is `None`, this method returns the entire response body.

    This method returns a coroutine. `read` is a regular method implementing the same functionality.

  `co_read_response`()

    Read response status line and headers

    Return a `HTTPResponse` instance containing information about response status, reason, and headers. The response body data must be retrieved separately (e.g. using `read` or `readall`).

    This method returns a coroutine. `read_response` is a regular method implementing the same functionality.

  `co_readall`()

    Read and return complete response body

    This method returns a coroutine. `readall` is a regular method implementing the same functionality.

  `co_readinto`(buf)

    Read response body data into buf

    Return the number of bytes written or zero if the response body has been read completely.

    buf must implement the memoryview protocol.

    This method returns a coroutine. `readinto` is a regular method implementing the same functionality.

  `co_send_request`(method, path, headers=None, body=None, expect100=False)

    Send a new HTTP request to the server

    The message body may be passed in the body argument or be sent separately. In the former case, body must be a bytes-like object. In the latter case, body must be an a `BodyFollowing` instance specifying the length of the data that will be sent. If no length is specified, the data will be send using chunked encoding.

    headers should be a mapping containing the HTTP headers to be send with the request. Multiple header lines with the same key are not supported. It is recommended to pass a `CaseInsensitiveDict` instance, other mappings will be converted to `CaseInsensitiveDict` automatically.

    If body is a provided as a bytes-like object, a `Content-MD5` header is generated automatically unless it has been provided in headers already.

    This method returns a coroutine. `send_request` is a regular method implementing the same functionality.

  `co_write`(buf)

    Write request body data

    `ExcessBodyData` will be raised when attempting to send more data than required to complete the request body of the active request.

    This method returns a coroutine. `write` is a regular method implementing the same functionality.

  `connect`()

    Connect to the remote server

    This method generally does not need to be called manually.

  `discard`()

    Read and discard current response body

    This method may block. `co_discard` provides a coroutine implementing the same functionality without blocking.

  `disconnect`()

    Close HTTP connection

  `get_ssl_cipher`()

    Get active SSL cipher

    If plain HTTP is used, return `None`. Otherwise, the call is delegated to the underlying SSL sockets `cipher` method.

  `get_ssl_peercert`(binary_form=False)

    Get peer SSL certificate

    If plain HTTP is used, return `None`. Otherwise, the call is delegated to the underlying SSL sockets `getpeercert` method.

  `read`(len_=None)

    Read up to len_ bytes of response body data

    This method may return less than len_ bytes, but will return `b''` only if the response body has been read completely.

    If len_ is `None`, this method returns the entire response body.

    This method may block. `co_read` provides a coroutine implementing the same functionality without blocking.

  `read_raw`(size)

    Read size bytes of uninterpreted data

    This method may be used even after `UnsupportedResponse` or `InvalidResponse` has been raised. It reads raw data from the socket without attempting to interpret it. This is probably only useful for debugging purposes to take a look at the raw data received from the server. This method blocks if no data is available, and returns `b''` if the connection has been closed.

    Calling this method will break the internal state and switch the socket to blocking operation. The connection has to be closed and reestablished afterwards.

    Don’t use this method unless you know exactly what you are doing.

  `read_response`()

    Read response status line and headers

    Return a `HTTPResponse` instance containing information about response status, reason, and headers. The response body data must be retrieved separately (e.g. using `read` or `readall`).

    This method may block. `co_read_response` provides a coroutine implementing the same functionality without blocking.

  `readall`()

    Read and return complete response body

    This method may block. `co_readall` provides a coroutine implementing the same functionality without blocking.

  `readinto`(buf)

    Read response body data into buf

    Return the number of bytes written or zero if the response body has been read completely.

    buf must implement the memoryview protocol.

    This method may block. `co_readinto` provides a coroutine implementing the same functionality without blocking.

  `reset`()

    Reset HTTP connection

    This method resets the status of the HTTP connection after an exception has occured. Any cached data and pending responses are discarded.

  `response_pending`()

    Return `True` if there are still outstanding responses

    This includes responses that have been partially read.

  `send_request`(method, path, headers=None, body=None, expect100=False)

    Send a new HTTP request to the server

    The message body may be passed in the body argument or be sent separately. In the former case, body must be a bytes-like object. In the latter case, body must be an a `BodyFollowing` instance specifying the length of the data that will be sent. If no length is specified, the data will be send using chunked encoding.

    headers should be a mapping containing the HTTP headers to be send with the request. Multiple header lines with the same key are not supported. It is recommended to pass a `CaseInsensitiveDict` instance, other mappings will be converted to `CaseInsensitiveDict` automatically.

    If body is a provided as a bytes-like object, a `Content-MD5` header is generated automatically unless it has been provided in headers already.

    This method may block. `co_send_request` provides a coroutine implementing the same functionality without blocking.

  `write`(buf)

    Write request body data

    `ExcessBodyData` will be raised when attempting to send more data than required to complete the request body of the active request.

    This method may block. `co_write` provides a coroutine implementing the same functionality without blocking.

  `proxy` = None

    a tuple `(hostname, port)` of the proxy server to use or `None`.

  `timeout` = None

    If a regular `HTTPConnection` method is unable to send or receive data for more than this period (in seconds), it will raise `ConnectionTimedOut`. Coroutines are not affected by this attribute.

class `HTTPResponse`(method, path, status, reason, headers, length=None)

  This class encapsulates information about HTTP response. Instances of this class are returned by the `HTTPConnection.read_response` method and have access to response status, reason, and headers. Response body data has to be read directly from the `HTTPConnection` instance.

  `headers` = None

    HTTP Response headers, a `email.message.Message` instance

  `length` = None

    Length of the response body or `None` if not known. This attribute contains the actual length of the transmitted response. That means that for responses where RFC 2616 mandates that no request body be sent (e.g. in response to HEAD requests or for 1xx response codes) this value is zero. In these cases, the length of the body that would have been send can be extracted from the `Content-Length` response header.

  `method` = None

    HTTP Method of the request this was response is associated with

  `path` = None

    Path of the request this was response is associated with

  `reason` = None

    HTTP reason phrase returned by the server

  `status` = None

    HTTP status code returned by the server

class `BodyFollowing`(length=None)

  Sentinel class for the body parameter of the `send_request` method. Passing an instance of this class declares that body data is going to be provided in separate method calls.

  If no length is specified in the constructor, the body data will be send using chunked encoding.

  `length`

    the length of the body data that is going to be send, or `None` to use chunked encoding.

class `CaseInsensitiveDict`(data=None, **kwargs)

  A case-insensitive `dict`-like object.

  Implements all methods and operations of `collections.abc.MutableMapping` as well as `copy`.

  All keys are expected to be strings. The structure remembers the case of the last key to be set, and `iter()`, `keys()` and `items()` will contain case-sensitive keys. However, querying and contains testing is case insensitive:

      cid = CaseInsensitiveDict()
      cid['Accept'] = 'application/json'
      cid['aCCEPT'] == 'application/json' # True
      list(cid) == ['Accept'] # True

  For example, `headers['content-encoding']` will return the value of a `'Content-Encoding'` response header, regardless of how the header name was originally stored.

  If the constructor, `update()`, or equality comparison operations are given multiple keys that have equal lower-case representions, the behavior is undefined.

  `lower_items`()

    Like `items()`, but with all lowercase keys.

class `PollNeeded`

  This class encapsulates the requirements for a IO operation to continue. `PollNeeded` instances are typically yielded by coroutines.

  `poll`(timeout=None)

    Wait until fd is ready for requested IO

    This is a convenince function that uses `poll` to wait until `fd` is ready for requested type of IO.

    If timeout is specified, return `False` if the timeout is exceeded without the file descriptor becoming ready.

  `fd`

    File descriptor that the IO operation depends on

  `mask`

    Event mask specifiying the type of required IO

    This attribute defines what type of IO the provider of the `PollNeeded` instance needs to perform on fd. It is expected that, when fd is ready for IO of the specified type, operation will continue without blocking.

    The type of IO is specified as a poll compatible event mask, i.e. a bitwise combination of `select.POLLIN` and `select.POLLOUT`.

class `AioFuture`(crt, loop=None)

  This class wraps a coroutine that yields `PollNeeded` instances into an `asyncio` compatible `Future`.

  This is done by registering a callback with the event loop that resumes the coroutine when the requested IO is available.

## Functions

`is_temp_network_error`(exc)

  Return true if exc represents a potentially temporary network problem

  DNS resolution errors (`socket.gaierror` or `socket.herror`) are considered permanent, because `HTTPConnection` employs a heuristic to convert these exceptions to `HostnameNotResolvable` or `DNSUnavailable` instead.

## Exceptions

Dugong functions may pass through any exceptions raised by the socket and `ssl.SSLSocket` methods. In addition to that, the following dugong-specific exceptions may be raised as well:

exception `ConnectionClosed`(msg=None)

  Raised if the connection was unexpectedly closed.

  This exception is raised also if the server declared that it will close the connection (by sending a `Connection: close` header). Such responses can still be read completely, but the next attempt to send a request or read a response will raise the exception. To re-use the connection after the server has closed the connection, call `HTTPConnection.reset` before further requests are send.

  This behavior is intentional, because the caller may have already issued other requests (i.e., used pipelining). By raising an exception, the caller is notified that any pending requests have been lost and need to be resend.

exception `InvalidResponse`(msg=None)

  Raised if the server produced an invalid response (i.e, something that is not proper HTTP 1.0 or 1.1).

exception `UnsupportedResponse`(msg=None)

  This exception is raised if the server produced a response that is not supported. This should not happen for servers that are HTTP 1.1 compatible.

  If an `UnsupportedResponse` exception has been raised, this typically means that synchronization with the server will be lost (i.e., dugong cannot determine where the current response ends and the next response starts), so the connection needs to be reset by calling the `disconnect()` method.

exception `ExcessBodyData`(msg=None)

  Raised when trying to send more data to the server than announced.

exception `StateError`(msg=None)

  Raised when attempting an operation that doesn’t make sense in the current connection state.

exception `ConnectionTimedOut`(msg=None)

  Raised if a regular `HTTPConnection` method (i.e., no coroutine) was unable to send or receive data for the timeout specified in the `HTTPConnection.timeout` attribute.

exception `HostnameNotResolvable`(hostname)

  Raised if a host name does not resolve to an ip address.

  Dugong raises this exception if a resolution attempt results in a `socket.gaierror` or `socket.herror` exception with errno `socket.EAI_AGAIN` or `socket.EAI_NONAME`, but at least one of the hostnames in `DNS_TEST_HOSTNAMES` can be resolved.

exception `DNSUnavailable`(hostname)

  Raised if the DNS server cannot be reached.

  Dugong raises this exception if a resolution attempt results in a `socket.gaierror` or `socket.herror` exception with errno `socket.EAI_AGAIN` or `socket.EAI_NONAME`, and none of the hostnames in `DNS_TEST_HOSTNAMES` can be resolved either.

## Constants

`MAX_LINE_SIZE` = 65535

  Maximal length of HTTP status line. If the server sends a line longer than this value, `InvalidResponse` will be raised.

`MAX_HEADER_SIZE` = 65535

  Maximal length of a response header (i.e., for all header lines together). If the server sends a header segment longer than this value, `InvalidResponse` will be raised.

`DNS_TEST_HOSTNAMES` = (('www.google.com', 80), ('www.iana.org', 80), ('C.root-servers.org', 53))

  Sequence of `(hostname, port)` tuples that are used by dugong to distinguish between permanent and temporary name resolution problems.

## Thread Safety

Dugong is not generally threadsafe. However, simultaneous use of the same `HTTPConnection` instance by two threads is supported if once thread is restricted to sending requests, and the other thread restricted to reading responses.

## Avoiding Deadlocks

The `HTTPConnection` class allows you to send an unlimited number of requests to the server before reading any of the responses. However, at some point the transmit and receive buffers on both the ends of the connection will fill up, and no more requests can be send before at least some of the responses are read, and attempts to send more data to the server will block. If the thread that attempts to send data is is also responsible for reading the responses, this will result in a deadlock.

There are several ways to avoid this:

- Do not send a new request before the last response has been read. This is the easiest solution, but it means that no HTTP pipelining can be used.
- Use different threads for sending requests and receiving responses.
- Use the coroutine based API (see Pipelining with Coroutines in the tutorial).