grab.document — Grab 0.6.41 documentation

# `grab.document`

The Document class is the result of network request made with Grab instance.

## Module Contents

### Classes

`Document` | Network response.

### Functions

`normalize_pairs`(→ collections.abc.Sequence[tuple[str, ...) |

### Attributes

`THREAD_STORAGE` |
`logger` |

grab.document.THREAD_STORAGE

grab.document.logger

grab.document.normalize_pairs(inp: Sequence[tuple[str, Any]] | Mapping[str, Any]) → collections.abc.Sequence[tuple[str, Any]]

class grab.document.Document(body: None | bytes = None, *, document_type: None | str = 'html', head: None | bytes = None, headers: None | email.message.Message = None, encoding: None | str = None, code: None | int = None, url: None | str = None, cookies: None | CookieJar = None)

  Network response.

  property status: None | int
  property json: Any

    Return response body deserialized into JSON object.

  property pyquery: Any

    Return pyquery handler.

  property body: None | bytes
  property tree: lxml.etree._Element

    Return DOM tree of the document built with HTML DOM builder.

  property form: lxml.html.FormElement

    Return default document’s form.

    If form was not selected manually then select the form which has the biggest number of input elements.

    The form value is just an lxml.html form element.

    Example:

        g.request('some URL')
        # Choose form automatically
        print g.form

        # And now choose form manually
        g.choose_form(1)
        print g.form

  __slots__ = ['document_type', 'code', 'head', '_bytes_body', 'headers', 'url', 'cookies', 'encoding',...
  __call__(query: str) → selection.SelectorList[lxml.etree._Element]
  select(*args: Any, **kwargs: Any) → selection.SelectorList[lxml.etree._Element]
  process_encoding(encoding: None | str = None) → str

    Process explicitly defined encoding or auto-detect it.

    If encoding is explicitly defined, ensure it is a valid encoding the python can deal with. If encoding is not specified, auto-detect it.

    Raises unicodec.InvalidEncodingName if explicitly set encoding is invalid.

  copy() → Document
  save(path: str) → None

    Save response body to file.

  url_details() → urllib.parse.SplitResult

    Return result of urlsplit function applied to response url.

  query_param(key: str) → str

    Return value of parameter in query string.

  browse() → None

    Save response in temporary file and open it in GUI browser.

  __getstate__() → collections.abc.Mapping[str, Any]

    Reset cached lxml objects which could not be pickled.

  __setstate__(state: collections.abc.Mapping[str, Any]) → None
  text_search(anchor: str | bytes) → bool

    Search the substring in response body.

    Parameters

      - anchor – string to search

      - byte – if False then anchor should be the unicode string, and search will be performed in response.unicode_body() else anchor should be the byte-string and search will be performed in response.body

    If substring is found return True else False.

  text_assert(anchor: str | bytes) → None

    If anchor is not found then raise DataNotFound exception.

  text_assert_any(anchors: list[str | bytes]) → None

    If no anchors were found then raise DataNotFound exception.

  rex_text(regexp: str | bytes | Pattern[str] | Pattern[bytes], flags: int = 0, default: Any = NULL) → Any

    Return content of first matching group of regexp found in response body.

  rex_search(regexp: str | bytes | Pattern[str] | Pattern[bytes], flags: int = 0, default: Any = NULL) → Any

    Search the regular expression in response body.

    Return found match object or None

  rex_assert(rex: str | bytes | Pattern[str] | Pattern[bytes]) → None

    Raise DataNotFound exception if rex expression is not found.

  get_body_chunk() → None | bytes
  unicode_body() → None | str

    Return response body as unicode string.

  set_body(body: bytes) → None
  classmethod wrap_io(inp: bytes | str) → StringIO | BytesIO
  classmethod _build_dom(content: bytes | str, mode: str, encoding: str) → lxml.etree._Element
  build_html_tree() → lxml.etree._Element
  build_xml_tree() → lxml.etree._Element
  choose_form(number: None | int = None, xpath: None | str = None, name: None | str = None, **kwargs: Any) → None

    Set the default form.

    Parameters

      - number – number of form (starting from zero)

      - id – value of “id” attribute

      - name – value of “name” attribute

      - xpath – XPath query

    Raises

      `DataNotFound` if form not found

    Raises

      `GrabMisuseError` if method is called without parameters

    Selected form will be available via form attribute of Grab instance. All form methods will work with default form.

    Examples:

        # Select second form
        g.choose_form(1)

        # Select by id
        g.choose_form(id="register")

        # Select by name
        g.choose_form(name="signup")

        # Select by xpath
        g.choose_form(xpath='//form[contains(@action, "/submit")]')

  get_cached_form() → None | FormElement

    Get form which has been already selected.

    Returns None if form has not been selected yet.

    It is for testing mainly. To not trigger pylint warnings about accessing protected element.

  set_input(name: str, value: Any) → None

    Set the value of form element by its name attribute.

    Parameters

      - name – name of element

      - value – value which should be set to element

    To check/uncheck the checkbox pass boolean value.

    Example:

        g.set_input('sex', 'male')

        # Check the checkbox
        g.set_input('accept', True)

  set_input_by_id(_id: str, value: Any) → None

    Set the value of form element by its id attribute.

    Parameters

      - _id – id of element

      - value – value which should be set to element

  set_input_by_number(number: int, value: Any) → None

    Set the value of form element by its number in the form.

    Parameters

      - number – number of element

      - value – value which should be set to element

  set_input_by_xpath(xpath: str, value: Any) → None

    Set the value of form element by xpath.

    Parameters

      - xpath – xpath path

      - value – value which should be set to element

  process_extra_post(post_items: list[tuple[str, Any]], extra_post_items: collections.abc.Sequence[tuple[str, Any]]) → list[tuple[str, Any]]
  clean_submit_controls(post: collections.abc.MutableMapping[str, Any], submit_name: None | str) → None
  get_form_request(submit_name: None | str = None, url: None | str = None, extra_post: None | Mapping[str, Any] | Sequence[tuple[str, Any]] = None, remove_from_post: None | Sequence[str] = None) → tuple[str, str, bool, collections.abc.Sequence[tuple[str, Any]]]

    Submit default form.

    Parameters

      - submit_name – name of button which should be “clicked” to submit form

      - url – explicitly specify form action url

      - extra_post – (dict or list of pairs) additional form data which will override data automatically extracted from the form.

      - remove_from_post – list of keys to remove from the submitted data

    Following input elements are automatically processed:

    - input[type=”hidden”] - default value

    - select: value of last option

    - radio - ???

    - checkbox - ???

    Multipart forms are correctly recognized by grab library.

  build_fields_to_remove(fields: collections.abc.Mapping[str, Any], form_inputs: collections.abc.Sequence[lxml.html.HtmlElement]) → set[str]
  process_form_fields(fields: collections.abc.MutableMapping[str, Any]) → None
  form_fields() → collections.abc.MutableMapping[str, lxml.html.HtmlElement]

    Return fields of default form.

    Fill some fields with reasonable values.

  choose_form_by_element(xpath: str) → None