Data Extraction
===============

This section explains how to write the rules for extracting data
out of a document. We'll scrape the following HTML markup for the movie
"The Shining" in our examples:

.. literalinclude:: ../examples/shining.html
   :language: html

Assuming this file is saved as :file:`shining.html`, let's get its content:

.. code-block:: python

   >>> document = open('shining.html').read()

The main functionality of the module is provided
by the :func:`scrape <piculet.scrape>` function:

>>> from piculet import scrape

This function takes a document and the rule specifications as parameters.
The document has to be well-formed XML and it will be converted into a tree.
Then the rules will be applied to the root node of the tree. The function
returns a mapping where each item is generated by one of the rules.

Rules contain items where each item is itself a mapping. The ``key`` specifies
the key for the item in the output mapping and the ``value`` specifies
how to extract the data to set as the value for that item. Typically,
a value specifier consists of a path query and a reducing function. The query
is applied to the root and a list of strings is obtained. Then,
the reducing function converts this list into a single string. [#reducing]_

For example, to get the title of the movie from the example document,
we can write:

>>> rules = {
...     "items": [
...         {
...             "key": "title",
...             "value": {
...                 "path": ".//title/text()",
...                 "reduce": lambda xs: xs[0]
...             }
...         }
...     ]
... }
>>> scrape(document, rules)
{'title': 'The Shining'}

The ``.//title/text()`` path generates the list ``['The Shining']``
and the reducing function ``lambda xs: xs[0]`` selects the first element
from that list.

If the document is HTML but not well-formed XML, the ``content_format``
parameter can be used to convert it into XML first: [#xhtml]_

>>> scrape(document, rules, content_format='html')

.. note::

   Piculet uses the `ElementTree`_ module for building and querying
   XML trees. Therefore, the XPath queries are limited by
   `what ElementTree supports`_ (plus the ``text()`` and ``@attr``
   clauses which are added by Piculet). However, Piculet will make use of
   the `lxml`_ package if it's installed, and in that case,
   a `much wider range of XPath constructs`_ can be used.

.. _ElementTree: https://docs.python.org/3/library/xml.etree.elementtree.html
.. _what ElementTree supports: https://docs.python.org/3/library/xml.etree.elementtree.html#xpath-support
.. _lxml: http://lxml.de/
.. _much wider range of XPath constructs: http://lxml.de/xpathxslt.html#xpath

Multiple items can be collected in a single invocation:

>>> rules = {
...     "items": [
...         {
...             "key": "title",
...             "value": {
...                 "path": ".//title/text()",
...                 "reduce": lambda xs: xs[0]
...             }
...         },
...         {
...             "key": "year",
...             "value": {
...                 "path": ".//span[@class='year']/text()",
...                 "reduce": lambda xs: xs[0]
...             }
...         }
...     ]
... }
>>> scrape(document, rules)
{'title': 'The Shining', 'year': '1980'}

If a path doesn't match any element in the tree, the item will be excluded
from the output and no error will be issued. Note that in the following
example, the "foo" key doesn't get included:

>>> rules = {
...     "items": [
...         {
...             "key": "title",
...             "value": {
...                 "path": ".//title/text()",
...                 "reduce": lambda xs: xs[0]
...              }
...         },
...         {
...             "key": "foo",
...             "value": {
...                 "path": ".//foo/text()",
...                 "reduce": lambda xs: xs[0]
...             }
...         }
...     ]
... }
>>> scrape(document, rules)
{'title': 'The Shining'}

Reducing
--------

Piculet contains a few predefined reducing functions which can be accessed
through the ``reducers`` attribute::

>>> from piculet import reducers

The reducing function that returns the first element of a list
-as used in the examples above- is one of the predefined reducers:

.. code-block:: python

   {
       "path": ".//title/text()",
       "reduce": reducers.first
   }

Other commonly used predefined reducing operations are joining and cleaning.
The ``join`` reducer joins all selected strings into one:

>>> rules = {
...     "items": [
...         {
...             "key": "full_title",
...             "value": {
...                 "path": ".//h1//text()",
...                 "reduce": reducers.join
...             }
...         }
...     ]
... }
>>> scrape(document, rules)
{'full_title': 'The Shining (1980)'}

If you want to get rid of extra whitespace, you can use the ``clean`` reducer.
After joining the strings, this will remove leading and trailing whitespace
and replace multiple whitespace with a single space:

>>> rules = {
...     "items": [
...         {
...             "key": "review",
...             "value": {
...                 "path": ".//div[@class='review']//text()",
...                 "reduce": reducers.clean
...             }
...         }
...     ]
... }
>>> scrape(document, rules)
{'review': 'Fantastic movie. Definitely recommended.'}

In this example, the ``join`` reducer would have produced the value
``'\n            Fantastic movie.\n            Definitely recommended.\n        '``

The predefined reducers can also be accessed by name using the "reducer"
key instead of "reduce" in the specification. So the title rule can also
be written as:

.. code-block:: python

   {
       "path": ".//title/text()",
       "reducer": "first"
   }

This might be useful if you would like to keep the rule specification
in an external text file, such as a ``.json`` file. If both a "reduce"
and a "reducer" key are given, the "reduce" callable will be used
and the "reducer" key will be ignored.

As explained above, if a path query doesn't match any element, the item
gets automatically excluded. That means, Piculet doesn't try to apply
the reducing function on the result of the path query if it's an empty list.
Therefore, reducing functions can safely assume that the path result is
a non-empty list.

Transforming
------------

After the reduction operation, you can also apply a transformation
to the resulting string. A transformation function must take a string
as parameter and can return any value of any type. For example,
to get the year of the movie as an integer:

>>> rules = {
...     "items": [
...         {
...             "key": "year",
...             "value": {
...                 "path": ".//span[@class='year']/text()",
...                 "reduce": reducers.first,
...                 "transform": int
...             }
...         }
...     ]
... }
>>> scrape(document, rules)
{'year': 1980}

Multi-valued items
------------------

Data with multiple values can be created by using a ``foreach`` key
in the value specifier. This is a path expression to select elements
from the tree. [#multivalued]_ The path and reducing function will be applied
as before to each selected element and the obtained values will be the members
of the resulting list. For example, to get the genres of the movie,
we can write:

>>> rules = {
...     "items": [
...         {
...             "key": "genres",
...             "value": {
...                 "foreach": ".//ul[@class='genres']/li",
...                 "path": "./text()",
...                 "reduce": reducers.first
...             }
...         }
...     ]
... }
>>> scrape(document, rules)
{'genres': ['Horror', 'Drama']}

If the ``foreach`` key doesn't match any element the item will be excluded
from the result:

>>> rules = {
...     "items": [
...         {
...             "key": "foos",
...             "value": {
...                 "foreach": ".//ul[@class='foos']/li",
...                 "path": "./text()",
...                 "reduce": reducers.first
...             }
...         }
...     ]
... }
>>> scrape(document, rules)
{}

If a transformation is specified, it will be applied to every element
in the list:

>>> rules = {
...     "items": [
...         {
...             "key": "genres",
...             "value": {
...                 "foreach": ".//ul[@class='genres']/li",
...                 "path": "./text()",
...                 "reduce": reducers.first,
...                 "transform": lambda x: x.lower()
...             }
...         }
...     ]
... }
>>> scrape(document, rules)
{'genres': ['horror', 'drama']}

Subrules
--------

Nested structures can be created by writing subrules as value specifiers.
If the value specifier is a mapping that contains an ``items`` key,
then this will be interpreted as a subrule and the generated mapping
will be the value for the key.

>>> rules = {
...     "items": [
...         {
...             "key": "director",
...             "value": {
...                 "items": [
...                     {
...                         "key": "name",
...                         "value": {
...                             "path": ".//div[@class='director']//a/text()",
...                             "reduce": reducers.first
...                         }
...                     },
...                     {
...                         "key": "link",
...                         "value": {
...                             "path": ".//div[@class='director']//a/@href",
...                             "reduce": reducers.first
...                         }
...                     }
...                 ]
...             }
...         }
...     ]
... }
>>> scrape(document, rules)
{'director': {'link': '/people/1', 'name': 'Stanley Kubrick'}}

Subrules can be combined with lists:

>>> rules = {
...     "items": [
...         {
...             "key": "cast",
...             "value": {
...                 "foreach": ".//table[@class='cast']/tr",
...                 "items": [
...                     {
...                         "key": "name",
...                         "value": {
...                             "path": "./td[1]/a/text()",
...                             "reduce": reducers.first
...                         }
...                     },
...                     {
...                         "key": "link",
...                         "value": {
...                             "path": "./td[1]/a/@href",
...                             "reduce": reducers.first
...                         }
...                     },
...                     {
...                         "key": "character",
...                         "value": {
...                             "path": "./td[2]/text()",
...                             "reduce": reducers.first
...                         }
...                     }
...                 ]
...             }
...         }
...     ]
... }
>>> scrape(document, rules)
{'cast': [{'character': 'Jack Torrance',
   'link': '/people/2',
   'name': 'Jack Nicholson'},
  {'character': 'Wendy Torrance',
   'link': '/people/3',
   'name': 'Shelley Duvall'}]}

Items generated by subrules can also be transformed. The transformation
function is always applied as the last step in a "value" definition. But
transformers for subitems take mappings (as opposed to strings) as parameter.

>>> rules = {
...     "items": [
...         {
...             "key": "cast",
...             "value": {
...                 "foreach": ".//table[@class='cast']/tr",
...                 "items": [
...                     {
...                         "key": "name",
...                         "value": {
...                             "path": "./td[1]/a/text()",
...                             "reduce": reducers.first
...                         }
...                     },
...                     {
...                         "key": "character",
...                         "value": {
...                             "path": "./td[2]/text()",
...                             "reduce": reducers.first
...                         }
...                     }
...                 ],
...                 "transform": lambda x: x['name'] + ' as ' + x['character']
...             }
...         }
...     ]
... }
>>> scrape(document, rules)
{'cast': ['Jack Nicholson as Jack Torrance',
  'Shelley Duvall as Wendy Torrance']}

Generating keys from content
----------------------------

You can generate items where the key value also comes from the content.
Consider for example how you would get the runtime and the language
of the movie. Instead of writing multiple items for each ``h3`` element
under an "info" class ``div``, we can write only one item that will select
these divs and use the h3 text as the key. Section elements like these divs
can be selected using ``foreach`` specifications in the keys. This will cause
a new item to be generated for each selected element. To get the key value,
we can use paths, reducers -and also transformers- that will be applied
to the selected section element:

>>> rules = {
...     "items": [
...         {
...             "foreach": ".//div[@class='info']",
...             "key": {
...                 "path": "./h3/text()",
...                 "reduce": reducers.first
...             },
...             "value": {
...                 "path": "./p/text()",
...                 "reduce": reducers.first
...             }
...         }
...     ]
... }
>>> scrape(document, rules)
{'Language:': 'English', 'Runtime:': '144 minutes'}

The ``normalize`` predefined reducer joins the strings, converts it
to lowercase, replaces spaces with underscores and strips other
non-alphanumeric characters:

>>> rules = {
...     "items": [
...         {
...             "foreach": ".//div[@class='info']",
...             "key": {
...                 "path": "./h3/text()",
...                 "reduce": reducers.normalize
...             },
...             "value": {
...                 "path": "./p/text()",
...                 "reduce": reducers.first
...             }
...         }
...     ]
... }
>>> scrape(document, rules)
{'language': 'English', 'runtime': '144 minutes'}

You could also give a string instead of a path and reducer for the key.
In this case, the elements would still be traversed; only the last one would
set the final value for the item. This could be OK if you are sure
that there is only one element that matches the ``foreach`` path of the key.

Lower-level functions
---------------------

If the same document will be scraped multiple times with different rules,
calling the ``scrape`` function repeatedly will cause the document to be parsed
repeatedly. A more efficient way would be to convert the document into a tree
once using the :func:`build_tree <piculet.build_tree>` function and then
calling the :func:`extract <piculet.extract>` function with the rule items:

>>> from piculet import build_tree, extract
>>> root = build_tree(document)
>>> items = [
...     {
...         "key": "title",
...         "value": {
...             "path": ".//title/text()",
...             "reduce": reducers.first
...         }
...     }
... ]
>>> extract(root, items)

If the document needs to be converted from HTML to XML, you can use
the :func:`html_to_xhtml <piculet.html_to_xhtml>` function:

>>> from piculet import html_to_xhtml
>>> converted = html_to_xhtml(document)
>>> root = build_tree(converted)

.. [#xhtml]

   Note that the example document is well-formed XML.

.. [#reducing]

   This means that the query has to end with either ``text()`` or some
   attribute value as in ``@attr``. And the reducing function should be
   implemented so that it takes a list of strings and returns a string.

.. [#multivalued]

   This implies that the ``foreach`` query should **not** end in ``text()``
   or ``@attr``.
