Data Extraction
===============

Although applications will mostly use the higher-level ``scrape`` function,
it is the ``extract`` function that handles the data extraction. So we'll
start by explaining how that works:

>>> from piculet import extract

We'll use the following (X)HTML document in our examples:

.. literalinclude:: shining.html
   :language: html

Assuming this file is saved as :file:`shining.html`, let's get its content:

.. code-block:: python

   >>> with open('shining.html') as f:
   ...     document = f.read()
   ...

The extraction function takes an XML document and a list of extraction rules.
It converts the document into an XML tree and applies the rules to the root
of the tree. The result will be a mapping where each item is generated by
one of the rules.

Each rule is itself a mapping. The ``key`` specifies the key for the entry
in the output mapping and the ``value`` specifies how to extract the data
to set as the value for the entry. Typically, a value specifier consists
of a path query and a reducer function. The query gets applied to the root
of the tree and produces a list of strings. Then the reducer function
transforms this list into a single string. [#]_

.. note::
      Piculet uses the `ElementTree`_ module for building and querying
      XML trees. Therefore, the XPath queries you write are limited by
      `whatever ElementTree supports`_. However, Piculet will make use of
      the `lxml`_ package if it is installed and in that case, you can use
      a `much wider range of XPath constructs`_.

.. _ElementTree: https://docs.python.org/3/library/xml.etree.elementtree.html
.. _whatever ElementTree supports: https://docs.python.org/3/library/xml.etree.elementtree.html#xpath-support
.. _lxml: http://lxml.de/
.. _much wider range of XPath constructs: http://lxml.de/xpathxslt.html#xpath

For example, to extract the title and year out of the sample document, we can
write the following rules where ``first`` is a predefined reducer that selects
the first element of a list:

>>> items = [
...     {
...         "key": "title",
...         "value": {
...             "path": ".//title/text()",
...             "reducer": "first"
...         }
...     },
...     {
...         "key": "year",
...         "value": {
...             "path": ".//span[@class='year']/text()",
...             "reducer": "first"
...         }
...     }
... ]
>>> extract(document, items)
{'title': 'The Shining', 'year': '1980'}

Other common predefined reducing operations are joining and cleaning.
The ``join`` reducer joins all selected strings into one:

>>> items = [
...     {
...         "key": "full_title",
...         "value": {
...             "path": ".//h1//text()",
...             "reducer": "join"
...         }
...     }
... ]
>>> extract(document, items)
{'full_title': 'The Shining (1980)'}

Joining the strings will cause extra whitespace to be preserved in the value.
If you want to get rid of these, use the ``clean`` reducer. After joining
the strings, this will remove leading and trailing whitespace and replace
multiple whitespace characters with a single space:

>>> items = [
...     {
...         "key": "review",
...         "value": {
...             "path": ".//div[@class='review']//text()",
...             "reducer": "clean"
...         }
...     }
... ]
{'review': 'Fantastic movie.'}

In this example, using the ``join`` reducer would have produced the value
``'\n      Fantastic movie.\n    '``

To create a list of values, a ``foreach`` key can be given to the value
specifier. This is a path expression to select elements from the tree.
The ``path`` and ``reducer`` will be applied as before to each selected
element and the obtained values will be the members of the result list. [#]_
For example, to get the genres of the movie, we can write:

>>> items = [
...     {
...         "key": "genres",
...         "value": {
...             "foreach": ".//ul[@class='genres']/li",
...             "path": "./text()",
...             "reducer": "first"
...         }
...     }
... ]
>>> extract(document, items)
{'genres': ['Horror', 'Drama']}

Subrules
--------

Nested structures can be created by writing subrules as value specifiers. If
the value specifier is a mapping that contains an ``items`` key, then this will
be interpreted as a subrule and the generated mapping will be the value for
the key.

>>> items = [
...     {
...         "key": "title",
...         "value": {
...             "path": ".//title/text()",
...             "reducer": "first"
...         }
...     },
...     {
...         "key": "director",
...         "value": {
...             "items": [
...                 {
...                     "key": "name",
...                     "value": {
...                         "path": ".//div[@class='director']//a/text()",
...                         "reducer": "first"
...                     }
...                 },
...                 {
...                     "key": "link",
...                     "value": {
...                         "path": ".//div[@class='director']//a/@href",
...                         "reducer": "first"
...                     }
...                 }
...             ]
...         }
...     }
... ]
>>> extract(document, items)
{'title': 'The Shining', 'director': {'name': 'Stanley Kubrick', 'link': '/people/1'}}

Subrules can also be combined with lists:

>>> items = [
...     {
...         "key": "cast",
...         "value": {
...             "foreach": ".//table[@class='cast']/tr",
...             "items": [
...                 {
...                     "key": "name",
...                     "value": {
...                         "path": "./td[1]/a/text()",
...                         "reducer": "first"
...                     }
...                 },
...                 {
...                     "key": "link",
...                     "value": {
...                         "path": "./td[1]/a/@href",
...                         "reducer": "first"
...                     }
...                 },
...                 {
...                     "key": "character",
...                     "value": {
...                         "path": "./td[2]/text()",
...                         "reducer": "first"
...                     }
...                 }
...             ]
...         }
...     }
... ]
>>> extract(document, items)
{'cast': [{'name': 'Jack Nicholson', 'link': '/people/2', 'character': 'Jack Torrance'}, {'name': 'Shelley Duvall', 'link': '/people/3', 'character': 'Wendy Torrance'}]}

Generating Keys from Content
----------------------------

You can generate items in the result mapping where the key value also comes
from the content. Consider for example how you would get the runtime and
the language from the sample document. One way of achieving this would be
to select the ``div`` elements with the ``info`` class and using the ``h3``
text as the key. Such sections in the document can be specified using
``foreach`` specifications in the keys. This will cause a new item to be
generated for each of the selected elements. To get the key value, we can use
paths and reducers that will be applied to each selected element. In the below
example, the ``normalize`` reducer joins the strings, converts it to lowercase,
replaces spaces with underscores and strips other non-alphanumeric characters:

>>> items = [
...     {
...         "foreach": ".//div[@class='info']",
...         "key": {
...             "path": "./h3/text()",
...             "reducer": "normalize"
...         },
...         "value": {
...             "path": "./div//text()",
...             "reducer": "clean"
...         }
...     }
... ]
>>> extract(document, items)
{'language': 'English', 'runtime': '144 minutes'}

You could also give a string instead of a path and reducer for the key.
In this case, the elements would still be traversed; only the last one would
set the final value for the item. This could make sense if you are sure
that there is only one element that matches the ``foreach`` path of the key.

.. [#]
      This means that the query has to end with either ``text()`` or some
      attribute value as in ``@attr``. And the reducer function should be
      implemented so that it takes a list of strings and returns a string.

.. [#]
      This means that the ``foreach`` query has to select elements, therefore
      **not** end in ``text()`` or ``@attr``.
