# Amara XML Toolkit manual (for version 1.2)

User Manual

## Introduction

Amara XML Tools is a collection of Pythonic tools for XML data binding. Not just tools that happen to be written in Python, but tools built from the ground up to use Python idioms and take advantage of the many advantages of Python over other programming languages.

Amara builds on 4Suite, but whereas 4Suite focuses more on literal implementation of XML standards in Python, Amara adds a much more Pythonic face to these capabilities. The combination ensures standards compliance within expressive Python form.

The main component of Amara is:

- Bindery: a data binding tool (fancy way of saying it's a very Pythonic XML API)

Other components are:

- Scimitar: an implementation of the ISO Schematron schema language for XML, which converts Schematron files to Python scripts

- domtools: a set of tools to augment Python DOMs

- saxtools: a set of tools to make SAX easier to use in Python

## Getting started

The easiest way to get started is to install Amara and all its dependencies at a stroke using EasyInstall. Just type `easy_install amara` and you're all set. If you run into trouble you may not have EasyInstall. It's very easy to set up. If you still run into problems, please report them on the mailing list.

You might choose not to use EasyInstall. If so, grab the minimally required 4Suite-XML package, install that, then grab Amara and install that using using the usual `python setup.py install`, a Windows installer or some other method.

## Amara Bindery: XML as easy as py

The following example shows how to create a binding from a simple XML file, monty.xml. (All example files mentioned are available in the demo directory of the Amara package.)

First, the contents of monty.xml:

    <?xml version="1.0" encoding="utf-8"?>
    <monty>
      <python spam="eggs">
        What do you mean "bleh"
      </python>
      <python ministry="abuse">
        But I was looking for argument
      </python>
    </monty>

Now the code to create the binding:

    import amara
    doc = amara.parse('monty.xml')

`doc` is the data binding result, an object representing the XML. Since I fed the binder a full XML document, I get back an object representing the document itself which then has a member representing the top-level element.

It's that simple. In order to get the value "eggs" (as a Python Unicode object) you can write:

    doc.monty.python.spam

Or in order to get the element object with the contents "But I was looking for argument" you can write:

    doc.monty.python[1]

The `parse` function can be passed a file-like object (stream) as well as a string. There is also a `parse_path` function for parsing XML retrieved from local files and URLs.

You can pass `amara.parse` a string (not Unicode object) with the XML content, an open-file-like object, a file path or a URI.

### More complex example

The following example shows how to create a binding from an XBEL file (a popular XML format for bookmarks that was developed by Python's very own XML-SIG). There is a sample XBEL file you can use in the demo directory.

    doc = amara.parse('xbel.xml')

The following sample code prints the first and second bookmark titles:

    print doc.xbel.folder.bookmark.title
    print doc.xbel.folder.bookmark[1].title

Note that Bindery tries to make things as natural as possible. You can access child elements by just using their name, usually. If there is more than one with the same name it grabs the first one. You can use list indices to specify one of multiple child elements with the same name. Naturally, a more explicit way of getting the first bookmark's title is:

    print doc.xbel.folder.bookmark[0].title

`title` is an element containing only text. the expression `doc.xbel.folder.bookmark[0].title` returns the binding object representing the element. Such objects have unicode conversion methods so that you can get their descendant text (all text nodes in the element's subtree), which is printed in the above line. The following four lines are all equivalent:

    print doc.xbel.folder.bookmark.title
    print doc.xbel.folder.bookmark[0].title
    print unicode(doc.xbel.folder.bookmark.title)
    print unicode(doc.xbel.folder.bookmark[0].title)

Calling the `unicode` conversion on an element node in Amara is very similar to calling the `string` conversion on an element node in XPath.

The following snippet is a recursive function that prints out all bookmark URLs in the file:

    def all_titles_in_folder(folder):
        #Warning: folder.bookmark will raise an AttributeError if there are no bookmarks
        for bookmark in folder.bookmark:
            print bookmark.href
        if hasattr(folder, "folder"):
            #There are sub-folders
            for folder in folder.folder:
                all_titles_in_folder(folder)
        return

    for folder in doc.xbel.folder:
        all_titles_in_folder(folder)

You would probably not do this in actual usage, though. You can perform this task with much less code, without recursion, and with a speed boost using XPath, which is covered in a later section.

If you want to count the number of elements of the same name, use `len`.

    len(doc.xbel.folder)

This gives the number of top-level folders.

### The workings of Bindery

In the default binding XML elements turn into specialized objects. For each generic identifier (element name) a class is generated that is derived from `bindery.element_base`. Attributes become simple data members whose value is a Unicode object containing the attribute value.

Element objects are specially constructed so they can be treated as single objects (in which case the first child element of the corresponding name is selected, or one can use list item access notation or even iterate.

Going back to the example, `binding.xbel.folder.bookmark` is the same as `binding.xbel.folder.bookmark[0]`. Both return the first bookmark in the first folder. To get the second bookmark in the first folder, use `binding.xbel.folder.bookmark[1]`.

### Complex element children

Bindery by default preserves ordering information from the source XML. You can access this through the children list of element and document objects:

    folder.xml_children

Within the children list, child elements are represented using the corresponding binding objects, child text becomes simple Unicode objects. Notice that in the default binding text children are normalized, meaning that the binding will never place two text nodes next to each other in `xml_children`.

If an element node contains text as well as child elements, be aware of the how descendant text nodes are accessed. You can get the accumulated text children of an element using the `xml_child_text property`. Given the document `<a>1<b>2</b>3<c/></a>`, `a.xml_child_text` would return `u'13'`. On the other hand, converting to Unicode (`unicode(a)`) would return `u'123'`.

### Checking the structure of nodes

You can get some information about the structure of most bindery objects by calling the `xml_doc method`.

    >>> import amara
    >>> doc = amara.parse('monty.xml')
    >>> print doc.xml_doc()
    Object references based on XML child elements:
    monty (1 element) based on 'monty' in XML
    >>> print doc.monty.xml_doc()
    Object references based on XML child elements:
    python (2 elements) based on 'python' in XML
    >>> print doc.monty.python.xml_doc()
    Object references based on XML attributes:
    spam based on 'spam' in XML
    Object references based on XML child elements:
    >>> print doc.monty.python.spam
    eggs

This is human-readable material, for convenience, but you can also get node structure information in machine-readable form. `xml_properties` returns a dictionary whose keys represent the object reference names from the XML attributes and elements.

    >>> import amara
    >>> doc = amara.parse("<a x='1'>hello<b/>lovely<c/>world</a>")
    >>> doc.a.xml_properties
    {u'x': u'1', u'c': &lt;amara.bindery.c object at 0xb7bbcdcc>, u'b': &lt;amara.bindery.b object at 0xb7bbcb8c>}

`xml_child_elements` returns similar dictionary, but reduced to only the child element.

    >>> #Continuing from the above snippet
    >>> doc.a.xml_child_elements

{u'c': <amara.bindery.c object at 0xb7bbcdcc>, u'b': <amara.bindery.b object at 0xb7bbcb8c>}

### Writing XML back out

The preservation of ordering information means that Bindery does a pretty good job of allowing you to render binding objects back to XML form. Use the `xml` method for this.

    print doc.xml()

The `xml` method returns encoded text, not Unicode. The default encoding is UTF-8. You can also serialize a portion of the document.

    print doc.xbel.folder.xml() #Just the first folder

You can pass in a stream for the output:

    doc.xml(sys.stdout)

You can control such matters as the output encoding, whether the output is pretty-printed, whether there is an output XML declaration, etc. by using parameters that control the output, based on the XSLT output control attributes. As an example, if you pas in `omitXmlDeclaration=u"yes"` the output of the XML declaration is suppressed. Here are the other parameters you can set.

- encoding - the character encoding to use (default UTF-8). The writer will automatically use character entities where necessary.

- omitXmlDeclaration - "yes" to suppress output of the XML declaration. Default "no".

- standalone - "yes" to set standalone in the XML declaration.

- mediaType - sets the media type of the output. You'll probably never need this.

- cdataSectionElements - a list of element names whose output will be wrapped in a CDATA section. This can provide for friendlier output in some cases.

You can force Amara to add namespace declarations to an element (or to the root node) even if it's not associated with an element or attribute. This is especially useful when dealing with QNames in content (A.K.A. hidden namespaces).

    doc.xml(force_nsdecls={u'x': u'http://example.com/'})

`force_nsdecls` is a dictionary with unicode prefix as the key and unicode namespace URI as the value. This makes it easy, for example, to force all the known namespace declarations to be at the top (a "sane" document).

    doc.xml(force_nsdecls=doc.prefixes)

Warning: namespace declarations in the document always trump those forced in this way. This includes the implicit declaration of the default namespace to null. In other words you cannot turn a document that doesn't use namespaces into one with a default namespace in one easy stroke as above. You would have to go through and set the `namespaceURI` of each element in the document.

### XPath

Bindery supports an XPath subset that covers almost all of that standard. The XPath features that are not supported are very rare, and for pracctical purposes you can assume it's complete XPath support. The folowing example retrieves all top-level folders:

    tl_folders = doc.xbel.xml_xpath(u'folder')
    for folder in tl_folders:
        print folder.title

You invoke the `xml_xpath` method on the object you wish to serve as the context for the XPath query. To get the first element child (regardless of node name) of the first bookmark of the first folder, use:

    doc.xbel.folder.bookmark.xml_xpath(u'*[1]')

or

    doc.xbel.xml_xpath(u'folder[1]/bookmark[1]/*[1]')

or

    doc.xbel.xml_xpath(u'/folder[1]/bookmark[1]/*[1]')

or

    doc.xml_xpath(u'xbel/folder[1]/bookmark[1]/*[1]')

etc.

Warning: in Python, lists indices start with 0 while they start with 1 in XPath.

Notice: this XPath returns a node set, rendered in Python as a list of nodes. It happens to be a list of one node, but you still have to extract it with [0].

The return value depends on the XPath expression (expr)

- If expr reults in an XPath string, the return value is a Python Unicode object

- If expr reults in an XPath number, the return value is a Python float

- If expr reults in an XPath boolean, the return value is a Python bool object

- If expr reults in an XPath node set, the return value is a Python list (always a list, even if it is empty or contains only one node)

The following example prints out all bookmark URLs in the file, but is much simpler and more compact than the equivalent code earlier in this document:

    bookmarks = doc.xml_xpath(u'//bookmark')
    for bookmark in bookmarks:
        print bookmark.href

The following just returns all hrefs wherever they appear in the document, using an attribute query:

    hrefs = doc.xml_xpath(u'//@href')
    for href in hrefs:
        print unicode(href)

The following prints the title of the bookmark for the 4Suite project:

    url = u"http://4suite.org/"
    title_elements = doc.xml_xpath('//bookmark[@url="%s"]/title'%url)
    #XPath node set expression always returns a list
    print unicode(title_elements[0])

### Applying a transform (XSLT) to a bindery node

You can apply XSLT directly to a bindery document or element. You could of course `xml()` to serialize it and apply XSLT against that, but it's probably easier and a more efficient to work directly on the node. To do so, use the `xml_xslt()` method. Warning: This function is quite limited, and will only handle the simplest transforms. If you find your transform does not work with it, serialize using xml() then use Ft.Xml.Xslt.transform, which is fully XSLT compliant.

    result_string = doc.xml_xslt('transform.xslt')

The one required argument is a reference to a transform. It can be a a string (not Unicode object), file-like object (stream), file path, URI or an Ft.Xml.InputSource.InputSource instance. If string or stream it must be self-contained XML (i.e. not requiring access to any other resource such as external entities or includes).

The optional `param` argument is a dictionary of stylesheet parameters, the keys of which may be given as unicode objects if they have no namespace, or as (uri, localname) tuples if they do.

The optional `output` argument is a file-like object to which output is written (incrementally, as processed).

See the following examples:

    params = params={u'id' : u'2', (u'http://example.com/ns', u'tags') : [u'a', u'b', u'c']}
    result_string = doc.xml_xslt('transform.xslt', params=params)

This would execute the XSLT with overridden values for the `id` (unprefixed) top level parameter and for the `x:tags` top level parameter where x is a prefix for the `http://example.com/ns` namespace. The latter case provides a node set of text nodes as the parameter.

    doc.xml_xslt(TRANSFORM_STRING, output=open('/tmp/foo.txt', 'w'))

Would execute the XSLT provided in the string, with output streaming into teh file `/tmp/foo.txt` (remember that XSLT output can be text, XML or HTML). There is no return value since `output` is specified.

    result_string = doc.xml_xslt(open('transform.xslt'))

Read the transform from an open file object.

### Namespaces

Bindery supports documents with namespaces. The following example displays a summary of the contents of an RSS 1.0 feed:

    import amara

    #Set up customary namespace bindings for RSS
    #These are used in XPath query and XPattern rules
    RSS10_NSS = {
        u'rdf': u'http://www.w3.org/1999/02/22-rdf-syntax-ns#',
        u'dc': u'http://purl.org/dc/elements/1.1/',
        u'rss': u'http://purl.org/rss/1.0/',
        }

    doc = amara.parse('rss10.rdf', prefixes=RSS10_NSS)

    #Create a dictionary of RSS items
    items = {}
    item_list = doc.xml_xpath(u'//rss:item')
    items = dict( [ ( item.about, item) for item in item_list ] )
    print items

    for channel in doc.RDF.channel:
        print "Channel:", channel.about
        print "Title:", channel.title
        print "Items:"
        for item_ref in channel.items.Seq.li:
            item = items[item_ref.resource]
            print "\t", item.link
            print "\t", item.title

The following illustrates how namespace details are maintained in bindery objects:

    #Show the namespace particulars of the rdf:RDF element
    print doc.RDF.namespaceURI
    print doc.RDF.localName
    print doc.RDF.prefix

For attributes, `xml_attributes` is a dictionary containing namespace information for all of an element's attributes. See the Attributes section below for more details.

Namespaces work naturally with XPath as well:

    #Get the RSS item with a given URL
    item_url = u'http://www.oreillynet.com/cs/weblog/view/wlg/532'
    matching_items = doc.RDF.xml_xpath(u'//rss:item[@rdf:about="%s"]'%item_url)
    print matching_items
    assert matching_items[0].about == item_url

In the above example I manually set the mapping from namespace prefixes to namespace names, but if your use of XML namespaces is "sane", you might not need this step. Bindery document objects automatically remember the namespace declarations made on the top level element of any document you parse.

### Naming things

The Python object reference and class names used in Amara bindings are based on the corresponding XML IDs, but there are limits to such mappings. For one thing, XML allows characters that are not allowed in Python IDs, such as `-`. In such cases Amara mangles the name, using underscores by default. In the case of a document such as `<a-1 b-1=""/>` you would access the element using `doc.a_1`, and the attribute using `doc.a_1.b_1`.

This can be problematic when writing generic code to address XML, because any element's name might be mangled in a different way depending on document context. You always have the option of accessing objects using their proper XML names in XPath:

`doc.xml_xpath(u"a-1/b-1)`

You can also use Python mapping protocol to access objects:

`doc[u'a-1'][u'b-1']`

This is equivalent to:

`doc[None, u'a-1'][None, u'b-1']`

The `None` is the namespace name, which of course you should specify as a Unicode object if the desired object is in fact in a namespace in the original document. Amara allows one further level of disambiguation:

`from xml.dom import Node E = Node.ELEMENT_NODE doc[E, None, u'a-1'][E, None, u'b-1']`

This allows you to handle the very rare situation where an attribute name clashes with that of a child element of its owner. For example, in `<a-1 b-1=""><b-1/></a-1>` you can access the element `b-1` as above, and the attribute of the same name using `doc[E, None, u'a-1'][A, None, u'b-1']`, assuming you have `A = Node.ATTRIBUTE_NODE`.

You can get the original XML name information from any object using properties named after DOMs:

- `element.nodeName`

- `element.localName`

- `element.namespaceURI`

- `element.prefix`

And you can get similar information on attributes by reading the `xml_attributes` dictionary on the element node.

### Push binding

If you're dealing with a large XML file, you may not want the entire data binding in memory at the same time. You may want to instantiate it bit by bit. If you have a clear pattern for how you want to break up the document, you can use the function `amara.pushbind`. See the following example:

    import amara

    for folder in amara.pushbind('xbel.xml', u'/xbel/folder'):
        title = folder.title
        bm_count = len(list(folder.bookmark))
        print "Folder", title, "has", bm_count, "top level bookmarks"

The neat thing is that this program doesn't have the entire binding in memory at time. In each iteration it loads just enough XML to represent each top-level folder element. `pushbind` is a generator that yields each little subtree each time it's invoked.

In general `pushbind` subtrees are Amara bindery objects based on the XSLT pattern that is passed in as its first argument. You also pass in an XML source, either a string (use the `string= keyword`) or a URI or file-name (use the `source= keyword`).

An interactive session helps illustrate:

    >>> XML="""\
    ... <doc>
    ...   <one><a>0</a><a>1</a></one>
    ...   <two><a>10</a><a>11</a></two>
    ... </doc>
    ... """
    >>> import amara
    >>> chunks = amara.pushbind(XML, u'a')
    >>> a = chunks.next()
    >>> print a
    0
    >>> print a.xml()
    <a>0</a>
    >>> a = chunks.next()
    >>> print a.xml()
    <a>1</a>
    >>> a = chunks.next()
    >>> print a.xml()
    <a>10</a>
    >>> a = chunks.next()
    >>> print a.xml()
    <a>11</a>
    >>> a = chunks.next()
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    StopIteration

In the above case the XSLT pattern "a" matched all the elements with this name in the file. All the XML outside the "a" elements is essentially discarded by this code, so you have to tailor the XSLT pattern to include all the material you're interested in. In the following example, only the first two "a" elements are included in the generator yield.

    >>> chunks = amara.pushbind(XML, u'one/a')
    >>> a = chunks.next()
    >>> print a.xml()
    <a>0</a>
    >>> a = chunks.next()
    >>> print a.xml()
    <a>1</a>
    >>> a = chunks.next()
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    StopIteration

The pattern `/doc/one/a` would result in similar behavior for the sample document.

You can use namespaces in these patterns by using prefixes defined in a dictionary passed to `pushbind`.

    import amara
    #Set up customary namespace bindings for RSS
    #These are used in XPath query and XPattern rules
    RSS10_NSS = {
    u'rdf': u'http://www.w3.org/1999/02/22-rdf-syntax-ns#',
    u'dc': u'http://purl.org/dc/elements/1.1/',
    u'rss': u'http://purl.org/rss/1.0/',
    }

    #Print out all titles in the RSS feed
    items = amara.pushbind('demo/rss10.rdf', u'rss:item',
                                   prefixes=RSS10_NSS)
    for item in items:
        print item.title
    #Print out all titles in the RSS feed, slightly different approach
    chunks = amara.pushbind('demo/rss10.rdf', u'rss:item/rss:title',
                                   prefixes=RSS10_NSS)
    for title in chunks:
        print title

### Modification

Modifying a binding is pretty straightforward. You can create or modify an attribute by simple assignment:

    import amara
    doc = amara.parse('monty.xml')
    doc.monty.foo = u'bar'
    doc.monty.spam = u'[attr modified]'

You can also replace an element's contents with a single text node, using similar notation:

    doc.monty.python = u'[elem 1 modified]\n'
    doc.monty.python[1] = u'[elem 2 modified]\n'

The resut of the above code is:

    <monty>
      <python spam="[attr modified]" foo="bar">[elem 1 modified]
    </python>
      <python ministry="abuse">[elem 2 modified]
    </python>
    </monty>

You can empty out all children of an element:

    doc.monty.python.xml_clear()

And add new elements and text. Create a new, empty element named `new` and append it as the last childof the first `python` element.

    doc.monty.python.xml_append(doc.xml_create_element(u'new'))

Append a text node.

    doc.monty.python.new.append(u'New Content')

You can insert new elements or text into specific locations, either before or after an existing child.

    #Create a new `python` element as the second element child of `monty`
    doc.monty.xml_insert_after(doc.monty.python,
                               doc.xml_create_element(u'python'))
    #Create a new `python` element as the first element child of `monty`
    doc.monty.xml_insert_before(doc.monty.python,
                                doc.xml_create_element(u'python'))

You can also delete a specific element, attribute, or text child:

    del doc.monty.python

which does the same thing as

    del doc.monty.python[0]

You can also use the `xml_remove_child` method:

    child = doc.monty.python
    doc.monty.xml_remove_child(child)

Just to exhaustively list the approaches to deletion, you can use the position of an object among its siblings to remove it from its parent. You can get this position using the `xml_index_on_parent` property.

    ix = doc.monty.python.xml_index_on_parent

This assigns ix the value 1 since the first python element is the second child of monty.

    ix = doc.monty.python[1].xml_index_on_parent

This assigns `ix` the value 3 since the second python element is the fourth child of `monty`. Once you have an index, you can use that index in order to delete a specific child by passing it to the `xml_remove_child_at` method:

    doc.monty.xml_remove_child_at(3)

This removes the second python element, using the index determined above. This works for text as well:

    doc.monty.xml_remove_child_at(0)

Removes the first text node. You can omit the index in the call to this method. By default it removes the last child:

    doc.monty.xml_remove_child_at()

If you run into the sort of identifier naming issues mentioned in the "Naming Things" section you can use the mapping protocol to make modifications. For example to remove the "b-1" child element from

    <a-1 b-1=""><b-1/></a-1>

One option is:

    from xml.dom import Node
    E = Node.ELEMENT_NODE
    del doc[E, None, u'a-1'][E, None, u'b-1']

### Creating elements (and attributes)

You can create a new element with a namespace:

    e = doc.xml_create_element(element_qname, element_ns)

which is equivalent to

    e = doc.xml_create_element(element_qname, namespace=element_ns)

You can easily add attributes while you're creating elements.

    import amara
    doc = amara.parse('monty.xml')
    #Create a third python element
    e = doc.xml_create_element(u'python', attributes={u'life': u'brian'})
    doc.monty.xml_append(e)
    print doc.xml()

This gives the following output:

    <?xml version="1.0" encoding="UTF-8"?>
    <monty>
      <python spam="eggs">
        What do you mean "bleh"
      </python>
      <python ministry="abuse">
        But I was looking for argument
      </python>
    <python life="brian"/></monty>

You can see how the attribute is manifested on the new element.

The dictionary `attributes={u'life': u'brian'}` is actually a short-hand. The full form is:

    {
      (<attr1_qname>, <attr1_namespace>): attr1_value},
      (<attr2_qname>, <attr2_namespace>): attr2_value},
      ...
      (<attrN_qname>, <attrN_namespace>): attrN_value},
    }

But you can abbreviate `(<attrN_qname>, <attrN_namespace>)` to just `<attrN_qname>` if the namespace is None (i.e. the attribute is not in a namespace).

So you could add a namespace qualified attribute as follows:

    import amara
    NS = u'urn:bogus'
    doc = amara.parse('monty.xml')
    #Create a third python element
    e = doc.xml_create_element(
         u'python',
         attributes={(u'ns:life', NS): u'brian'},
         content=u'unfortunate'
         )
    doc.monty.xml_append(e)
    print doc.xml()

Notice that this time I threw in some content as well. The result:

    <?xml version="1.0" encoding="UTF-8"?>
    <monty>
      <python spam="eggs">
        What do you mean "bleh"
      </python>
      <python ministry="abuse">
        But I was looking for argument
      </python>
    <python xmlns:ns="urn:bogus" ns:life="brian">unfortunate</python></monty>

### Modifying attributes

To create an attribute after the element is created, use the `xml_set_attribute` method. To add the namespace qualified attribute from the example above, use the following:

    import amara
    NS = u'urn:bogus'
    doc = amara.parse('monty.xml')
    #Create a third python element
    e = doc.xml_create_element(
              u'python',
              content=u'unfortunate'
              )
    doc.monty.xml_append(e)
    doc.monty.python[2].xml_set_attribute((u'ns:life', NS), u'brian')
    print doc.xml()

This produces the same XML as above. If no namespace is required, the first argument to `xml_set_attribute` is just the attribute name.

The `xml_set_attribute` method returns the name of the resulting attribute.

You can set an attribute's value using Python idiom as well:

    doc.monty.python[2].life = u'Pi'

Information about an element's attributes, including namespace information, is kept in `xml_attributes`. This is a dictionary keyed by the local name of each attribute, with the value being a tuple of the namespace qualified attribute name and the namespace URL. For example, given the following code:

    import amara
    NS = u'urn:bogus'
    doc = amara.parse('monty.xml')
    #Create a third python element
    e = doc.xml_create_element(
      u'python',
      content=u'unfortunate'
    )
    doc.monty.xml_append(e)
    #Add a namespace qualified attribute
    doc.monty.python[2].xml_set_attribute((u'ns:life', NS), u'brian')
    #Add an attribute with no namespace
    doc.monty.python[2].xml_set_attribute(u'foo', u'bar')

`doc.monty.python[2].xml_attributes` gives the value:

    {u'life': (u'ns:life', u'urn:bogus'), u'foo': (u'foo', None)}

Just for fun, here's an interesting variation that illustrates the special status of the XML namespace.

    import amara
    from xml.dom import XML_NAMESPACE as XML_NS
    doc = amara.parse('monty.xml')
    #Create a third python element
    e = doc.xml_create_element(
              u'python',
              attributes={(u'xml:lang', XML_NS): u'en'},
              content=u'Ni!'
              )
    doc.monty.xml_append(e)
    print doc.xml()

This gives the following output:

    <?xml version="1.0" encoding="UTF-8"?>
    <monty>
      <python spam="eggs">
        What do you mean "bleh"
      </python>
      <python ministry="abuse">
        But I was looking for argument
      </python>
    <python xml:lang="en">Ni!</python></monty>

Notice that there's no declaration for the XML namespace, as allowed by the standard.

### Appending XML fragments

Amara offers a powerful facility for adding to XML documents through the `xml_append_fragment` method on elements. You pass this method a string (not Unicode, since this is a parse operation) with a fragment of literal XML which is parsed and added to the element. The XML fragment must be a well formed external parsed entity. Basically multiple root elements are allowed, but they must be properly balanced, special characters escaped, and so on. Doctype declaration is prohibited. According to XML rules, the encoded string is assumed to be UTF-8 or UTF-16, but you can override this with an XML text declaration (`<?xml version="1.0" encoding="ENC"?>`) or by passing in an encoding parameter to this function.

    import amara
    doc = amara.parse('monty.xml')
    doc.monty.xml_append_fragment('<py3 x="1">p</py3><py4 y="2">q</py4>')
    print doc.monty.xml_child_elements.keys()

The output is `[u'python', u'py3', u'py4']`, as two elements are added in the `xml_append_fragment` call.

The optional encoding is a string with the encoding to be used in parsing the XML fragment. If this parameter is specified, it overrrides any text declaration in the XML fragment.

    #Latin-1 ordinal 230 is the a-e ligature
    doc.monty.xml_append_fragment('<q>P%an</q>'%chr(230), 'latin-1')

### Creating full documents

You can create entire documents from scratch

    doc = amara.create_document()
    doc.xml_append(doc.xml_create_element(u"hello"))
    doc.xml()

Yields

    <?xml version="1.0" encoding="UTF-8"?>
    <hello/>

You can specify a root element to be created in the empty document. The following code:

    doc = amara.create_document(u"hello")
    doc.hello.xml_append(doc.xml_create_element(u"world"))
    doc.xml()

Yields

    <?xml version="1.0" encoding="UTF-8"?>
    <hello><world/></hello>

This is equivalent to:

    doc = amara.create_document()
    doc.xml_append(doc.xml_create_element(u"hello"))
    doc.hello.xml_append(doc.xml_create_element(u"world"))
    doc.xml()

You can set a namespace on the root element using the `ns` parameter. There are many other useful parameters. Try `import amara; help(amara.create_document)` at a Python prompt for more details.

### Making deep copies of nodes

If you're familiar with XSLT, you might wonder how to do the equivalent of xsl:copy-of in Amara. You can create such a deep copy using the `copy` module in the Python standard library.

    import copy
    import amara
    doc = amara.parse('monty.xml')
    #Clone the document
    doc2 = copy.deepcopy(doc)
    #This modification only affects the clone
    doc2.monty.python.spam = u"abcd"

    #This output will just like what was read in
    print doc.xml()
    #This output will show the change from "eggs" to "abcd"
    print doc2.xml()

### Processing instructions and comments

Bindery supports these XML constructs in a fairly natural way. If you have PIs or comments in a source document parsed into a binding, it will have objects representing the PIs and comments:

    import amara
    DOC = """\
    <?xml-stylesheet url="xxx.css" type="text/css"?>
    <!--A greeting for all-->
    <hello-world/>
    """
    doc = amara.parse(DOC)
    print doc.xml_children

shows the following list:

    [
    <amara.bindery.pi_base instance at 0x433f6c>,
    <amara.bindery.comment_base instance at 0x433fac>,
    <amara.bindery.hello_world object at 0x433fec>
    ]

The first item is a bound PI object and the second a bound comment. You can dig deeper if you like:

    pi = doc.xml_children[0]
    comment = doc.xml_children[1]
    print repr(pi.target) #shows u'xml-stylesheet'
    print repr(pi.data) #shows u'url="xxx.css" type="text/css"'
    print repr(comment.data) #shows u'A greeting for all'

You can also create or mutate these objects.

    doc = amara.create_document()
    pi = bindery.pi_base(u"xml-stylesheet", u'url="xxx.css" type="text/css"')
    doc.xml_append(pi)
    doc.xml_append(doc.xml_create_element(u"A"))

### Document types

There is also an API for document type declarations (DTDecls).

To create a document with a DTDecl, do something like the following:

    doc = amara.create_document(
        u"xsa",
        pubid=u"-//LM Garshol//DTD XML Software Autoupdate 1.0//EN//XML",
        sysid=u"http://www.garshol.priv.no/download/xsa/xsa.dtd"
        )

Which results in a tree equivalent to:

    <?xml version="1.0" encoding="utf-8"?>
    <!DOCTYPE xsa PUBLIC "-//LM Garshol//DTD XML Software Autoupdate 1.0//EN//XML"
                         "http://www.garshol.priv.no/download/xsa/xsa.dtd">
    <xsa/>

Notice how this automatically creates the document element for you (no need for a separate `xml_append`). You can expand on this to create attributes and content for the document element:

    from xml.dom import XML_NAMESPACE as XML_NS

    doc = amara.create_document(
        u"xsa",
        attributes={(u'xml:lang', XML_NS): u'en'},
        content=u' ',
        pubid=u"-//LM Garshol//DTD XML Software Autoupdate 1.0//EN//XML",
        sysid=u"http://www.garshol.priv.no/download/xsa/xsa.dtd"
        )

Which results in:

    <?xml version="1.0" encoding="utf-8"?>
    <!DOCTYPE xsa PUBLIC "-//LM Garshol//DTD XML Software Autoupdate 1.0//EN//XML"                     "http://www.garshol.priv.no/download/xsa/xsa.dtd">
    <xsa xml:lang="en"> </xsa>

You can then access the DTDecl details:

    assert doc.xml_pubid == u"-//LM Garshol//DTD XML Software Autoupdate 1.0//EN//XML"
    assert doc.xml_sysid == u"http://www.garshol.priv.no/download/xsa/xsa.dtd"
    assert doc.xml_doctype_name == u"xsa"

#### Future doctype support from parse

The idea is that if you parse a document with a DTDecl, the root node contains the document element QName, the public ID and the system ID. This does not work yet because of idiosyncracies of the Python/XML libraries. This should be fixed when I remove dependencies on these libraries in a coming version. When it does work, you should be able to do:

    #DOES NOT YET WORK

    XSA = """\
    <?xml version="1.0" encoding="utf-8"?>
    <!DOCTYPE xsa PUBLIC "-//LM Garshol//DTD XML Software Autoupdate 1.0//EN//XML"                     "http://www.garshol.priv.no/download/xsa/xsa.dtd">
    <xsa>
      <vendor>
        <name>Fourthought, Inc.</name>
        <email>info@fourthought.com</email>
        <url>http://fourthought.com</url>
      </vendor>
      <product id="FourSuite">
        <name>4Suite</name>
        <version>1.0a1</version>
        <last-release>20030327</last-release>
        <info-url>http://4suite.org</info-url>
        <changes>
     - Begin the 1.0 release cycle
        </changes>
      </product>
    </xsa>
    """
    doc = amara.parse(XSA)
    assert doc.xml_pubid == u"-//LM Garshol//DTD XML Software Autoupdate 1.0//EN//XML"
    assert doc.xml_sysid == u"http://www.garshol.priv.no/download/xsa/xsa.dtd"
    assert doc.xml_doctype_name == u"xsa"

Internal DTD subset constructs are not preserved in the binding.

### Customizing the binding

Bindery works by iterating over XML nodes and firing off a set of rules triggered by the node type and other details. The default binding is the result of the default rules that are registered for each node type, but Bindery makes this easy to tweak by letting you register your own rules.

Bindery comes bundled with 3 easy rule frameworks to handle some common binding needs.

#### Treating some elements as simple string values

The title elements in XBEL are always simple text, and creating full Python objects for them is overkill in most cases. They could be just as easily simple data members with the Unicode value of the element's content. To make this adjustment in Bindery, register an instance of the `simple_string_element_rule` rule. This rule takes an list of XSLT pattern expressions which indicate which elements are to be simplified. So to simplify all title elements:

    import amara
    from amara import binderytools
    #Specify (using XSLT patterns) elements to be treated similarly to attributes
    rules = [
        binderytools.simple_string_element_rule(u'title')
        ]
    #Execute the binding
    doc = amara.parse('xbel.xml', rules=rules)

    #title is now simple unicode
    print doc.xbel.folder.bookmark.title.__class__

#### Omitting certain elements entirely

Perhaps you want to focus on only part of a document, and to save memory and hassle, you want to omit certain elements that are not of interest in the binding. You can use the `omit_element_rule` in this case.

The following example does not create bindings for folder titles at all (but bookmark titles are preserved):

    import amara
    from amara import binderytools
    #Specify (using XSLT patterns) elements to be ignored
    rules = [
        binderytools.omit_element_rule(u'folder/title')
        ]
    #Execute the binding
    doc = amara.parse('xbel.xml', rules=rules)

    #Following would now raise an exception:
    #print doc.xbel.folder.title

#### Omitting certain node types entirely

You can also omit processing instructions, comments or text nodes from a document using the `omit_nodetype_rule` rule. The rule takes a single parameter which can be one of the three node types `xml.dom.Node.COMMENT_NODE`, `xml.dom.Node.PROCESSING_INSTRUCTION_NODE` or `xml.dom.Node.TEXT_NODE`.

In the following example there are no bindings created for comments:

    import amara
    from amara import binderytools
    from xml.dom import Node

    DOCUMENT = """<!-- Printing the best language --> <language>Python</language>"""

    rules = [binderytools.omit_nodetype_rule(Node.COMMENT_NODE)]

    doc = amara.parse(DOCUMENT, rules = rules)

    print doc.xml()

With the resulting output:

    <?xml version="1.0" encoding="UTF-8"?>
    <language>Python</language>

#### Stripping whitespace

A common need is to strip out pure whitespace nodes so that they don't clutter up "children" lists. Bindery bundles the `ws_strip_element_rule` rule for this purpose. Elements that match the pattern are stripped of whitespace.

    import amara
    from amara import binderytools
    #Specify (using XSLT patterns) elements to be stripped
    #In this case select all top-level elements for stripping
    rules = [
        binderytools.ws_strip_element_rule(u'/*')
        ]
    #Execute the binding
    doc = amara.parse('xbel.xml', rules=rules)

You can combine rules, such as stripping white space while still omitting certain elements.

#### Creating an element skeleton

If all you care about is the structure of elements and attributes, and not the text content you can use the `element_skeleton_rule`.

Elements that match the pattern have all character data stripped.

    import amara
    from amara import binderytools
    #Specify (using XSLT patterns) elements to be bound as skeletons
    #In this case select all elements
    rules = [
        binderytools.element_skeleton_rule(u'*')
        ]
    #Execute the binding
    doc = amara.parse('xbel.xml', rules=rules)

#### The type inferencer

The basic idea behind data binding is translating XML into native data types. Amara provides a rule that looks at each XML node to see if it can infer a native Python data type for the value, in particular int, float or datetime.

    TYPE_MIX = """\
    <?xml version="1.0" encoding="utf-8"?>
    <a a1="1">
      <b b1="2.1"/>
      <c c1="2005-01-31">
        <d>5</d>
      <e>2003-01-30T17:48:07.848769Z</e>
      </c>
      <g>good</g>
    </a>"""

    import amara
    from amara import binderytools
    rules=[binderytools.type_inference()]
    doc = amara.parse(TYPE_MIX, rules=rules)
    doc.a.a1 == 1     #type int
    doc.a.b.b1 == 2.1 #type float
    doc.a.c.c1 == datetime.datetime(2005, 1, 31) #type datetime.

#### Using namespaces in custom rules

The built-in custom rules use XSLT patterns, which use prefixes to specify namespaces. You may have to let the binder tools know what namespace bindings are in effect:

    import amara
    from amara import binderytools
    #Set up customary namespace bindings for RSS
    #These are used in XPath query and XPattern rules
    RSS10_NSS = {
        'rdf': 'http://www.w3.org/1999/02/22-rdf-syntax-ns#',
        'dc': 'http://purl.org/dc/elements/1.1/',
        'rss': 'http://purl.org/rss/1.0/',
        }

    rules = [
        binderytools.simple_string_element_rule(u'title')
        ]
    #Execute the binding
    doc = amara.parse('rss10.rdf', prefixes=RSS10_NSS, rules=rules)

Remember, however, as dicussed above, if you declare all the namespaces you use on top-level elements, you do not need to repeat them in the explicit `prefixes` dictionary.

### Push binding and rules

You can also use rules with pushbind. If all you ever want from "a" elements is to extract the text content, you could do something like the following. Look carefully: the sample document is slightly different this time.

    >>> XML="""\
    ... <doc>
    ...   <one><a>0</a><b>1</b></one>
    ...   <two><a>10</a><b>11</b></two>
    ... </doc>
    ... """
    >>> import amara
    >>> from amara import binderytools
    >>> #This rule says "treat all elements at the third level of depth as simple strings"
    >>> rule = binderytools.simple_string_element_rule(u'/*/*/*')
    >>> #Push back bindings of all second level elements ('one' and 'two')
    >>> chunks = amara.pushbind(XML, u'/*/*', rules=[rule])
    >>> elem = chunks.next()
    >>> print elem.a.__class__
    <type 'unicode'>
    >>> print elem.a
    u'0'
    >>> print elem.b.__class__
    <type 'unicode'>
    >>> print elem.b
    u'1'

### Using custom binding classes

If you need more sophisticated tweaking, you proably want to register your own customized binding class. The following example gives bookmark elements a method, `retrieve`, which retrieves the body of the Web page:

    import urllib
    from xml.dom import Node
    import amara
    from amara import bindery

    #Subclass from the default binding class
    #We're adding a specialized method for accessing a bookmark on the net
    class specialized_bookmark(bindery.element_base):
        def retrieve(self):
            try:
                stream = urllib.urlopen(self.href)
                content = stream.read()
                stream.close()
                return content
            except IOError:
                import sys; sys.stderr.write("Unable to access %s\n"%self.href)

    doc = amara.parse(XML, binding_classes={(None, u'bookmark'): specialized_bookmark})

    #Show specialized instance
    print doc.xbel.folder.bookmark.__class__

    #Exercise the custom method
    print "Content of first bookmark:"
    print doc.xbel.folder.bookmark.retrieve()

Focus on the line:

    doc = amara.parse(XML, binding_classes={(None, u'bookmark'): specialized_bookmark})

When you register classes to use in binding a given elements type you do so by specifying namespace URI and local name of the element. If you know that the element is not in a namespace, as in the XBEL example, you use `None`. `None` is the Right Way to signal "not in a namespace" in most Python/XML tools, and not the empty string `""`.

See also the `accesstrigger.py` demo.

#### General warning about customized bindings

Bindery tries to manage things so that writing back the XML from a binding makes sense, and that XPath gives expected results, but it is easy to bring about odd results if you customize the binding.

As an exampe, if you use `simple_string_element_rule` and then reserialize using the `xml` method, the elements that were simplified will be written back out as XML attributes rather than child elements. If you do run into such artifacts after customizing a binding the usual remedy is to write a custoized `xml` method or add specialized XPath wrapper code (see `bonderyxpath.xpath_wrapper_mixin` for the default XPath wrappering).

### Bindery extension guide

Bindery is designed to be extensible, but this is not a simple proposition given the huge flexibility of XML expression, and the many different ways developpers might want to generate resulting Python objects (and vice versa). You can pretty much do whatever you need to by writing Bindery extensions, but in order to keep things basically manageable, there are some ground rules.

#### Bindery laws:

1. Binding objects corresponding to an XML document have a single root object from which all other objects can be reached through navigating attributes (no, fancy method calls don't count)

[TODO: more on this section to come. If you try tweaking bindery extensions and have some useful notes, please pitch in by sending them along.]

## Scimitar: the most flexible schema language for the most flexible programming language

Scimitar is an implementation of ISO Schematron that compiles a Schematron schema into a Python validator script.

Scimitar supports all of the draft ISO Schematron specification. See the TODO file for known gaps in Scimitar convenience.

You typically use scimitar in two phases. Say you have a schematron schema schema1.stron and you want to validate multiple XML files against it, instance1.xml, instance2.xml, instance3.xml.

First you run schema1.stron through the scimitar compiler script, scimitar.py:

    scimitar.py schema1.stron

A file, schema1.py (same file stem with the "py" extension sunstituted), is generated in the current working directory. If you'd prefer a different location or file name, use the "-o" option. The generated file is a validator script in Python. It checks the schematron rules specified in schema1.stron.

You now run the generated validator script on each XML file you wish to validate:

    python schema1.py instance1.xml

The validation report is generated on standard output by default, or you can use the "-o" option to redirect it to a file.

The validation report is an XML external parsed entity, in other words a file that is much like a well-formed XML document, but with some restrictions loosened so that it's effectively text with possible embedded tags.

To elaborate using the example from the schematron 1.5 specification:

    $ cat simple1.stron
    <?xml version="1.0" encoding="UTF-8"?>
    <sch:schema xmlns:sch="http://www.ascc.net/xml/schematron" version="ISO">
     <sch:title>Example Schematron Schema</sch:title>
     <sch:pattern>
       <sch:rule context="dog">
        <sch:assert test="count(ear) = 2"
        >A 'dog' element should contain two 'ear' elements.</sch:assert>
        <sch:report test="bone"
        >This dog has a bone.</sch:report>
       </sch:rule>
      </sch:pattern>
    </sch:schema>

    $ scimitar.py simple1.stron
    $ ls simple*.py
    simple1-stron.py
    $ cat instance2.xml
    <dog><ear/></dog>

    $ python simple1-stron.py instance2.xml
    <?xml version="1.0" encoding="UTF-8"?>
    Processing schema: Example Schematron Schema

    Processing pattern: [unnamed]

    Assertion failure:
    A 'dog' element should contain two 'ear' elements.

## Amara DOM Tools: giving DOM a more Pythonic face

DOM came from the Java world, and hardly the most Pythonic API possible (see the Bindery above for a good step forward). Some DOM-like implementations such as 4Suite's Domlettes mix in some Pythonic idiom. Amara DOM Tools goes even further in this regard.

### The pushdom

You're probably familiar with `xml.dom.pulldom`, which offers a nice hybrid between SAX and DOM, allowing you to efficiently isolate important parts of a document in a SAX-like manner, and then using DOM for finer-grained manipulation. Amara's pushdom makes this process even more convenient You give it a set of XPatterns, and it provides a generator yielding a series of DOM chunks according to the patterns.

In this way you can process huge files with very little memory usage, but most of the convenience of DOM.

    for docfrag in domtools.pushdom('demo/labels.xml', u'/labels/label'):
        label = docfrag.firstChild
        name = label.xpath('string(name)')
        city = label.xpath('string(address/city)')
        if name.lower().find('eliot') != -1:
            print city.encode('utf-8')

Prints "Stamford".

See also this XML-DEV message.

### Generator tools

For more on the generator tools see the article "Generating DOM Magic".

### Getting an XPath for a given node

`domtools.abs_path` allows you to get the absolute path for a node. The following code:

    from amara import domtools
    from Ft.Xml.Domlette import NonvalidatingReader
    from Ft.Lib import Uri
    file_uri = Uri.OsPathToUri('labels.xml', attemptAbsolute=1)
    doc = NonvalidatingReader.parseUri(file_uri)

    print domtools.abs_path(doc)
    print domtools.abs_path(doc.documentElement)
    for node in doc.documentElement.childNodes:
        print domtools.abs_path(node)

Displays:

    /
    /labels[1]
    /labels[1]/text()[1]
    /labels[1]/label[1]
    /labels[1]/text()[2]
    /labels[1]/label[2]
    /labels[1]/text()[3]
    /labels[1]/label[3]
    /labels[1]/text()[4]
    /labels[1]/label[4]
    /labels[1]/text()[5]

For more on abs_path tools see the article "Location, Location, Location".

## Amara SAX Tools: SAX without the brain explosion

Tenorsax (`amara.saxtools.tenorsax`) is a framework for "linerarizing" SAX logic so that it flows a bit more naturally, and needs a lot less state machine wizardry.

I haven't yet had time to document it heavily, but see test/saxtools/xhtmlsummary.py for an example.

See also this XML-DEV message.

## Flextyper: user-defined datatypes in Python for XML processing

Flextyper is an implementation of Jeni Tennison's Data Type Library Language (DTLL) (on track to become part 5 of ISO Document Schema Definition Languages (DSDL). You can use Flextyper to generate Python modules containing data types classes that can be used with 4Suite's RELAX NG library

Flextyper is currently experimental. It won't come into its full usefulness until the next release of 4Suite, although you can use it with current CVS releases of 4Suite.

Flextyper compiles a DTLL file into a collection of Python modules implementing the contained data types. Run it as follows:

    flextyper.py dtll.xml

A set of files, one per data type namespace defined in the DTLL, is created. By default the output file names are based on the input, e.g. dtll-datatypes1.py, dtll-datatypes2.py, etc.

You can now register these data types modules with a processor instance of 4Suite's RELAX NG implementation.