                                 Howto
                                 =====

   XIST is an extensible HTML/XML generator written in Python. It was
   developed as a replacement for an HTML preprocessor named HSC and
   borrows some features and ideas from it. It also borrows the basic
   ideas (XML/HTML elements as Python objects) from HTMLgen or
   HyperText.

   (If you're impatient, there's also a list of examples that shows
   what can be done with XIST.)

Overview
========

   XIST can be used as a compiler that reads an input XML file and
   generates a transformed output file, or it could be used for
   generating XML dynamically inside a web server (but note that
   handling object trees is slower than simply sending string
   fragments). In either case generating the final HTML or XML output
   requires the following three steps:

     * Generating a source XML tree: This can be done either by
       parsing an XML file, or by directly constructing the tree --
       as HTMLgen and HyperText do -- as a tree of Python objects.
       XIST provides a very natural and pythonic API for that.
     * Converting the source tree into a target tree: This target
       tree can be a HTML tree or a SVG tree or XSL-FO tree or any
       other XML tree you like. Every node class provides a convert
       method for performing this conversion. For your own XML
       element types you have to define your own element classes and
       implement an appropriate convert method. This is possible for
       processing instructions and entity references too.
     * Publishing the target tree: For generating the final output a
       Publisher object is used that generates the encoded byte
       string fragments that can be written to an output stream (or
       yielded from a WSGI application, etc.).

Constructing XML trees
======================

   Like any other XML tree API, XIST provides the usual classes:

     * Element for XML elements;
     * Attr for attributes;
     * Attrs for attribute mappings;
     * Text for text data;
     * Frag for document fragments, (a Frag object is simply a list
       of nodes);
     * Comment for XML comments (e.g. <!-- the comment -->);
     * ProcInst for processing instructions (e.g. <?php echo
       $spam;?>);
     * Entity for entity references (e.g. &parrot;) and
     * DocType for document type declarations (e.g. <!DOCTYPE html
       PUBLIC ...>).

  XML trees as Python objects
  ---------------------------

   XIST works somewhat different from a normal DOM API. Instead of
   only one element class, XIST has one class for every element type.
   All the elements from different XML vocabularies known to XIST are
   defined in modules in the ll.xist.ns subpackage. (Of course it's
   possible to define additional namespaces for your own XML
   vocabulary). The definition of HTML can be found in
   ll.xist.ns.html for example.

   Every element class has a constructor of the form:

 __init__(self, *content, **attrs)

   Positional arguments (i.e. items in content) will be the child
   nodes of the element node. Keyword arguments will be attributes.
   You can pass most of Python's builtin types to such a constructor.
   Strings (str and unicode) and integers will be automatically
   converted to Text objects. Constructing an HTML element works like
   this:

 from ll.xist.ns import html

 node = html.div(
    "Hello ",
    html.a("Python", href="http://www.python.org/"),
    " world!"
 )

   For attribute names that collide with Python keywords or are not
   legal identifiers (most notably class in HTML) the attribute name
   must be slightly modified, so that it's a legal Python identifier
   (for class an underscore is appended):

 node = html.div(
    "Hello world!",
    class_="greeting"
 )

   (Don't worry: This modified attribute name will be mapped to the
   real official attribute name once the output is generated.)

   You can pass attributes as a dictionary too:

 node = html.div(
    "Hello world!",
    {
       "class_": "greeting",
       "id": 42,
       "title": "Greet the world"
    }
 )

  Generating XML trees from XML files
  -----------------------------------

   XML trees can also be generated by parsing XML files. For this the
   module ll.xist.parsers provides several functions:

 def parseString(text, base=None, sysid=None, **parserargs)
 def parseURL(url, base=None, sysid=None, **parserargs)
 def parseFile(filename, base=None, sysid=None, **parserargs)
 def parse(stream, base=None, sysid=None, **parserargs)

   parseString is for parsing strings (str and unicode) and parseURL
   is for parsing resources from URLs. With parseFile you can parse
   local files and with parse you can directly parse from a file-like
   object.

   All four functions create a parser internally, parse the supplied
   source document and return the resulting object tree.

   For example, parsing a string can be done like this:

 from ll.xist import parsers
 from ll.xist.ns import html

 node = parsers.parseString(
    "<p>Hello <a href="http://www.python.org/">Python</a> world!</p>"
 )

   For further info about the rest of the arguments to the parsing
   functions, see the documentation for ll.xist.parsers.Parser.

Defining new elements and converting XML trees
==============================================

   To be able to parse an XML file, you have to provide an element
   class for every element type that appears in the file. These
   classes either come from modules provided by XIST or you can
   define your own. Defining your own element class for an element
   named cool works like this:

 class cool(xsc.Element):
    def convert(self, converter):
       node = html.b(self.content, u" is cool!")
       return node.convert(converter)

   You have to derive your new class from xsc.Element. The name of
   the class will be the element name. For element type names that
   are invalid Python identifiers, you can use the class attribute
   xmlname in the element class to overwrite the element name.

   To be able to convert an element of this type to a new XML tree
   (probably HTML in most cases), you have to implement the convert
   method. In this method you can build a new XML tree from the
   content and attributes of the object.

   Using this new element is simple

 >>> node = cool("Python")
 >>> print node.conv().asBytes()
 <b>Python is cool!</b>

   conv simply calls convert with a default converter argument. We'll
   come to converters in a minute. asBytes is a method that converts
   the node to a byte string. This method will be explained when we
   discuss the publishing interface.

   Note that it is vital for your own convert methods that you
   recursively call convert on your own content, because otherwise
   some unconverted nodes might remain in the tree. Let's define a
   new element:

 class python(xsc.Element):
    def convert(self, converter):
       return html.a(u"Python", href=u"http://www.python.org/")

   Now we can do the following:

 >>> node = cool(python())
 >>> print node.conv().asBytes()
 <b><a href="http://www.python.org/">Python</a> is cool!</b>

   But if we forget to call convert for our own content, i.e. if the
   element cool was written like this:

 class cool(xsc.Element):
    def convert(self, converter):
       return html.b(self.content, " is cool!")

   we would get:

 >>> node = cool(python())
 >>> print node.conv().asBytes()
 <b><python /> is cool!</b>

   Furthermore convert should never modify self, because convert
   might be called multiple times for the same node.

  Converters
  ----------

   conv is a convenience method that creates a default converter for
   you and calls convert. This converter is created once and is
   passed to all convert calls. It is used to store parameters for
   the conversion process and it allows elements to pass information
   to other nodes. You can also call convert yourself, which would
   look like this:

 from ll.xist import converters
 from ll.xist.ns import html

 node = cool(python())
 node = node.convert(converters.Converter())

   You can pass the following arguments to the Converter constructor:

   root
           root (which defaults to None) is the root URL for the
           conversion process. When you want to resolve a link in
           some of your own convert methods, the URL must be
           interpreted relative to this root URL (You can use
           URLAttr.forInput for that).

   mode
           mode (which defaults to None) works the same way as modes
           in XSLT. You can use this for implementing different
           conversion modes.

   stage
           stage (which defaults to "deliver") allows you to
           implement multi stage conversion: Suppose that you want to
           deliver a dynamically constructed web page with XIST that
           contains results from a database query and the current
           time. The data in the database changes infrequently, so it
           doesn't make sense to do the query on every request. The
           query is done every few minutes and the resulting HTML
           tree is stored in the servlet (using any of the available
           Python servlet technologies). For this conversion the
           stage would be "cache" and your database XML element would
           do the query when stage=="cache". Your time display
           element would do the conversion when stage=="deliver" and
           simply returns itself when stage=="cache", so it would
           still be part of the cached XML tree and would be
           converted to HTML on every request.

   target
           target (which defaults to ll.xist.ns.html) specifies what
           the output should be. Values must be namespace subclasses
           (see below for an explanation of namespaces).

   lang

           lang (which defaults to None) is the language in which the
           result tree should be. This can be used in the convert
           method to implement different conversions for different
           languages, e.g.:

 class note(xsc.Element):
    def convert(self, converter):
       if converter.lang==u"de":
          title = u"Anmerkung"
       elif converter.lang==u"en":
          title = u"Note"
       else:
          title = u"???"
       node = xsc.Frag(
          html.h1(title),
          html.div(self.content)
       )
       return node.convert(converter)

   Additional arguments are passed when a converter is created in the
   context of a make script.

  Attributes
  ----------

   Setting and accessing the attributes of an element works via the
   dictionary interface:

 >>> node = html.a(u"Python", href=u"http://www.python.org/")
 >>> print node[u"href"].asBytes()
 http://www.python.org/
 >>> del node[u"href"]
 >>> print node[u"href"].asBytes()

 >>> node[u"href"] = u"http://www.python.org"
 >>> print node[u"href"].asBytes()
 http://www.python.org/

   All attribute values are instances of subclasses of the class
   Attr. Available subclasses are:

     * TextAttr, for normal text attributes;
     * URLAttr, for attributes that are URLs;
     * BoolAttr, for boolean attributes (for such an attribute only
       its presence is important, it's value will always be the same
       as the attribute name when publishing);
     * IntAttr, for integer attributes;
     * ColorAttr, for color attributes (e.g. #ffffff).

   IntAttr and ColorAttr mostly serve as documentation of the
   attributes purpose. Both classes have no added functionality.

   Attr itself is derived from Frag so it is possible to use all the
   sequence methods on an attribute.

   Unset attributes will be treated like empty ones so the following
   is possible:

 del node["spam"]
 node["spam"].append("ham")

   This also means that after del node["spam"][:] the attribute will
   be empty again and will be considered to be unset. Such attributes
   will be skipped when publishing.

   The main purpose of this is to allow you to construct values
   conditionally and then use those values as attribute values:

 import random

 if random.random() < 0.5:
    class_ = None
 else:
    class_ = u"foo"

 node = html.div(u"foo", class_=class_)

   In 50% of the cases the generated div element will not have a
   class attribute.

    Defining attributes
    -------------------

   When you define a new element you have to specify the attributes
   allowed for this element. For this use the class attribute Attrs
   (which must be a class derived from xsc.Element.Attrs) and define
   the attributes by deriving them from one of the existing attribute
   classes. We could extend our example element in the following way:

 class cool(xsc.Element):
    class Attrs(xsc.Element.Attrs):
       class adj(xsc.TextAttr): pass

    def convert(self, converter):
       node = xsc.Frag(self.content, u" is")
       if u"adj" in self.attrs:
          node.append(u" ", html.em(self[u"adj"]))
       node.append(u" cool!")
       return node.convert(converter)

   and use it like this:

 >>> node = cool(python(), adj=u"totally")
 >>> node.conv().asBytes()
 <a href="http://www.python.org/">Python</a> is <em>totally</em> cool!

    Default attributes
    ------------------

   It is possible to define default values for attributes via the
   class attribute default:

 class cool(xsc.Element):
    class Attrs(xsc.Element.Attrs):
       class adj(xsc.TextAttr):
          default = u"absolutely"

    def convert(self, converter):
       node = xsc.Frag(self.content, u" is")
       if u"adj" in self.attrs:
          node.append(u" ", html.em(self[u"adj"]))
       node.append(u" cool!")
       return node.convert(converter)

   Now if we instantiate the class without specifying adj we'll get
   the default:

 >>> node = cool(python())
 >>> print node.conv().asBytes()
 <a href="http://www.python.org/">Python</a> is <em>absolutely</em> cool!

   If we want a cool instance without an adj attribute, we can pass
   None as the attribute value:

 >>> node = cool(python(), adj=None)
 >>> print node.conv().asBytes()
 <a href="http://www.python.org/">Python</a> is cool!

    Attribute value sets
    --------------------

   It's possible to specify that an attribute has a fixed set of
   allowed values. This can be done with the class attribute values.
   We could extend our example to look like this:

 class cool(xsc.Element):
    class Attrs(xsc.Element.Attrs):
       class adj(xsc.TextAttr):
          default = "absolutely"
          values = (u"absolutely", u"totally", u"very")

    def convert(self, converter):
       node = xsc.Frag(self.content, u" is")
       if u"adj" in self.attrs:
          node.append(" ", html.em(self[u"adj"]))
       node.append(u" cool!")
       return node.convert(converter)

   These values won't be checked when we create our cool instance.
   Only when this node is parsed from a file will the warning be
   issued:

 >>> s = '<cool adj="pretty"><python/></cool>'
 >>> node = parsers.parseString(s)
 /home/walter/pythonroot/ll/xist/xsc.py:1665: IllegalAttrValueWarning: Attribute value 'pretty' not allowed for __main__:cool.Attrs.adj.
   warnings.warn(errors.IllegalAttrValueWarning(self))

   The warning will also be issued if we publish such a node, but
   note that for warnings Python's warning framework is used, so the
   warning will be printed only once (but of course you can change
   that with warnings.filterwarnings):

 >>> node = cool(python(), adj=u"pretty")
 >>> print node.asBytes()
 /home/walter/pythonroot/ll/xist/xsc.py:1665: IllegalAttrValueWarning: Attribute value 'pretty' not allowed for __main__:cool.Attrs.adj.
   warnings.warn(errors.IllegalAttrValueWarning(self))
 <cool adj="very"><python /></cool>

    Required attributes
    -------------------

   Finally it's possible to specify that an attribute is required.
   This again will only be checked when parsing or publishing. To
   specify that an attribute is required simply add the class
   attribute required with the value True. The attribute alt of the
   class ll.xist.ns.html.img is such an attribute, so we'll get:

 >>> from ll.xist.ns import html
 >>> node = html.img(src="eggs.png")
 >>> print node.asBytes()
 /home/walter/pythonroot/ll/xist/xsc.py:2046: RequiredAttrMissingWarning: Required attribute 'alt' missing in ll.xist.ns.html:img.Attrs.
   warnings.warn(errors.RequiredAttrMissingWarning(self, attrs.keys()))
 <img src="eggs.png" />

  Namespaces
  ----------

   Now that you've defined your own elements, you have to tell the
   parser about them, so they can be instantiated when a file is
   parsed. This is done with namespace classes.

   Namespace classes can be thought of as object oriented versions of
   XML namespaces. Two class attributes can be used to configure the
   namespace: xmlname specifies the default namespace prefix to use
   for the namespace and xmlurl is the namespace name. All element
   classes nested inside the namespace class belong to the namespace.

   It's also possible to define your namespace class without any
   nested element classes and later add those classes to the
   namespace by attribute assignment or with the class method update,
   which expects a dictionary as an argument. All objects found in
   the values of the dictionary will be added to the namespace as
   attributes. So you can put the namespace class at the end of your
   Python module after all the element classes are defined and add
   all the objects from the local scope to the namespace. Your
   complete namespace might looks like this:

 class python(xsc.Element):
    def convert(self, converter):
       return html.a(
          u"Python",
          href=u"http://www.python.org/"
       )

 class cool(xsc.Element):
    def convert(self, converter):
       node = html.b(self.content, u" is cool!")
       return node.convert(converter)

 class __ns__(xsc.Namespace):
    xmlname = "foo"
    xmlurl = "http://www.example.com/foo"
 __ns__.update(vars())

   All defined namespace classes will be registered with the parser
   automatically, so all elements belonging to the namespace will be
   used when parsing files.

    Namespaces as modules
    ---------------------

   It is convenient to define all classes that belong to one
   namespace in one Python module. However this means that the
   resulting module will only contain one "interesting" object: the
   namespace class. To make using this class more convenient, it's
   possible to turn the namespace class into a module by using the
   class method makemod instead of update:

 class python(xsc.Element):
    def convert(self, converter):
       return html.a(
          u"Python",
          href=u"http://www.python.org/"
       )

 class cool(xsc.Element):
    def convert(self, converter):
       node = html.b(self.content, u" is cool!")
       return node.convert(converter)

 class __ns__(xsc.Namespace):
    xmlname = "foo"
    xmlurl = "http://www.example.com/foo"
 __ns__.makemod(vars())

   Suppose that the above code is in the file foo.py. Doing an import
   foo will then give you the namespace class instead of the module:

 >>> import foo
 >>> foo
 <foo:__ns__ namespace name=u"foo" url=u"http://www.example.com/foo" with 2 elements from "foo.py" at 0x87654321>
 >>> foo.python
 <foo:python element at 0x87654321>

    Global attributes
    -----------------

   You can define global attributes belonging to a certain namespace
   in the same way as defining local attributes belonging to a
   certain element type: Define a nested Attrs class inside the
   namespace class (derived from ll.xist.xsc.Namespace.Attrs):

 from ll.xist import xsc

 class __ns__(xsc.Namespace):
    xmlname = "foo"
    xmlurl = "http://www.example.com/foo"

    class Attrs(xsc.Namespace.Attrs):
       class foo(xsc.TextAttr): pass
 __ns__.makemod(vars())

   Setting and accessing such an attribute can be done like this:

 >>> from ll.xist.ns import html
 >>> import foo
 >>> node = html.div(u"foo", {(foo, u"foo"): u"bar")
 >>> str(node[foo, u"foo"])
 'bar'

   An alternate way of specifying a global attribute in a constructor
   looks like this:

 >>> from ll.xist.ns import html
 >>> import foo
 >>> node = html.div(u"foo", foo.Attrs(foo=u"baz"))
 >>> str(node[foo, u"foo"])
 'baz'

    Subclassing namespaces
    ----------------------

   Each element class that belongs to a namespace can access its
   namespace via the class attribute __ns__. When you're subclassing
   namespace classes, the elements in the base namespace will be
   automatically subclassed too. Of course you can explicitly
   subclass an element class too. The following example shows the
   usefulness of this feature. Define your base namespace like this
   and put it into navns.py:

 from ll.xist import xsc
 from ll.xist.ns import html

 languages = [
    (u"Python", u"http://www.python.org/"),
    (u"Perl", u"http://www.perl.org/"),
    (u"PHP", u"http://www.php.net/"),
    (u"Java", u"http://java.sun.com/")
 ]

 class navigation(xsc.Element):
    def convert(self, converter):
       node = self.__ns__.links()
       for (name, url) in languages:
          node.append(self.__ns__.link(name, href=url))
       return node.convert(converter)

 class links(xsc.Element):
    def convert(self, converter):
       node = self.content
       return node.convert(converter)

 class link(xsc.Element):
    class Attrs(xsc.Element.Attrs):
       class href(xsc.URLAttr): pass

    def convert(self, converter):
       node = html.div(html.a(self.content, href=self[u"href"]))
       return node.convert(converter)

 class __ns__(xsc.Namespace):
    xmlname = "nav"
    xmlurl = "http://www.example.com/nav"
 __ns__.makemod(vars())

   This namespace defines a navigation element that generates divs
   with links to various homepages for programming languages. We can
   use it like this:

 >>> import navns
 >>> print navns.navigation().conv().asBytes()
 <div><a href="http://www.python.org/">Python</a></div>
 <div><a href="http://www.perl.org/">Perl</a></div>
 <div><a href="http://www.php.net/">PHP</a></div>
 <div><a href="http://java.sun.com/">Java</a></div>

   (Of course the output will all be on one line.)

   Now we can define a derived namespace (in the file nav2ns.py) that
   overwrites the element classes links and link to change how the
   navigation looks:

 from ll.xist import xsc
 from ll.xist.ns import html

 import navns

 class __ns__(navns):
    class links(navns.links):
       def convert(self, converter):
          node = html.table(
             self.content,
             border=0,
             cellpadding=0,
             cellspacing=0,
             class_=u"navigation",
          )
          return node.convert(converter)

    class link(navns.link):
       def convert(self, converter):
          node = html.tr(
             html.td(
                html.a(
                   self.content,
                   href=self[u"href"],
                )
             )
          )
          return node.convert(converter)
 __ns__.makemod(vars())

   When we use the navigation element from the derived namespace
   we'll get the following output:

 >>> import nav2ns
 >>> print nav2ns.navigation().conv().asBytes()
 <table border="0" cellpadding="0" cellspacing="0" class="navigation">
 <tr><td><a href="http://www.python.org/">Python</a></td></tr>
 <tr><td><a href="http://www.perl.org/">Perl</a></td></tr>
 <tr><td><a href="http://www.php.net/">PHP</a></td></tr>
 <tr><td><a href="http://java.sun.com/">Java</a></td></tr>
 </table>

   (again all on one line.)

   Notice that we automatically got an element class
   nav2ns.navigation, that this element class inherited the convert
   method from its base class and that the call to convert on the
   derived class did instantiate the link classes from the derived
   namespace.

    Namespaces as conversion targets
    --------------------------------

   The converter argument passed to the convert method has an
   attribute target which is a namespace class and specifies the
   target namespace to which self should be converted.

   You can check which conversion is wanted with issubclass. Once
   this is determined you can use element classes from the target to
   create the required XML object tree. This makes it possible to
   customize the conversion by passing a derived namespace to the
   convert method. To demonstrate this, we change our example
   namespace to use the conversion target like this:

 import navns

 class __ns__(navns):
    class links(nav.links):
       def convert(self, converter):
          node = converter.target.table(
             self.content,
             border=0,
             cellpadding=0,
             cellspacing=0,
             class_=u"navigation",
          )
          return node.convert(converter)

    class link(nav.link):
       def convert(self, converter):
          target = converter.target
          node = target.tr(
             target.td(
                target.a(
                   self.content,
                   href=self[u"href"],
                )
             )
          )
          return node.convert(converter)

   What we might want to do is have all links (i.e. all
   ll.xist.ns.html.a elements) generated with an attribute
   target="_top". For this we derive a new namespace from
   ll.xist.ns.html and overwrite the a element:

 from ll.xist.ns import html

 class __ns__(html):
    class a(html.a):
       def convert(self, converter):
          node = html.a(self.content, self.attrs, target=u"_top")
          return node.convert(converter)

   Now we can pass this namespace as the conversion target and all
   links will have a target="_top".

  Validation and content models
  -----------------------------

   When generating HTML you might want to make sure that your
   generated code doesn't contain any illegal tag nesting (i.e.
   something bad like <p><p>Foo</p></p>). The module ll.xist.ns.html
   does this automatically:

 >>> from ll.xist.ns import html
 >>> node = html.p(html.p(u"foo"))
 >>> print node.asBytes()
 /home/walter/pythonroot/ll/xist/sims.py:238: WrongElementWarning: element <ll.xist.ns.html:p> may not contain element <ll.xist.ns.html:p>
   warnings.warn(WrongElementWarning(node, child, self.elements))
 <p><p>foo</p></p>

   For your own elements you can specify the content model too. This
   is done by setting the class attribute model inside the element
   class. model must be an object that provides a checkvalid method.
   This method will be called during parsing or publishing with the
   element as an argument. When a validation violation is detected,
   the Python warning framework should be used to issue a warning.

   The module ll.xist.sims contains several classes that provide
   simple validation methods: Empty can be used to ensure that the
   element doesn't have any content (like br and img in HTML). Any
   does allow any content. NoElements will warn about elements from
   the same namespace (elements from other namespaces will be OK).
   NoElementsOrText will warn about elements from the same namespace
   and non-whitespace text content. Elements will only allow the
   elements specified in the constructor. ElementsOrText will only
   allow the elements specified in the constructor and text.

   None of these classes will check the number of child elements or
   their order.

   For more info see the sims module.

  Entities
  --------

   In the same way as defining new element types, you can define new
   entities. But to be able to use the new entities in an XML file
   you have to use a parser that supports reporting undefined
   entities to the application via skippedEntity (SGMLOPParser and
   ExpatParser in the module ll.xist.parsers do that). The following
   example is from the module ll.xist.ns.abbr:

 from ll.xist import xsc
 from ll.xist.ns import html

 class html(xsc.Entity):
    def convert(self, converter):
       return html.abbr(
          u"HTML",
          title=u"Hypertext Markup Language",
          lang=u"en"
       )

   You can use this entity in your XML files like this:

 <cool adj="very">&html;</cool>

  Processing instructions
  -----------------------

   Defining processing instructions works just like elements and
   entities. Derive a new class from ll.xist.xsc.ProcInst and
   implement convert. The following example implements a processing
   instruction that returns an uppercase version of its content as a
   text node.

 class upper(xsc.ProcInst):
    def convert(self, converter):
       return xsc.Text(self.content.upper())

   It can be used in an XML file like this:

 <cool><?upper foo?></cool>

   There are namespaces containing processing instruction classes
   that don't provide a convert method. These processing instruction
   objects will then be published as XML processing instructions. One
   example is the namespace ll.xist.ns.php.

   Other namespaces (like ll.xist.ns.jsp) contain processing
   instruction classes, but they will be published in a different
   (not XML compatible) format. For example
   ll.xist.ns.jsp.expression("foo") will be published as <%= foo>.

Publishing XML trees
====================

   After creating the XML tree and converting the tree into its final
   output form, you have to write the resulting tree to a file. This
   can be done with the publishing API. Three methods that use the
   publishing API are bytes, asBytes and write. bytes is a generator
   that will yield the complete 8-bit XML string in fragments.
   asBytes returns the complete 8-bit XML string.

   You can specify the encoding with the parameter encoding (with
   "utf-8" being the default). Unencodable characters will be escaped
   with character references when possible (i.e. inside text nodes,
   for comments or processing instructions you'll get an exception):

 >>> from ll.xist import xsc
 >>> from ll.xist.ns import html
 >>> s = u"A\e4\u03a9\u8a9e"
 >>> node = html.div(s)
 >>> print node.asBytes(encoding="ascii")
 <div>A&#228;&#937;&#35486;>
 >>> print node.asBytes(encoding="iso-8859-1")
 <div>Aä&#937;&#35486;>
 >>> print xsc.Comment(s).asBytes()
 Traceback (most recent call last):
   File "<stdin>", line 1, in ?
   File "/home/walter/pythonroot/ll/xist/xsc.py", line 600, in asBytes
     publisher.publish(stream, self, base)
   File "/home/walter/pythonroot/ll/xist/publishers.py", line 205, in publish
     self.node.publish(self)
   File "/home/walter/pythonroot/ll/xist/xsc.py", line 1305, in publish
     publisher.write(self.content)
   File "/usr/local/lib/python2.3/codecs.py", line 178, in write
     data, consumed = self.encode(object, self.errors)
 UnicodeEncodeError: 'ascii' codec can't encode characters in position 1-4: ordinal not in range(128)

   When you include an XML header or an HTML meta header, XIST will
   automatically insert the correct encoding when publishing:

 >>> from ll.xist import xsc
 >>> from ll.xist.ns import xml, meta
 >>> e = xsc.Frag(xml.XML10(), u"\n", meta.contenttype())
 >>> print e.conv().asBytes(encoding="iso-8859-15")
 <?xml version='1.0' encoding='iso-8859-15'?>
 <meta content="text/html; charset=iso-8859-15" http-equiv="Content-Type" />

   Another useful parameter is xhtml, it specifies whether you want
   pure HTML or XHTML as output:

   xhtml==0
           This will give you pure HTML, i.e. no final / for elements
           with an empty content model, so you'll get e.g. <br> in
           the output. Elements that don't have an empty content
           model, but are empty will be published with a start and
           end tag (i.e. <div></div>).

   xhtml==1
           This gives HTML compatible XHTML. Elements with an empty
           content model will be published like this: <br /> (This is
           the default).

   xhtml==2
           This gives full XML output. Every empty element will be
           published with an empty tag (without an additional space):
           <br/> or <div/>.

   Writing a node to a file can be done with the method write:

 >>> from ll.xist.ns import html
 >>> node = html.div(u"äöü", html.br(), u"ÄÖÜ")
 >>> node.write(open("foo.html", "wb"), encoding="ascii")

   All these methods use the method publish internally. publish gets
   passed an instance of ll.xist.publisher.Publisher.

Searching trees
===============

   There are two methods available for iterating through an XML tree
   and finding nodes in the tree: The walk method and XFind
   expressions.

  The walk method
  ---------------

   The method walk is a generator. You pass a callable object to walk
   which is used for determining which part of the tree should be
   searched and which nodes should be returned.

   ll.xist.xsc provides several useful predefined classes for
   specifying what should be returned from walk: FindType will search
   only the first level of the tree and will return any node that is
   an instance of one of the classes passed to the constructor. So if
   you have an instance of ll.xist.ns.html.ul named node you could
   search for all ll.xist.ns.html.li elements inside with the
   following code:

 for cursor in node.content.walk(xsc.FindType(html.li)):
    print unicode(cursor.node)

   FindTypeAll can be used when you want to search the complete tree.
   The following example extracts all the links on the Python home
   page:

 from ll.xist import xsc, parsers
 from ll.xist.ns import html

 node = parsers.parseURL("http://www.python.org/", tidy=True)

 for cursor in node.walk(xsc.FindTypeAll(html.a)):
    print cursor.node[u"href"]

   This gives the output:

 http://www.python.org/
 http://www.python.org/search/
 http://www.python.org/download/
 http://www.python.org/doc/
 http://www.python.org/Help.html
 http://www.python.org/dev/
 ...

   The following example will find all external links on the Python
   home page:

 from ll.xist import xsc, parsers
 from ll.xist.ns import html

 node = parsers.parseURL("http://www.python.org/", tidy=True)

 def isextlink(cursor):
    if isinstance(cursor.node, html.a) and not unicode(cursor.node[u"href"]).startswith(u"http://www.python.org"):
       return (True, xsc.entercontent)
    return (xsc.entercontent,)

 for cursor in node.walk(isextlink):
    print cursor.node[u"href"]

   This gives the output:

 http://www.jython.org/
 http://sourceforge.net/tracker/?atid=105470&group%5fid=5470
 http://sourceforge.net/tracker/?atid=305470&group%5fid=5470
 http://sourceforge.net/cvs/?group%5fid=5470
 http://www.python-in-business.org/
 http://www.europython.org/
 mailto:webmaster@python.org
 ...

   The callable (isextlink in the example) will be called for each
   node visited. The cursor argument has an attribute node that is
   the node in question. For the other attributes see the Cursor
   class.

   The callable must return a sequence with the following entries:

   ll.xist.xsc.entercontent
           enter the content of this element and continue searching;

   ll.xist.xsc.enterattrs
           enter the attributes of this element and continue
           searching;

   boolean value
           If true, the node will be part of the result.

   The sequence will be "executed" in the order you specify. To
   change the top down traversal from our example to a bottom up
   traversal we could change isextlink to the following (note the
   swapped tuple entries):

 def isextlink(node):
    if isinstance(node, html.a) and not unicode(node[u"href"]).startswith(u"http://www.python.org"):
       return (xsc.entercontent, True)
    return (xsc.entercontent,)

   Note that the cursor yielded from walk will be reused by
   subsequent next calls, so you should not modify the cursor and you
   can't rely on attributes of the cursor after reentry to walk.

  XFind expressions
  -----------------

   A second method exists for iterating through a tree: XFind
   expressions. An XFind expression looks somewhat like an XPath
   expression, but is implemented as a pure Python expression
   (overloading the division operators).

   Our example from above that searched for lis inside uls can be
   rewritten as follows:

 for child in node/html.li:
    print unicode(child)

   A XFind expression returns an iterator for certain parts of the
   XML tree. In an XFind expression a/b, a must be either a node or
   an iterator producing nodes (note that an XFind expression itself
   is such an iterator, so a itself might be a XFind expression). b
   must be an XFind operator.

   Every subclass of ll.xist.xsc.Node is a XFind operator. If b is
   such a subclass, a/b will produce any child nodes of the nodes
   from a that is an instance of b. If b is an attribute class, you
   will get attribute nodes instead of child nodes. Other XFind
   operators can be found in the module ll.xist.xfind. The all
   operator will produce every node in the tree (except for
   attributes):

 from ll.xist import xfind
 from ll.xist.ns import html

 node = html.div(
    html.div(
       html.div(id=3),
       html.div(id=4),
       id=2,
    ),
    html.div(
       html.div(id=6),
       html.div(id=7),
       id=5,
    ),
    id=1
 )

 for child in node/xfind.all:
    print child["id"]

   The output of this is:

 1
 2
 3
 4
 5
 6
 7

   The following example demonstrates how to find all links on the
   Python homepage via an XFind expression:

 from ll.xist import xfind, parsers
 from ll.xist.ns import html

 node = parsers.parseURL("http://www.python.org/", tidy=True)
 for link in node/xfind.all/html.a:
    print link["href"]

   An all operator in the middle of an XFind expression can be
   abbreviated. The XFind expression from the last example
   (node/xfind.all/html.a) can be rewritten like this: node//html.a.

   Another XFind operator is contains. It acts as a filter, i.e. the
   nodes produced by a/xfind.contains(b) are a subset of the nodes
   produced by a, those that contain child nodes of type b. Searching
   for all links on the Python home page that contain images can be
   done like this:

 >>> from ll.xist import xfind, parsers
 >>> from ll.xist.ns import html
 >>> node = parsers.parseURL("http://www.python.org/", tidy=True)
 >>> for link in node//html.a/xfind.contains(html.img):
 ...     print link["href"]
 ...
 http://www.python.org/
 http://www.python.org/psf/donations.html
 http://www.opensource.org/

   Note that using the all operator twice in an XFind expression
   currently won't give you the expected result, as nodes might be
   produced twice.

   Calling __getitem__ on an XFind operator gives you an item
   operator. Such an item operator only returns a specific item (or
   slice) of those nodes returned by the base iterator. An example:

 >>> from ll.xist.ns import html
 >>> e = html.table(html.tr(html.td(j) for j in xrange(i, i+3)) for i in xrange(1, 10, 3))
 >>> print e.pretty().asBytes()
 <table>
    <tr>
       <td>1</td>
       <td>2</td>
       <td>3</td>
    </tr>
    <tr>
       <td>4</td>
       <td>5</td>
       <td>6</td>
    </tr>
    <tr>
       <td>7</td>
       <td>8</td>
       <td>9</td>
    </tr>
 </table>
 >>> # Every cell
 >>> for td in e/html.tr/html.td:
 ...     print td
 ...
 1
 2
 3
 4
 5
 6
 7
 8
 9
 >>> # Every first cell in each row
 >>> for td in e/html.tr/html.td[0]:
 ...     print td
 1
 4
 7
 >>> # Every cell in the first row
 >>> for td in e/html.tr[0]/html.td:
 ...     print td
 1
 2
 3
 >>> # The first of all cells
 >>> for td in e/(html.tr/html.td)[0]:
 ...     print td
 1

Manipulating trees
==================

   XIST provides many methods for manipulating an XML tree.

   The method withsep can be used to put a seperator node between the
   child nodes of an Element or a Frag:

 >>> from ll.xist import xsc
 >>> from ll.xist.ns import html
 >>> node = html.div(*xrange(10))
 >>> print node.withsep(", ").asBytes()
 <div>0, 1, 2, 3, 4, 5, 6, 7, 8, 9</div>

   The method shuffled returns a shuffled version of the Element or
   Frag:

 >>> from ll.xist import xsc
 >>> from ll.xist.ns import html
 >>> node = html.div(*xrange(10))
 >>> print node.shuffled().withsep(", ").asBytes()
 <div>8, 1, 3, 6, 7, 5, 2, 9, 4, 0</div>

   There are methods named reversed and sorted that return a reversed
   or sorted version of an element or fragment:

 >>> from ll.xist import xsc
 >>> from ll.xist.ns import html
 >>> def key(n):
 ...    return unicode(n)
 >>> node = html.div(8,4,2,1,9,6,3,0,7,5)
 >>> print node.sorted(key=key).reversed().withsep(",").asBytes()
 <div>9,8,7,6,5,4,3,2,1,0</div>

   The method mapped recursively walks the tree and generates a new
   tree, where all the nodes are mapped through a function. An
   example: To replace Python with Parrot in every text node on the
   Python page, do the following:

 from ll.xist import xsc, parsers

 def p2p(node, converter):
    if isinstance(node, xsc.Text):
       node = node.replace(u"Python", u"Parrot")
       node = node.replace(u"python", u"parrot")
    return node

 node = parsers.parseURL("http://www.python.org/", tidy=True)
 node = node.mapped(p2p)
 node.write(open("parrot_index.html", "wb"))

   The function must either return a new node, in which case this new
   node will be used instead of the old one, or return the old node
   to tell mapped that it should recursively continue with the
   content of the node.

URLs
====

   For URL handling XIST uses the module ll.url. Refer to its
   documentation for the basic functionality (especially regarding
   the methods __div__ and relative).

   When XIST parses an XML resource it uses a so called "base" URL.
   This base URL can be passed to all parsing functions. If it isn't
   specified it defaults to the URL of the resource being parsed.
   This base URL will be prepended to all URLs that are read during
   parsing:

 >>> from ll.xist import parsers
 >>> from ll.xist.ns import html
 >>> node = parsers.parseString('<img src="eggs.png"/>', base="root:spam/index.html")
 >>> print node.asBytes()
 <img src="root:spam/eggs.png" />

   For publishing a base URL can be specified too. URLs will be
   published relative to this base URL with the exception of relative
   URLs in the tree. This means:

     * When you have a relative URL (e.g. #top) generated by a
       convert call, this URL will stay the same when publishing.
     * Base URLs for parsing should never be relative: Relative base
       URLs will be prepended to all relative URLs in the file, but
       this will not be reverted for publishing. In most cases the
       base URL should be a root URL when you parse local files.
     * When you parse remote web pages you can either omit the base
       argument, so it will default to the URL being parsing, so that
       links, images, etc. on the page will still point back to their
       original location, or you might want to use the empty URL
       URL() as the base, so you'll get all URLs in the page as they
       are.
     * When XIST is used as a compiler for static pages, you're going
       to read source XML files, do a conversion and write the result
       to a new target file. In this case you should probably use the
       URL of the target file for both parsing and publishing. Let's
       assume we have an URL #top in the source file. When we use the
       "real" file names for parsing and publishing like this:

 node = parsers.parseFile("spam.htmlxsc", base="root:spam.htmlxsc")
 node = node.conv()
 node.write(open("spam.html", "wb"), base="root:spam.html")

       the following will happen: The URL #top will be parsed as
       root:spam.htmlxsc#top. After conversion this will be written
       to spam.html relative to the URL root:spam.html, which results
       in spam.html#top, which works, but is not what you want.

       When you use root:spam.html both for parsing and publishing,
       #top will be written to the target file as expected.

Pretty printing XML
===================

   The method pretty can be used for pretty printing XML. It returns
   a new version of the node, with additional white space between the
   elements:

 from ll.xist.ns import html
 node = html.html(
    html.head(
       html.title(u"foo"),
    ),
    html.body(
       html.div(
          html.h1(u"The ", html.em(u"foo"), u" page!"),
          html.p(u"Welcome to the ", html.em(u"foo"), u" page."),
       ),
    ),
 )

 print node.pretty().asBytes()

   This will print:

 <html>
    <head>
       <title>foo</title>
    </head>
    <body>
       <div>
          <h1>The <em>foo</em> page!</h1>
          <p>Welcome to the <em>foo</em> page.</p>
       </div>
    </body>
 </html>

   Element content will only be modified if it doesn't contain Text
   nodes, so mixed content will not be touched.

Automatic generation of image size attributes
=============================================

   The module ll.xist.ns.htmlspecials contains an element autoimg
   that extends ll.xist.ns.html.img. When converted to HTML via the
   convert method the size of the image will be determined and the
   height and width attributes will be set accordingly (if those
   attributes are not set already).

Embedding Python code
=====================

   It's possible to embed Python code into XIST XML files. For this
   XIST supports two new processing instructions: pyexec and pyeval
   (in the module ll.xist.ns.code). The content of pyexec will be
   executed when the processing instruction node is converted.

   The result of a call to convert for a pyeval processing
   instruction is whatever the Python code in the content returns.
   The processing instruction content is treated as the body of a
   function, so you can put multiple return statements there. The
   converter is available as the parameter converter inside the
   processing instruction. For example, consider the following XML
   file:

 <?pyexec
    # sum
    def gauss(top=100):
       sum = 0
       for i in xrange(top+1):
          sum += i
       return sum
 ?>
 <b><?pyeval return gauss()?></b>

   Parsing this file and calling convert results in the following:

 <b>5050</b>