pyTenjin FAQ

> # pyTenjin FAQ

> release: 0.8.0
> last update: $Date$

> Table of contents:

> - Basic
>   - I got an SyntaxError exception.
>   - Is there any way to 'escape' or 'remove' newline at the end of line?
>   - '#{_content}' includes extra newline at end. Can I delete it?
>   - Can I change 'escape()' and 'to_str()' function name?
>   - Can I change '_buf' variable name?
>   - Is it able to specify default value if a variable is not set?
>   - Is pyTenjin ready for Google App Engine?
> - Template
>   - Is it able to specify variables passed to template?
>   - It is hard to specify all template arguments. Any good idea?
>   - Is it able to change embedded expression pattern?
>   - Does pyTenjin support M17N?
>   - Is it possible to create separated template chaches for each language?
> - Layout Template
>   - Can I change layout template name in a template file?
>   - Can I nest layout templates for any depth?
>   - Can I disable default layout template for a certain template?
>   - Is Django-like "Template Inheritance" supported?
> - Encoding
>   - How to specify template encoding?
>   - Can I specify encoding name in template file?
>   - I got 'SyntaxError: encoding declaration in Unicode string'
>   - I got UnicodeDecodeError, but I can't find what is wrong
> - Preprocessing
>   - What is preprocessing?
>   - What is the merit of preprocessing?
>   - Is there any examples of preprocessing?
>     - Loop expantion
>     - Helper methods execution in advance
>     - M17N (Multilingualization)
> - Performance
>   - How fast is pyTenjin compared with other solutions?
>   - Why pyTenjin is so fast?
>   - Is there any way to get more speed?

> ## Basic

> ### I got an SyntaxError exception.

> Command-line option '-z' checks syntax of template file. You should check template by it.

> File 'ex1.pyhtml':

>     <?py for i in range(0, 10): ?>
>     <?py     if i % 2 == 0: ?>
>     #{i} is even.
>     <?py     else ?>
>     #{i} is odd.
>     <?py     #end ?>
>     <?py #end ?>

> Result:

>     $ pytenjin -z ex1.pyhtml
>     ex1.pyhtml:4:9: invalid syntax
>       4:     else
>                  ^

> ### Is there any way to 'escape' or 'remove' newline at the end of line?

> Yes, but it is not beautiful very much.

> Assume that you want to generate CSV file. The following is a wrong example.

> File 'ex2a.pycsv': (wrong)

>     <?py
>     table = [
>       ( "A",  10,  20,  30, ),
>       ( "B",  11,  21,  31, ),
>       ( "C",  12,  22,  23, ),
>     ]
>     ?>
>     <?py for line in table: ?>
>     <?py     sep = '' ?>
>     <?py     for cell in line: ?>
>     #{sep}#{cell}
>     <?py         sep = ', ' ?>
>     <?py     #end ?>
>     <?py #end ?>

> Result: (wrong)

>     $ pytenjin ex2a.pycsv
>     A
>     , 10
>     , 20
>     , 30
>     B
>     , 11
>     , 21
>     , 31
>     C
>     , 12
>     , 22
>     , 23

> The following is corrected template.

> File 'ex2b.pycsv':

>     <?py
>     table = [
>       ( "A",  10,  20,  30, ),
>       ( "B",  11,  21,  31, ),
>       ( "C",  12,  22,  23, ),
>     ]
>     ?>
>     <?py
>     for line in table:
>         sep = ''
>         for cell in line:
>     ?>#{sep}#{cell}<?py
>             sep = ', '
>         #end
>     ?>

>     <?py
>     #end
>     ?>

> Result:

>     $ pytenjin ex2b.pycsv
>     A, 10, 20, 30
>     B, 11, 21, 31
>     C, 12, 22, 23

> But it is a little complex and not beautiful. In this case, you may prefer to use '_buf' variable directly.

> File 'ex2c.pycsv':

>     <?py
>     table = [
>       ( "A",  10,  20,  30),
>       ( "B",  11,  21,  31),
>       ( "C",  12,  22,  23),
>     ]
>     ?>
>     <?py
>     for line in table:
>         sep = ''
>         for cell in line:
>             if sep: _buf.append(sep)
>             _buf.append(to_str(cell))
>             sep = ', '
>         #end
>         _buf.append("\n")
>     #end
>     ?>

> Result:

>     $ pytenjin ex2c.pycsv
>     A, 10, 20, 30
>     B, 11, 21, 31
>     C, 12, 22, 23

> ### '#{_content}' includes extra newline at end. Can I delete it?

> Yes. You can use '`<?py echo(_content) ?>`' or '`<?py _buf.append(_content)` ?>' instead of '`#{_content}`'.

> File 'ex3-layout.pyhtml':

>     <!-- -->
>     #{_content}
>     <!-- -->

>     <!-- -->
>     <?py echo(_content) ?>
>     <!-- -->

>     <!-- -->
>     <?py _buf.append(_content) ?>
>     <!-- -->

> File 'ex3-content.pyhtml':

>     foo
>     bar
>     baz

> Result:

>     $ pytenjin --layout=ex3-layout.pyhtml ex3-content.pyhtml
>     <!-- -->
>     foo
>     bar
>     baz

>     <!-- -->

>     <!-- -->
>     foo
>     bar
>     baz
>     <!-- -->

>     <!-- -->
>     foo
>     bar
>     baz
>     <!-- -->

> [experimental] If you pass 'smarttrim=True' option to tenjin.Template() or tenjin.Engine(), "\n#{expr}\n" will be trimmed into "\n#{expr}". And command-line option '--smarttrim' is the same as 'smarttrim=True' option.

> The following example shows that an empty line is not appread when '--smarttrim' is specified.

> Result:

>     $ pytenjin --smarttrim --layout=ex3-layout.pyhtml ex3-content.pyhtml
>     <!-- -->
>     foo
>     bar
>     baz
>     <!-- -->

>     <!-- -->
>     foo
>     bar
>     baz
>     <!-- -->

>     <!-- -->
>     foo
>     bar
>     baz
>     <!-- -->

> ### Can I change 'escape()' and 'to_str()' function name?

> Yes. You can change them by setting 'escapefunc' and 'tostrfunc' options for tenjin.Template() or tenjin.Engine().

> File 'ex4.py':

>     import tenjin
>     engine = tenjin.Engine(escapefunc='cgi.escape', tostrfunc='str')
>     template = engine.get_template('ex4.pyhtml')
>     print template.script,

> File 'ex4.pyhtml':

>     Hello ${name}!
>     <?py for item in items: ?>
>     #{item}
>     <?py #end ?>

> Result:

>     $ python ex4.py
>     _buf.extend(('''Hello ''', cgi.escape(str(name)), '''!\n''', ));
>     for item in items:
>         _buf.extend((str(item), '''\n''', ));
>     #end

> Command-line option '--escapefunc=name' and '--tostrfunc=name' is equivarent to the above.

> Result:

>     $ pytenjin -sb --escapefunc=cgi.escape --tostrfunc=str ex4.pyhtml
>     _buf.extend(('''Hello ''', cgi.escape(str(name)), '''!\n''', ));
>     for item in items:
>         _buf.extend((str(item), '''\n''', ));
>     #end

> ### Can I change '_buf' variable name?

> No. Variable name '_buf' should not and will never be changed.

> ### Is it able to specify default value if a variable is not set?

> Yes. It is able to specify default value by `_context.get('varname', defaultvalue)`.

> File 'ex5.pyhtml':

>     Hello ${_context.get('username', 'Guest')}!

> Result:

>     $ pytenjin -c 'username="Tenjin"' ex5.pyhtml
>     Hello Tenjin!
>     $ pytenjin ex5.pyhtml
>     Hello Guest!

> ### Is pyTenjin ready for Google App Engine?

> Yes. You can use pyTenjin in Google App Engine (GAE).

> Compared to Django template engine, there are some merits to use pyTenjin in Google AppEngine:

> - pyTenjin runs much faster than Django template engine. If you are facing with CPU quota, you should try pyTenjin.
> - pyTenjin library is consist of only one file. If you are facing with number of files quota, you should try pyTenjin.

> The following is an example to use Tenjin in Google AppEngine with memcache.

> gae-example1.py

>     from google.appengine.ext import webapp
>     from google.appengine.ext.webapp.util import run_wsgi_app

>     import tenjin
>     from tenjin.helpers import *
>     shared_cache = tenjin.GaeMemcacheCacheStorage()
>     engine = tenjin.Engine(cache=shared_cache)

>     ## it is recommended to configure logging
>     import logging
>     logging.basicConfig(level=logging.DEBUG)
>     tenjin.logger = logging

>     class MainPage(webapp.RequestHandler):
>       def get(self):
>         context = {'title': 'Tenjin Example',
>                    'items': ['<AAA>','B&B','"CCC"'] }
>         html = engine.render("index.pyhtml", context)
>         self.response.out.write(html)

>     application = webapp.WSGIApplication([('/', MainPage)], debug=True)

>     def main():
>       run_wsgi_app(application)

>     if __name__ == "__main__":
>       main()

> ## Template

> ### Is it able to specify variables passed to template?

> Yes. You can specify template arguments by '`<?py #@ARGS arg1, arg2, arg3 ?>`'.

> File 'ex6-layout.pyhtml'

>     <?xml version="1.0 ?>
>     <?py #@ARGS x, y ?>
>     <p>
>       x = #{x}
>       y = #{y}
>       z = #{z}
>     </p>

> Template arguments line is converted into local variable assignment statements.

> Source code

>     $ pytenjin -s ex6.pyhtml
>     _buf = []; _buf.extend(('''<?xml version="1.0 ?>\n''', ));
>     x = _context.get('x'); y = _context.get('y');
>     _buf.extend(('''<p>
>       x = ''', to_str(x), '''
>       y = ''', to_str(y), '''
>       z = ''', to_str(z), '''
>     </p>\n''', ));
>     print ''.join(_buf)

> Undeclared arguments are not available even when they are passed via context object.

> Result:

>     $ pytenjin -c 'x=10; y=20; z=30' ex6.pyhtml
>       File "ex6.pyhtml", line 6, in <module>
>         z = #{z}
>     NameError: name 'z' is not defined

> ### It is hard to specify all template arguments. Any good idea?

> There may be some cases that it is hard to specify all template arguments. But `_context.get('varname', None)` is much hard and pain.

> One idea is to expand `@varname` into `_context.get('varname', None)` automaticaly.

> File 'ex7-expandargs.py':

>     import tenjin
>     from tenjin.helpers import *
>     import re

>     def _expand(code):
>         """expand '@var' into '_context.get("var", None)'"""
>         return re.sub(r"@(\w+)", r"_context.get('\1', None)", code)

>     class MyTemplate(tenjin.Template):

>         def add_stmt(self, buf, code):
>             tenjin.Template.add_stmt(self, buf, _expand(code))

>         def add_expr(self, buf, code, flag_escape=None):
>             tenjin.Template.add_expr(self, buf, _expand(code), flag_escape)

>     print("----- script -----")
>     print(MyTemplate('ex7-expandargs.pyhtml').script)

>     print("----- result -----")
>     tenjin.Engine.templateclass = MyTemplate
>     engine = tenjin.Engine()
>     html = engine.render('ex7-expandargs.pyhtml')
>     print(html)

> File 'ex7-expandargs.pyhtml':

>     <p>Hello #{@user or 'guest'}!<p>
>     <?py if @message: ?>
>     <p>#{@message}</p>
>     <?py #endif ?>

> Result:

>     $ python ex7-expandargs.py
>     ----- script -----
>     _buf.extend(('''<p>Hello ''', to_str(_context.get('user', None) or 'guest'), '''!<p>\n''', ));
>     if _context.get('message', None):
>         _buf.extend(('''<p>''', to_str(_context.get('message', None)), '''</p>\n''', ));
>     #endif

>     ----- result -----
>     <p>Hello guest!<p>

> ### Is it able to change embedded expression pattern?

> Yes, you can create subclass of Template class and override embedded expression pattern.

> ex8-expr-pattern.pyhtml:

>     <p>HTML escaped: [|value|]</p>
>     <p>not escaped:  [:value:]</p>

> ex8-expr-pattern.py:

>     import tenjin, re
>     from tenjin.helpers import *

>     class MyTemplate(tenjin.Template):

>         ## '[|expr|]' escapes HTML and '[:expr:]' doesn't
>         EXPR_PATTERN = re.compile('\[(\|(.*?)\||:(.*?):)\]', re.S);

>         ## return pattern object for embedded expressions
>         def expr_pattern(self):
>             return MyTemplate.EXPR_PATTERN

>         ## return expression string and flag whether escape or not from matched object
>         def get_expr_and_escapeflag(self, match):
>             expr = match.group(2) or match.group(3)
>             escapeflag = match.group(2) and True or False
>             return expr, escapeflag

>     if __name__ == '__main__':
>         context = {'value': 'AAA&BBB'}
>         engine = tenjin.Engine(templateclass=MyTemplate)
>         output = engine.render('ex8-expr-pattern.pyhtml', context)
>         print output,

> Result:

>     $ python ex8-expr-pattern.py
>     <p>HTML escaped: AAA&amp;BBB</p>
>     <p>not escaped:  AAA&BBB</p>

> ### Does pyTenjin support M17N?

> No. pyTenjin doesn't provide M17 feature.

> pyTenjin doesn't provide M17N feature directly because requirements for M17N are different for each applications or frameworks. Some applications or frameworks adapt GetText library and others use their original M17N library. What pyTenjin should do is not to provide M17N feature but to show an example to support M17N.

> But using preprocessing, you can make your M17N template files much faster. See the next section for details.

> ### Is it possible to create separated template chaches for each language?

> Yes.

> The point is:

> - Change cache filename according to language. For example, create cache file 'file.pyhtml.en.cache', 'file.pyhtml.fr.cache', 'file.pyhtml.it.cache', and so on from template file 'file.pyhtml'. This can be done by overriding CacheStorage#_cachename().
> - Create CacheStorage and Engine object for each language.
> - Use preprocessing to create different cache content for each language.

> The following is an example to generate M17N pages from a template file.

> ex9-m18n.pyhtml:

>     <div>
>     <?PY ## '_()' represents translator method ?>
>      <p>${{_('Hello')}} ${username}!</p>
>     </div>

> ex9-m18n.py:

>     # -*- coding: utf-8 -*-
>     import tenjin
>     from tenjin.helpers import *
>     import re

>     ##
>     ## message catalog to translate message
>     ##
>     MESSAGE_CATALOG = {
>         'en': { 'Hello': 'Hello',
>                 'Good bye': 'Good bye',
>               },
>         'fr': { 'Hello': 'Bonjour',
>                 'Good bye': 'Au revoir',
>               },
>     }

>     ##
>     ## create translation function and return it.
>     ## ex.
>     ##    _ = create_translation_func('fr')
>     ##    print _('Hello')   #=> 'Bonjour'
>     ##
>     def create_translation_func(lang):
>         dict = MESSAGE_CATALOG.get(lang)
>         if not dict:
>             raise ValueError("%s: unknown lang." % lang)
>         def func(message_key):
>             return dict.get(message_key)
>         return func

>     ##
>     ## cache storage class to cache template object for each language
>     ##
>     class M17nCacheStorage(tenjin.MarshalCacheStorage):

>         lang = 'en'       # default language

>         def __init__(self, *args, **kwargs):
>             if 'lang' in kwargs:
>                 lang = kwargs.pop('lang')
>                 if lang:
>                     self.lang = lang
>             tenjin.MarshalCacheStorage.__init__(self, *args, **kwargs)

>         ## change cache filename to 'file.pyhtml.lang.cache'
>         def _cachename(self, fullpath):
>             return "%s.%s.cache" % (fullpath, self.lang)

>     ##
>     ## test program
>     ##
>     if __name__ == '__main__':

>         template_name = 'ex9-m18n.pyhtml'
>         common_context = { 'username': 'World' }

>         ## create cache storage and engine for English
>         m17ncache = M17nCacheStorage(lang='en')
>         engine = tenjin.Engine(preprocess=True, cache=m17ncache)

>         ## render html for English
>         context = common_context.copy()
>         context['_'] = create_translation_func('en')
>         html = engine.render(template_name, context)
>         print("--- lang: en ---")
>         print(html)

>         ## create cache storage and engine for French
>         m17ncache = M17nCacheStorage(lang='fr')
>         engine = tenjin.Engine(preprocess=True, cache=m17ncache)

>         ## render html for French
>         context = common_context.copy()
>         context['_'] = create_translation_func('fr')
>         html = engine.render(template_name, context)
>         print("--- lang: fr ---")
>         print(html)

> Result:

>     $ python ex9-m18n.py
>     --- lang: en ---
>     <div>
>      <p>Hello World!</p>
>     </div>

>     --- lang: fr ---
>     <div>
>      <p>Bonjour World!</p>
>     </div>

> After that, you can find two cache files are created.

>     $ ls ex9-m18n.pyhtml*
>     ex9-m18n.pyhtml  ex9-m18n.pyhtml.en.cache  ex9-m18n.pyhtml.fr.cache

> And each cache files have different content.

>     ### "_('Hello')" is translated into "Hello" in Engilish cache file
>     $ pytenjin -a dump ex9-m18n.pyhtml.en.cache
>     _buf.extend(('''<div>
>      <p>Hello ''', escape(to_str(username)), '''!</p>
>     </div>\n''', ));

>     ### "_('Hello')" is translated into "Bonjour" in French cache file
>     $ pytenjin -a dump ex9-m18n.pyhtml.fr.cache
>     _buf.extend(('''<div>
>      <p>Bonjour ''', escape(to_str(username)), '''!</p>
>     </div>\n''', ));

> ## Layout Template

> ### Can I change layout template name in a template file?

> Yes. If you set `_context['_layout']`, its value is regarded as layout template name.

> - You can specify template file name (ex. 'user_list.pyhtml') or template short name (ex. ':list').
> - If you set True to '_context['_layout']', default layout template name is used instead.
> - It is able to N-th level nested template.

> See the next section for details.

> ### Can I nest layout templates for any depth?

> Yes. If you set `_context['_layout']`, you can nest layout templates in any depth.

> The following example shows that:

> - 'ex9-content.pyhtml' uses 'ex9-mylayout.pyhtml' as layout template.
> - 'ex9-mylayout.pyhtml' uses 'ex9-baselayout.pyhtml' as layout template.
> File 'ex10-content.pyhtml':

>     <?py _context['title'] = 'Changing Layout Template Test' ?>
>     <?py ## specify layout template name ?>
>     <?py _context['_layout'] = 'ex10-mylayout.pyhtml' ?>
>     foo
>     bar
>     baz

> File 'ex10-mylayout.pyhtml':

>     <?py ## use default layout template name ?>
>     <?py _context['_layout'] = True ?>
>     <div id="content">
>     <?py _buf.append(_content) ?>
>     </div>

> File 'ex10-baselayout.pyhtml':

>     <html>
>       <body>
>     <?py if 'title' in locals(): ?>
>         <h1>${title}</h1>
>     <?py #end ?>
>     <?py _buf.append(_content) ?>
>       </body>
>     </html>

> Result:

>     $ pytenjin --layout=ex10-baselayout.pyhtml ex10-content.pyhtml
>     <html>
>       <body>
>         <h1>Changing Layout Template Test</h1>
>     <div id="content">
>     foo
>     bar
>     baz
>     </div>
>       </body>
>     </html>

> ### Can I disable default layout template for a certain template?

> Yes. If you set False to _context['_layout'], default layout template will not be applied.

> ### Is Django-like "Template Inheritance" supported?

> No, but you can emulate it partially by combination of template capturing and '_context['_layout']'.

> File 'ex11-baselayout.pyhtml':

>     <html>
>      <body>

>     <?py ## if variable 'header_part' is defined then print it, ?>
>     <?py ## else print default header part. ?>
>       <div id="header">
>     <?py if not captured_as('header_part'): ?>
>        <img src="img/logo.png" alt="logo" ?>
>     <?py #end ?>
>       </div>

>     <?py ## main content part ?>
>       <div id="content">
>     <?py _buf.append(content_part) ?>
>       </div>

>     <?py ## if variable 'footer_part' is defined then print it, ?>
>     <?py ## else print default footer part. ?>
>       <div id="footer">
>     <?py if not captured_as('footer_part'): ?>
>        <hr />
>        <em>webmaster@example.com</em>
>     <?py #end ?>
>       </div>

>      </body>
>     </html>

> File 'ex11-customlayout.pyhtml':

>     <?py ## '_context["_layout"]' is equivarent to '{% extends "foobar.html" %}' ?>
>     <?py ## in Django template engine. ?>
>     <?py _context['_layout'] = 'ex11-baselayout.pyhtml' ?>

>     <?py ## you can override header or footer by capturing. ?>
>     <?py start_capture('footer_part') ?>
>     <address style="text-align:right">
>       copyright&copy; 2007 kuwata-lab all rights reserved<br />
>       <a href="webmaster&#64;kuwata-lab.com">webmaster&#64;kuwata-lab.com</a>
>     </address>
>     <?py stop_capture() ?>

> File 'ex11-content.pyhtml':

>     <?py ## '_context["_layout"]' is equivarent to '{% extends "foobar.html" %}' ?>
>     <?py ## in Django template engine. ?>
>     <?py _context['_layout'] = 'ex11-customlayout.pyhtml' ?>

>     <?py ## main content part ?>
>     <?py start_capture('content_part') ?>
>     <ul>
>     <?py for item in items: ?>
>       <li>${item}</li>
>     <?py #end ?>
>     </ul>
>     <?py stop_capture() ?>

> '`captured_as()`' is a pre-defined helper function. For example,

>     <?py if not captured_as('header_part'): ?>
>        <img src="img/logo.png" alt="logo" ?>
>     <?py #end ?>

> is equivarent to the following.

>     <?py if 'header_part' in _context: ?>
>     <?py     _buf.append(_context['header_part']) ?>
>     <?py else: ?>
>        <img src="img/logo.png" alt="logo" ?>
>     <?py #end ?>

> The following is the result. It shows that footer part in baselayout is overrided by other templates.

> Result:

>     $ pytenjin -c "items=['AAA', 'BBB', 'CCC']" ex11-content.pyhtml
>     <html>
>      <body>

>       <div id="header">
>        <img src="img/logo.png" alt="logo" ?>
>       </div>

>       <div id="content">
>     <ul>
>       <li>AAA</li>
>       <li>BBB</li>
>       <li>CCC</li>
>     </ul>
>       </div>

>       <div id="footer">
>     <address style="text-align:right">
>       copyright&copy; 2007 kuwata-lab all rights reserved<br />
>       <a href="webmaster&#64;kuwata-lab.com">webmaster&#64;kuwata-lab.com</a>
>     </address>
>       </div>

>      </body>
>     </html>

> ## Encoding

> ### How to specify template encoding?

> pyTenjin supports two approaches to handle template encoding.

> One approach is binary(=str)-based: to handle template content as string (byte sequence). In this approach, you must decode unicode object to str in 'to_str()' function. Command-line option '-k encoding' is equivarent to this.

> The other one is unicode-based: to convert template content into unicode object. In this approach, you must decode binary(=str) data into unicode object in 'to_str()' function. Command-line option '--encoding=encoding' is equivarent the second way.

> See User's Guide for details.

> In Python 3.0, pyTenjin supports only unicode-based approach because str object represents unicode.

> ### Can I specify encoding name in template file?

> Yes. You can contain encoding declaration in template.

> File 'ex12.pyhtml':

>     <?py # -*- coding: utf-8 -*- ?>
>     <?py s1 = '..non-ascii characters..' ?>
>     <?py s2 = u'..non-ascii characters..' ?>
>     s1 = ${s1}  # OK
>     s2 = ${s2}  # OK

> ### I got 'SyntaxError: encoding declaration in Unicode string'

> This is because you added encoding declaration in template file AND you specified encoding option to tenjin.Engine() or tenjin.Template().

> Solution:

> - If you specify encoding option to tenjin.Engine() or tenjin.Template(), remove encoding declaration from your template file.
> - If you want to add encoding declaration into your template file, don't specify encoding option to tenjin.Engine() or tenjin.Template().

> ### I got UnicodeDecodeError, but I can't find what is wrong

> If you got UnicodeDecodeError, you should do the following solutions.

> - Set logger to `tenjin.logger`. If you set, pyTenjin will report the content of `_buf`.

>       import logging
>       logging.basicConfig(level=logging.DEBUG)
>       tenjin.logger = logging

> - Render tempalte with specifying `_buf` and check it directly.

>       _buf = []
>       try:
>           engine.get_template("index.pyhtml").render(context, _buf=_buf)
>           print(''.join(_buf))
>       except UnicodeDecodeError:
>           for item in _buf:
>               if isinstance(item, str):
>                   try:
>                       str.decode('ascii')
>                   except UnicodeDecodeError:
>                       print("*** failed to decode: %s" % repr(item))

> ## Preprocessing

> ### What is preprocessing?

> Preprocessing is a feature to evaluate a part of logics embedded in template files at when template is loaded.

> Tenjin has two stages for rendering:

> Convertion stage
>   Convert template into Python script. This stage is invoked only once for each template files.
> Evaluation stage
>   Evaluate converted script with given context data. This stage is invoked every time when template is rendered.

> Normally, embedded logics in template files are evaluated at Evaluation stage. pyTenjin can also evaluate a part of logics at convertion stage. It is called preprocessing.

> Preprocessed logics are evaluated only once because it is evaluated at convertion stage. It means that preprocessed logics are not evaluated at rendering template.

> kind | non-preprocessing | preprocessing
> --- | --- | ---
> statements | <?py ... ?> | <?PY ... ?>
> expression (with escape) | ${...} | ${{...}}
> expression (without escape) | #{...} | #{{...}}

> ### What is the merit of preprocessing?

> The merit of preprocessing is the speed of rendering templates.

> Preprocessed logics are evaluated only once because it is evaluated at convertion stage and not evaluated at rendering templates. It means that preprocessed logics are no-weight when rendering time.

> For example, assume an helper function 'link_to()' which generates <a></a> tag. If you embed it to your template file such as '#{link_to("Create", action="new")}', this function will be evaluated whenever template is rendered.

> Without preprocessing

>     template file:
>       #{link_to("Create", action="new")}

>     converted script:
>       _buf.extend((to_str(link_to("Create", action="new")), ))

>     output:
>       <a href="/new">Create</a>

> However, if you use preprocessing such as '#{{link_to("Create", action="new")}}', this function will be evaluated only once when template is loaded.

> With preprocessing

>     template file:
>       #{{link_to('Create', action='new')}}

>     converted script:
>       _buf.extend(('''<a href="/new">Create</a>''', ))

>     output:
>       <a href="/new">Create</a>

> In the result, rendering template will be much faster because function evaluation is eliminated when rendering.

> ### Is there any examples of preprocessing?

> #### Loop expantion

> Using preprocessing, it is able to expand loop in advance. It makes rendering speed much faster.

> weekday1.pyhtml:

>     <?PY WEEKDAYS = ['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat'] ?>
>     <select name="weekday">
>       <option>-</option>
>     <?PY i = 0 ?>
>     <?PY for wday in WEEKDAYS: ?>
>     <?PY     i += 1?>
>       <option value="#{{i}}">#{{wday}}</option>
>     <?PY #end ?>
>     </select>

> weekday1.py:

>     ## import all helper methods
>     import tenjin
>     from tenjin.helpers import *
>     ## render with preprocessing
>     engine = tenjin.Engine(preprocess=True)
>     print '***** preprocessed *****'
>     print engine.get_template('weekday1.pyhtml').script,
>     print '***** output *****'
>     print engine.render('weekday1.pyhtml'),

> result:

>     $ python weekday1.py
>     ***** preprocessed *****
>     _buf.extend(('''<select name="weekday">
>       <option>-</option>
>       <option value="1">Sun</option>
>       <option value="2">Mon</option>
>       <option value="3">Tue</option>
>       <option value="4">Wed</option>
>       <option value="5">Thu</option>
>       <option value="6">Fri</option>
>       <option value="7">Sat</option>
>     </select>\n''', ));
>     ***** output *****
>     <select name="weekday">
>       <option>-</option>
>       <option value="1">Sun</option>
>       <option value="2">Mon</option>
>       <option value="3">Tue</option>
>       <option value="4">Wed</option>
>       <option value="5">Thu</option>
>       <option value="6">Fri</option>
>       <option value="7">Sat</option>
>     </select>

> If you want to add selected attribute (' selected="selected"') dinamically, see the following.

> weekday2.pyhtml:

>     <?PY WEEKDAYS = ['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat'] ?>
>     <select name="weekday">
>     <?py selected = { str(params.get('weekday')): ' selected="selected"' } ?>
>       <option>-</option>
>     <?PY i = 0 ?>
>     <?PY for wday in WEEKDAYS: ?>
>     <?PY     i += 1?>
>       <option value="#{{i}}"#{selected.get('#{{i}}')}>#{{wday}}</option>
>     <?PY #end ?>
>     </select>

> weekday2.py:

>     ## import all helper methods
>     import tenjin
>     from tenjin.helpers import *
>     ## render with preprocessing
>     engine = tenjin.Engine(preprocess=True)
>     context = { 'params': { 'weekday': 3, 'day': 19 } }
>     print('***** preprocessed *****')
>     print(engine.get_template('weekday2.pyhtml').script)
>     print('***** output *****')
>     print(engine.render('weekday2.pyhtml', context))

> result:

>     $ python weekday2.py
>     ***** preprocessed *****
>     _buf.extend(('''<select name="weekday">\n''', ));
>     selected = { str(params.get('weekday')): ' selected="selected"' }
>     _buf.extend(('''  <option>-</option>
>       <option value="1"''', to_str(selected.get('1')), '''>Sun</option>
>       <option value="2"''', to_str(selected.get('2')), '''>Mon</option>
>       <option value="3"''', to_str(selected.get('3')), '''>Tue</option>
>       <option value="4"''', to_str(selected.get('4')), '''>Wed</option>
>       <option value="5"''', to_str(selected.get('5')), '''>Thu</option>
>       <option value="6"''', to_str(selected.get('6')), '''>Fri</option>
>       <option value="7"''', to_str(selected.get('7')), '''>Sat</option>
>     </select>\n''', ));

>     ***** output *****
>     <select name="weekday">
>       <option>-</option>
>       <option value="1">Sun</option>
>       <option value="2">Mon</option>
>       <option value="3" selected="selected">Tue</option>
>       <option value="4">Wed</option>
>       <option value="5">Thu</option>
>       <option value="6">Fri</option>
>       <option value="7">Sat</option>
>     </select>

> It is possible to make helper function to generate <select> and <option> tags.

> weekday3.pyhtml:

>     <form>
>     #{{pp_select_weekday_tag("params.get('weekday')", name='weekday')}}
>     </form>

> weekday3.py:

>     ## helper function
>     WEEKDAYS = ['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat']
>     def pp_select_weekday_tag(expr_str, name='weekday', indent=''):
>         buf = []
>         buf.append('<select name="%s">' % name)
>         attr = 'selected="selected"'
>         buf.append('%s<?py _selected = { str(%s): \' %s\' } ?>' % \
>                           (indent, expr_str, attr))
>         buf.append('  <option>-</option>')
>         i = 0
>         for wday in WEEKDAYS:
>             i += 1
>             expr = '_selected.get("%s")' % i
>             buf.append('  <option value="%s"#{%s}>%s</option>' % \
>                           (i, expr, wday))
>         buf.append('</select>')
>         return "\n".join(buf)

>     ## import all helper methods
>     import tenjin
>     from tenjin.helpers import *

>     ##
>     engine = tenjin.Engine(preprocess=True)
>     context = { 'params': { 'weekday': 3, 'day': 19 } }
>     print '***** preprocessed *****'
>     print engine.get_template('weekday3.pyhtml').script,
>     print '***** output *****'
>     print engine.render('weekday3.pyhtml', context),

> result:

>     $ python weekday3.py
>     ***** preprocessed *****
>     _buf.extend(('''<form>
>     <select name="weekday">\n''', ));
>     _selected = { str(params.get('weekday')): ' selected="selected"' }
>     _buf.extend(('''  <option>-</option>
>       <option value="1"''', to_str(_selected.get("1")), '''>Sun</option>
>       <option value="2"''', to_str(_selected.get("2")), '''>Mon</option>
>       <option value="3"''', to_str(_selected.get("3")), '''>Tue</option>
>       <option value="4"''', to_str(_selected.get("4")), '''>Wed</option>
>       <option value="5"''', to_str(_selected.get("5")), '''>Thu</option>
>       <option value="6"''', to_str(_selected.get("6")), '''>Fri</option>
>       <option value="7"''', to_str(_selected.get("7")), '''>Sat</option>
>     </select>
>     </form>\n''', ));
>     ***** output *****
>     <form>
>     <select name="weekday">
>       <option>-</option>
>       <option value="1">Sun</option>
>       <option value="2">Mon</option>
>       <option value="3" selected="selected">Tue</option>
>       <option value="4">Wed</option>
>       <option value="5">Thu</option>
>       <option value="6">Fri</option>
>       <option value="7">Sat</option>
>     </select>
>     </form>

> #### Helper methods execution in advance

> Many web frameworks provides their own helper functions for view layer. Some of them can be executed in advance. Using preprocessing, it is able to execute these helper functions in advance, and view layer will be much faster in the result.

> helpers1.pyhtml:

>     <p>
>     #{link_to('Create', action='new')}
>     #{{link_to('Create', action='new')}}
>     </p>

> helpers1.py:

>     ## define helper method
>     def link_to(label, href=None, action=None):
>         if not href and action:
>             href = "/%s/%s" % (controller_name, action)
>         return '<a href="%s">%s</a>' % (href, label)

>     ## import all helper methods to use preprocessing
>     import tenjin
>     from tenjin.helpers import *

>     ##
>     controller_name = 'user'
>     engine = tenjin.Engine(preprocess=True)
>     print '***** preprocessed *****'
>     print engine.get_template('helpers1.pyhtml').script,
>     print '***** output *****'
>     print engine.render('helpers1.pyhtml'),

> result:

>     $ python helpers1.py
>     ***** preprocessed *****
>     _buf.extend(('''<p>
>     ''', to_str(link_to('Create', action='new')), '''
>     <a href="/user/new">Create</a>
>     </p>\n''', ));
>     ***** output *****
>     <p>
>     <a href="/user/new">Create</a>
>     <a href="/user/new">Create</a>
>     </p>

> It is able to embed expression which should be evaluated at rendering stage.

> - `_p("...")` is equivarent to `#{...}`
> - `_P("...")` is equivarent to `${...}`
> helpers2.pyhtml:

>     <p>
>     #{link_to(escape(user['name']), action='show', id=user['id'])}
>     #{{link_to(_P("user['name']"), action='show', id=_p("user['id']"))}}
>     </p>

> helpers2.py:

>     ## define helper method
>     def link_to(label, href=None, action=None, id=None):
>         if not href and action:
>             if id:
>                 href = "/%s/%s/%s" % (controller_name, action, id)
>             else:
>                 href = "/%s/%s" % (controller_name, action)
>         return '<a href="%s">%s</a>' % (href, label)

>     ## import all helper methods to use preprocessing
>     import tenjin
>     from tenjin.helpers import *

>     ##
>     controller_name = 'user'
>     context = { 'user': {'id': 123, 'name': 'Tom&Jerry'} }
>     engine = tenjin.Engine(preprocess=True)
>     print '***** preprocessed *****'
>     print engine.get_template('helpers2.pyhtml', context).script,
>     print '***** output *****'
>     print engine.render('helpers2.pyhtml', context),

> result:

>     $ python helpers2.py
>     ***** preprocessed *****
>     _buf.extend(('''<p>
>     ''', to_str(link_to(escape(user['name']), action='show', id=user['id'])), '''
>     <a href="/user/show/''', to_str(user['id']), '''">''', escape(to_str(user['name'])), '''</a>
>     </p>\n''', ));
>     ***** output *****
>     <p>
>     <a href="/user/show/123">Tom&amp;Jerry</a>
>     <a href="/user/show/123">Tom&amp;Jerry</a>
>     </p>

> #### M17N (Multilingualization)

> Preprocessing is also effective for M17N (Multilingualization), because the runtime cost of M17N can be almost zero by preprocessing.

> See this section for M17N example using preprocessing.

> ## Performance

> ### How fast is pyTenjin compared with other solutions?

> pyTenjin contains benchmark script. This shows that pyTenjin works much faster than other solutions.

> MacOS X 10.4 Tiger, Intel CoreDuo 1.83GHz, Memory 2GB

>     $ cd pyTenjin-X.X.X/benchmark
>     $ python -V
>     Python 2.5.1
>     $ python bench.py -q -n 10000
>     Compiling bench_cheetah.tmpl -> bench_cheetah.py (backup bench_cheetah.py.bak)
>     *** ntimes=10000
>                                utime      stime      total       real
>     tenjin                   6.47000    0.49000    6.96000    6.98909
>     tenjin-reuse             5.54000    0.06000    5.61000    5.63055
>     tenjin-nocache          20.14000    0.41000   20.55000   20.60475
>     django                  69.99000    1.34000   71.33000   71.57211
>     django-reuse            58.92000    0.88000   59.80000   59.94480
>     cheetah                 20.33000    0.03000   20.36000   20.41416
>     cheetah-reuse           19.80000    0.02000   19.82000   19.86858
>     myghty                 106.25000    1.63000  107.88000  108.16097
>     myghty-reuse            18.70000    0.60000   19.30000   19.35395
>     kid                    379.64000    0.60000  380.24000  381.11728
>     kid-reuse              378.52000    0.44000  378.96000  379.64911
>     genshi                 557.29000    3.00000  560.30000  561.71955
>     genshi-reuse           270.47000    1.22000  271.69000  272.26885
>     mako                    16.82000    0.96000   17.78000   18.36388
>     mako-reuse              13.47000    0.02000   13.49000   13.51232
>     mako-nocache           236.10000    1.67000  237.77000  238.38705
>     templetor              424.03000    4.15000  428.19000  429.59667
>     templetor-reuse         61.46000    0.07000   61.53000   61.68483

> Versions:

> - Python 2.5.1
> - Tenjin 0.6.1
> - Django 0.95
> - Cheetah 2.0
> - Myghty 1.1
> - Kid 0.9.6
> - Genshi 0.4.4
> - Mako 0.1.9
> - Templetor (web.py) 0.22

> In addition, module size of pyTenjin is small, and it is very light-weight to import it. This is important for CGI program. Other solutions may be very heavy to import the module and suitable only for apache module or FastCGI.

> ### Why pyTenjin is so fast?

> Because it doesn't use template engine original language.

> Other template engines, such as Template-Toolkit(perl), Django(python), or Smarty(php), has their original languages. This is not good idea for script language because:

> - They are slow.
> - Implementation will be complex.
> - Learning cost is high.

> In addition, pyTenjin is faster than Jakarta Velocity which is a very popular template engine in Java. (It means that dynamic Java is slower than script languages!)

> Template engine should use their host language directly unless there are some kind of reasons.

> ### Is there any way to get more speed?

> You can get more speed by including 'escape()' and 'to_str()' functions to context data.

> File 'ex13a.py':

>     import tenjin
>     from tenjin.helpers import *

>     ## include 'escape()' and 'to_str()' functions to context data
>     context = { 'title': 'Example', 'items': ['A', 'B', 'C'] }
>     context['escape'] = escape
>     context['to_str'] = to_str

>     engine = tenjin.Engine()
>     output = engine.render('ex13a.pyhtml', context)

> You can get more and more speed by deleting implicit call of 'to_str()' function. Of course, you have to call it properly in your templates.

> File 'ex13b.py':

>     import tenjin
>     from tenjin.helpers import *

>     ## include 'escape()' and 'to_str()' functions to context data
>     context = { 'title': 'Example', 'items': ['A', 'B', 'C'] }
>     context['escape'] = escape
>     context['to_str'] = to_str

>     ## delete implicit call of 'to_str()' function
>     engine = tenjin.Engine(tostrfunc='')

>     ## show python code and output
>     filename = 'ex13b.pyhtml'
>     template = engine.get_template(filename)
>     output = engine.render(filename, context)
>     print "--- python code ---"
>     print template.script
>     print "--- output ---"
>     print output,

> File 'ex13b.pyhtml':

>     <h1>${title}</h1>
>     <ul>
>     <?py for i, item in enumerate(items): ?>
>       <li>#{to_str(i)}: #{item}</li>
>     <?py #end ?>
>     </ul>

> Result:

>     $ python ex13b.py
>     --- python code ---
>     _buf.extend(('''<h1>''', escape((title)), '''</h1>
>     <ul>\n''', ));
>     for i, item in enumerate(items):
>         _buf.extend(('''  <li>''', (to_str(i)), ''': ''', (item), '''</li>\n''', ));
>     #end
>     _buf.extend(('''</ul>\n''', ));

>     --- output ---
>     <h1>Example</h1>
>     <ul>
>       <li>0: A</li>
>       <li>1: B</li>
>       <li>2: C</li>
>     </ul>