tokens
Dict of ``{'state': [(regex, tokentype, new_state), ...], ...}``
The initial state is 'root'. ``new_state`` can be omitted to signify
no state transition. If it is a string, the state is pushed on the stack
and changed. If it is a tuple of strings, all states are pushed on the
stack and the current state will be the topmost. It can also be
``combined('state1', 'state2', ...)`` to signify a new, anonymous state
combined from the rules of two or more existing ones. Furthermore, it can
be '#pop' to signify going back one step in the state stack, or '#push'
to push the current state on the stack again.
The tuple can also be replaced with ``include('state')``, in which
case the rules from the state named by the string are included in the
current one.
- Value:
{'comments': [('(\\s*#.*)', Token.Comment)],
'objList': [('\\s*\\)', Token.Text, '#pop'), 'object'],
'object': [('\\s*\\[', Token.Text, 'predObj'),
('\\s*<[^> ]*>', Token.Name.Attribute),
('\\s*("""(?:.|\\n)*?""")(\\@[a-z]{2-4}|\\^\\^<?[a-zA-Z0-9
\\-\\:_#/\\.]*>?)?\\s*',
<function callback at 0x850a33c>),
('\\s*".*?[^\\\\]"(?:\\@[a-z]{2-4}|\\^\\^<?[a-zA-Z0-9\\-\\
...
|
|