tokens
Dict of ``{'state': [(regex, tokentype, new_state), ...], ...}``
The initial state is 'root'. ``new_state`` can be omitted to signify
no state transition. If it is a string, the state is pushed on the stack
and changed. If it is a tuple of strings, all states are pushed on the
stack and the current state will be the topmost. It can also be
``combined('state1', 'state2', ...)`` to signify a new, anonymous state
combined from the rules of two or more existing ones. Furthermore, it can
be '#pop' to signify going back one step in the state stack, or '#push'
to push the current state on the stack again.
The tuple can also be replaced with ``include('state')``, in which
case the rules from the state named by the string are included in the
current one.
- Value:
{'comments': [('(\\s*#.*)', Token.Comment)],
'filterExp': ['variable',
'object',
('\\s*[+*/<>=~!%&|-]+\\s*', Token.Operator),
('\\s*\\)', Token.Text, '#pop')],
'graph': [('\\s*(<[^>]*\\>)',
Token.Name.Class,
('triple', 'predObj')),
...
|
|