Lexical analysis — ptk 1.0.0 documentation

# Lexical analysis

`@``token`(regex, types=None)

  Decorator for token definitions in classes derived from `LexerBase`.

  Parameters: | rx (str) – A regular expression defining the possible token values types (List of strings) – A list of token types that this method can recognize. If omitted, the token type is assumed to be the method’s name.

  Basic usage:

      class MyLexer(ReLexer):
          @token(r'[a-zA-Z_][a-zA-Z0-9_]*')
          def identifier(self, tok):
              pass

  This will define an identifier token type, which value is the recognized string. The tok parameter holds two attributes, type and value. You can modify the value in place:

      class MyLexer(ReLexer):
          @token(r'[1-9][0-9]*')
          def number(self, tok):
              tok.value = int(tok.value)

  In some cases it may be necessary to change the token’s type as well; for instance to disambiguate between identifiers that are builtins and other ones. In order for the lexer to know which token types can be generated, you should pass a list of strings as the types parameter:

      class MyLexer(ReLexer):
          @token(r'[a-zA-Z_][a-zA-Z0-9_]*', types=['builtin', 'identifier'])
          def identifier_or_builtin(self, tok):
              tok.type = 'builtin' in tok.value in ['len', 'id'] else 'identifier'

  In this case the default value of the type attribute is None and you must set it. Letting None as token type (or setting it to None) will cause the token to be ignored.

  Note

  The type of token values depends on the type of the strings used to define the regular expressions. Unicode expressions will hold Unicode values, and bytes expressions will hold bytes values.

  Note

  Disambiguation is done the regular way: if several regular expressions match the input, the longest match is choosen. If the matches are of equal length, the first (in source code order) declaration wins.

exception `ptk.lexer.` `LexerError`(char, colno, lineno)

  Unrecognized token in input

  Variables: | lineno – Line in input colno – Column in input

class `ptk.lexer.` `LexerBase`

  This defines the interface for lexer classes. For concrete implementations, see `ProgressiveLexer` and `ReLexer`.

  class `Token`(type, value)
    `type`

      Alias for field number 0

    `value`

      Alias for field number 1

  `LexerBase.` `position`()

    Returns: | The current position in stream as a 2-tuple (column, line).

  `LexerBase.` `advanceColumn`(count=1)

    Advances the current position by count columns.

  `LexerBase.` `advanceLine`(count=1)

    Advances the current position by count lines.

  static `LexerBase.` `ignore`(char)

    Override this to ignore characters in input stream. The default is to ignore spaces and tabs.

    Parameters: | char – The character to test
    Returns: | True if char should be ignored

  `LexerBase.` `setConsumer`(consumer)

    Sets the current consumer. A consumer is an object with a feed method; all characters seen on the input stream after the consumer is set are passed directly to it. When the feed method returns a 2-tuple (type, value), the corresponding token is generated and the consumer reset to None. This may be handy to parse tokens that are not easily recognized by a regular expression but easily by code; for instance the following lexer recognizes C strings without having to use negative lookahead:

        class MyLexer(ReLexer):
            @token('"')
            def cstring(self, tok):
                class CString(object):
                    def __init__(self):
                        self.state = 0
                        self.value = StringIO.StringIO()
                    def feed(self, char):
                        if self.state == 0:
                            if char == '"':
                                return 'cstring', self.value.getvalue()
                            if char == '\\':
                                self.state = 1
                            else:
                                self.value.write(char)
                        elif self.state == 1:
                            self.value.write(char)
                            self.state = 0
                self.setConsumer(CString())

  `LexerBase.` `parse`(string)

    Parses the whole string

  `LexerBase.` `newToken`(tok)

    This method will be invoked as soon as a token is recognized on input.

    Parameters: | tok – The token. This is a named tuple with type and value attributes.

  classmethod `LexerBase.` `tokenTypes`()

    Returns: | the set of all token names, as strings.

class `ptk.lexer.` `ReLexer`

  Concrete lexer based on Python regular expressions. this is way faster than `ProgressiveLexer` but it can only tokenize whole strings.

class `ptk.lexer.` `ProgressiveLexer`

  Concrete lexer based on a simple pure-Python regular expression engine. This lexer is able to tokenize an input stream in a progressive fashion; just call the `ProgressiveLexer.feed()` method with whatever bytes are available when they’re available. Useful for asynchronous contexts.

  This is slow as hell.

  `feed`(char, charPos=None)

    Handle a single input character. When you’re finished, call this with EOF as argument.

`ptk.lexer.` `EOF`

  This is a singleton used to indicate end of stream. It may be used as a token, a token type and a token value. In the first case it is its own type and value.