translate.misc.sparse.SimpleParser

Package translate :: Package misc :: Module sparse :: Class SimpleParser |
[hide private]
[frames] | no frames]

# Class SimpleParser

source code

this is a simple parser

Instance Methods | [hide private]
__init__(self, defaulttokenlist=None, whitespacechars=`'``\t\r\n``'`, includewhitespacetokens=0) | source code
stringtokenize(self, text) makes strings in text into tokens... | source code
keeptogether(self, text) checks whether a token should be kept together | source code
isstringtoken(self, text) checks whether a token is a string token | source code
separatetokens(self, text, tokenlist=None) this separates out tokens in tokenlist from whitespace etc | source code
removewhitespace(self, text) this removes whitespace but lets it separate things out into separate tokens | source code
applytokenizer(self, inputlist, tokenizer) apply a tokenizer to a set of text, flattening the result | source code
applytokenizers(self, inputlist, tokenizers) apply a set of tokenizers to a set of text, flattening each time | source code
tokenize(self, source, tokenizers=None) tokenize the text string with the standard tokenizers | source code
findtokenpos(self, tokennum) finds the position of the given token in the text | source code
getlinepos(self, tokenpos) finds the line and character position of the given character | source code
raiseerror(self, message, tokennum) raises a ParserError | source code