Token — super-collator 0.0.2 documentation

# Token

This module implements token.

class Token

  The token API.

  Implemented by SingleToken and MultiToken.

  __init__()

    Initialize the token.

  strategy_data: Any

    A data store reserved for use by strategies.

  __len__() → int

    Return how many tokens are in the tree.

  __iter__() → Iterator[Optional[super_collator.token.Token[super_collator.token.TT]]]

    Iterate the tokens in the tree.

  __getitem__(n) → Optional[super_collator.token.Token[super_collator.token.TT]]

    Return the nth token in the iteration.

  __str__() → str

    Return a string represenation of the token tree.

class SingleToken(data: super_collator.token.TT, user_data: Optional[Any] = None)

  Represents one input token.

  __init__(data: super_collator.token.TT, user_data: Optional[Any] = None)

    Initialize the token.

    Parameters

      user_data (Any) – user data that will be round-tripped unchanged.

  data: super_collator.token.TT

    The data to align.

  user_data: Any

    A data store reserved for user data.

  __len__() → int

    Return how many tokens are in the tree.

  __iter__() → Iterator[Optional[super_collator.token.Token[super_collator.token.TT]]]

    Iterate the tokens in the tree.

  __getitem__(n: int) → Optional[super_collator.token.Token[super_collator.token.TT]]

    Return the nth token in the iteration.

  __str__() → str

    Return a string represenation of the token tree.

class MultiToken(*args: Optional[super_collator.token.Token[super_collator.token.TT]])

  A container for one or two tokens.

  A MultiToken is a node in a binary tree that mimics the order in which multiple aligments were done.

  An alignment produces a sequence of MultiToken trees. Each tree contains all the tokens that were aligned at this position plus any gaps represented as None.

  __init__(*args: Optional[super_collator.token.Token[super_collator.token.TT]])

    Initialize the token.

  __len__() → int

    Return how many tokens are in the tree.

  __iter__() → Iterator[Optional[super_collator.token.Token[super_collator.token.TT]]]

    Iterate the tokens in the tree.

  __getitem__(n: int) → Optional[super_collator.token.Token[super_collator.token.TT]]

    Return the nth token in the iteration.

  __str__() → str

    Return a string represenation of the token tree.