RnaTokenizer - MultiMolecule
Author: MultiMolecule
Description: Neural Networks for RNA, DNA, and Protein

跳转至

# RnaTokenizer

RnaTokenizer is smart, it tokenizes raw RNA nucleotides into tokens, no matter if the input is in uppercase or lowercase, uses U (Uracil) or U (Thymine), and with or without special tokens. It also supports tokenization into nmers and codons, so you don’t have to write complex code to preprocess your data.

By default, `RnaTokenizer` uses the standard alphabet. If `nmers` is greater than `1`, or `codon` is set to `True`, it will instead use the streamline alphabet.

## `` multimolecule.tokenisers.RnaTokenizer

Bases: `Tokenizer`

Tokenizer for RNA sequences.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`alphabet` | `Alphabet | str | List[str] | None` | alphabet to use for tokenization. If is `None`, the standard RNA alphabet will be used. If is a `string`, it should correspond to the name of a predefined alphabet. The options include `standard` `iupac` `streamline` `nucleobase` If is an alphabet or a list of characters, that specific alphabet will be used. | `None`
`nmers` | `int` | Size of kmer to tokenize. | `1`
`codon` | `bool` | Whether to tokenize into codons. | `False`
`replace_T_with_U` | `bool` | Whether to replace T with U. | `True`
`do_upper_case` | `bool` | Whether to convert input to uppercase. | `True`

Examples:

Python Console Session

    >>> from multimolecule import RnaTokenizer
    >>> tokenizer = RnaTokenizer()
    >>> tokenizer('<pad><cls><eos><unk><mask><null>ACGUNIXVHDBMRWSYK.*-')["input_ids"]
    [1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 2]
    >>> tokenizer('acgu')["input_ids"]
    [1, 6, 7, 8, 9, 2]
    >>> tokenizer('acgt')["input_ids"]
    [1, 6, 7, 8, 9, 2]
    >>> tokenizer = RnaTokenizer(replace_T_with_U=False)
    >>> tokenizer('acgt')["input_ids"]
    [1, 6, 7, 8, 3, 2]
    >>> tokenizer = RnaTokenizer(nmers=3)
    >>> tokenizer('uagcuuauc')["input_ids"]
    [1, 83, 17, 64, 49, 96, 84, 22, 2]
    >>> tokenizer = RnaTokenizer(codon=True)
    >>> tokenizer('uagcuuauc')["input_ids"]
    [1, 83, 49, 22, 2]
    >>> tokenizer('uagcuuauca')["input_ids"]
    Traceback (most recent call last):
    ValueError: length of input sequence  must be a multiple of 3 for codon tokenization, but got 10

Source code in `multimolecule/tokenisers/rna/tokenization_rna.py`

Python
     30
     31
     32
     33
     34
     35
     36
     37
     38
     39
     40
     41
     42
     43
     44
     45
     46
     47
     48
     49
     50
     51
     52
     53
     54
     55
     56
     57
     58
     59
     60
     61
     62
     63
     64
     65
     66
     67
     68
     69
     70
     71
     72
     73
     74
     75
     76
     77
     78
     79
     80
     81
     82
     83
     84
     85
     86
     87
     88
     89
     90
     91
     92
     93
     94
     95
     96
     97
     98
     99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    class RnaTokenizer(Tokenizer):
        """
        Tokenizer for RNA sequences.

        Args:
            alphabet: alphabet to use for tokenization.

                - If is `None`, the standard RNA alphabet will be used.
                - If is a `string`, it should correspond to the name of a predefined alphabet. The options include
                    + `standard`
                    + `iupac`
                    + `streamline`
                    + `nucleobase`
                - If is an alphabet or a list of characters, that specific alphabet will be used.
            nmers: Size of kmer to tokenize.
            codon: Whether to tokenize into codons.
            replace_T_with_U: Whether to replace T with U.
            do_upper_case: Whether to convert input to uppercase.

        Examples:
            >>> from multimolecule import RnaTokenizer
            >>> tokenizer = RnaTokenizer()
            >>> tokenizer('<pad><cls><eos><unk><mask><null>ACGUNIXVHDBMRWSYK.*-')["input_ids"]
            [1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 2]
            >>> tokenizer('acgu')["input_ids"]
            [1, 6, 7, 8, 9, 2]
            >>> tokenizer('acgt')["input_ids"]
            [1, 6, 7, 8, 9, 2]
            >>> tokenizer = RnaTokenizer(replace_T_with_U=False)
            >>> tokenizer('acgt')["input_ids"]
            [1, 6, 7, 8, 3, 2]
            >>> tokenizer = RnaTokenizer(nmers=3)
            >>> tokenizer('uagcuuauc')["input_ids"]
            [1, 83, 17, 64, 49, 96, 84, 22, 2]
            >>> tokenizer = RnaTokenizer(codon=True)
            >>> tokenizer('uagcuuauc')["input_ids"]
            [1, 83, 49, 22, 2]
            >>> tokenizer('uagcuuauca')["input_ids"]
            Traceback (most recent call last):
            ValueError: length of input sequence  must be a multiple of 3 for codon tokenization, but got 10
        """

        model_input_names = ["input_ids", "attention_mask"]

        def __init__(
            self,
            alphabet: Alphabet | str | List[str] | None = None,
            nmers: int = 1,
            codon: bool = False,
            replace_T_with_U: bool = True,
            additional_special_tokens: List | Tuple | None = None,
            do_upper_case: bool = True,
            **kwargs,
        ):
            if codon and nmers > 1:
                raise ValueError("Codon and nmers cannot be used together.")
            if codon:
                nmers = 3  # set to 3 to get correct vocab
            if not isinstance(alphabet, Alphabet):
                alphabet = get_alphabet(alphabet, nmers=nmers)
            super().__init__(
                alphabet=alphabet,
                additional_special_tokens=additional_special_tokens,
                do_upper_case=do_upper_case,
                **kwargs,
            )
            self.replace_T_with_U = replace_T_with_U
            self.nmers = nmers
            self.condon = codon

        def _tokenize(self, text: str, **kwargs):
            if self.do_upper_case:
                text = text.upper()
            if self.replace_T_with_U:
                text = text.replace("T", "U")
            if self.condon:
                if len(text) % 3 != 0:
                    raise ValueError(
                        f"length of input sequence  must be a multiple of 3 for codon tokenization, but got {len(text)}"
                    )
                return [text[i : i + 3] for i in range(0, len(text), 3)]
            if self.nmers > 1:
                return [text[i : i + self.nmers] for i in range(len(text) - self.nmers + 1)]  # noqa: E203
            return list(text)

2024-05-27 09:26:37

#### Cookie Consent

We use cookies to recognize your repeated visits and preferences, as well as to measure the effectiveness of our documentation and whether users find what they're searching for. With your consent, you're helping us to make our documentation better. Please check out our Privacy Notice for more information.

- Google Analytics
- GitHub

管理设定