Metadata-Version: 2.1
Name: totokenizers
Version: 1.2.2
Summary: Text tokenizers.
Home-page: https://github.com/TeiaLabs/totokenizers
Author: TeiaLabs
Author-email: contato@teialabs.com
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: tiktoken
Requires-Dist: tokenizers
Provides-Extra: test
Requires-Dist: pytest; extra == "test"

# totokenizers

A model-agnostic library to encode text into tokens and couting them using different tokenizers.

## install

`pip install totokenizers`

## usage

```python
from totokenizers.factories import TotoModelInfo, Totokenizer

mdoel = "openai/gpt-3.5-turbo-0613"
desired_max_tokens = 250
tokenizer = Totokenizer.from_model(model)
model_info = TotoModelInfo.from_model(model)

thread_length = tokenizer.count_chatml_tokens(thread, functions)
if thread_length + desired_max_tokens > model_info.max_tokens:
    raise YourException(thread_length, desired_max_tokens, model_info.max_tokens)
```
