# BTok

A Python multilingual tokenizer trained on modern web corpus with [SentencePiece](https://github.com/google/sentencepiece).

![](https://img.shields.io/pypi/v/btok?label=btok&color=blue&cacheSeconds=60)

## Install

```sh
pip install btok --upgrade
```

## Usage

Run example:

```sh
python example.py
```

See: [example.py](https://github.com/Hansimov/btok/blob/main/example.py)

```python
```
