# rag-tokenizer

# Upload to pypi.org 

```shell
uv build;
uv publish --token $PYPI_API_TOKEN
```

# Install
```shell
pip install rag-tokenizer
```

# Usage

```python
import rag_tokenizer
res = rag_tokenizer.tokenize("Hello world!")
print(res)
```