Metadata-Version: 2.4
Name: ion-tokenizer
Version: 0.0.1
Summary: Phrase-first tokenizer for LLMs. 54-0 vs BPE.
Author-email: Creayo <support@creayodev.com>
License: Proprietary
Project-URL: Homepage, https://iontokenizer.com
Project-URL: Documentation, https://iontokenizer.com/docs
Project-URL: Repository, https://github.com/creayo-dev/ion
Keywords: tokenizer,nlp,llm,bpe,phrases
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# Ion Tokenizer

Phrase-first tokenizer for LLMs. 54-0 vs BPE in head-to-head benchmarks.

**Coming soon.** Visit [iontokenizer.com](https://iontokenizer.com) for updates.

## Installation
```bash
pip install ion-tokenizer
```

## Links

- Website: https://iontokenizer.com
- GitHub: https://github.com/creayo-dev/ion
- Company: https://creayodev.com
