Metadata-Version: 2.4
Name: smoltok
Version: 0.1.1
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Rust
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Summary: Byte-Pair Encoding tokenizer implemented in Rust
Keywords: bpe,tokenizer,nlp,rust,encoding,text-processing
Home-Page: https://github.com/Winston-503/smoltok
Author-email: Dmytro Nikolaiev <dmytro.nikolaiev.ai@gmail.com>
License-Expression: Apache-2.0
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://github.com/Winston-503/smoltok
Project-URL: Repository, https://github.com/Winston-503/smoltok
Project-URL: Documentation, https://github.com/Winston-503/smoltok#readme
Project-URL: Issues, https://github.com/Winston-503/smoltok/issues

# smoltok 🦀

Byte-Pair Encoding tokenizer implemented in Rust.

See [GitHub Repository](https://github.com/Winston-503/smoltok) for documentation.

