Metadata-Version: 2.1
Name: tokenizer-hub
Version: 0.0.1
Summary: Yoctol Natural Language Tokenizer
Home-page: https://github.com/Yoctol/tokenizer-hub
Author: Solumilken
License: MIT
Platform: UNKNOWN
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Requires-Python: >=3.5
Requires-Dist: jieba (==0.39)
Requires-Dist: nltk (==3.3.0)
Requires-Dist: purewords (==0.1.1)

# tokenizer-hub

yoctol 乂卍oO煞氣ㄟtokenizerOo卍乂

Tokenizers have the same interface of Jieba:

```python
from tokenizer_hub import XXX_tokenizer
tokenizer = XXX_tokenizer()
tokenizer.lcut('我来到北京清华大学')
> ['我', '来到', '北京', '清华大学']
```


