# DOM tokenizers

HTML DOM-aware tokenizers for Hugging Face language models.

## Setup for development

```sh
git clone https://github.com/gbenson/dom-tokenizers.git
cd dom-tokenizers
python3 -m venv .venv
. .venv/bin/activate
pip install --upgrade pip
pip install -e .[dev,train]
```
