# README

These are tutorial notes for `tokenizer`.

## Generating Word Ngrams

The "standard" method is to create a spaCy doc and then call `textacy.extract.basics.ngrams`:

```python
import spacy
import textacy.extract.basics.ngrams as ng
nlp = spacy.load("xx_sent_ud_sm")
text = "The end is nigh."
doc = nlp(text)
ngrams = list(ng(doc, 2, min_freq=1))
```

This will produce `[The end, end is, is nigh]`. The output is a list of spaCy tokens. An additional `[token.text for token in ngrams]` is required to ensure that you have quoted strings: `["The end", "end is", "is nigh"]`.

Textacy has a lot of additional options, which are documented at [https://textacy.readthedocs.io/en/latest/api_reference/extract.html#textacy.extract.basics.ngrams](https://textacy.readthedocs.io/en/latest/api_reference/extract.html#textacy.extract.basics.ngrams). However, if you do not need these options, you can use Lexos' helper function `lexos.tokenizer.ngrams_from_doc`:

```python
import spacy
nlp = spacy.load("xx_sent_ud_sm")
text = "The end is nigh."
doc = nlp(text)
ngrams = ngrams_from_doc(doc, size=2)
```

Notice that in both cases, the output will be a list of overlapping ngrams generated by a rolling window across the pre-tokenised document. If you want your document to contain ngrams _as_ tokens, you will need to create a new document using Lexos' `lexos.tokenizer.doc_from_ngrams` function:

```python
doc = doc_from_ngrams(ngrams, strict=True)
```

Setting `strict=False` will preserve all the whitespace in the ngrams; otherwise, your language model may modify the output by doing things like splitting punctuation into separate tokens.

There is also a `lexos.tokenizer.docs_from_ngrams` function to which you can feed multiple lists of ngrams.

A possible workflow might call Textacy directly to take advantage of some its filters, when generating ngrams and then calling `lexos.tokenizer.doc_from_ngrams` to pipe the extracted tokens back into a doc. `textacy.extract.basics.ngrams` has sister functions that do things like extract noun chunks, making this a very powerful approach generating ngrams with semantic information.

## Generating Character Ngrams

Character ngrams at their most basic level split the _untokenised_ string every n characters. So "The end is nigh." would produce something like `["Th", "e ", "nd", " i", "s ", "ni", "gh", "."]` (if we wanted to preserve the whitespace). Lexos has the `lexos.tokenizer.generate_ngrams` function to do this:

```python
text = "The end is nigh."
ngrams = generate_character_ngrams(text, 2, drop_whitespace=False)
```

This will produce the output shown above. If we wanted to output `["Th", "en", "di", "sn", "ig", "h."]`, we would use `drop_whitespace=True` (which is the default). Note that this `lexos.tokenizer.generate_character_ngrams` is a wrapper for Python's `textwrap.wrap` method, which can also be called directly.

Once you have produced a list of ngrams, you can create a doc from them using `lexos.tokenizer.ngrams_from_doc``, as shown above.

Use `lexos.tokenizer.generate_character_ngrams` (a) when you simply want a list of non-overlapping ngrams, or (b) when you want to produce docs with non-overlapping ngrams as tokens.

Note that your language model may not be able apply labels effectively to ngram tokens, so working with character ngrams is primarily useful if you are planning to work with the token forms only, or if the ngram size you use maps closely to character lengths of words in the language you are working in.
