---

title: NLP

keywords: fastai
sidebar: home_sidebar

summary: "API details."
description: "API details."
---
<!--

#################################################
### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ###
#################################################
# file to edit: notebooks/05_nlp.ipynb
# command to build the docs after a change: nbdev_build_docs

-->

<div class="container" id="notebook-container">
        
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%</span><span class="k">load_ext</span> autoreload
<span class="o">%</span><span class="k">autoreload</span> 2
<span class="o">%</span><span class="k">matplotlib</span> inline
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1"># Only needed for testing.</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="kn">from</span> <span class="nn">string</span> <span class="k">import</span> <span class="n">ascii_lowercase</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1"># Nonsense sample text.</span>
<span class="n">text</span> <span class="o">=</span> <span class="p">[</span>
    <span class="n">f</span><span class="s2">&quot;Row </span><span class="si">{i}</span><span class="s2">: I went, yesterday; she wasn&#39;t here after school? Today. --2&quot;</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">25_000</span><span class="p">)</span>
<span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="p">[</span><span class="s1">&#39;a&#39;</span><span class="p">])</span>
<span class="n">df</span><span class="o">.</span><span class="n">tail</span><span class="p">()</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>a</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>24995</td>
      <td>Row 24995: I went, yesterday; she wasn't here ...</td>
    </tr>
    <tr>
      <td>24996</td>
      <td>Row 24996: I went, yesterday; she wasn't here ...</td>
    </tr>
    <tr>
      <td>24997</td>
      <td>Row 24997: I went, yesterday; she wasn't here ...</td>
    </tr>
    <tr>
      <td>24998</td>
      <td>Row 24998: I went, yesterday; she wasn't here ...</td>
    </tr>
    <tr>
      <td>24999</td>
      <td>Row 24999: I went, yesterday; she wasn't here ...</td>
    </tr>
  </tbody>
</table>
</div>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="tokenize" class="doc_header"><code>tokenize</code><a href="https://github.com/hdmamin/incendio/tree/master/incendio/nlp.py#L25" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>tokenize</code>(<strong><code>text</code></strong>, <strong><code>nlp</code></strong>=<em><code>&lt;spacy.lang.en.English object at 0x10ed70190&gt;</code></em>)</p>
</blockquote>

<pre><code>Word tokenize a single string.

Parameters
----------
x: str
    A piece of text to tokenize.
nlp: spacy tokenizer, e.g. spacy.lang.en.English
    By default, a spacy tokenizer with a small English vocabulary
    is used. NER, parsing, and tagging are disabled. Any spacy
    tokenzer can be passed in, but keep in mind other configurations
    may slow down this function dramatically.

Returns
-------
list[str]: List of word tokens from a single input string.</code></pre>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="tokenize_many" class="doc_header"><code>tokenize_many</code><a href="https://github.com/hdmamin/incendio/tree/master/incendio/nlp.py#L46" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>tokenize_many</code>(<strong><code>rows</code></strong>, <strong><code>chunk</code></strong>=<em><code>1000</code></em>, <strong><code>nlp</code></strong>=<em><code>&lt;spacy.lang.en.English object at 0x10ed70190&gt;</code></em>)</p>
</blockquote>

<pre><code>Word tokenize a sequence of strings using multiprocessing. The max
number of available processes are used.

Parameters
----------
rows: Iterable[str]
    A sequence of strings to tokenize. This could be a list, a column of
    a DataFrame, etc.
chunk: int
    This determines how many items to send to multiprocessing at a time.
    The default of 1,000 is usually fine, but if you have extremely
    long pieces of text and memory is limited, you can always decrease it.
    Very small chunk sizes may increase processing time. Note that larger
    values will generally cause the progress bar to update more choppily.
nlp: spacy tokenizer, e.g. spacy.lang.en.English
    By default, a spacy tokenizer with a small English vocabulary
    is used. NER, parsing, and tagging are disabled. Any spacy
    tokenzer can be passed in, but keep in mind other configurations
    may slow down this function dramatically.

Returns
-------
list[list[str]]: Each nested list of word tokens corresponds to one
of the input strings.</code></pre>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1"># ~5-6 seconds</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">a</span><span class="o">.</span><span class="n">apply</span><span class="p">(</span><span class="n">tokenize</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1"># ~1-2 seconds</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">tokenize_many</span><span class="p">(</span><span class="n">df</span><span class="o">.</span><span class="n">a</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="Vocabulary" class="doc_header"><code>class</code> <code>Vocabulary</code><a href="https://github.com/hdmamin/incendio/tree/master/incendio/nlp.py#L79" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>Vocabulary</code>(<strong><code>w2idx</code></strong>, <strong><code>w2vec</code></strong>=<em><code>None</code></em>, <strong><code>idx_misc</code></strong>=<em><code>None</code></em>, <strong><code>corpus_counts</code></strong>=<em><code>None</code></em>, <strong><code>all_lower</code></strong>=<em><code>True</code></em>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="Embeddings" class="doc_header"><code>class</code> <code>Embeddings</code><a href="https://github.com/hdmamin/incendio/tree/master/incendio/nlp.py#L458" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>Embeddings</code>(<strong><code>mat</code></strong>, <strong><code>w2i</code></strong>, <strong><code>mat_2d</code></strong>=<em><code>None</code></em>)</p>
</blockquote>

<pre><code>Embeddings object. Lets us easily map word to index, index to
word, and word to vector. We can use this to find similar words,
build analogies, or get 2D representations for cdting.</code></pre>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="back_translate" class="doc_header"><code>back_translate</code><a href="https://github.com/hdmamin/incendio/tree/master/incendio/nlp.py#L845" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>back_translate</code>(<strong><code>text</code></strong>, <strong><code>to</code></strong>, <strong><code>from_lang</code></strong>=<em><code>'en'</code></em>)</p>
</blockquote>

<pre><code>Parameters
----------

Returns
-------
str: Same language and basically the same content as the original text,
    but usually with slightly altered grammar, sentence structure, and/or
    vocabulary.</code></pre>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">text</span> <span class="o">=</span> <span class="s2">&quot;&quot;&quot;</span>
<span class="s2">Visit ESPN to get up-to-the-minute sports news coverage, scores, highlights and commentary for NFL, MLB, NBA, College Football, NCAA Basketball and more.</span>
<span class="s2">&quot;&quot;&quot;</span>
<span class="n">back_translate</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="s1">&#39;es&#39;</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>&#39;Visit ESPN to get coverage of sports news, scores, highlights and comments from the NFL, MLB, NBA, college football, NCAA basketball and more.&#39;</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">text</span> <span class="o">=</span> <span class="s2">&quot;&quot;&quot;</span>
<span class="s2">Visit ESPN to get up-to-the-minute sports news coverage, scores, highlights and commentary for NFL, MLB, NBA, College Football, NCAA Basketball and more.</span>
<span class="s2">&quot;&quot;&quot;</span>
<span class="n">back_translate</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="s1">&#39;fr&#39;</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>&#39;Visit ESPN for up-to-date sports information, scores, highlights and commentary for the NFL, MLB, NBA, college football, NCAA basketball and more.&#39;</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="postprocess_embeddings" class="doc_header"><code>postprocess_embeddings</code><a href="https://github.com/hdmamin/incendio/tree/master/incendio/nlp.py#L864" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>postprocess_embeddings</code>(<strong><code>emb</code></strong>, <strong><code>d</code></strong>=<em><code>None</code></em>)</p>
</blockquote>

<pre><code>Implements the algorithm from the paper:

All-But-The-Top: Simple and Effective Post-Processing
for Word Representations (https://arxiv.org/pdf/1702.01417.pdf)

There are three steps:
1. Compute the mean embedding and subtract this from the
original embedding matrix.
2. Perform PCA and extract the top d components.
3. Eliminate the principal components from the mean-adjusted
embeddings.

Parameters
----------
emb: np.array
    Embedding matrix of size (vocab_size, embedding_length).
d: int
    Number of components to use in PCA. Defaults to
    embedding_length/100 as recommended by the paper.</code></pre>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="compress_embeddings" class="doc_header"><code>compress_embeddings</code><a href="https://github.com/hdmamin/incendio/tree/master/incendio/nlp.py#L892" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>compress_embeddings</code>(<strong><code>emb</code></strong>, <strong><code>new_dim</code></strong>, <strong><code>d</code></strong>=<em><code>None</code></em>)</p>
</blockquote>

<pre><code>Reduce embedding dimension as described in the paper:

Simple and Effective Dimensionality Reduction for Word Embeddings
(https://lld-workshop.github.io/2017/papers/LLD_2017_paper_34.pdf)

Parameters
----------
emb: np.array
    Embedding matrix of size (vocab_size, embedding_length).
d: int
    Number of components to use in the post-processing
    method described here: https://arxiv.org/pdf/1702.01417.pdf
    Defaults to embedding_length/100 as recommended by the paper.

Returns
-------
np.array: Compressed embedding matrix of shape (vocab_size, new_dim).</code></pre>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

</div>
 

