---

title: Extraction Utilities


keywords: fastai
sidebar: home_sidebar

summary: "Contains helpful functions for extracting embeddings and preparing data for it."
description: "Contains helpful functions for extracting embeddings and preparing data for it."
nb_path: "nbs/01_extract.ipynb"
---
<!--

#################################################
### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ###
#################################################
# file to edit: nbs/01_extract.ipynb
# command to build the docs after a change: nbdev_build_docs

-->

<div class="container" id="notebook-container">
        
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">emb_szs</span> <span class="o">=</span> <span class="p">((</span><span class="mi">3</span><span class="p">,</span> <span class="mi">10</span><span class="p">),</span> <span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="mi">8</span><span class="p">))</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">embed</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">ModuleList</span><span class="p">([</span><span class="n">nn</span><span class="o">.</span><span class="n">Embedding</span><span class="p">(</span><span class="n">ni</span><span class="p">,</span> <span class="n">nf</span><span class="p">)</span> <span class="k">for</span> <span class="n">ni</span><span class="p">,</span><span class="n">nf</span> <span class="ow">in</span> <span class="n">emb_szs</span><span class="p">])</span>
<span class="n">embed</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>ModuleList(
  (0): Embedding(3, 10)
  (1): Embedding(4, 8)
)</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="JSONizerWithBool" class="doc_header"><code>class</code> <code>JSONizerWithBool</code><a href="https://github.com/manikyabard/transfertab/tree/master/transfertab/extract.py#L14" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>JSONizerWithBool</code>(<strong><code>skipkeys</code></strong>=<em><code>False</code></em>, <strong><code>ensure_ascii</code></strong>=<em><code>True</code></em>, <strong><code>check_circular</code></strong>=<em><code>True</code></em>, <strong><code>allow_nan</code></strong>=<em><code>True</code></em>, <strong><code>sort_keys</code></strong>=<em><code>False</code></em>, <strong><code>indent</code></strong>=<em><code>None</code></em>, <strong><code>separators</code></strong>=<em><code>None</code></em>, <strong><code>default</code></strong>=<em><code>None</code></em>) :: <code>JSONEncoder</code></p>
</blockquote>

<pre><code>Extensible JSON &lt;http://json.org&gt; encoder for Python data structures.

Supports the following objects and types by default:

+-------------------+---------------+
| Python            | JSON          |
+===================+===============+
| dict              | object        |
+-------------------+---------------+
| list, tuple       | array         |
+-------------------+---------------+
| str               | string        |
+-------------------+---------------+
| int, float        | number        |
+-------------------+---------------+
| True              | true          |
+-------------------+---------------+
| False             | false         |
+-------------------+---------------+
| None              | null          |
+-------------------+---------------+

To extend this to recognize other objects, subclass and implement a
``.default()`` method with another method that returns a serializable
object for ``o`` if possible, otherwise it should call the superclass
implementation (to raise ``TypeError``).</code></pre>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">({</span><span class="s2">&quot;cat1&quot;</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">],</span> <span class="s2">&quot;cat2&quot;</span><span class="p">:</span> <span class="p">[</span><span class="s1">&#39;a&#39;</span><span class="p">,</span> <span class="s1">&#39;b&#39;</span><span class="p">,</span> <span class="s1">&#39;c&#39;</span><span class="p">,</span> <span class="s1">&#39;b&#39;</span><span class="p">,</span> <span class="s1">&#39;a&#39;</span><span class="p">],</span> <span class="s2">&quot;cat3&quot;</span><span class="p">:</span> <span class="p">[</span><span class="s1">&#39;A&#39;</span><span class="p">,</span> <span class="s1">&#39;B&#39;</span><span class="p">,</span> <span class="s1">&#39;C&#39;</span><span class="p">,</span> <span class="s1">&#39;D&#39;</span><span class="p">,</span> <span class="s1">&#39;A&#39;</span><span class="p">]})</span>
<span class="n">df</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>cat1</th>
      <th>cat2</th>
      <th>cat3</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>a</td>
      <td>A</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>b</td>
      <td>B</td>
    </tr>
    <tr>
      <th>2</th>
      <td>3</td>
      <td>c</td>
      <td>C</td>
    </tr>
    <tr>
      <th>3</th>
      <td>4</td>
      <td>b</td>
      <td>D</td>
    </tr>
    <tr>
      <th>4</th>
      <td>5</td>
      <td>a</td>
      <td>A</td>
    </tr>
  </tbody>
</table>
</div>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">catdict</span> <span class="o">=</span> <span class="n">getcatdict</span><span class="p">(</span><span class="n">df</span><span class="p">,</span> <span class="p">(</span><span class="s2">&quot;cat2&quot;</span><span class="p">,</span> <span class="s2">&quot;cat3&quot;</span><span class="p">))</span>
<span class="n">cats</span> <span class="o">=</span> <span class="p">(</span><span class="s2">&quot;cat2&quot;</span><span class="p">,</span> <span class="s2">&quot;cat3&quot;</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">embdict</span> <span class="o">=</span> <span class="n">extractembeds</span><span class="p">(</span><span class="n">embed</span><span class="p">,</span> <span class="n">df</span><span class="p">,</span> <span class="n">transfercats</span><span class="o">=</span><span class="n">cats</span><span class="p">,</span> <span class="n">allcats</span><span class="o">=</span><span class="n">cats</span><span class="p">,</span> <span class="n">path</span><span class="o">=</span><span class="s2">&quot;tempwtbson&quot;</span><span class="p">,</span> <span class="n">kind</span><span class="o">=</span><span class="s2">&quot;bson&quot;</span><span class="p">)</span>
<span class="n">embdict</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>{&#39;cat2&#39;: {&#39;classes&#39;: [&#39;a&#39;, &#39;b&#39;, &#39;c&#39;],
  &#39;embeddings&#39;: [[0.31570491194725037,
    -0.07632226496934891,
    1.5683248043060303,
    -0.417350172996521,
    -0.10798821598291397,
    1.4268646240234375,
    -0.22982962429523468,
    -0.16915012896060944,
    0.002859442261978984,
    -0.4939035475254059],
   [0.6530274748802185,
    -0.5577511191368103,
    -0.9275949001312256,
    -0.06805138289928436,
    -2.2739336490631104,
    0.1566399186849594,
    -0.0531904362142086,
    -0.43463948369026184,
    -0.0794961154460907,
    0.4645240008831024],
   [1.0870261192321777,
    -0.22893156111240387,
    -0.253396600484848,
    -0.3393022119998932,
    -2.0341274738311768,
    -0.31127995252609253,
    0.3499477803707123,
    -1.9891204833984375,
    0.674164891242981,
    -1.3391718864440918]]},
 &#39;cat3&#39;: {&#39;classes&#39;: [&#39;A&#39;, &#39;B&#39;, &#39;C&#39;, &#39;D&#39;],
  &#39;embeddings&#39;: [[1.3585036993026733,
    0.024397719651460648,
    0.4804745614528656,
    1.1160022020339966,
    0.8734705448150635,
    0.784949004650116,
    -0.5678505897521973,
    0.33350786566734314],
   [0.3660596013069153,
    -0.9798707962036133,
    -0.2037343829870224,
    -0.22464202344417572,
    -1.0697559118270874,
    -0.6113787889480591,
    -0.9179865717887878,
    -0.8937533497810364],
   [1.278954029083252,
    0.18886688351631165,
    1.2901731729507446,
    0.5247588157653809,
    1.2530524730682373,
    -0.898102343082428,
    1.1512700319290161,
    1.5226550102233887],
   [-0.6129401922225952,
    -0.38670244812965393,
    0.7002972364425659,
    -1.2176426649093628,
    0.5013972520828247,
    -1.5657707452774048,
    -2.1267337799072266,
    0.5773623585700989]]}}</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">embdict</span> <span class="o">=</span> <span class="n">extractembeds</span><span class="p">(</span><span class="n">embed</span><span class="p">,</span> <span class="n">df</span><span class="p">,</span> <span class="n">transfercats</span><span class="o">=</span><span class="n">cats</span><span class="p">,</span> <span class="n">allcats</span><span class="o">=</span><span class="n">cats</span><span class="p">,</span> <span class="n">path</span><span class="o">=</span><span class="s2">&quot;tempwtjson&quot;</span><span class="p">,</span> <span class="n">kind</span><span class="o">=</span><span class="s2">&quot;json&quot;</span><span class="p">)</span>
<span class="n">embdict</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>{&#39;cat2&#39;: {&#39;classes&#39;: [&#39;a&#39;, &#39;b&#39;, &#39;c&#39;],
  &#39;embeddings&#39;: [[0.31570491194725037,
    -0.07632226496934891,
    1.5683248043060303,
    -0.417350172996521,
    -0.10798821598291397,
    1.4268646240234375,
    -0.22982962429523468,
    -0.16915012896060944,
    0.002859442261978984,
    -0.4939035475254059],
   [0.6530274748802185,
    -0.5577511191368103,
    -0.9275949001312256,
    -0.06805138289928436,
    -2.2739336490631104,
    0.1566399186849594,
    -0.0531904362142086,
    -0.43463948369026184,
    -0.0794961154460907,
    0.4645240008831024],
   [1.0870261192321777,
    -0.22893156111240387,
    -0.253396600484848,
    -0.3393022119998932,
    -2.0341274738311768,
    -0.31127995252609253,
    0.3499477803707123,
    -1.9891204833984375,
    0.674164891242981,
    -1.3391718864440918]]},
 &#39;cat3&#39;: {&#39;classes&#39;: [&#39;A&#39;, &#39;B&#39;, &#39;C&#39;, &#39;D&#39;],
  &#39;embeddings&#39;: [[1.3585036993026733,
    0.024397719651460648,
    0.4804745614528656,
    1.1160022020339966,
    0.8734705448150635,
    0.784949004650116,
    -0.5678505897521973,
    0.33350786566734314],
   [0.3660596013069153,
    -0.9798707962036133,
    -0.2037343829870224,
    -0.22464202344417572,
    -1.0697559118270874,
    -0.6113787889480591,
    -0.9179865717887878,
    -0.8937533497810364],
   [1.278954029083252,
    0.18886688351631165,
    1.2901731729507446,
    0.5247588157653809,
    1.2530524730682373,
    -0.898102343082428,
    1.1512700319290161,
    1.5226550102233887],
   [-0.6129401922225952,
    -0.38670244812965393,
    0.7002972364425659,
    -1.2176426649093628,
    0.5013972520828247,
    -1.5657707452774048,
    -2.1267337799072266,
    0.5773623585700989]]}}</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">load_bson</span><span class="p">(</span><span class="s2">&quot;tempwtbson&quot;</span><span class="p">)</span> <span class="o">==</span> <span class="n">embdict</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>True</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">os</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="s2">&quot;tempwtbson&quot;</span><span class="p">)</span>
<span class="n">os</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="s2">&quot;tempwtjson&quot;</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Export">Export<a class="anchor-link" href="#Export"> </a></h3>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

</div>
 

