---

title: Transfer


keywords: fastai
sidebar: home_sidebar

summary: "Contains methods for transferring."
description: "Contains methods for transferring."
nb_path: "nbs/02_transfer.ipynb"
---
<!--

#################################################
### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ###
#################################################
# file to edit: nbs/02_transfer.ipynb
# command to build the docs after a change: nbdev_build_docs

-->

<div class="container" id="notebook-container">
        
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>We'll create collections of Embedding layers, which will be used to test our transfer methods.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">emb_szs1</span> <span class="o">=</span> <span class="p">((</span><span class="mi">3</span><span class="p">,</span> <span class="mi">10</span><span class="p">),</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">8</span><span class="p">))</span>
<span class="n">emb_szs2</span> <span class="o">=</span> <span class="p">((</span><span class="mi">2</span><span class="p">,</span> <span class="mi">10</span><span class="p">),</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">8</span><span class="p">))</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">embed1</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">ModuleList</span><span class="p">([</span><span class="n">nn</span><span class="o">.</span><span class="n">Embedding</span><span class="p">(</span><span class="n">ni</span><span class="p">,</span> <span class="n">nf</span><span class="p">)</span> <span class="k">for</span> <span class="n">ni</span><span class="p">,</span><span class="n">nf</span> <span class="ow">in</span> <span class="n">emb_szs1</span><span class="p">])</span>
<span class="n">embed2</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">ModuleList</span><span class="p">([</span><span class="n">nn</span><span class="o">.</span><span class="n">Embedding</span><span class="p">(</span><span class="n">ni</span><span class="p">,</span> <span class="n">nf</span><span class="p">)</span> <span class="k">for</span> <span class="n">ni</span><span class="p">,</span><span class="n">nf</span> <span class="ow">in</span> <span class="n">emb_szs2</span><span class="p">])</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">embed1</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>ModuleList(
  (0): Embedding(3, 10)
  (1): Embedding(2, 8)
)</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Now, we'll create collections containing required metadata.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">newcatcols</span> <span class="o">=</span> <span class="p">(</span><span class="s2">&quot;new_cat1&quot;</span><span class="p">,</span> <span class="s2">&quot;new_cat2&quot;</span><span class="p">)</span>
<span class="n">oldcatcols</span> <span class="o">=</span> <span class="p">(</span><span class="s2">&quot;old_cat2&quot;</span><span class="p">,</span> <span class="s2">&quot;old_cat3&quot;</span><span class="p">)</span>

<span class="n">newcatdict</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&quot;new_cat1&quot;</span> <span class="p">:</span> <span class="p">[</span><span class="s2">&quot;new_class1&quot;</span><span class="p">,</span> <span class="s2">&quot;new_class2&quot;</span><span class="p">,</span> <span class="s2">&quot;new_class3&quot;</span><span class="p">],</span> <span class="s2">&quot;new_cat2&quot;</span> <span class="p">:</span> <span class="p">[</span><span class="s2">&quot;new_class1&quot;</span><span class="p">,</span> <span class="s2">&quot;new_class2&quot;</span><span class="p">]}</span>
<span class="n">oldcatdict</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&quot;old_cat2&quot;</span> <span class="p">:</span> <span class="p">[</span><span class="s2">&quot;a&quot;</span><span class="p">,</span> <span class="s2">&quot;b&quot;</span><span class="p">],</span> <span class="s2">&quot;old_cat3&quot;</span> <span class="p">:</span> <span class="p">[</span><span class="s2">&quot;A&quot;</span><span class="p">,</span> <span class="s2">&quot;B&quot;</span><span class="p">]}</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">json_file_path</span> <span class="o">=</span> <span class="s2">&quot;../data/jsons/metadict.json&quot;</span>

<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">json_file_path</span><span class="p">,</span> <span class="s1">&#39;r&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">j</span><span class="p">:</span>
     <span class="n">metadict</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">j</span><span class="o">.</span><span class="n">read</span><span class="p">())</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p><code>metadict</code> is a <code>Dict</code> with the keys as the classes in dest. model's data, and value is another <code>Dict</code> where <code>mapped_cat</code> corresponds to the class in src model's data, along with information about how the classes map from dest. data to src data.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">metadict</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>{&#39;new_cat1&#39;: {&#39;mapped_cat&#39;: &#39;old_cat2&#39;,
  &#39;classes_info&#39;: {&#39;new_class1&#39;: [&#39;a&#39;, &#39;b&#39;],
   &#39;new_class2&#39;: [&#39;b&#39;],
   &#39;new_class3&#39;: []}},
 &#39;new_cat2&#39;: {&#39;mapped_cat&#39;: &#39;old_cat3&#39;,
  &#39;classes_info&#39;: {&#39;new_class1&#39;: [&#39;A&#39;], &#39;new_class2&#39;: []}}}</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">({</span><span class="s2">&quot;old_cat1&quot;</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">],</span> <span class="s2">&quot;old_cat2&quot;</span><span class="p">:</span> <span class="p">[</span><span class="s1">&#39;a&#39;</span><span class="p">,</span> <span class="s1">&#39;b&#39;</span><span class="p">,</span> <span class="s1">&#39;b&#39;</span><span class="p">,</span> <span class="s1">&#39;b&#39;</span><span class="p">,</span> <span class="s1">&#39;a&#39;</span><span class="p">],</span> <span class="s2">&quot;old_cat3&quot;</span><span class="p">:</span> <span class="p">[</span><span class="s1">&#39;A&#39;</span><span class="p">,</span> <span class="s1">&#39;B&#39;</span><span class="p">,</span> <span class="s1">&#39;B&#39;</span><span class="p">,</span> <span class="s1">&#39;B&#39;</span><span class="p">,</span> <span class="s1">&#39;A&#39;</span><span class="p">]})</span>
<span class="n">cats</span> <span class="o">=</span> <span class="p">(</span><span class="s2">&quot;old_cat2&quot;</span><span class="p">,</span> <span class="s2">&quot;old_cat3&quot;</span><span class="p">)</span>
<span class="n">embdict</span> <span class="o">=</span> <span class="n">extractembeds</span><span class="p">(</span><span class="n">embed2</span><span class="p">,</span> <span class="n">df</span><span class="p">,</span> <span class="n">transfercats</span><span class="o">=</span><span class="n">cats</span><span class="p">,</span> <span class="n">allcats</span><span class="o">=</span><span class="n">cats</span><span class="p">,</span> <span class="n">path</span><span class="o">=</span><span class="s2">&quot;tempwtbson&quot;</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="get_metadict_skeleton" class="doc_header"><code>get_metadict_skeleton</code><a href="https://github.com/manikyabard/transfertab/tree/master/transfertab/transfer.py#L19" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>get_metadict_skeleton</code>(<strong><code>df</code></strong>:<code>DataFrame</code>, <strong><code>catcols</code></strong>=<em><code>None</code></em>, <strong><code>path</code></strong>=<em><code>None</code></em>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">get_metadict_skeleton</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>{&#39;old_cat2&#39;: {&#39;mapped_cat&#39;: &#39;&#39;, &#39;classes_info&#39;: {&#39;a&#39;: [], &#39;b&#39;: []}},
 &#39;old_cat3&#39;: {&#39;mapped_cat&#39;: &#39;&#39;, &#39;classes_info&#39;: {&#39;A&#39;: [], &#39;B&#39;: []}}}</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">transferembeds_</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>(Module,Module) -&gt; transferembeds_
(Module,dict) -&gt; transferembeds_
(Module,PosixPath) -&gt; transferembeds_
(Module,str) -&gt; transferembeds_</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Embeddings before transfer:</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">embed1</span><span class="o">.</span><span class="n">state_dict</span><span class="p">()</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>OrderedDict([(&#39;0.weight&#39;,
              tensor([[ 0.1291,  0.9989, -0.1258,  0.4697, -1.8180, -0.4062,  0.7807, -2.4058,
                        2.2032,  1.8388],
                      [ 0.1456,  0.2293,  0.2135,  0.4504, -1.4981, -0.2788,  0.9045,  0.1295,
                       -0.9927, -0.0125],
                      [-1.6132,  0.8939, -0.2192, -0.7470, -0.5318, -2.4357, -0.0404, -0.8680,
                        0.2412,  1.8898]])),
             (&#39;1.weight&#39;,
              tensor([[-0.0732, -1.5366,  0.6748,  1.9617, -0.7229,  1.9168, -0.2036, -0.5741],
                      [ 0.6987, -1.2535, -0.2394, -0.3216,  0.9821,  1.1238,  2.2877, -0.7127]]))])</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">embed2</span><span class="o">.</span><span class="n">state_dict</span><span class="p">()</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>OrderedDict([(&#39;0.weight&#39;,
              tensor([[-1.7019, -0.6068,  1.3590,  3.2759,  1.5049,  1.1870,  0.5087,  0.6172,
                        0.0863, -0.5930],
                      [-1.8109, -0.6033,  1.1796,  0.6103,  0.6482,  1.4825, -1.3552,  1.0069,
                        0.1493, -1.3304]])),
             (&#39;1.weight&#39;,
              tensor([[ 2.3631,  0.1950, -1.3559,  0.0663,  0.1289,  0.5940, -0.0549,  0.2415],
                      [ 0.6715, -1.1929, -0.2372, -1.3345, -1.2651, -0.0468, -0.0934, -2.2118]]))])</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">transfer_cats</span> <span class="o">=</span> <span class="p">(</span><span class="s2">&quot;new_cat1&quot;</span><span class="p">,</span> <span class="s2">&quot;new_cat2&quot;</span><span class="p">)</span>
<span class="n">transferembeds_</span><span class="p">(</span><span class="n">embed1</span><span class="p">,</span> <span class="n">embdict</span><span class="p">,</span> <span class="n">metadict</span><span class="p">,</span> <span class="n">transfer_cats</span><span class="p">,</span> <span class="n">newcatcols</span><span class="o">=</span><span class="n">newcatcols</span><span class="p">,</span> <span class="n">oldcatcols</span><span class="o">=</span><span class="n">oldcatcols</span><span class="p">,</span> <span class="n">newcatdict</span><span class="o">=</span><span class="n">newcatdict</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">transfer_cats</span> <span class="o">=</span> <span class="p">(</span><span class="s2">&quot;new_cat1&quot;</span><span class="p">,</span> <span class="s2">&quot;new_cat2&quot;</span><span class="p">)</span>
<span class="n">transferembeds_</span><span class="p">(</span><span class="n">embed1</span><span class="p">,</span> <span class="n">embed2</span><span class="p">,</span> <span class="n">metadict</span><span class="p">,</span> <span class="n">transfer_cats</span><span class="p">,</span> <span class="n">newcatcols</span><span class="o">=</span><span class="n">newcatcols</span><span class="p">,</span> <span class="n">oldcatcols</span><span class="o">=</span><span class="n">oldcatcols</span><span class="p">,</span> <span class="n">oldcatdict</span><span class="o">=</span><span class="n">oldcatdict</span><span class="p">,</span> <span class="n">newcatdict</span><span class="o">=</span><span class="n">newcatdict</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">transfer_cats</span> <span class="o">=</span> <span class="p">(</span><span class="s2">&quot;new_cat1&quot;</span><span class="p">,</span> <span class="s2">&quot;new_cat2&quot;</span><span class="p">)</span>
<span class="n">transferembeds_</span><span class="p">(</span><span class="n">embed1</span><span class="p">,</span> <span class="n">pathlib</span><span class="o">.</span><span class="n">Path</span><span class="p">(</span><span class="s2">&quot;tempwtbson&quot;</span><span class="p">),</span> <span class="n">metadict</span><span class="p">,</span> <span class="n">transfer_cats</span><span class="p">,</span> <span class="n">newcatcols</span><span class="o">=</span><span class="n">newcatcols</span><span class="p">,</span> <span class="n">oldcatcols</span><span class="o">=</span><span class="n">oldcatcols</span><span class="p">,</span> <span class="n">newcatdict</span><span class="o">=</span><span class="n">newcatdict</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Embeddings after transfer:</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">embed1</span><span class="o">.</span><span class="n">state_dict</span><span class="p">()</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>OrderedDict([(&#39;0.weight&#39;,
              tensor([[-1.7564, -0.6051,  1.2693,  1.9431,  1.0765,  1.3347, -0.4232,  0.8120,
                        0.1178, -0.9617],
                      [-1.8109, -0.6033,  1.1796,  0.6103,  0.6482,  1.4825, -1.3552,  1.0069,
                        0.1493, -1.3304],
                      [-1.7564, -0.6051,  1.2693,  1.9431,  1.0765,  1.3347, -0.4232,  0.8120,
                        0.1178, -0.9617]])),
             (&#39;1.weight&#39;,
              tensor([[ 2.3631,  0.1950, -1.3559,  0.0663,  0.1289,  0.5940, -0.0549,  0.2415],
                      [ 1.5173, -0.4990, -0.7966, -0.6341, -0.5681,  0.2736, -0.0741, -0.9852]]))])</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">embed2</span><span class="o">.</span><span class="n">state_dict</span><span class="p">()</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>OrderedDict([(&#39;0.weight&#39;,
              tensor([[-1.7019, -0.6068,  1.3590,  3.2759,  1.5049,  1.1870,  0.5087,  0.6172,
                        0.0863, -0.5930],
                      [-1.8109, -0.6033,  1.1796,  0.6103,  0.6482,  1.4825, -1.3552,  1.0069,
                        0.1493, -1.3304]])),
             (&#39;1.weight&#39;,
              tensor([[ 2.3631,  0.1950, -1.3559,  0.0663,  0.1289,  0.5940, -0.0549,  0.2415],
                      [ 0.6715, -1.1929, -0.2372, -1.3345, -1.2651, -0.0468, -0.0934, -2.2118]]))])</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">os</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="s2">&quot;tempwtbson&quot;</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Export">Export<a class="anchor-link" href="#Export"> </a></h3>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

</div>
 

