---

title: MovieLens Dataset Transformation


keywords: fastai
sidebar: home_sidebar

summary: "Implementation of transformation functions specific to movielens datasets."
description: "Implementation of transformation functions specific to movielens datasets."
nb_path: "nbs/transforms/datasets/transforms.datasets.movielens.ipynb"
---
<!--

#################################################
### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ###
#################################################
# file to edit: nbs/transforms/datasets/transforms.datasets.movielens.ipynb
# command to build the docs after a change: nbdev_build_docs

-->

<div class="container" id="notebook-container">
        
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="sparseFeature" class="doc_header"><code>sparseFeature</code><a href="https://github.com/RecoHut-Projects/recohut/tree/master/recohut/transforms/datasets/movielens.py#L13" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>sparseFeature</code>(<strong><code>feat</code></strong>, <strong><code>feat_num</code></strong>, <strong><code>embed_dim</code></strong>=<em><code>4</code></em>)</p>
</blockquote>
<p>create dictionary for sparse feature
:param feat: feature name
:param feat_num: the total number of sparse features that do not repeat
:param embed_dim: embedding dimension
:return:</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="create_ml_1m_dataset" class="doc_header"><code>create_ml_1m_dataset</code><a href="https://github.com/RecoHut-Projects/recohut/tree/master/recohut/transforms/datasets/movielens.py#L24" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>create_ml_1m_dataset</code>(<strong><code>file</code></strong>, <strong><code>trans_score</code></strong>=<em><code>2</code></em>, <strong><code>embed_dim</code></strong>=<em><code>8</code></em>, <strong><code>test_neg_num</code></strong>=<em><code>100</code></em>)</p>
</blockquote>
<p>:param file: A string. dataset path.
:param trans_score: A scalar. Greater than it is 1, and less than it is 0.
:param embed_dim: A scalar. latent factor.
:param test_neg_num: A scalar. The number of test negative samples
:return: user_num, item_num, train_df, test_df</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="create_implicit_ml_1m_dataset" class="doc_header"><code>create_implicit_ml_1m_dataset</code><a href="https://github.com/RecoHut-Projects/recohut/tree/master/recohut/transforms/datasets/movielens.py#L87" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>create_implicit_ml_1m_dataset</code>(<strong><code>file</code></strong>, <strong><code>trans_score</code></strong>=<em><code>2</code></em>, <strong><code>embed_dim</code></strong>=<em><code>8</code></em>, <strong><code>maxlen</code></strong>=<em><code>40</code></em>)</p>
</blockquote>
<p>:param file: A string. dataset path.
:param trans_score: A scalar. Greater than it is 1, and less than it is 0.
:param embed_dim: A scalar. latent factor.
:param maxlen: A scalar. maxlen.
:return: user_num, item_num, train_df, test_df</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">!</span>wget -q --show-progress https://files.grouplens.org/datasets/movielens/ml-1m.zip
<span class="o">!</span>unzip ml-1m.zip
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>ml-1m.zip           100%[===================&gt;]   5.64M  3.71MB/s    in 1.5s    
Archive:  ml-1m.zip
   creating: ml-1m/
  inflating: ml-1m/movies.dat        
  inflating: ml-1m/ratings.dat       
  inflating: ml-1m/README            
  inflating: ml-1m/users.dat         
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">file</span> <span class="o">=</span> <span class="s1">&#39;ml-1m/ratings.dat&#39;</span>
<span class="n">test_neg_num</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">embed_dim</span> <span class="o">=</span> <span class="mi">64</span>
<span class="n">trans_score</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">maxlen</span> <span class="o">=</span> <span class="mi">200</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">feature_columns</span><span class="p">,</span> <span class="n">train</span><span class="p">,</span> <span class="n">val</span><span class="p">,</span> <span class="n">test</span> <span class="o">=</span> <span class="n">create_ml_1m_dataset</span><span class="p">(</span><span class="n">file</span><span class="p">,</span> <span class="n">trans_score</span><span class="p">,</span> <span class="n">embed_dim</span><span class="p">,</span> <span class="n">test_neg_num</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>==========Data Preprocess Start=============
============Negative Sampling===============
</pre>
</div>
</div>

<div class="output_area">

<div class="output_subarea output_stream output_stderr output_text">
<pre>100%|██████████| 6040/6040 [00:44&lt;00:00, 134.33it/s]
</pre>
</div>
</div>

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>============Data Preprocess End=============
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">feature_columns</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>[{&#39;embed_dim&#39;: 64, &#39;feat&#39;: &#39;user_id&#39;, &#39;feat_num&#39;: 6041},
 {&#39;embed_dim&#39;: 64, &#39;feat&#39;: &#39;item_id&#39;, &#39;feat_num&#39;: 3953}]</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">train</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>[array([   1,    1,    1, ..., 6040, 6040, 6040]),
 array([1270, 1721, 1022, ..., 2917, 1921, 1784]),
 array([2152, 1229, 3617, ..., 2960, 3686, 1569])]</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">val</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>[array([   1,    2,    3, ..., 6038, 6039, 6040]),
 array([1907, 1544, 3868, ..., 2700, 1204,  161]),
 array([3132, 1812,  271, ..., 1697, 2718, 3572])]</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">test</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>[array([   1,    2,    3, ..., 6038, 6039, 6040]),
 array([  48, 1917, 2081, ..., 1183, 1254, 1221]),
 array([[ 426, 1915, 2201, ..., 1687, 2916, 1266],
        [3294, 2362,  167, ..., 1322, 2715, 3013],
        [2973, 3000, 1832, ...,  514, 2845, 1901],
        ...,
        [1258,  335, 3638, ..., 3582, 2221,  763],
        [1767, 2924,  691, ..., 1624, 2493,  371],
        [1106, 3048, 1940, ..., 3520, 2102, 2275]])]</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">feature_columns</span><span class="p">,</span> <span class="n">train</span><span class="p">,</span> <span class="n">val</span><span class="p">,</span> <span class="n">test</span> <span class="o">=</span> <span class="n">create_implicit_ml_1m_dataset</span><span class="p">(</span><span class="n">file</span><span class="p">,</span> <span class="n">trans_score</span><span class="p">,</span> <span class="n">embed_dim</span><span class="p">,</span> <span class="n">maxlen</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>==========Data Preprocess Start=============
</pre>
</div>
</div>

<div class="output_area">

<div class="output_subarea output_stream output_stderr output_text">
<pre>100%|██████████| 6040/6040 [00:35&lt;00:00, 170.13it/s]
</pre>
</div>
</div>

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>==================Padding===================
============Data Preprocess End=============
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">feature_columns</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>[{&#39;embed_dim&#39;: 64, &#39;feat&#39;: &#39;user_id&#39;, &#39;feat_num&#39;: 6041},
 {&#39;embed_dim&#39;: 64, &#39;feat&#39;: &#39;item_id&#39;, &#39;feat_num&#39;: 3953}]</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">train</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>([array([5534, 3031, 1764, ..., 3159, 2137, 3129]),
  array([[   0,    0,    0, ...,  349, 1356, 1580],
         [   0,    0,    0, ..., 3255, 2108,  507],
         [2322, 3316,    9, ..., 2502, 1476, 2759],
         ...,
         [   0,    0,    0, ..., 1258, 1240, 1270],
         [   0,    0,    0, ..., 2038, 1831,   24],
         [2379, 3846, 3041, ..., 2391,  866, 3476]], dtype=int32),
  array([1372, 2309, 3052, ..., 1285, 2668, 2143])],
 array([1, 0, 1, ..., 1, 1, 0]))</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">val</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>([array([3468, 1903, 1902, ..., 5215, 2977, 3597]),
  array([[   0,    0,    0, ..., 3114,  593, 2345],
         [   0,    0,    0, ..., 1201, 3671, 3681],
         [1754,   44,  247, ..., 1092, 3005, 2605],
         ...,
         [   0,    0,    0, ..., 1252,  720,  745],
         [   0,    0,    0, ..., 2581, 2724, 2763],
         [   0,    0,    0, ..., 2096, 2137, 1032]], dtype=int32),
  array([3951, 1202,  832, ..., 3177,  476, 1029])],
 array([0, 0, 1, ..., 0, 0, 1]))</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

</div>
 

