---

title: Dilated Recurrent Neural Networks


keywords: fastai
sidebar: home_sidebar

summary: "DRNN pytorch implementation."
description: "DRNN pytorch implementation."
nb_path: "nbs/models_components__drnn.ipynb"
---
<!--

#################################################
### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ###
#################################################
# file to edit: nbs/models_components__drnn.ipynb
# command to build the docs after a change: nbdev_build_docs

-->

<div class="container" id="notebook-container">
        
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Dilated Recurrent Neural Networks. <a href="https://papers.nips.cc/paper/6613-dilated-recurrent-neural-networks.pdf">https://papers.nips.cc/paper/6613-dilated-recurrent-neural-networks.pdf</a>
implementation from <a href="https://github.com/zalandoresearch/pytorch-dilated-rnn">https://github.com/zalandoresearch/pytorch-dilated-rnn</a>
Residual LSTM: Design of a Deep Recurrent Architecture for Distant Speech Recognition. <a href="https://arxiv.org/abs/1701.03360">https://arxiv.org/abs/1701.03360</a>
A Dual-Stage Attention-Based recurrent neural network for time series prediction. <a href="https://arxiv.org/abs/1704.02971">https://arxiv.org/abs/1704.02971</a></p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="LSTMCell" class="doc_header"><code>class</code> <code>LSTMCell</code><a href="https://github.com/Nixtla/nixtlats/tree/master/nixtlats/models/components/drnn.py#L11" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>LSTMCell</code>(<strong><code>input_size</code></strong>, <strong><code>hidden_size</code></strong>, <strong><code>dropout</code></strong>=<em><code>0.0</code></em>) :: <code>Module</code></p>
</blockquote>
<p>Base class for all neural network modules.</p>
<p>Your models should also subclass this class.</p>
<p>Modules can also contain other Modules, allowing to nest them in
a tree structure. You can assign the submodules as regular attributes::</p>

<pre><code>import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

</code></pre>
<p>Submodules assigned in this way will be registered, and will have their
parameters converted too when you call :meth:<code>to</code>, etc.</p>
<p>:ivar training: Boolean represents whether this module is in training or
                evaluation mode.
:vartype training: bool</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="ResLSTMCell" class="doc_header"><code>class</code> <code>ResLSTMCell</code><a href="https://github.com/Nixtla/nixtlats/tree/master/nixtlats/models/components/drnn.py#L39" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>ResLSTMCell</code>(<strong><code>input_size</code></strong>, <strong><code>hidden_size</code></strong>, <strong><code>dropout</code></strong>=<em><code>0.0</code></em>) :: <code>Module</code></p>
</blockquote>
<p>Base class for all neural network modules.</p>
<p>Your models should also subclass this class.</p>
<p>Modules can also contain other Modules, allowing to nest them in
a tree structure. You can assign the submodules as regular attributes::</p>

<pre><code>import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

</code></pre>
<p>Submodules assigned in this way will be registered, and will have their
parameters converted too when you call :meth:<code>to</code>, etc.</p>
<p>:ivar training: Boolean represents whether this module is in training or
                evaluation mode.
:vartype training: bool</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="ResLSTMLayer" class="doc_header"><code>class</code> <code>ResLSTMLayer</code><a href="https://github.com/Nixtla/nixtlats/tree/master/nixtlats/models/components/drnn.py#L80" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>ResLSTMLayer</code>(<strong><code>input_size</code></strong>, <strong><code>hidden_size</code></strong>, <strong><code>dropout</code></strong>=<em><code>0.0</code></em>) :: <code>Module</code></p>
</blockquote>
<p>Base class for all neural network modules.</p>
<p>Your models should also subclass this class.</p>
<p>Modules can also contain other Modules, allowing to nest them in
a tree structure. You can assign the submodules as regular attributes::</p>

<pre><code>import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

</code></pre>
<p>Submodules assigned in this way will be registered, and will have their
parameters converted too when you call :meth:<code>to</code>, etc.</p>
<p>:ivar training: Boolean represents whether this module is in training or
                evaluation mode.
:vartype training: bool</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="AttentiveLSTMLayer" class="doc_header"><code>class</code> <code>AttentiveLSTMLayer</code><a href="https://github.com/Nixtla/nixtlats/tree/master/nixtlats/models/components/drnn.py#L97" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>AttentiveLSTMLayer</code>(<strong><code>input_size</code></strong>, <strong><code>hidden_size</code></strong>, <strong><code>dropout</code></strong>=<em><code>0.0</code></em>) :: <code>Module</code></p>
</blockquote>
<p>Base class for all neural network modules.</p>
<p>Your models should also subclass this class.</p>
<p>Modules can also contain other Modules, allowing to nest them in
a tree structure. You can assign the submodules as regular attributes::</p>

<pre><code>import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

</code></pre>
<p>Submodules assigned in this way will be registered, and will have their
parameters converted too when you call :meth:<code>to</code>, etc.</p>
<p>:ivar training: Boolean represents whether this module is in training or
                evaluation mode.
:vartype training: bool</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="DRNN" class="doc_header"><code>class</code> <code>DRNN</code><a href="https://github.com/Nixtla/nixtlats/tree/master/nixtlats/models/components/drnn.py#L132" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>DRNN</code>(<strong><code>n_input</code></strong>, <strong><code>n_hidden</code></strong>, <strong><code>n_layers</code></strong>, <strong><code>dilations</code></strong>, <strong><code>dropout</code></strong>=<em><code>0</code></em>, <strong><code>cell_type</code></strong>=<em><code>'GRU'</code></em>, <strong><code>batch_first</code></strong>=<em><code>False</code></em>) :: <code>Module</code></p>
</blockquote>
<p>Base class for all neural network modules.</p>
<p>Your models should also subclass this class.</p>
<p>Modules can also contain other Modules, allowing to nest them in
a tree structure. You can assign the submodules as regular attributes::</p>

<pre><code>import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

</code></pre>
<p>Submodules assigned in this way will be registered, and will have their
parameters converted too when you call :meth:<code>to</code>, etc.</p>
<p>:ivar training: Boolean represents whether this module is in training or
                evaluation mode.
:vartype training: bool</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Tests">Tests<a class="anchor-link" href="#Tests"> </a></h2>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">n_time</span> <span class="o">=</span> <span class="mi">24</span>
<span class="n">batch_size</span> <span class="o">=</span> <span class="mi">3</span>
<span class="n">n_feats</span> <span class="o">=</span> <span class="mi">10</span>
<span class="n">n_output_feats</span> <span class="o">=</span> <span class="mi">20</span>
<span class="n">num_layers</span> <span class="o">=</span> <span class="mi">4</span>
<span class="n">dilations</span> <span class="o">=</span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">9</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Test-forward">Test forward<a class="anchor-link" href="#Test-forward"> </a></h3>
</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">for</span> <span class="n">cell_type</span> <span class="ow">in</span> <span class="p">[</span><span class="s1">&#39;GRU&#39;</span><span class="p">,</span> <span class="s1">&#39;RNN&#39;</span><span class="p">,</span> <span class="s1">&#39;LSTM&#39;</span><span class="p">,</span> <span class="s1">&#39;ResLSTM&#39;</span><span class="p">]:</span>
    <span class="n">model</span> <span class="o">=</span> <span class="n">DRNN</span><span class="p">(</span><span class="n">n_input</span><span class="o">=</span><span class="n">n_feats</span><span class="p">,</span> <span class="n">n_hidden</span><span class="o">=</span><span class="n">n_output_feats</span><span class="p">,</span> <span class="n">n_layers</span><span class="o">=</span><span class="n">num_layers</span><span class="p">,</span> 
             <span class="n">dilations</span><span class="o">=</span><span class="n">dilations</span><span class="p">,</span> <span class="n">cell_type</span><span class="o">=</span><span class="n">cell_type</span><span class="p">)</span>
    <span class="n">x</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">n_time</span><span class="p">,</span> <span class="n">batch_size</span><span class="p">,</span> <span class="n">n_feats</span><span class="p">)</span>
    <span class="n">output</span><span class="p">,</span> <span class="n">hidden</span> <span class="o">=</span> <span class="n">model</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
    <span class="n">test_eq</span><span class="p">(</span><span class="n">output</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="p">(</span><span class="n">n_time</span><span class="p">,</span> <span class="n">batch_size</span><span class="p">,</span> <span class="n">n_output_feats</span><span class="p">))</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

</div>
 

