---

title: Attention Layers


keywords: fastai
sidebar: home_sidebar

summary: "Implementation of Attention modules including Multihead attention etc.."
description: "Implementation of Attention modules including Multihead attention etc.."
nb_path: "nbs/layers/attention.ipynb"
---
<!--

#################################################
### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ###
#################################################
# file to edit: nbs/layers/attention.ipynb
# command to build the docs after a change: nbdev_build_docs

-->

<div class="container" id="notebook-container">
        
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="TokenEmbedding" class="doc_header"><code>class</code> <code>TokenEmbedding</code><a href="https://github.com/RecoHut-Projects/recohut/tree/master/recohut/layers/attention.py#L14" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>TokenEmbedding</code>(<strong><code>vocab_size</code></strong>, <strong><code>embed_size</code></strong>=<em><code>512</code></em>) :: <code>Embedding</code></p>
</blockquote>
<p>A simple lookup table that stores embeddings of a fixed dictionary and size.</p>
<p>This module is often used to store word embeddings and retrieve them using indices.
The input to the module is a list of indices, and the output is the corresponding
word embeddings.</p>
<p>Args:
    num_embeddings (int): size of the dictionary of embeddings
    embedding_dim (int): the size of each embedding vector
    padding_idx (int, optional): If specified, the entries at :attr:<code>padding_idx</code> do not contribute to the gradient;
                                 therefore, the embedding vector at :attr:<code>padding_idx</code> is not updated during training,
                                 i.e. it remains as a fixed "pad". For a newly constructed Embedding,
                                 the embedding vector at :attr:<code>padding_idx</code> will default to all zeros,
                                 but can be updated to another value to be used as the padding vector.
    max_norm (float, optional): If given, each embedding vector with norm larger than :attr:<code>max_norm</code>
                                is renormalized to have norm :attr:<code>max_norm</code>.
    norm_type (float, optional): The p of the p-norm to compute for the :attr:<code>max_norm</code> option. Default <code>2</code>.
    scale_grad_by_freq (boolean, optional): If given, this will scale gradients by the inverse of frequency of
                                            the words in the mini-batch. Default <code>False</code>.
    sparse (bool, optional): If <code>True</code>, gradient w.r.t. :attr:<code>weight</code> matrix will be a sparse tensor.
                             See Notes for more details regarding sparse gradients.</p>
<p>Attributes:
    weight (Tensor): the learnable weights of the module of shape (num_embeddings, embedding_dim)
                     initialized from :math:<code>\mathcal{N}(0, 1)</code></p>
<p>Shape:</p>

<pre><code>- Input: :math:`(*)`, IntTensor or LongTensor of arbitrary shape containing the indices to extract
- Output: :math:`(*, H)`, where `*` is the input shape and :math:`H=\text{embedding\_dim}`

</code></pre>
<p>.. note::
    Keep in mind that only a limited number of optimizers support
    sparse gradients: currently it's :class:<code>optim.SGD</code> (<code>CUDA</code> and <code>CPU</code>),
    :class:<code>optim.SparseAdam</code> (<code>CUDA</code> and <code>CPU</code>) and :class:<code>optim.Adagrad</code> (<code>CPU</code>)</p>
<p>.. note::
    When :attr:<code>max_norm</code> is not <code>None</code>, :class:<code>Embedding</code>'s forward method will modify the
    :attr:<code>weight</code> tensor in-place. Since tensors needed for gradient computations cannot be
    modified in-place, performing a differentiable operation on <code>Embedding.weight</code> before
    calling :class:<code>Embedding</code>'s forward method requires cloning <code>Embedding.weight</code> when
    :attr:<code>max_norm</code> is not <code>None</code>. For example::</p>

<pre><code>    n, d, m = 3, 5, 7
    embedding = nn.Embedding(n, d, max_norm=True)
    W = torch.randn((m, d), requires_grad=True)
    idx = torch.tensor([1, 2])
    a = embedding.weight.clone() @ W.t()  # weight must be cloned for this to be differentiable
    b = embedding(idx) @ W.t()  # modifies weight in-place
    out = (a.unsqueeze(0) + b.unsqueeze(1))
    loss = out.sigmoid().prod()
    loss.backward()

</code></pre>
<p>Examples::</p>

<pre><code>&gt;&gt;&gt; # an Embedding module containing 10 tensors of size 3
&gt;&gt;&gt; embedding = nn.Embedding(10, 3)
&gt;&gt;&gt; # a batch of 2 samples of 4 indices each
&gt;&gt;&gt; input = torch.LongTensor([[1,2,4,5],[4,3,2,9]])
&gt;&gt;&gt; embedding(input)
tensor([[[-0.0251, -1.6902,  0.7172],
         [-0.6431,  0.0748,  0.6969],
         [ 1.4970,  1.3448, -0.9685],
         [-0.3677, -2.7265, -0.1685]],

        [[ 1.4970,  1.3448, -0.9685],
         [ 0.4362, -0.4004,  0.9400],
         [-0.6431,  0.0748,  0.6969],
         [ 0.9124, -2.3616,  1.1151]]])


&gt;&gt;&gt; # example with padding_idx
&gt;&gt;&gt; embedding = nn.Embedding(10, 3, padding_idx=0)
&gt;&gt;&gt; input = torch.LongTensor([[0,2,0,5]])
&gt;&gt;&gt; embedding(input)
tensor([[[ 0.0000,  0.0000,  0.0000],
         [ 0.1535, -2.0309,  0.9315],
         [ 0.0000,  0.0000,  0.0000],
         [-0.1655,  0.9897,  0.0635]]])

&gt;&gt;&gt; # example of changing `pad` vector
&gt;&gt;&gt; padding_idx = 0
&gt;&gt;&gt; embedding = nn.Embedding(3, 3, padding_idx=padding_idx)
&gt;&gt;&gt; embedding.weight
Parameter containing:
tensor([[ 0.0000,  0.0000,  0.0000],
        [-0.7895, -0.7089, -0.0364],
        [ 0.6778,  0.5803,  0.2678]], requires_grad=True)
&gt;&gt;&gt; with torch.no_grad():
...     embedding.weight[padding_idx] = torch.ones(3)
&gt;&gt;&gt; embedding.weight
Parameter containing:
tensor([[ 1.0000,  1.0000,  1.0000],
        [-0.7895, -0.7089, -0.0364],
        [ 0.6778,  0.5803,  0.2678]], requires_grad=True)</code></pre>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="PositionalEmbedding" class="doc_header"><code>class</code> <code>PositionalEmbedding</code><a href="https://github.com/RecoHut-Projects/recohut/tree/master/recohut/layers/attention.py#L19" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>PositionalEmbedding</code>(<strong><code>max_len</code></strong>, <strong><code>d_model</code></strong>) :: <code>Module</code></p>
</blockquote>
<p>Base class for all neural network modules.</p>
<p>Your models should also subclass this class.</p>
<p>Modules can also contain other Modules, allowing to nest them in
a tree structure. You can assign the submodules as regular attributes::</p>

<pre><code>import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

</code></pre>
<p>Submodules assigned in this way will be registered, and will have their
parameters converted too when you call :meth:<code>to</code>, etc.</p>
<p>:ivar training: Boolean represents whether this module is in training or
                evaluation mode.
:vartype training: bool</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="GELU" class="doc_header"><code>class</code> <code>GELU</code><a href="https://github.com/RecoHut-Projects/recohut/tree/master/recohut/layers/attention.py#L33" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>GELU</code>() :: <code>Module</code></p>
</blockquote>
<p>Base class for all neural network modules.</p>
<p>Your models should also subclass this class.</p>
<p>Modules can also contain other Modules, allowing to nest them in
a tree structure. You can assign the submodules as regular attributes::</p>

<pre><code>import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

</code></pre>
<p>Submodules assigned in this way will be registered, and will have their
parameters converted too when you call :meth:<code>to</code>, etc.</p>
<p>:ivar training: Boolean represents whether this module is in training or
                evaluation mode.
:vartype training: bool</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="PositionwiseFeedForward" class="doc_header"><code>class</code> <code>PositionwiseFeedForward</code><a href="https://github.com/RecoHut-Projects/recohut/tree/master/recohut/layers/attention.py#L38" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>PositionwiseFeedForward</code>(<strong><code>d_model</code></strong>, <strong><code>d_ff</code></strong>) :: <code>Module</code></p>
</blockquote>
<p>Base class for all neural network modules.</p>
<p>Your models should also subclass this class.</p>
<p>Modules can also contain other Modules, allowing to nest them in
a tree structure. You can assign the submodules as regular attributes::</p>

<pre><code>import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

</code></pre>
<p>Submodules assigned in this way will be registered, and will have their
parameters converted too when you call :meth:<code>to</code>, etc.</p>
<p>:ivar training: Boolean represents whether this module is in training or
                evaluation mode.
:vartype training: bool</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="LayerNorm" class="doc_header"><code>class</code> <code>LayerNorm</code><a href="https://github.com/RecoHut-Projects/recohut/tree/master/recohut/layers/attention.py#L49" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>LayerNorm</code>(<strong><code>features</code></strong>, <strong><code>eps</code></strong>=<em><code>1e-06</code></em>) :: <code>Module</code></p>
</blockquote>
<p>Base class for all neural network modules.</p>
<p>Your models should also subclass this class.</p>
<p>Modules can also contain other Modules, allowing to nest them in
a tree structure. You can assign the submodules as regular attributes::</p>

<pre><code>import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

</code></pre>
<p>Submodules assigned in this way will be registered, and will have their
parameters converted too when you call :meth:<code>to</code>, etc.</p>
<p>:ivar training: Boolean represents whether this module is in training or
                evaluation mode.
:vartype training: bool</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="SublayerConnection" class="doc_header"><code>class</code> <code>SublayerConnection</code><a href="https://github.com/RecoHut-Projects/recohut/tree/master/recohut/layers/attention.py#L62" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>SublayerConnection</code>(<strong><code>size</code></strong>, <strong><code>dropout</code></strong>) :: <code>Module</code></p>
</blockquote>
<p>layer norm and dropout (dropout and then layer norm)</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="Attention" class="doc_header"><code>class</code> <code>Attention</code><a href="https://github.com/RecoHut-Projects/recohut/tree/master/recohut/layers/attention.py#L75" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>Attention</code>() :: <code>Module</code></p>
</blockquote>
<p>Base class for all neural network modules.</p>
<p>Your models should also subclass this class.</p>
<p>Modules can also contain other Modules, allowing to nest them in
a tree structure. You can assign the submodules as regular attributes::</p>

<pre><code>import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

</code></pre>
<p>Submodules assigned in this way will be registered, and will have their
parameters converted too when you call :meth:<code>to</code>, etc.</p>
<p>:ivar training: Boolean represents whether this module is in training or
                evaluation mode.
:vartype training: bool</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="MultiHeadedAttention" class="doc_header"><code>class</code> <code>MultiHeadedAttention</code><a href="https://github.com/RecoHut-Projects/recohut/tree/master/recohut/layers/attention.py#L96" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>MultiHeadedAttention</code>(<strong><code>h</code></strong>, <strong><code>d_model</code></strong>, <strong><code>head_size</code></strong>=<em><code>None</code></em>, <strong><code>dropout</code></strong>=<em><code>0.1</code></em>) :: <code>Module</code></p>
</blockquote>
<p>Base class for all neural network modules.</p>
<p>Your models should also subclass this class.</p>
<p>Modules can also contain other Modules, allowing to nest them in
a tree structure. You can assign the submodules as regular attributes::</p>

<pre><code>import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

</code></pre>
<p>Submodules assigned in this way will be registered, and will have their
parameters converted too when you call :meth:<code>to</code>, etc.</p>
<p>:ivar training: Boolean represents whether this module is in training or
                evaluation mode.
:vartype training: bool</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="TransformerBlock" class="doc_header"><code>class</code> <code>TransformerBlock</code><a href="https://github.com/RecoHut-Projects/recohut/tree/master/recohut/layers/attention.py#L131" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>TransformerBlock</code>(<strong><code>hidden</code></strong>, <strong><code>attn_heads</code></strong>, <strong><code>head_size</code></strong>, <strong><code>feed_forward_hidden</code></strong>, <strong><code>dropout</code></strong>, <strong><code>attn_dropout</code></strong>=<em><code>0.1</code></em>) :: <code>Module</code></p>
</blockquote>
<p>Base class for all neural network modules.</p>
<p>Your models should also subclass this class.</p>
<p>Modules can also contain other Modules, allowing to nest them in
a tree structure. You can assign the submodules as regular attributes::</p>

<pre><code>import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

</code></pre>
<p>Submodules assigned in this way will be registered, and will have their
parameters converted too when you call :meth:<code>to</code>, etc.</p>
<p>:ivar training: Boolean represents whether this module is in training or
                evaluation mode.
:vartype training: bool</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="SASMultiHeadedAttention" class="doc_header"><code>class</code> <code>SASMultiHeadedAttention</code><a href="https://github.com/RecoHut-Projects/recohut/tree/master/recohut/layers/attention.py#L148" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>SASMultiHeadedAttention</code>(<strong><code>h</code></strong>, <strong><code>d_model</code></strong>, <strong><code>head_size</code></strong>=<em><code>None</code></em>, <strong><code>dropout</code></strong>=<em><code>0.1</code></em>) :: <code>Module</code></p>
</blockquote>
<p>Base class for all neural network modules.</p>
<p>Your models should also subclass this class.</p>
<p>Modules can also contain other Modules, allowing to nest them in
a tree structure. You can assign the submodules as regular attributes::</p>

<pre><code>import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

</code></pre>
<p>Submodules assigned in this way will be registered, and will have their
parameters converted too when you call :meth:<code>to</code>, etc.</p>
<p>:ivar training: Boolean represents whether this module is in training or
                evaluation mode.
:vartype training: bool</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="SASPositionwiseFeedForward" class="doc_header"><code>class</code> <code>SASPositionwiseFeedForward</code><a href="https://github.com/RecoHut-Projects/recohut/tree/master/recohut/layers/attention.py#L184" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>SASPositionwiseFeedForward</code>(<strong><code>d_model</code></strong>, <strong><code>d_ff</code></strong>, <strong><code>dropout</code></strong>=<em><code>0.1</code></em>) :: <code>Module</code></p>
</blockquote>
<p>Base class for all neural network modules.</p>
<p>Your models should also subclass this class.</p>
<p>Modules can also contain other Modules, allowing to nest them in
a tree structure. You can assign the submodules as regular attributes::</p>

<pre><code>import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

</code></pre>
<p>Submodules assigned in this way will be registered, and will have their
parameters converted too when you call :meth:<code>to</code>, etc.</p>
<p>:ivar training: Boolean represents whether this module is in training or
                evaluation mode.
:vartype training: bool</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="SASTransformerBlock" class="doc_header"><code>class</code> <code>SASTransformerBlock</code><a href="https://github.com/RecoHut-Projects/recohut/tree/master/recohut/layers/attention.py#L198" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>SASTransformerBlock</code>(<strong><code>hidden</code></strong>, <strong><code>attn_heads</code></strong>, <strong><code>head_size</code></strong>, <strong><code>feed_forward_hidden</code></strong>, <strong><code>dropout</code></strong>, <strong><code>attn_dropout</code></strong>=<em><code>0.1</code></em>) :: <code>Module</code></p>
</blockquote>
<p>Base class for all neural network modules.</p>
<p>Your models should also subclass this class.</p>
<p>Modules can also contain other Modules, allowing to nest them in
a tree structure. You can assign the submodules as regular attributes::</p>

<pre><code>import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

</code></pre>
<p>Submodules assigned in this way will be registered, and will have their
parameters converted too when you call :meth:<code>to</code>, etc.</p>
<p>:ivar training: Boolean represents whether this module is in training or
                evaluation mode.
:vartype training: bool</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<blockquote><p>References</p>
<ol>
<li><a href="https://github.com/sparsh-ai/stanza/blob/S714864/model/attention.py">https://github.com/sparsh-ai/stanza/blob/S714864/model/attention.py</a></li>
</ol>
</blockquote>

</div>
</div>
</div>
</div>
 

