# Attention (machine learning)

In machine learning, attention is a method that determines the importance of each component in a sequence relative to the other components in that sequence. In natural language processing, importance is represented by "soft" weights assigned to each word in a sentence. More generally, attention encodes vectors called token embeddings across a fixed-width sequence that can range from tens to millions of tokens in size.
Unlike "hard" weights, which are computed during the backwards training pass, "soft" weights exist only in the forward pass and therefore change with every step of the input. Earlier designs implemented the attention mechanism in a serial recurrent neural network (RNN) language translation system, but a more recent design, namely the transformer, removed the slower sequential RNN and relied more heavily on the faster parallel attention scheme.
Inspired by ideas about attention in humans, the attention mechanism was developed to address the weaknesses of using information from the hidden layers of recurrent neural networks. Recurrent neural networks favor more recent information contained in words at the end of a sentence, while information earlier in the sentence tends to be attenuated. Attention allows a token equal access to any part of a sentence directly, rather than only through the previous state.


## History

Additional surveys of the attention mechanism in deep learning are provided by Niu et al. and Soydaner.
The major breakthrough came with self-attention, where each element in the input sequence attends to all others, enabling the model to capture global dependencies. This idea was central to the Transformer architecture, which replaced recurrence with attention mechanisms. As a result, Transformers became the foundation for models like BERT, T5 and generative pre-trained transformers (GPT). 


## Overview

The modern era of machine attention was revitalized by grafting an attention mechanism (Fig 1.  orange) to an Encoder-Decoder.

Figure 2 shows the internal step-by-step operation of the attention block (A) in Fig 1.


### Interpreting attention weights
In translating between languages, alignment is the process of matching words from the source sentence to words of the translated sentence. Networks that perform verbatim translation without regard to word order would show the highest scores along the (dominant) diagonal of the matrix. The off-diagonal dominance shows that the attention mechanism is more nuanced. 
Consider an example of translating I love you to French. On the first pass through the decoder, 94% of the attention weight is on the first English word I, so the network offers the word je. On the second pass of the decoder, 88% of the attention weight is on the third English word you, so it offers t'. On the last pass, 95% of the attention weight is on the second English word love, so it offers aime.
In the I love you example, the second word love is aligned with the third word aime. Stacking soft row vectors together for je, t', and aime yields an alignment matrix:

Sometimes, alignment can be multiple-to-multiple. For example, the English phrase look it up corresponds to cherchez-le. Thus, "soft" attention weights work better than "hard" attention weights (setting one attention weight to 1, and the others to 0), as we would like the model to make a context vector consisting of a weighted sum of the hidden vectors, rather than "the best one", as there may not be a best hidden vector.


## Variants

Many variants of attention implement soft weights, such as

fast weight programmers, or fast weight controllers (1992). A "slow" neural network outputs the "fast" weights of another neural network through outer products. The slow network learns by gradient descent. It was later renamed as "linearized self-attention".
Bahdanau-style attention, also referred to as additive attention,
Luong-style attention, which is known as multiplicative attention,
Early attention mechanisms similar to modern self-attention were proposed using recurrent neural networks. However, the highly parallelizable self-attention was introduced in 2017 and successfully used in the Transformer model,
positional attention and factorized positional attention.
For convolutional neural networks, attention mechanisms can be distinguished by the dimension on which they operate, namely: spatial attention, channel attention, or combinations.
These variants recombine the encoder-side inputs to redistribute those effects to each target output. Often, a correlation-style matrix of dot products provides the re-weighting coefficients.  In the figures below, W is the matrix of context attention weights, similar to the formula in Overview section above.


## Optimizations


### Flash attention
The size of the attention matrix is proportional to the square of the number of input tokens. Therefore, when the input is long, calculating the attention matrix requires a lot of GPU memory. Flash attention is an implementation that reduces the memory needs and increases efficiency without sacrificing accuracy. It achieves this by partitioning the attention computation into smaller blocks that fit into the GPU's faster on-chip memory, reducing the need to store large intermediate matrices and thus lowering memory usage while increasing computational efficiency.


### FlexAttention
FlexAttention is an attention kernel developed by Meta that allows users to modify attention scores prior to softmax and dynamically chooses the optimal attention algorithm.


## Applications
Attention is widely used in natural language processing, computer vision, and speech recognition. In NLP, it improves context understanding in tasks like question answering and summarization. In vision, visual attention helps models focus on relevant image regions, enhancing object detection and image captioning.


### Attention maps as explanations for vision transformers

From the original paper on vision transformers (ViT), visualizing attention scores as a heat map (called saliency maps or attention maps) has become an important and routine way to inspect the decision making process of ViT models. One can compute the attention maps with respect to any attention head at any layer, while the deeper layers tend to show more semantically meaningful visualization. Attention rollout is a recursive algorithm to combine attention scores across all layers, by computing the dot product of successive attention maps.
Because vision transformers are typically trained in a self-supervised manner, attention maps are generally not class-sensitive. When a classification head attached to the ViT backbone, class-discriminative attention maps (CDAM) combines attention maps and gradients with respect to the class [CLS] token. Some class-sensitive interpretability methods originally developed for convolutional neural networks can be also applied to ViT, such as GradCAM, which back-propagates the gradients to the outputs of the final attention layer.
Using attention as basis of explanation for the transformers in language and vision is not without debate. While some pioneering papers analyzed and framed attention scores as explanations, higher attention scores do not always correlate with greater impact on model performances. 


## Mathematical representation


### Standard scaled dot-product attention
For matrices: 
  
    
      
        Q
        ∈
        
          
            R
          
          
            m
            ×
            
              d
              
                k
              
            
          
        
        ,
        K
        ∈
        
          
            R
          
          
            n
            ×
            
              d
              
                k
              
            
          
        
      
    
    {\displaystyle Q\in \mathbb {R} ^{m\times d_{k}},K\in \mathbb {R} ^{n\times d_{k}}}
  
 and 
  
    
      
        V
        ∈
        
          
            R
          
          
            n
            ×
            
              d
              
                v
              
            
          
        
      
    
    {\displaystyle V\in \mathbb {R} ^{n\times d_{v}}}
  
, the scaled dot-product, or QKV attention, is defined as:

  
    
      
        
          Attention
        
        (
        Q
        ,
        K
        ,
        V
        )
        =
        
          softmax
        
        
          (
          
            
              
                Q
                
                  K
                  
                    T
                  
                
              
              
                
                  d
                  
                    k
                  
                
              
            
          
          )
        
        V
        ∈
        
          
            R
          
          
            m
            ×
            
              d
              
                v
              
            
          
        
      
    
    {\displaystyle {\text{Attention}}(Q,K,V)={\text{softmax}}\left({\frac {QK^{T}}{\sqrt {d_{k}}}}\right)V\in \mathbb {R} ^{m\times d_{v}}}
  

where 
  
    
      
        
          

          
          
            T
          
        
      
    
    {\displaystyle {}^{T}}
  
 denotes transpose and the softmax function is applied independently to every row of its argument. The matrix 
  
    
      
        Q
      
    
    {\displaystyle Q}
  
 contains 
  
    
      
        m
      
    
    {\displaystyle m}
  
 queries, while matrices 
  
    
      
        K
        ,
        V
      
    
    {\displaystyle K,V}
  
 jointly contain an unordered set of 
  
    
      
        n
      
    
    {\displaystyle n}
  
 key-value pairs. Value vectors in matrix 
  
    
      
        V
      
    
    {\displaystyle V}
  
 are weighted using the weights resulting from the softmax operation, so that the rows of the 
  
    
      
        m
      
    
    {\displaystyle m}
  
-by-
  
    
      
        
          d
          
            v
          
        
      
    
    {\displaystyle d_{v}}
  
 output matrix are confined to the convex hull of the points in 
  
    
      
        
          
            R
          
          
            
              d
              
                v
              
            
          
        
      
    
    {\displaystyle \mathbb {R} ^{d_{v}}}
  
 given by the rows of 
  
    
      
        V
      
    
    {\displaystyle V}
  
.
To understand the permutation invariance and permutation equivariance properties of QKV attention, let 
  
    
      
        A
        ∈
        
          
            R
          
          
            m
            ×
            m
          
        
      
    
    {\displaystyle A\in \mathbb {R} ^{m\times m}}
  
 and 
  
    
      
        B
        ∈
        
          
            R
          
          
            n
            ×
            n
          
        
      
    
    {\displaystyle B\in \mathbb {R} ^{n\times n}}
  
 be permutation matrices; and 
  
    
      
        D
        ∈
        
          
            R
          
          
            m
            ×
            n
          
        
      
    
    {\displaystyle D\in \mathbb {R} ^{m\times n}}
  
 an arbitrary matrix. The softmax function is permutation equivariant in the sense that:

  
    
      
        
          softmax
        
        (
        A
        D
        B
        )
        =
        A
        
        
          softmax
        
        (
        D
        )
        B
      
    
    {\displaystyle {\text{softmax}}(ADB)=A\,{\text{softmax}}(D)B}
  

By noting that the transpose of a permutation matrix is also its inverse, it follows that:

  
    
      
        
          Attention
        
        (
        A
        Q
        ,
        B
        K
        ,
        B
        V
        )
        =
        A
        
        
          Attention
        
        (
        Q
        ,
        K
        ,
        V
        )
      
    
    {\displaystyle {\text{Attention}}(AQ,BK,BV)=A\,{\text{Attention}}(Q,K,V)}
  

which shows that QKV attention is equivariant with respect to re-ordering the queries (rows of 
  
    
      
        Q
      
    
    {\displaystyle Q}
  
); and invariant to re-ordering of the key-value pairs in 
  
    
      
        K
        ,
        V
      
    
    {\displaystyle K,V}
  
. These properties are inherited when applying linear transforms to the inputs and outputs of QKV attention blocks. For example, a simple self-attention function defined as:

  
    
      
        X
        ↦
        
          Attention
        
        (
        X
        
          T
          
            q
          
        
        ,
        X
        
          T
          
            k
          
        
        ,
        X
        
          T
          
            v
          
        
        )
      
    
    {\displaystyle X\mapsto {\text{Attention}}(XT_{q},XT_{k},XT_{v})}
  

is permutation equivariant with respect to re-ordering the rows of the input matrix 
  
    
      
        X
      
    
    {\displaystyle X}
  
 in a non-trivial way, because every row of the output is a function of all the rows of the input. Similar properties hold for multi-head attention, which is defined below.


### Masked attention
When QKV attention is used as a building block for an autoregressive decoder, and when at training time all input and output matrices have 
  
    
      
        n
      
    
    {\displaystyle n}
  
 rows, a masked attention variant is used:

  
    
      
        
          Attention
        
        (
        Q
        ,
        K
        ,
        V
        )
        =
        
          softmax
        
        
          (
          
            
              
                
                  Q
                  
                    K
                    
                      T
                    
                  
                
                
                  
                    d
                    
                      k
                    
                  
                
              
            
            +
            M
          
          )
        
        V
      
    
    {\displaystyle {\text{Attention}}(Q,K,V)={\text{softmax}}\left({\frac {QK^{T}}{\sqrt {d_{k}}}}+M\right)V}
  

where the mask, 
  
    
      
        M
        ∈
        
          
            R
          
          
            n
            ×
            n
          
        
      
    
    {\displaystyle M\in \mathbb {R} ^{n\times n}}
  
 is a strictly upper triangular matrix, with zeros on and below the diagonal and 
  
    
      
        −
        ∞
      
    
    {\displaystyle -\infty }
  
 in every element above the diagonal. The softmax output, also in 
  
    
      
        
          
            R
          
          
            n
            ×
            n
          
        
      
    
    {\displaystyle \mathbb {R} ^{n\times n}}
  
 is then lower triangular, with zeros in all elements above the diagonal. The masking ensures that for all 
  
    
      
        1
        ≤
        i
        <
        j
        ≤
        n
      
    
    {\displaystyle 1\leq i<j\leq n}
  
, row 
  
    
      
        i
      
    
    {\displaystyle i}
  
 of the attention output is independent of row 
  
    
      
        j
      
    
    {\displaystyle j}
  
 of any of the three input matrices. The permutation invariance and equivariance properties of standard QKV attention do not hold for the masked variant.


### Multi-head attention

Multi-head attention

  
    
      
        
          MultiHead
        
        (
        Q
        ,
        K
        ,
        V
        )
        =
        
          Concat
        
        (
        
          
            head
          
          
            1
          
        
        ,
        .
        .
        .
        ,
        
          
            head
          
          
            h
          
        
        )
        
          W
          
            O
          
        
      
    
    {\displaystyle {\text{MultiHead}}(Q,K,V)={\text{Concat}}({\text{head}}_{1},...,{\text{head}}_{h})W^{O}}
  

where each head is computed with QKV attention as:

  
    
      
        
          
            head
          
          
            i
          
        
        =
        
          Attention
        
        (
        Q
        
          W
          
            i
          
          
            Q
          
        
        ,
        K
        
          W
          
            i
          
          
            K
          
        
        ,
        V
        
          W
          
            i
          
          
            V
          
        
        )
      
    
    {\displaystyle {\text{head}}_{i}={\text{Attention}}(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V})}
  

and 
  
    
      
        
          W
          
            i
          
          
            Q
          
        
        ,
        
          W
          
            i
          
          
            K
          
        
        ,
        
          W
          
            i
          
          
            V
          
        
      
    
    {\displaystyle W_{i}^{Q},W_{i}^{K},W_{i}^{V}}
  
, and 
  
    
      
        
          W
          
            O
          
        
      
    
    {\displaystyle W^{O}}
  
 are parameter matrices.
The permutation properties of (standard, unmasked) QKV attention apply here also. For permutation matrices, 
  
    
      
        A
        ,
        B
      
    
    {\displaystyle A,B}
  
:

  
    
      
        
          MultiHead
        
        (
        A
        Q
        ,
        B
        K
        ,
        B
        V
        )
        =
        A
        
        
          MultiHead
        
        (
        Q
        ,
        K
        ,
        V
        )
      
    
    {\displaystyle {\text{MultiHead}}(AQ,BK,BV)=A\,{\text{MultiHead}}(Q,K,V)}
  

from which we also see that multi-head self-attention:

  
    
      
        X
        ↦
        
          MultiHead
        
        (
        X
        
          T
          
            q
          
        
        ,
        X
        
          T
          
            k
          
        
        ,
        X
        
          T
          
            v
          
        
        )
      
    
    {\displaystyle X\mapsto {\text{MultiHead}}(XT_{q},XT_{k},XT_{v})}
  

is equivariant with respect to re-ordering of the rows of input matrix 
  
    
      
        X
      
    
    {\displaystyle X}
  
.


### Bahdanau (additive) attention

  
    
      
        
          Attention
        
        (
        Q
        ,
        K
        ,
        V
        )
        =
        
          softmax
        
        (
        tanh
        ⁡
        (
        
          W
          
            Q
          
        
        Q
        +
        
          W
          
            K
          
        
        K
        )
        V
        )
      
    
    {\displaystyle {\text{Attention}}(Q,K,V)={\text{softmax}}(\tanh(W_{Q}Q+W_{K}K)V)}
  

where 
  
    
      
        
          W
          
            Q
          
        
      
    
    {\displaystyle W_{Q}}
  
 and 
  
    
      
        
          W
          
            K
          
        
      
    
    {\displaystyle W_{K}}
  
 are learnable weight matrices.


### Luong attention (general)

  
    
      
        
          Attention
        
        (
        Q
        ,
        K
        ,
        V
        )
        =
        
          softmax
        
        (
        Q
        W
        
          K
          
            T
          
        
        )
        V
      
    
    {\displaystyle {\text{Attention}}(Q,K,V)={\text{softmax}}(QWK^{T})V}
  

where 
  
    
      
        W
      
    
    {\displaystyle W}
  
 is a learnable weight matrix.


### Self-attention
Self-attention is essentially the same as cross-attention, except that query, key, and value vectors all come from the same model. Both encoder and decoder can use self-attention, but with subtle differences.
For encoder self-attention, we can start with a simple encoder without self-attention, such as an "embedding layer", which simply converts each input word into a vector by a fixed lookup table. This gives a sequence of hidden vectors 
  
    
      
        
          h
          
            0
          
        
        ,
        
          h
          
            1
          
        
        ,
        …
      
    
    {\displaystyle h_{0},h_{1},\dots }
  
. These can then be applied to a dot-product attention mechanism, to obtain
  
    
      
        
          
            
              
                
                  h
                  
                    0
                  
                  ′
                
              
              
                
                =
                
                  A
                  t
                  t
                  e
                  n
                  t
                  i
                  o
                  n
                
                (
                
                  h
                  
                    0
                  
                
                
                  W
                  
                    Q
                  
                
                ,
                H
                
                  W
                  
                    K
                  
                
                ,
                H
                
                  W
                  
                    V
                  
                
                )
              
            
            
              
                
                  h
                  
                    1
                  
                  ′
                
              
              
                
                =
                
                  A
                  t
                  t
                  e
                  n
                  t
                  i
                  o
                  n
                
                (
                
                  h
                  
                    1
                  
                
                
                  W
                  
                    Q
                  
                
                ,
                H
                
                  W
                  
                    K
                  
                
                ,
                H
                
                  W
                  
                    V
                  
                
                )
              
            
            
              
              
                
                ⋯
              
            
          
        
      
    
    {\displaystyle {\begin{aligned}h_{0}'&=\mathrm {Attention} (h_{0}W^{Q},HW^{K},HW^{V})\\h_{1}'&=\mathrm {Attention} (h_{1}W^{Q},HW^{K},HW^{V})\\&\cdots \end{aligned}}}
  
or more succinctly, 
  
    
      
        
          H
          ′
        
        =
        
          A
          t
          t
          e
          n
          t
          i
          o
          n
        
        (
        H
        
          W
          
            Q
          
        
        ,
        H
        
          W
          
            K
          
        
        ,
        H
        
          W
          
            V
          
        
        )
      
    
    {\displaystyle H'=\mathrm {Attention} (HW^{Q},HW^{K},HW^{V})}
  
. This can be applied repeatedly, to obtain a multilayered encoder. This is the "encoder self-attention", sometimes called the "all-to-all attention", as the vector at every position can attend to every other.


### Masking
For decoder self-attention, all-to-all attention is inappropriate, because during the autoregressive decoding process, the decoder cannot attend to future outputs that has yet to be decoded. This can be solved by forcing the attention weights 
  
    
      
        
          w
          
            i
            j
          
        
        =
        0
      
    
    {\displaystyle w_{ij}=0}
  
 for all 
  
    
      
        i
        <
        j
      
    
    {\displaystyle i<j}
  
, called "causal masking". This attention mechanism is the "causally masked self-attention".


## See also
Recurrent neural network
seq2seq
Transformer (deep learning architecture)
Attention
Dynamic neural network


## References


## External links
Olah, Chris; Carter, Shan (September 8, 2016). "Attention and Augmented Recurrent Neural Networks". Distill. 1 (9). Distill Working Group. doi:10.23915/distill.00001.
Dan Jurafsky and James H. Martin (2022). Speech and Language Processing (3rd ed. draft, January 2022) — Chapter 10.4 (Attention) and Chapter 9.7 (Self-Attention Networks: Transformers)
Alex Graves (2020). Attention and Memory in Deep Learning — video lecture from DeepMind / UCL