Attention Mechanisms

An attention mechanism is a differentiable operation that lets a neural network compute each output as a weighted combination of a set of input vectors, where the weights are themselves computed from the content of the inputs and a query. Attention replaces a fixed, position-based connection pattern with a learned, content-dependent one, and it is the central operation of the transformer architecture.

Attention was introduced to solve a concrete problem in recurrent sequence-to-sequence models (see [TECH-RNN-001]): the encoder had to compress an entire source sentence into one fixed-length vector from which the decoder generated the translation. Bahdanau, Cho and Bengio (2014) conjectured that this fixed-length vector was a bottleneck and proposed letting the decoder (soft-)search for the parts of the source relevant to each target word.

Two features explain the significance of attention. First, it gives any position direct access to any other, so the path length for information between distant positions does not grow with their separation. Second, when used without recurrence, it can be computed for all positions of a sequence at once, which fits parallel hardware. Vaswani et al. (2017) built the Transformer entirely on this operation, and the architecture is covered in [TECH-ARCH-001]. This entry describes the mechanism and its variants; it does not repeat the architectural account.

Alignment in translation. Bahdanau, Cho and Bengio (2014) added an attention (alignment) model to an RNN encoder-decoder, allowing the model to learn soft alignments between source and target words while translating. Luong, Pham and Manning (2015) examined two classes of attention for translation, a global approach that attends to all source words and a local approach that looks at a subset, and reported gains over non-attentional systems on WMT English-German tasks.

Beyond text. Xu et al. (2015) applied visual attention to image captioning, where the decoder attends to regions of an image while generating each word, showing that the idea was not specific to language.

Self-attention and the Transformer. Vaswani et al. (2017) proposed the Transformer, "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely", and reported superior translation quality with greater parallelizability and shorter training time on the tasks they tested. Attention in which queries, keys and values all come from the same sequence is called self-attention.

Efficiency and variants. Subsequent work addressed the cost of attention over long sequences, the memory cost of the key-value cache during generation, and positional treatment. Tay et al. (2020) survey a range of "efficient transformer" variants. Shazeer (2019) proposed multi-query attention to speed decoding, Ainslie et al. (2023) proposed grouped-query attention, and Dao et al. (2022) proposed an input/output-aware exact attention algorithm. Katharopoulos et al. (2020) formulated a linear-attention variant with an RNN form.

#4.1 General form: query, keys, values

Given a query $q$, a set of keys $k_1,\dots,k_n$ and values $v_1,\dots,v_n$, attention computes a weighted sum of the values:

The function $s$ is a score (compatibility) function; the softmax turns scores into non-negative weights that sum to one. All operations are differentiable, so the whole mechanism is trained end to end by backpropagation (see [TECH-BACKPROP-001]).

#4.2 Score functions

The additive (MLP) score of Bahdanau et al. (2014) feeds the decoder state and an encoder state through a small feedforward network. Luong et al. (2015) compared simpler multiplicative forms, including a dot product. The Transformer uses a scaled dot product (Vaswani et al., 2017):

where $d_k$ is the key dimension. The scaling prevents large dot products from pushing the softmax into regions of very small gradients.

#4.3 Matrix form and self-attention

For a sequence of $N$ tokens with representations $X \in \mathbb{R}^{N\times d}$, self-attention forms $Q = XW_Q$, $K = XW_K$, $V = XW_V$ and computes

The matrix $QK^\top \in \mathbb{R}^{N\times N}$ holds all pairwise scores, so time and memory in the naive algorithm grow quadratically with $N$ (Vaswani et al., 2017).

#4.4 Multi-head attention

Rather than one attention function, the Transformer runs $h$ of them in parallel on different learned linear projections and concatenates the results, which allows the model to attend to different kinds of relations at once (Vaswani et al., 2017). The head count and dimensions are architecture hyperparameters.

#4.5 Masking and cross-attention

In causal (masked) self-attention, scores to future positions are set to $-\infty$ before the softmax so each position depends only on earlier ones; this is how autoregressive generation is trained. In cross-attention, queries come from one sequence (such as the decoder) and keys and values from another (such as the encoder or an image encoder), as in the original translation setting and in image captioning (Xu et al., 2015).

#4.6 Positional information

Attention is permutation-equivariant: without extra information, reordering the inputs only reorders the outputs. Order must be injected, as in the sinusoidal encodings of Vaswani et al. (2017), or through learned or relative position schemes. The details of positional schemes are covered in [TECH-ARCH-001].

#4.7 Soft versus hard attention

The mechanisms above are soft: every input contributes with some weight, and gradients flow through the weights. Hard attention selects discrete locations and requires gradient estimators other than ordinary backpropagation; Xu et al. (2015) studied both a soft and a hard variant for captioning.

• Machine translation. The setting in which attention was introduced and where it gave reported gains over non-attentional models (Bahdanau et al., 2014; Luong et al., 2015).
• Image captioning. Attention over image regions (Xu et al., 2015).
• Large language models. Transformer language models are built from stacked self-attention layers (see [TECH-LLM-001]).
• Vision and multimodal models. Attention over image patches and across modalities (see [TECH-ARCH-001], [TECH-MM-001]; compare convolution in [TECH-CNN-001]).
• Retrieval-like use. Attention can be read as a soft lookup in a key-value memory, and this view links to retrieval augmentation (see [TECH-RAG-001]).

#Advantages

• Direct long-range access. Any position can interact with any other in one layer.
• Parallelism. With no recurrence, all positions can be processed in parallel during training (Vaswani et al., 2017).
• Flexibility. The same operation applies to text, images, audio and structured inputs.

#Limitations

• Quadratic cost. Standard self-attention scales quadratically in sequence length in time and memory (Vaswani et al., 2017; Tay et al., 2020).
• Generation-time memory. Autoregressive decoding stores keys and values for earlier tokens; Shazeer (2019) and Ainslie et al. (2023) target this cost.
• Weak built-in order and locality bias. Position information must be supplied, and the model must learn locality rather than receiving it by design.

#Unresolved questions

1. Are attention weights explanations? Jain and Wallace (2019) reported experiments in which attention weights were frequently uncorrelated with gradient-based measures of feature importance and in which very different attention distributions gave equivalent predictions, and concluded that attention weights largely do not provide meaningful explanations. Wiegreffe and Pinter (2019) argued that the claim depends on one's definition of explanation and on the experimental design, and proposed alternative tests. The question is Contested and is relevant to interpretability work (see [SAFE-INTERP-001]).
2. Do cheaper attention variants preserve quality? Efficient variants trade exactness or expressiveness for cost (Tay et al., 2020). Which trade-offs hold at very large scale is an empirical question. The sources cited here do not include a comparative study at frontier scale.
3. Can attention-free or hybrid models match full attention? See [TECH-RNN-001], on state-space and linear-attention models.

As of 2026-10-19, dot-product multi-head self-attention in transformer-style networks is the standard component of the large language and multimodal models discussed in this encyclopedia. The definition and mathematics of attention are Established. That attention improved translation over fixed-vector encoder-decoders is Well Supported by the original papers' experiments (Bahdanau et al., 2014; Luong et al., 2015). Exact attention can now be computed with considerably less memory traffic through IO-aware kernels without changing results (Dao et al., 2022); this is an implementation-level improvement. The interpretability status of attention weights is Contested (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019). Specific context-window lengths and cost figures of deployed systems change quickly and are tracked in MODEL-001.

This section describes plausible directions and does not register forecasts.

• Near term (0–3 years). Continued refinement of memory-efficient and cache-efficient attention variants and their kernels (see [TECH-INFER-001]).
• Medium term (3–10 years). The relative roles of full attention, sparse or windowed attention and recurrent or state-space layers in long-context models remain to be settled by empirical comparison.
• Long term (10+ years). Whether attention as currently defined is a lasting component of general-purpose models, or one stage in the evolution of architectures, is an open question discussed in [TECH-ARCH-001].

• Foundational Concepts:
  • TECH-RNN-001 Recurrent Networks and LSTM: the encoder-decoder models in which attention first appeared.
  • TECH-NN-001 Artificial Neural Networks: background on layers and training.
  • TECH-BACKPROP-001 Backpropagation: how attention weights are trained.
• Related Methods & Technologies:
  • TECH-ARCH-001 The Transformer Architecture: the full architecture built on attention.
  • TECH-CNN-001 Convolutional Neural Networks: the competing local-structure operation.
  • TECH-LLM-001 Large Language Models: the main use of attention at scale.
  • TECH-MM-001 Multimodal Foundation Models: cross-modal attention.
  • TECH-RAG-001 Retrieval-Augmented Generation: external lookup related to attention-as-lookup.
  • TECH-INFER-001 Inference, Quantization and Efficiency: serving-time cost of attention.
• Advanced Implications & Future Trajectories:
  • SAFE-INTERP-001 Interpretability and Mechanistic Interpretability: whether attention patterns explain model behaviour.