---
id: TECH-ATTN-001
title: "Attention Mechanisms"
alternative_names:
  - "Neural Attention"
  - "Soft Attention"
  - "Self-Attention (as a special case)"
category: "Technical → Neural Network Components"
volume: III
chapter: 19
entry_type: "Standard Article"
level: 3
status: "Draft"
last_reviewed: 2026-10-15
version: "0.1.0"
prerequisites:
  - "TECH-RNN-001"
related_entries:
  - "TECH-ARCH-001"
  - "TECH-NN-001"
  - "TECH-LLM-001"
  - "TECH-CNN-001"
  - "TECH-MM-001"
  - "SAFE-INTERP-001"
---

# Attention Mechanisms

## 1. Concise Definition
An **attention mechanism** is a differentiable operation that lets a neural network compute each output as a weighted combination of a set of input vectors, where the weights are themselves computed from the content of the inputs and a query. Attention replaces a fixed, position-based connection pattern with a learned, content-dependent one, and it is the central operation of the transformer architecture.

## 2. Overview & Significance
Attention was introduced to solve a concrete problem in recurrent sequence-to-sequence models (see [TECH-RNN-001]): the encoder had to compress an entire source sentence into one fixed-length vector from which the decoder generated the translation. Bahdanau, Cho and Bengio (2014) conjectured that this fixed-length vector was a bottleneck and proposed letting the decoder (soft-)search for the parts of the source relevant to each target word.

Two features explain the significance of attention. First, it gives any position direct access to any other, so the path length for information between distant positions does not grow with their separation. Second, when used without recurrence, it can be computed for all positions of a sequence at once, which fits parallel hardware. Vaswani et al. (2017) built the Transformer entirely on this operation, and the architecture is covered in [TECH-ARCH-001]. This entry describes the mechanism and its variants; it does not repeat the architectural account.

## 3. Historical Development
**Alignment in translation.** Bahdanau, Cho and Bengio (2014) added an attention (alignment) model to an RNN encoder-decoder, allowing the model to learn soft alignments between source and target words while translating. Luong, Pham and Manning (2015) examined two classes of attention for translation, a *global* approach that attends to all source words and a *local* approach that looks at a subset, and reported gains over non-attentional systems on WMT English-German tasks.

**Beyond text.** Xu et al. (2015) applied visual attention to image captioning, where the decoder attends to regions of an image while generating each word, showing that the idea was not specific to language.

**Self-attention and the Transformer.** Vaswani et al. (2017) proposed the Transformer, "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely", and reported superior translation quality with greater parallelizability and shorter training time on the tasks they tested. Attention in which queries, keys and values all come from the same sequence is called **self-attention**.

**Efficiency and variants.** Subsequent work addressed the cost of attention over long sequences, the memory cost of the key-value cache during generation, and positional treatment. Tay et al. (2020) survey a range of "efficient transformer" variants. Shazeer (2019) proposed multi-query attention to speed decoding, Ainslie et al. (2023) proposed grouped-query attention, and Dao et al. (2022) proposed an input/output-aware exact attention algorithm. Katharopoulos et al. (2020) formulated a linear-attention variant with an RNN form.

## 4. Technical / Conceptual Explanation

### 4.1 General form: query, keys, values
Given a **query** $q$, a set of **keys** $k_1,\dots,k_n$ and **values** $v_1,\dots,v_n$, attention computes a weighted sum of the values:

$$ \text{out} = \sum_{j=1}^{n} \alpha_j\, v_j, \qquad \alpha_j = \frac{\exp\big(s(q, k_j)\big)}{\sum_{m=1}^{n}\exp\big(s(q, k_m)\big)}. $$

The function $s$ is a **score** (compatibility) function; the softmax turns scores into non-negative weights that sum to one. All operations are differentiable, so the whole mechanism is trained end to end by backpropagation (see [TECH-BACKPROP-001]).

### 4.2 Score functions
The additive (MLP) score of Bahdanau et al. (2014) feeds the decoder state and an encoder state through a small feedforward network. Luong et al. (2015) compared simpler multiplicative forms, including a dot product. The Transformer uses a scaled dot product (Vaswani et al., 2017):

$$ s(q, k) = \frac{q^\top k}{\sqrt{d_k}}, $$

where $d_k$ is the key dimension. The scaling prevents large dot products from pushing the softmax into regions of very small gradients.

### 4.3 Matrix form and self-attention
For a sequence of $N$ tokens with representations $X \in \mathbb{R}^{N\times d}$, self-attention forms $Q = XW_Q$, $K = XW_K$, $V = XW_V$ and computes

$$ \text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V. $$

The matrix $QK^\top \in \mathbb{R}^{N\times N}$ holds all pairwise scores, so time and memory in the naive algorithm grow quadratically with $N$ (Vaswani et al., 2017).

### 4.4 Multi-head attention
Rather than one attention function, the Transformer runs $h$ of them in parallel on different learned linear projections and concatenates the results, which allows the model to attend to different kinds of relations at once (Vaswani et al., 2017). The head count and dimensions are architecture hyperparameters.

### 4.5 Masking and cross-attention
In **causal (masked) self-attention**, scores to future positions are set to $-\infty$ before the softmax so each position depends only on earlier ones; this is how autoregressive generation is trained. In **cross-attention**, queries come from one sequence (such as the decoder) and keys and values from another (such as the encoder or an image encoder), as in the original translation setting and in image captioning (Xu et al., 2015).

### 4.6 Positional information
Attention is permutation-equivariant: without extra information, reordering the inputs only reorders the outputs. Order must be injected, as in the sinusoidal encodings of Vaswani et al. (2017), or through learned or relative position schemes. The details of positional schemes are covered in [TECH-ARCH-001].

### 4.7 Soft versus hard attention
The mechanisms above are **soft**: every input contributes with some weight, and gradients flow through the weights. **Hard** attention selects discrete locations and requires gradient estimators other than ordinary backpropagation; Xu et al. (2015) studied both a soft and a hard variant for captioning.

## 5. Types, Components & Classifications
| Variant | Idea | Source |
| :--- | :--- | :--- |
| Additive / MLP attention | Score from a small feedforward network | Bahdanau et al. (2014) |
| Global and local attention | Attend to all or a window of source positions | Luong et al. (2015) |
| Scaled dot-product attention | Dot-product score scaled by $\sqrt{d_k}$ | Vaswani et al. (2017) |
| Self-attention / cross-attention | Queries and keys from the same or different sequences | Vaswani et al. (2017) |
| Multi-head attention | Several attention functions in parallel | Vaswani et al. (2017) |
| Multi-query and grouped-query attention | Share key/value heads across query heads to cut cache size | Shazeer (2019); Ainslie et al. (2023) |
| Exact IO-aware attention (FlashAttention) | Same result as standard attention, computed with fewer slow-memory accesses | Dao et al. (2022) |
| Linear attention | Replace the softmax with a feature-map formulation, giving a recurrent form | Katharopoulos et al. (2020) |
| Efficient / sparse variants | Restrict or approximate the attention pattern | Surveyed in Tay et al. (2020) |
| Visual attention | Attend over image regions | Xu et al. (2015) |

## 6. Applications & Concrete Examples
- **Machine translation.** The setting in which attention was introduced and where it gave reported gains over non-attentional models (Bahdanau et al., 2014; Luong et al., 2015).
- **Image captioning.** Attention over image regions (Xu et al., 2015).
- **Large language models.** Transformer language models are built from stacked self-attention layers (see [TECH-LLM-001]).
- **Vision and multimodal models.** Attention over image patches and across modalities (see [TECH-ARCH-001], [TECH-MM-001]; compare convolution in [TECH-CNN-001]).
- **Retrieval-like use.** Attention can be read as a soft lookup in a key-value memory, and this view links to retrieval augmentation (see [TECH-RAG-001]).

## 7. Advantages, Limitations & Unresolved Questions

### Advantages
- **Direct long-range access.** Any position can interact with any other in one layer.
- **Parallelism.** With no recurrence, all positions can be processed in parallel during training (Vaswani et al., 2017).
- **Flexibility.** The same operation applies to text, images, audio and structured inputs.

### Limitations
- **Quadratic cost.** Standard self-attention scales quadratically in sequence length in time and memory (Vaswani et al., 2017; Tay et al., 2020).
- **Generation-time memory.** Autoregressive decoding stores keys and values for earlier tokens; Shazeer (2019) and Ainslie et al. (2023) target this cost.
- **Weak built-in order and locality bias.** Position information must be supplied, and the model must learn locality rather than receiving it by design.

### Unresolved questions
1. **Are attention weights explanations?** Jain and Wallace (2019) reported experiments in which attention weights were frequently uncorrelated with gradient-based measures of feature importance and in which very different attention distributions gave equivalent predictions, and concluded that attention weights largely do not provide meaningful explanations. Wiegreffe and Pinter (2019) argued that the claim depends on one's definition of explanation and on the experimental design, and proposed alternative tests. The question is Contested and is relevant to interpretability work (see [SAFE-INTERP-001]).
2. **Do cheaper attention variants preserve quality?** Efficient variants trade exactness or expressiveness for cost (Tay et al., 2020). Which trade-offs hold at very large scale is an empirical question. [VERIFICATION NEEDED: no comparative study at frontier scale was confirmed for this draft.]
3. **Can attention-free or hybrid models match full attention?** See [TECH-RNN-001], on state-space and linear-attention models.

## 8. Current State of Knowledge
As of 2026-10-15, dot-product multi-head self-attention in transformer-style networks is the standard component of the large language and multimodal models discussed in this encyclopedia. The definition and mathematics of attention are Established. That attention improved translation over fixed-vector encoder-decoders is Well Supported by the original papers' experiments (Bahdanau et al., 2014; Luong et al., 2015). Exact attention can now be computed with considerably less memory traffic through IO-aware kernels without changing results (Dao et al., 2022); this is an implementation-level improvement. The interpretability status of attention weights is Contested (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019). Specific context-window lengths and cost figures of deployed systems change quickly and are tracked in `MODEL-001`.

## 9. Future Trajectory & Strategic Outlook
This section describes plausible directions and does not register forecasts.

- **Near term (0–3 years).** Continued refinement of memory-efficient and cache-efficient attention variants and their kernels (see [TECH-INFER-001]).
- **Medium term (3–10 years).** The relative roles of full attention, sparse or windowed attention and recurrent or state-space layers in long-context models remain to be settled by empirical comparison.
- **Long term (10+ years).** Whether attention as currently defined is a lasting component of general-purpose models, or one stage in the evolution of architectures, is an open question discussed in [TECH-ARCH-001].

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[TECH-RNN-001] Recurrent Networks and LSTM](./TECH-RNN-001_recurrent_networks_and_lstm.md): the encoder-decoder models in which attention first appeared.
  - [[TECH-NN-001] Artificial Neural Networks](./TECH-NN-001_artificial_neural_networks.md): background on layers and training.
  - [[TECH-BACKPROP-001] Backpropagation](./TECH-BACKPROP-001_backpropagation.md): how attention weights are trained.
- **Related Methods & Technologies**:
  - [[TECH-ARCH-001] The Transformer Architecture](./TECH-ARCH-001_transformer_architecture.md): the full architecture built on attention.
  - [[TECH-CNN-001] Convolutional Neural Networks](./TECH-CNN-001_convolutional_neural_networks.md): the competing local-structure operation.
  - [[TECH-LLM-001] Large Language Models](./TECH-LLM-001_large_language_models.md): the main use of attention at scale.
  - [[TECH-MM-001] Multimodal Foundation Models](./TECH-MM-001_multimodal_foundation_models.md): cross-modal attention.
  - [[TECH-RAG-001] Retrieval-Augmented Generation](./TECH-RAG-001_retrieval_augmented_generation.md): external lookup related to attention-as-lookup.
  - [[TECH-INFER-001] Inference, Quantization and Efficiency](./TECH-INFER-001_inference_quantization_and_efficiency.md): serving-time cost of attention.
- **Advanced Implications & Future Trajectories**:
  - [[SAFE-INTERP-001] Interpretability and Mechanistic Interpretability](../vol08_safety_governance/SAFE-INTERP-001_interpretability_and_mechanistic_interpretability.md): whether attention patterns explain model behaviour.

## 11. References & Foundational Literature
1. Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv:1409.0473. https://arxiv.org/abs/1409.0473. [Supports the introduction of soft attention to an RNN encoder-decoder and the fixed-length-vector bottleneck conjecture.]
2. Luong, M.-T., Pham, H., & Manning, C. D. (2015). Effective approaches to attention-based neural machine translation. arXiv:1508.04025. https://arxiv.org/abs/1508.04025. [Supports the global and local attention classes and their reported translation gains.]
3. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015). Show, attend and tell: Neural image caption generation with visual attention. arXiv:1502.03044. https://arxiv.org/abs/1502.03044. [Supports visual attention for captioning and the soft/hard distinction.]
4. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. arXiv:1706.03762. https://arxiv.org/abs/1706.03762. [Supports scaled dot-product, multi-head and self-attention and the attention-only Transformer.]
5. Shazeer, N. (2019). Fast transformer decoding: One write-head is all you need. arXiv:1911.02150. https://arxiv.org/abs/1911.02150. [Supports multi-query attention to reduce decoding cost.]
6. Jain, S., & Wallace, B. C. (2019). Attention is not explanation. arXiv:1902.10186. https://arxiv.org/abs/1902.10186. [Supports the finding that attention weights are frequently uncorrelated with gradient-based importance.]
7. Wiegreffe, S., & Pinter, Y. (2019). Attention is not not explanation. arXiv:1908.04626. https://arxiv.org/abs/1908.04626. [Supports the counter-argument that the claim depends on the definition of explanation and the test design.]
8. Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F. (2020). Transformers are RNNs: Fast autoregressive transformers with linear attention. arXiv:2006.16236. https://arxiv.org/abs/2006.16236. [Supports linear attention with a recurrent formulation.]
9. Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2020). Efficient transformers: A survey. arXiv:2009.06732. https://arxiv.org/abs/2009.06732. [Supports the survey of efficient attention variants.]
10. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022). FlashAttention: Fast and memory-efficient exact attention with IO-awareness. arXiv:2205.14135. https://arxiv.org/abs/2205.14135. [Supports exact attention computed with fewer memory accesses.]
11. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., & Sanghai, S. (2023). GQA: Training generalized multi-query transformer models from multi-head checkpoints. arXiv:2305.13245. https://arxiv.org/abs/2305.13245. [Supports grouped-query attention.]

Note: all 11 references were confirmed against arXiv abstract pages (title, authors, date). Claims about their contents are limited to what the abstracts state.

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Query-key-value softmax-weighted sum definition | Established | Bahdanau et al. (2014); Vaswani et al. (2017) | High |
| Scaled dot-product and multi-head formulations | Established | Vaswani et al. (2017) | High |
| Quadratic time/memory of standard self-attention | Established | Vaswani et al. (2017); Tay et al. (2020) | High |
| Attention improved translation over fixed-vector encoders | Well Supported | Bahdanau et al. (2014); Luong et al. (2015) | High |
| Exact attention can be computed with less memory traffic | Well Supported | Dao et al. (2022) | High |
| Grouped/multi-query attention reduces cache cost | Well Supported | Shazeer (2019); Ainslie et al. (2023) | Moderate to High |
| Linear attention has a recurrent form | Established (derivation) | Katharopoulos et al. (2020) | High |
| Attention weights are faithful explanations of model decisions | Contested | Jain & Wallace (2019); Wiegreffe & Pinter (2019) | Open |
| Efficient variants match full attention at frontier scale | Preliminary | Tay et al. (2020) (survey only) | Low to Moderate |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.