Deep Learning and Neural Networks

The Transformer Architecture is a deep neural network topology introduced by Vaswani et al. (2017) that relies entirely on self-attention mechanisms to compute representations of input and output sequences, dispensing with recurrence and convolutions. By computing pairwise interactions between all tokens in a sequence simultaneously, the architecture enables massive parallelization, highly efficient scaling, and robust modeling of long-range dependencies, forming the foundational substrate for modern large language models and multimodal foundation models.

Prior to the Transformer, sequence modeling in artificial intelligence was dominated by Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks. RNNs process tokens sequentially, inherently creating a computational bottleneck that prevents parallelization across time steps and leads to vanishing gradients over long sequences.

The significance of the Transformer lies in its structural capacity to overcome these sequential constraints:

1. Parallelization: It computes the interactions between all positions in a sequence simultaneously using matrix multiplication, making it exceptionally well-suited for highly parallel hardware like GPUs and TPUs.
2. Global Receptive Field: Every token attends to every other token directly, reducing the path length for learning long-range dependencies from $O(N)$ (in RNNs) to $O(1)$.
3. Scaling Behavior: For Transformer language models, studies report that cross-entropy loss falls approximately as a power law in parameter count, dataset size and compute over the ranges studied (Kaplan et al., 2020; Hoffmann et al., 2022), which has supported the development of large foundation models (e.g., BERT, GPT-3, Llama).

Historically, sequence-to-sequence learning relied on encoder-decoder RNNs (Sutskever et al., 2014). To improve alignment between the input sequence and the generated output, Bahdanau et al. (2014) introduced the attention mechanism, which allowed the decoder to dynamically weight parts of the input sequence during translation. However, the underlying sequential RNN backbone remained.

In 2017, researchers at Google Brain and Google Research (Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, & Polosukhin) published Attention Is All You Need. They demonstrated that the recurrence could be eliminated entirely. Their model, the Transformer, achieved state-of-the-art results on English-to-German translation tasks while requiring significantly less training time.

Subsequent developments bifurcated the architecture:

• Encoder-only models (e.g., BERT, 2018; Devlin et al., 2019) focused on bidirectional context for natural language understanding and masked language modeling.
• Decoder-only models (e.g., GPT, 2018, Radford et al., 2018; Llama, Touvron et al., 2023) utilized causal masking for autoregressive generative tasks, becoming the dominant paradigm for Large Language Models (LLMs).
• Encoder-decoder models (e.g., T5, 2019; Raffel et al., 2020) retained the original structure for translation and sequence-to-sequence mapping.

#4.1 Tokenization and Embeddings

Input sequences (text, image patches, or audio frames) are first discretized into tokens. Each token is mapped to a continuous vector $x_i \in \mathbb{R}^d$, forming an input matrix $X \in \mathbb{R}^{N \times d}$, where $N$ is the sequence length and $d$ is the model dimension.

Because the model lacks recurrence, positional information must be injected explicitly. The original Transformer used fixed sinusoidal positional encodings:

These encodings are added to the token embeddings before entering the network. Modern implementations frequently use learned absolute embeddings or relative mechanisms like Rotary Positional Embeddings (RoPE) (Su et al., 2021).

#4.2 Scaled Dot-Product Attention

The core mathematical operation is scaled dot-product attention. The input sequence $X$ is linearly projected into three matrices: Queries ($Q$), Keys ($K$), and Values ($V$):

The attention function maps queries and keys to a probability distribution, which is used to compute a weighted sum of the values:

The scaling factor $\frac{1}{\sqrt{d_k}}$ counteracts the tendency of the dot product to grow large and push the softmax function into regions with vanishing gradients.

#4.3 Multi-Head Attention

Instead of a single attention function, the Transformer linearly projects the queries, keys, and values $h$ times (heads). Attention is performed in parallel on each projected version, concatenated, and projected again:

where $\text{head}_i = \text{Attention}(Q W_{Q_i}, K W_{K_i}, V W_{V_i})$. This allows the model to jointly attend to information from different representation subspaces at different positions.

#4.4 Causal Masking

For autoregressive generation (decoder models), a causal mask is applied to the $Q K^\top$ matrix before the softmax operation. The mask sets all upper-triangular entries (representing future tokens) to $-\infty$, ensuring that the prediction for position $t$ can depend only on positions less than or equal to $t$.

#4.5 Feed-Forward Networks and Residuals

Each attention block is followed by a position-wise feed-forward network (FFN), typically a two-layer perceptron with a non-linearity (e.g., ReLU, GELU, or SwiGLU):

Both the multi-head attention and the FFN are surrounded by residual connections (He et al., 2016) and Layer Normalization (Ba et al., 2016):

This is the post-LN arrangement of the original paper; many recent large language models instead use pre-normalization, often with RMSNorm.

The Transformer ecosystem is classified by architectural topology:

1. Decoder-Only (Autoregressive): (e.g., GPT-1, GPT-3, Llama). Utilizes causal masking to predict the next token. Widely used for generative language models, including few-shot use (Brown et al., 2020). Several proprietary frontier systems are widely believed to follow this design, but their architectures are undisclosed.
2. Encoder-Only (Bidirectional): (e.g., BERT, RoBERTa; Liu et al., 2019). Removes causal masking, allowing tokens to attend to the entire sequence simultaneously. Trained via masked language modeling. Commonly used for classification, semantic search, and information extraction.
3. Encoder-Decoder: (Original 2017 Transformer, T5, BART; Lewis et al., 2020). The encoder processes the input bidirectionally, and the decoder generates output autoregressively while cross-attending to the encoder representations. Used for translation and summarization.

• Natural Language: Machine translation, interactive chatbots, code synthesis, and document summarization.
• Computer Vision: The Vision Transformer (ViT) treats image patches as a sequence of tokens, matching or exceeding strong convolutional baselines on ImageNet and other image-recognition benchmarks when pre-trained on large datasets (Dosovitskiy et al., 2021).
• Structural Biology: AlphaFold 2 relies heavily on Evoformer blocks (a variant of transformer attention) to process multiple sequence alignments and predict 3D protein structures (Jumper et al., 2021).
• Multimodal AI: CLIP aligns text and image embeddings in a shared space (Radford et al., 2021); other systems extend multimodal modeling to audio and video.

#Demonstrated Advantages

• Scalability: Transformer language models have been trained at very large scale; models with trillions of total parameters have been reported as sparse mixture-of-experts models (Fedus et al., 2021).
• General-Purpose Use: Increasingly used across language, vision and other modalities, although convolutional and other architectures remain in use.
• In-Context Learning: Demonstrates few-shot in-context learning at sufficient scale, adapting to new tasks from examples in the prompt without gradient updates (Brown et al., 2020).

#Core Limitations

• Quadratic Complexity: The standard attention matrix $Q K^\top$ requires $O(N^2)$ time and memory with respect to sequence length $N$, making exceedingly long contexts computationally prohibitive.
• Inference Latency (KV Bottleneck): During autoregressive generation, past Keys and Values must be cached (KV cache) to prevent recomputation. As sequence length grows, the memory bandwidth required to read the KV cache becomes the primary inference bottleneck (Shazeer, 2019).
• Lack of Inherent Structural Bias: Unlike CNNs (which assume spatial locality), Transformers must learn all relational biases from scratch, requiring massive pretraining datasets to avoid overfitting.

#Unresolved Scientific Questions

1. Expressivity versus Practice: Theoretical results depend on idealizations. Pérez et al. (2021) show that a hard-attention Transformer variant is Turing complete, whereas Merrill et al. (2022) show that saturated Transformers with floating-point values can be simulated by constant-depth threshold circuits (TC$^0$), an upper bound. How these results relate to multi-step algorithmic reasoning in trained models remains debated.
2. Beyond Quadratic Attention: Can linear-time state-space models (e.g., Mamba; Gu & Dao, 2023) or recurrent architectures match the empirical scaling curves of exact softmax attention?

The Transformer is the dominant architecture for large-scale language modeling, though alternatives such as state-space models are an active area of research (Gu & Dao, 2023). Optimization improvements such as FlashAttention (Dao et al., 2022) reduce reads and writes to GPU high-bandwidth memory, computing exact attention in $O(N^2)$ time with $O(N)$ additional memory via hardware-aware tiling.

To mitigate the KV cache bottleneck during inference, some architectures employ Multi-Query Attention (MQA) (Shazeer, 2019) or Grouped-Query Attention (GQA) (Ainslie et al., 2023), which share key/value heads across multiple query heads; GQA interpolates between multi-head and multi-query attention, with reported quality close to multi-head attention and speed comparable to multi-query attention. Rotary Positional Embeddings (RoPE) (Su et al., 2021; journal version 2024) are widely used for relative position encoding; extending models beyond their trained context length typically requires further techniques such as positional interpolation (Chen et al., 2023) or YaRN (Peng et al., 2023).

The following are speculative outlooks, not forecasts with formal records (see docs/EVIDENCE_FRAMEWORK.md section 2 and FCST-REG-001).

• Near-Term (0–3 Years): Speculative: longer context windows through techniques such as ring attention (Liu et al., 2023) and sparse variants, and continued use of Mixture-of-Experts routing to decouple parameter count from active compute (see [TECH-MOE-001]).
• Medium-Term (3–10 Years): Speculative: hybrid architectures combining Transformer attention (for high-fidelity recall) with linear state-space models or recurrent networks (for unbounded context streams and constant-time inference).
• Long-Term (10+ Years): Highly speculative: replacement by other architectures is possible, but no specific successor has been established.

• Foundational Concepts:
  • TECH-ML-001 Machine Learning: General principles of empirical risk minimization.
  • TECH-DL-001 Deep Learning: Backpropagation and deep computational graphs.
• Related Methods & Technologies:
  • TECH-MOE-001 Mixture of Experts Models: Sparse expert routing used with Transformer blocks.
  • TECH-LLM-001 Large Language Models: The primary instantiation of the architecture.
  • CAP-GENAI-001 Generative Artificial Intelligence: Generative mechanics utilizing autoregressive transformers.
• Advanced Implications & Future Trajectories:
  • AGI-001 Artificial General Intelligence: Implications of scale and the limits of the transformer paradigm.

Backpropagation is an algorithm that computes the gradient of a scalar loss with respect to every parameter of a neural network in a single backward sweep, by applying the chain rule of calculus layer by layer from the output toward the input and reusing intermediate results. It is a special case of reverse-mode automatic differentiation. Backpropagation computes gradients only; the parameter update is performed by a separate optimization rule such as gradient descent.

Training a neural network (see [TECH-NN-001]) means minimizing a loss $\mathcal{L}(\theta)$ over a very large parameter vector $\theta$. First-order methods need $\nabla_\theta \mathcal{L}$ at every step. Computing each partial derivative separately, for example by perturbing one parameter at a time, would cost one extra forward evaluation per parameter. For a network with $P$ parameters, that is $P$ additional passes per step, which is impractical for modern values of $P$.

Backpropagation removes this factor. Because the loss is a scalar and the network is a composition of differentiable operations, a single forward pass followed by a single backward pass yields all $P$ partial derivatives, at a cost of the same order as the forward pass (Baydin et al., 2018). This property, not any particular network design, is what made gradient-based training of deep models routine. Nearly all systems discussed in this encyclopedia that are trained by gradient descent rely on it, including large language models (see [TECH-LLM-001]).

The distinction between backpropagation and the optimizer is a frequent source of confusion. Backpropagation answers "in which direction does each parameter change the loss?". The optimizer answers "how far and in what manner do we move?" (see [FOUND-010]).

The chain rule itself is classical. The history of its use for efficiently differentiating long compositions is a matter of documented priority disputes, and this article states only what the cited sources support.

Reverse-mode differentiation. Baydin et al. (2018) survey automatic differentiation (AD) and describe backpropagation as a special case of reverse-mode AD, noting that the machine learning and AD communities developed related ideas largely independently. Reverse-mode AD for general computational graphs was developed in the numerical analysis literature before its widespread use in neural networks (Baydin et al., 2018). [VERIFICATION NEEDED: the commonly cited attribution of the first reverse-mode formulation to Linnainmaa (1970) was not confirmed against a primary or publisher source for this draft.]

Application to neural networks. Werbos (1982; year [VERIFICATION NEEDED]: Crossref record carries no date) described the use of backward sensitivity calculations for nonlinear systems, and later accounts treat this as an early proposal to train multilayer networks this way. [VERIFICATION NEEDED: the exact content of Werbos's earlier 1974 thesis, which is sometimes cited for the neural network application, was not checked.] Rumelhart, Hinton and Williams (1986) demonstrated in Nature that multilayer networks trained by this procedure learn useful internal representations, and this paper is the usual point of reference for the connectionist revival described in [HIST-008]. LeCun et al. (1989) applied the algorithm to a convolutional network for handwritten digit recognition (see [TECH-CNN-001]).

Difficulties in depth and time. Bengio, Simard and Frasconi (1994) analyzed why gradient-based learning of long-term dependencies is difficult, identifying exponentially vanishing or growing gradient contributions. This motivated gated recurrent units such as the LSTM (Hochreiter and Schmidhuber, 1997; see [TECH-RNN-001]) and, later, residual connections (He et al., 2015).

Software. From the 2010s, deep learning frameworks implement reverse-mode AD generically, so that practitioners define the forward computation and obtain gradients automatically. Baydin et al. (2018) describe the "dynamic computational graph" and "differentiable programming" approaches in this setting. Schmidhuber (2015) provides a broader historical survey of the field.

#4.1 Setup

Consider a feedforward network with layers $\ell = 1, \dots, L$. Let $a^{0} = x$ be the input. For each layer,

and let the loss for a single example be $\mathcal{L} = \ell\big(a^{L}, y\big)$. The goal is $\partial \mathcal{L} / \partial W^{\ell}$ and $\partial \mathcal{L} / \partial b^{\ell}$ for every $\ell$. (Batches are handled by averaging; the derivation below is for one example.)

#4.2 The error signal

Define the error signal (sensitivity) of layer $\ell$ as the gradient of the loss with respect to the pre-activation:

By the chain rule, with $\odot$ denoting elementwise multiplication and $\phi'$ the elementwise derivative:

The second equation is the backward recursion: the error at layer $\ell$ is the error at layer $\ell+1$ propagated through the transposed weights and then gated by the local activation derivative. Once $\delta^{\ell}$ is known, the parameter gradients follow directly, since $z^{\ell}$ depends on $W^{\ell}$ and $b^{\ell}$ linearly:

#4.3 The algorithm

1. Forward pass. Compute and store $z^{\ell}$ and $a^{\ell}$ for all layers, then evaluate the loss.
2. Output error. Compute $\delta^{L}$.
3. Backward pass. For $\ell = L-1$ down to $1$, compute $\delta^{\ell}$ from $\delta^{\ell+1}$.
4. Gradients. Form $\partial\mathcal{L}/\partial W^{\ell}$ and $\partial\mathcal{L}/\partial b^{\ell}$ for all $\ell$.
5. Update (outside backpropagation). Apply an optimizer step, for example $\theta \leftarrow \theta - \eta \nabla_\theta \mathcal{L}$.

The stored forward quantities are needed in the backward pass, so memory grows with depth and batch size. The activation-memory cost is a principal constraint in large-model training (see [TECH-DIST-001]).

#4.4 Computational graph view and reverse-mode AD

Backpropagation generalizes to any computation expressed as a directed acyclic graph of differentiable operations. For each node $v$ with output $u_v$, define the adjoint $\bar u_v = \partial \mathcal{L} / \partial u_v$. If $u_v$ feeds into several downstream nodes $w \in \mathrm{succ}(v)$, then

Adjoints are computed in reverse topological order, starting from $\bar{\mathcal{L}} = 1$. Each operation only needs to supply a rule for multiplying an incoming adjoint by its local Jacobian (a vector-Jacobian product); the full Jacobian is never formed. This is reverse-mode AD (Baydin et al., 2018).

Why reverse rather than forward mode. Forward-mode AD propagates derivatives with respect to one input direction per sweep. For a function from $\mathbb{R}^{P}$ to a scalar, obtaining the full gradient takes $P$ forward-mode sweeps but one reverse-mode sweep. Since neural network losses are scalar functions of many parameters, reverse mode is the efficient choice. Forward mode is preferable when a function has few inputs and many outputs (Baydin et al., 2018).

#4.5 Gradient flow through depth and time

Unrolling the recursion shows that the error signal at layer $\ell$ contains a product of per-layer Jacobians,

where $D_k$ holds the activation derivatives of layer $k$ on its diagonal. If the norms of these factors are consistently below one, gradients shrink exponentially with depth (vanishing); if above one, they grow (exploding). The same product structure arises in recurrent networks across time steps, which is the case analyzed by Bengio et al. (1994) and, for clipping and regularization remedies, by Pascanu, Mikolov and Bengio (2012). Common mitigations include non-saturating activations, careful initialization, normalization (Ioffe and Szegedy, 2015), gated architectures, residual connections (He et al., 2015), and gradient clipping.

#4.6 Backpropagation through time

For a recurrent network unrolled over $T$ steps with shared weights, backpropagation applied to the unrolled graph is called backpropagation through time (BPTT). Gradients for shared weights are summed over steps. Because memory and compute scale with $T$, truncated BPTT limits the backward window. See [TECH-RNN-001].

#4.7 Checkpointing and the memory-compute trade

Activation storage can be reduced by saving only a subset of activations and recomputing the rest during the backward pass. This trades additional forward computation for lower memory. The technique is widely used in large-model training, and its details belong to [TECH-DIST-001].

• Reverse-mode AD (standard backpropagation). One backward sweep per scalar output; memory proportional to stored intermediates.
• Forward-mode AD. One sweep per input direction. Efficient for few inputs and used for Jacobian-vector products.
• Backpropagation through time and truncated variants. For recurrent models.
• Second-order information. Hessian-vector products can be computed with AD-based techniques and underlie some optimization methods (Baydin et al., 2018). Full second-order methods are rarely used at large scale.
• Local learning and backprop alternatives. Several proposals compute weight updates without a global backward pass, including forward-only and local-learning rules, motivated by hardware efficiency or biological plausibility. [VERIFICATION NEEDED: no specific alternative-algorithm papers were confirmed for this draft, so none are named.]
• Stochastic estimators for non-differentiable steps. Where a computation includes sampling, gradients require either the reparameterization trick (see [TECH-VAE-001]) or score-function estimators.

• Supervised training of classifiers and regressors. The algorithm in section 4.3 is the standard training loop of deep networks (LeCun, Bengio and Hinton, 2015).
• Convolutional networks. The convolution operation is linear, so its backward pass is also a convolution-like operation, and weight sharing means gradients are summed over positions (LeCun et al., 1989; see [TECH-CNN-001]).
• Sequence models. BPTT trains LSTM and GRU networks ([TECH-RNN-001]); attention-based networks are trained by backpropagating through the attention weights ([TECH-ATTN-001]).
• Generative models. Variational autoencoders rely on a differentiable sampling formulation so that gradients pass through a stochastic node ([TECH-VAE-001]); adversarial training backpropagates through a discriminator into a generator ([TECH-GAN-001]).
• Deep learning frameworks. Automatic differentiation libraries apply reverse-mode AD to arbitrary programs built from differentiable primitives (Baydin et al., 2018).

#Advantages

• Efficiency. One backward sweep produces all parameter gradients at a cost of the same order as the forward pass (Baydin et al., 2018).
• Exactness. Backpropagation yields the exact gradient of the computed function (up to floating-point error), unlike finite-difference estimates.
• Modularity. Any differentiable operation that supplies a vector-Jacobian product can be inserted into a network.

#Limitations

• Memory. Stored activations scale with depth and batch size.
• Gradient pathologies. Vanishing and exploding gradients (Bengio et al., 1994; Pascanu et al., 2012).
• Local information only. Gradients describe the loss locally. They do not by themselves guarantee convergence to a good solution on a non-convex loss.
• Sequential dependency. The backward pass cannot begin until the forward pass has finished, which constrains some forms of parallelism (see [TECH-DIST-001]).
• Differentiability requirement. Discrete or non-differentiable operations need relaxations or estimators.

#Unresolved questions

1. Biological plausibility. Whether the brain implements anything equivalent to backpropagation, given requirements such as symmetric forward and backward weights, is an open question in neuroscience and machine learning. [VERIFICATION NEEDED: no specific primary source on this debate was confirmed; the statement is a description of an open question, not a cited finding.]
2. Why gradient descent with backpropagation finds solutions that generalize. See [TECH-NN-001], section 7.
3. Efficient alternatives at scale. Whether training methods that avoid a global backward pass can match backpropagation-based training at frontier scale is undetermined.

The mathematics of backpropagation as exact reverse-mode differentiation is Established and is covered in standard texts and in the AD literature (Baydin et al., 2018). Its practical use is universal in gradient-trained deep networks. Open research concerns the surrounding system: how to hold activation memory in check, how to stabilize gradient flow in very deep or long-context models, and whether credit assignment can be done more cheaply or more locally. As of 2026-10-15, no alternative has been confirmed in the sources reviewed for this draft to replace backpropagation in frontier-scale training. This is a statement about the sources reviewed, not a survey of all current research.

This section describes plausible directions and does not register forecasts.

• Near term (0–3 years). Continued engineering of memory-saving backward passes, kernel fusion and recomputation strategies, tied closely to accelerator characteristics (see [TECH-HW-001]).
• Medium term (3–10 years). Research on local or forward-only learning rules may be driven by hardware energy constraints. Whether any such method becomes competitive at scale is an open empirical question.
• Long term (10+ years). Whether continual or lifelong learning systems (see [AGI-CL-001]) will continue to use backpropagation over stored data, or move to online credit-assignment mechanisms, is unresolved.

• Foundational Concepts:
  • TECH-NN-001 Artificial Neural Networks: the model class whose parameters are differentiated.
  • FOUND-010 Optimization and Gradient Descent: the update rules that consume backpropagated gradients.
  • TECH-ML-001 Machine Learning: the empirical risk minimization setting.
• Related Methods & Technologies:
  • TECH-DL-001 Deep Learning: the broader practice built on this algorithm.
  • TECH-RNN-001 Recurrent Networks and LSTM: backpropagation through time and gradient pathologies.
  • TECH-CNN-001 Convolutional Neural Networks: early application of backpropagation to a structured network.
  • TECH-VAE-001 Autoencoders and Variational Autoencoders: gradient estimation through stochastic nodes.
  • TECH-DIST-001 Distributed Training: memory and parallelism constraints of the backward pass.
  • TECH-HW-001 AI Hardware & Accelerator Systems: hardware efficiency of forward and backward passes.
• Advanced Implications & Future Trajectories:
  • HIST-008 Backpropagation and the Connectionist Revival: historical account.
  • TECH-ARCH-001 The Transformer Architecture: a model family trained by backpropagation at very large scale.
  • AGI-CL-001 Continual Learning and Persistent Memory: open problems in learning without forgetting.

A convolutional neural network (CNN) is a neural network in which at least some layers apply the same small learned filter at every position of a grid-structured input, such as an image, so that each output depends only on a local neighbourhood and the filter's weights are shared across positions. This design builds in the assumption that useful patterns are local and can appear anywhere in the input.

A fully connected layer applied to an image treats each pixel as an unrelated input, so a layer connecting a modest image to a modest hidden layer already needs a very large weight matrix, and it has no built-in notion that neighbouring pixels are related. Convolutional layers address both issues with two structural constraints: local connectivity (each unit sees a small window) and weight sharing (the same filter is reused at all positions). Together they cut the number of parameters sharply and make the layer's response translation equivariant: shifting the input shifts the output in the same way.

CNNs were the dominant architecture for computer vision from the early 2010s, and the 2012 ImageNet result of Krizhevsky, Sutskever and Hinton (2017) is commonly treated as the event that brought deep learning to wide attention in the field (see [HIST-005]). Since 2020, attention-based vision models (Dosovitskiy et al., 2020) have been competitive, and convolutional networks remain in wide use, often in hybrid designs. The CNN is also the clearest example of how a structural assumption (an inductive bias) can be built into a network (see [TECH-NN-001]).

Biological and early computational precursors. Hubel and Wiesel (1962) described cells in the cat visual cortex with localized receptive fields and a layered organization. Fukushima (1980) proposed the neocognitron, a multilayer network with alternating feature-extracting and position-tolerance stages, inspired by that work. It was trained with a self-organizing procedure rather than backpropagation.

Trained convolutional networks. LeCun et al. (1989) applied backpropagation (see [TECH-BACKPROP-001]) to a network with constrained, shared weights for handwritten zip code digit recognition. LeCun, Bottou, Bengio and Haffner (1998) described LeNet-style networks for document recognition and argued for training the whole recognition pipeline end to end with gradient-based learning.

Scale-up. Krizhevsky, Sutskever and Hinton (2017; the journal version of the 2012 NeurIPS paper) trained a deep convolutional network with five convolutional layers on the ImageNet data set using GPUs. Their abstract and results reported performance considerably better than the previous state of the art on the benchmark they used. He et al. (2015) introduced residual connections, which allowed much deeper convolutional networks to be trained.

Challenge from attention. Dosovitskiy et al. (2020) applied a standard transformer to sequences of image patches and reported that, when pre-trained on large amounts of data and transferred to recognition benchmarks, it attains excellent results compared with state-of-the-art convolutional networks while requiring substantially fewer computational resources to train. Liu et al. (2022) re-examined the convolutional design space, built a modernized pure-convolutional model (ConvNeXt), and reported that it competes favorably with transformers in accuracy and scalability.

#4.1 The convolution operation

For a single-channel two-dimensional input $X$ and a filter (kernel) $K$ of size $k \times k$, a convolutional layer computes, for each output position $(i,j)$,

Strictly, this formula is cross-correlation; true convolution flips the kernel. Deep learning libraries conventionally implement cross-correlation and still call it convolution, which is harmless because the kernel is learned.

With multiple channels, each filter spans all $C_{\text{in}}$ input channels, and a layer has $C_{\text{out}}$ such filters, producing $C_{\text{out}}$ output feature maps. The number of weights is $C_{\text{out}}\, C_{\text{in}}\, k^2$ (plus biases), independent of the image height and width.

#4.2 Stride, padding and receptive field

Stride $s$ moves the filter by $s$ positions between evaluations, reducing the output resolution. Padding adds border values so that the output size can match the input size. For input size $n$, kernel size $k$, padding $p$ and stride $s$ per dimension, the output size is

The receptive field of a unit is the region of the input that can influence it. It grows with each stacked layer, so deeper units integrate over larger regions. Stacking small filters is a common way to obtain a large receptive field with few parameters.

#4.3 Nonlinearity and pooling

A nonlinearity (commonly ReLU) follows each convolution. Pooling summarizes small windows, for example with their maximum or average, which reduces resolution and provides some tolerance to small shifts. Many modern networks replace pooling with strided convolutions.

#4.4 Weight sharing, equivariance and invariance

Because the same filter is used at every position, shifting the input produces a correspondingly shifted feature map (equivariance, ignoring boundary effects). Invariance to position, as needed for classification, comes from subsequent operations such as pooling or a final global aggregation. A convolutional network is not automatically invariant to rotation, scale or other transformations unless it is trained on varied examples or designed for them.

#4.5 A typical classification network

A standard design alternates convolution, nonlinearity and downsampling, with the number of channels increasing as spatial resolution decreases, and ends with a pooled feature vector fed to a linear classifier with softmax. Residual networks add skip connections of the form $y = x + F(x)$ around blocks of convolutions (He et al., 2015). The overall network is trained end to end by minimizing a classification loss with backpropagation, where the gradient for a shared filter is the sum of its gradients over all positions (LeCun et al., 1989).

#4.6 Other dimensions

The same construction applies to one-dimensional data (audio waveforms, text, time series) and three-dimensional data (volumes, video). In sequence modelling, convolutions provide an alternative to recurrence (see [TECH-RNN-001]).

• Classic stacked designs. LeNet-style networks (LeCun et al., 1998) and AlexNet-style networks (Krizhevsky et al., 2017) stack convolution, nonlinearity and pooling.
• Residual networks. Skip connections ease optimization of very deep networks (He et al., 2015).
• Fully convolutional networks. Networks with no dense layers produce spatial outputs, which suits dense prediction tasks such as segmentation.
• Dilated and depthwise-separable convolutions. Variants that enlarge the receptive field without extra parameters or factor the convolution to reduce cost. [VERIFICATION NEEDED: no primary source for these variants was checked for this draft.]
• Modernized convolutional networks. ConvNeXt (Liu et al., 2022) revisits convolutional design using practices associated with transformer training.
• Hybrid models. Designs that combine convolutional stages with attention (see [TECH-ATTN-001]).
• Beyond grids. Graph neural networks generalize local aggregation from grids to arbitrary graphs (see [TECH-GNN-001]).

• Handwritten digit and document recognition. An early deployed use was recognition of handwritten characters; LeCun et al. (1998) describe a document recognition system built from such networks.
• Image classification, detection and segmentation. CNN backbones are standard components in many image pipelines, and residual networks (He et al., 2015) were widely adopted as such backbones. [VERIFICATION NEEDED: no survey of current deployment shares was checked, so "widely adopted" is based on the cited paper's influence only.]
• Generative models. Convolutional generators and discriminators are used in adversarial image synthesis (see [TECH-GAN-001]) and in the encoders and decoders of variational autoencoders (see [TECH-VAE-001]).
• Audio and signal processing. One-dimensional convolutions are applied to waveforms and spectrograms. [VERIFICATION NEEDED: no source was confirmed; included as a general description.]
• Multimodal systems. Image encoders used inside multimodal models may be convolutional or transformer-based (see [TECH-MM-001]).

For current benchmark leaderboards and model-by-model numbers, see the Living Reference collection MODEL-001; they are intentionally not tabulated here.

#Advantages

• Parameter efficiency and data efficiency. Weight sharing and locality reduce the number of parameters relative to dense layers and encode a useful prior for images.
• Computational efficiency. Cost grows linearly with image area for fixed filter sizes, in contrast to the quadratic cost of global self-attention over all positions (see [TECH-ARCH-001]).
• Hardware fit. Convolutions map efficiently onto accelerators (see [TECH-HW-001]).

#Limitations

• Limited long-range interaction per layer. Information travels across the image only by stacking layers or downsampling.
• Limited built-in invariances. The architecture handles translation but not rotation, scale or viewpoint changes by design.
• Fixed grid assumption. The standard operation assumes regular, grid-structured data.
• Interpretability. Learned filters are inspectable in early layers, but the behaviour of deep stacks is difficult to explain (see [SAFE-INTERP-001]).

#Unresolved questions

1. Convolution versus attention. Whether convolutional inductive biases remain advantageous at very large data and compute scales, or are largely absorbed by sufficiently large attention models, is not settled. Dosovitskiy et al. (2020) and Liu et al. (2022) offer evidence pointing in different directions; differences in training recipes and data complicate comparison, and this article does not attempt to adjudicate.
2. Robustness. How reliably CNN predictions transfer under distribution shift is an ongoing research concern. [VERIFICATION NEEDED: no source on robustness evaluations was confirmed for this draft.]

As of 2026-10-15, convolutional and attention-based models both see substantial use in vision. The mathematical description of the convolution layer, weight sharing and equivariance is Established. The empirical claim that convolutional networks trained end to end outperformed earlier hand-engineered pipelines on large-scale image recognition is Well Supported by the ImageNet results (Krizhevsky et al., 2017) and the subsequent residual network work (He et al., 2015). The relative merits of convolutional and attention-based designs are Contested; head-to-head results depend on pre-training data, scale and training recipe (Dosovitskiy et al., 2020; Liu et al., 2022).

This section describes plausible directions and does not register forecasts.

• Near term (0–3 years). Continued use of convolutional components in efficiency-sensitive and embedded settings, and in hybrid designs.
• Medium term (3–10 years). The balance among convolutional, attention-based and state-space designs for vision and video depends on empirical comparisons at scale that are not yet resolved.
• Long term (10+ years). Whether strong structural priors such as convolution matter for general-purpose systems trained on very large multimodal datasets, or are replaced by learned structure, is an open question tied to the broader architecture debate in [TECH-ARCH-001].

• Foundational Concepts:
  • TECH-NN-001 Artificial Neural Networks: layers, activations and training.
  • TECH-BACKPROP-001 Backpropagation: gradient computation with shared weights.
• Related Methods & Technologies:
  • TECH-DL-001 Deep Learning: the wider practice in which CNNs rose to prominence.
  • TECH-RNN-001 Recurrent Networks and LSTM: the other early structured network family.
  • TECH-GAN-001 Generative Adversarial Networks: convolutional generators and discriminators.
  • TECH-VAE-001 Autoencoders and Variational Autoencoders: convolutional encoders and decoders.
  • TECH-GNN-001 Graph Neural Networks: generalizing local aggregation to graphs.
  • TECH-HW-001 AI Hardware & Accelerator Systems: GPUs and the scale-up of CNN training.
  • HIST-005 The Deep Learning Breakthrough: the 2012 ImageNet result in context.
• Advanced Implications & Future Trajectories:
  • TECH-ARCH-001 The Transformer Architecture: the Vision Transformer and the convolution-attention comparison.
  • TECH-MM-001 Multimodal Foundation Models: image encoders in multimodal systems.

Diffusion Models are a class of deep generative models that synthesize high-fidelity data by learning to reverse a gradual, stochastic noising process. Mathematically, they progressively corrupt training data with Gaussian noise through a forward Markov chain, and then train a neural network (typically a U-Net or Transformer) to estimate and subtract that noise—effectively learning the score (the gradient of the log-probability density) of the data distribution to iteratively denoise pure random noise into coherent images, audio, or continuous data structures (Ho, Jain, & Abbeel, 2020).

The introduction of diffusion models revolutionized the field of generative artificial intelligence, particularly in visual and continuous modalities. Before diffusion models, Generative Adversarial Networks (GANs) were the leading approach to image synthesis; Dhariwal and Nichol (2021) reported that diffusion models could surpass GANs on image-synthesis benchmarks. GANs generate in a single forward pass, but their adversarial minimax training is widely reported to be unstable and prone to mode collapse (failing to represent the full diversity of the dataset); see TECH-GAN-001.

The significance of the diffusion paradigm includes:

1. Training Stability & Diversity: Diffusion models optimize a simplified, reweighted variational lower bound (Ho et al., 2020), avoiding adversarial training dynamics.
2. Compositional Conditioning: Through cross-attention mechanisms and classifier-free guidance, diffusion models show improved adherence to compositional natural language prompts, with known failures on spatial relations and counting (see Section 7).
3. Continuous Domains: While autoregressive Transformers dominate discrete sequence modeling (text), diffusion models are widely used for continuous data such as images, video and protein backbones (see Section 6).

The theoretical groundwork for diffusion models was laid in non-equilibrium thermodynamics. In 2015, Sohl-Dickstein et al. introduced diffusion probabilistic models for machine learning, drawing on non-equilibrium thermodynamics (Sohl-Dickstein et al., 2015).

Separately, Yang Song and Stefano Ermon (2019) developed Score-Based Generative Modeling, which trained networks via score matching with Langevin dynamics. In 2020, Ho, Jain, and Abbeel introduced Denoising Diffusion Probabilistic Models (DDPM), establishing an explicit connection between diffusion models and denoising score matching and reporting an FID of 3.17 on CIFAR-10 (Ho et al., 2020). Song et al. (2021) later unified score-based models and diffusion models through stochastic differential equations.

In late 2021, OpenAI's GLIDE demonstrated text-conditioned diffusion and compared CLIP guidance with Classifier-Free Guidance (Ho & Salimans, 2022), which human evaluators preferred (Nichol et al., 2021); DALL-E 2 followed in 2022 (Ramesh et al., 2022). Running diffusion directly in high-resolution pixel space is computationally expensive (Rombach et al., 2022).

The critical architectural breakthrough occurred in 2022 with Latent Diffusion Models (LDM) (Rombach et al., 2022). By compressing images into a lower-dimensional latent space using an autoencoder and running the diffusion process within that space, LDMs reduced compute requirements relative to pixel-space diffusion (Rombach et al., 2022). The open-weight Stable Diffusion model is based on this approach.

#4.1 The Forward Process (Noising)

Given an uncorrupted data sample $x_0 \sim q(x)$, the forward process adds Gaussian noise in $T$ discrete timesteps according to a predefined variance schedule $\beta_1, \dots, \beta_T$:

Because the noise is Gaussian, any arbitrary timestep $t$ can be sampled directly from $x_0$ without simulating the intermediate steps, using $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$:

As $T \to \infty$, $x_T$ becomes isotropic Gaussian noise $\mathcal{N}(0, \mathbf{I})$.

#4.2 The Reverse Process (Denoising)

The goal is to learn the reverse distribution $p_\theta(x_{t-1} \mid x_t)$ to denoise the data. This is approximated by a neural network (often a U-Net) parameterized by $\theta$. Ho et al. (2020) reported that, instead of predicting the mean of the distribution, training the network $\epsilon_\theta(x_t, t)$ to predict the noise vector $\epsilon$ added to $x_0$ gave good sample quality; the resulting loss is a simplified, reweighted variational bound rather than the exact likelihood bound:

#4.3 Conditioning and Classifier-Free Guidance (CFG)

To generate an image from text, the network must be conditioned on a text embedding $c$ (typically extracted from a CLIP text encoder). The network becomes $\epsilon_\theta(x_t, t, c)$. Classifier-Free Guidance balances image fidelity and prompt adherence. During training, the conditioning vector $c$ is randomly dropped out (replaced with a null token $\emptyset$). During inference, the model predicts both the conditioned and unconditioned noise, and the update is extrapolated away from the unconditional prediction by a guidance scale $w$:

Higher values of $w$ force the image to adhere more strictly to the prompt, often at the cost of diversity. This $w$ equals $1 + w$ in the convention of Ho and Salimans (2022), whose sampler uses $(1+w)\epsilon_\theta(z_t, c) - w\,\epsilon_\theta(z_t)$.

1. Pixel-Space Diffusion: (e.g., the original DDPM; Ho et al., 2020). Operates directly on pixel values, which is computationally expensive at high resolution (Rombach et al., 2022).
2. Latent Diffusion: (e.g., Stable Diffusion). Operates in a compressed latent space. In Stable Diffusion v1, an autoencoder with downsampling factor 8 maps $512 \times 512$ images to $64 \times 64$ latents, where diffusion occurs, and a decoder reconstructs the final image (CompVis, 2022; Rombach et al., 2022).
3. Transformer-Based Diffusion Backbones: DiT replaces the convolutional U-Net backbone with a Transformer operating on latent patches (Peebles & Xie, 2023). Stable Diffusion 3 uses a transformer-based backbone (Esser et al., 2024), and OpenAI's Sora report describes a transformer operating on spacetime patches of latent codes (OpenAI, 2024).

• Text-to-Image Generation: Stable Diffusion (Rombach et al., 2022), GLIDE, DALL-E 2 and Imagen (Saharia et al., 2022) are diffusion-based text-to-image systems with published descriptions; the architectures of some commercial systems (for example Midjourney) are undisclosed.
• Video Generation: OpenAI's technical report (OpenAI, 2024) states that Sora is a text-conditional diffusion model with a transformer operating on spacetime patches of video and image latent codes; this is the developer's own description, and the report states that model and implementation details are not included.
• Biomolecular Design: RFdiffusion generates protein backbones by retraining the RoseTTAFold network on denoising tasks, with experimental validation reported for several design tasks (Watson et al., 2023).

#Core Advantages

• Distributional Coverage: Diffusion models are widely described as less prone to mode collapse than GANs [VERIFICATION NEEDED: a source comparing sample diversity or recall of diffusion models and GANs].
• Compositionality: Can combine disjoint concepts (e.g., "an astronaut riding a horse in the style of Van Gogh") in conditioned generation, although failures occur (see below).

#Fundamental Limitations

• Inference Latency: GANs generate in a single forward pass, whereas autoregressive models require one pass per token. Diffusion models require iterative denoising, from about 1000 steps in the original DDPM setup (Ho et al., 2020) to tens of steps with accelerated samplers, making inference computationally intensive.
• Compositional Semantic Failures: Despite improvements, diffusion models can struggle with precise spatial relations ("a red cube under a blue sphere"), counting, exact text rendering and complex anatomy. [VERIFICATION NEEDED: a systematic source for spatial-relation, counting and text-rendering failures of diffusion models]
• Memorization and Provenance: Diffusion models can memorize and regenerate individual training images; Carlini et al. (2023) extracted over a thousand training examples from state-of-the-art models, so training-data provenance matters for privacy and copyright. The legal status of training on copyrighted data is not assessed in this article.

#Unresolved Scientific Questions

1. Consistency vs. Iteration: Can distillation methods (like Consistency Models; Song et al., 2023) reduce inference to one or a few steps without losing sample quality relative to many-step diffusion?
2. Unifying Discrete and Continuous: Can diffusion processes be natively adapted to discrete token sequences (natural language) to effectively compete with autoregressive LLMs?

Diffusion is a widely used paradigm for image, video and other continuous-data generation. Some recent systems use Transformer backbones in place of convolutional U-Nets; Peebles and Xie (2023) report that sample quality (FID) improves consistently as the forward-pass compute (Gflops) of DiT models increases. Inference-acceleration techniques, such as Consistency Models (Song et al., 2023) and Latent Consistency Models (Luo et al., 2023), aim to reduce the number of sampling steps.

The following are speculative outlooks, not forecasts with formal records (see docs/EVIDENCE_FRAMEWORK.md section 2 and FCST-REG-001).

• Near-Term (0–3 Years): Speculative: faster sampling through distillation methods, and wider use of diffusion in video and 3D generation.
• Medium-Term (3–10 Years): Speculative: video generative models may be explored as simulators for robot planning; this is an open research question.
• Long-Term (10+ Years): Highly speculative: autoregressive and diffusion methods may converge into a unified paradigm; no evidence establishes this outcome.

• Foundational Concepts:
  • TECH-ML-001 Machine Learning: General principles of loss objectives.
• Related Methods & Technologies:
  • CAP-GENAI-001 Generative Artificial Intelligence: The overarching category of generative systems.
  • TECH-GAN-001 Generative Adversarial Networks: The adversarial alternative to diffusion.
  • TECH-DL-001 Deep Learning: Optimization and representations underlying the U-Net and Transformer backbones.
  • TECH-ARCH-001 The Transformer Architecture: The architecture displacing U-Nets in modern DiT models.

Deep Learning (DL) is a subfield of machine learning based on artificial neural networks with multiple computational processing layers. These architectures apply parameterized differentiable non-linear transformations to raw inputs, automatically learning hierarchical representations that range from low-level sensory features to abstract, invariant semantic concepts (LeCun, Bengio, & Hinton, 2015).

Prior to the rise of deep learning, machine learning pipelines relied heavily on manual feature engineering—domain experts had to design handcrafted mathematical feature extractors (such as SIFT for computer vision or MFCCs for speech processing) before applying a shallow classifier. Deep learning superseded this paradigm by introducing end-to-end representation learning, wherein the internal representations and the terminal task predictions are optimized simultaneously via continuous gradient descent.

Its structural significance includes:

1. Hierarchical Abstraction: Successive layers compose simpler features into increasingly complex geometric manifolds, enabling direct processing of raw audio waveforms, pixels, genomic strings, and text tokens.
2. Computational Differentiability: Formulates entire system graphs as differentiable programs, enabling global optimization via the chain rule of calculus.
3. Empirical Power-Law Scaling: For language models, test loss has been reported to improve approximately as a power law in parameters, data and compute over the ranges studied (Kaplan et al., 2020).

The mathematical foundations of artificial neural networks began with Warren McCulloch and Walter Pitts's (1943) formalization of binary threshold neurons (McCulloch & Pitts, 1943) and Frank Rosenblatt's (1958) Perceptron (Rosenblatt, 1958). In 1969, Marvin Minsky and Seymour Papert published Perceptrons, proving that single-layer perceptrons cannot represent non-linearly separable functions such as exclusive-or ($\text{XOR}$) (Minsky & Papert, 1969); the book is often credited with contributing to reduced interest in connectionism in the 1970s, though the causal weight of the book relative to other factors is debated.

Although multi-layer networks could theoretically resolve non-linear tasks, training them remained intractable until the popularization of Backpropagation by David Rumelhart, Geoffrey Hinton, and Ronald Williams in 1986 (Rumelhart et al., 1986), building on earlier reverse-mode differentiation work (Linnainmaa, 1976; Werbos, 1982). In 1989, Yann LeCun and colleagues applied backpropagation to weight-shared convolutional networks for handwritten zip-code recognition (LeCun et al., 1989). In the same year Cybenko (1989) and Hornik, Stinchcombe, and White (1989) independently showed universal approximation results: feedforward networks with a single hidden layer of non-linear units can approximate any continuous function on compact subsets of $\mathbb{R}^n$ (Cybenko for sigmoidal units; Hornik et al. for a broader class of activation functions).

Through the 1990s and early 2000s, deep networks were difficult to train, with vanishing and exploding gradients, limited data and limited compute commonly cited as obstacles, and kernel methods such as SVMs were widely used.

The turning point occurred in 2012 with AlexNet (Krizhevsky, Sutskever, & Hinton, 2012), which leveraged graphics processing units (NVIDIA GPUs), the ImageNet dataset (Deng et al., 2009), non-saturating rectified linear activation units ($\text{ReLU}$), and dropout regularization to win the ILSVRC-2012 visual recognition competition. This breakthrough initiated rapid architectural innovations: Residual Networks (ResNet; He et al., 2016), Generative Adversarial Networks (GANs; Goodfellow et al., 2014), and the Transformer architecture (Vaswani et al., 2017).

#4.1 Computational Graph & Forward Pass

A deep neural network parameterized by weights and biases $\theta = \{W_l, b_l\}_{l=1}^L$ computes a composite function:

At layer $l$, the pre-activation vector $z_l$ and activation vector $a_l$ are computed as:

where $a_0 = x$ (the input tensor), $W_l \in \mathbb{R}^{d_l \times d_{l-1}}$ is the weight matrix, $b_l \in \mathbb{R}^{d_l}$ is the bias vector, and $\sigma(\cdot)$ is an element-wise non-linear activation function (such as $\text{ReLU}(z) = \max(0, z)$, $\text{GELU}(z) = z \Phi(z)$, or $\text{SiLU}(z) = z \cdot \mathrm{sigmoid}(z)$).

#4.2 Optimization via Backpropagation

Given a scalar loss function $\mathcal{L}(a_L, y)$, gradients are propagated backward through the network using the multivariate chain rule. The error vector for layer $l$ is defined as $\delta_l = \frac{\partial \mathcal{L}}{\partial z_l}$:

The parameter gradients are then computed directly from the local activations and error vectors:

#4.3 Gradient Descent Optimizers

Parameters are updated iteratively using Stochastic Gradient Descent (SGD) with momentum, or adaptive learning rate algorithms such as AdamW (Loshchilov & Hutter, 2019):

where $\eta$ is the learning rate, $\beta_1, \beta_2$ are moving average coefficients, and $\lambda$ represents the decoupled weight decay factor.

• Autonomous Vision: Convolutional and Transformer-based networks are used for perception in driver-assistance and autonomous-driving research and products; the architectures of deployed commercial systems are generally not fully disclosed.
• Foundation Language Models: Autoregressive Transformers perform text generation, few-shot task adaptation and dialogue; GPT-3 had 175 billion parameters (Brown et al., 2020, as reported in 2020; see MODEL-001 for current figures).
• Biomedical Sciences: Attention-based architectures (AlphaFold 2; Jumper et al., 2021) predict protein 3-D structure from sequence.
• Acoustic Speech Synthesis: Neural generative models of raw audio, such as WaveNet (van den Oord et al., 2016), synthesize speech waveforms.

#Core Advantages

• Automatic Feature Learning: Substantially reduces the need for handcrafted feature engineering in high-dimensional perceptual data, although feature design persists in tabular and scientific settings.
• High Capacity & Scalability: Can be trained at very large parameter counts and data volumes; reported loss improvements follow power laws with diminishing returns per unit of scale (Kaplan et al., 2020).
• Differentiable Composability: Facilitates modular combinations of vision, text, and audio encoders into unified end-to-end multimodal systems.

#Core Limitations

• Compute and Energy Footprint: Training large models typically requires large clusters of specialized accelerators (GPUs/TPUs); exact hardware and energy figures for recent frontier training runs are mostly undisclosed.
• Opacity & Mechanistic Uninterpretability: Highly distributed representation makes extracting exact causal explanations for individual outputs difficult (the "black box" challenge).
• Adversarial Fragility: Susceptible to human-imperceptible adversarial perturbations that cause drastic classification failures (Szegedy et al., 2013).
• Data Inefficiency: Requires vast datasets compared to biological learning systems.

#Unresolved Research Questions

1. Mechanistic Interpretability: Can internal circuits of deep networks be reverse-engineered into human-auditable computational algorithms (Olah et al., 2020; Elhage et al., 2021)?
2. Non-Convex Optimization: Under what conditions does gradient descent find generalizable solutions in high-dimensional non-convex loss landscapes?
3. Out-of-Distribution Generalization: How can neural representations be made robust against distribution shifts that violate the i.i.d. assumption?

Deep learning is the reigning empirical paradigm across natural language, computer vision, and multimodal AI. Contemporary focus centers on Transformer-based foundation models and Diffusion architectures.

Large-scale training commonly combines tensor (intra-layer) parallelism (Shoeybi et al., 2019), pipeline parallelism (Huang et al., 2019; Narayanan et al., 2021), activation checkpointing (Chen et al., 2016), and reduced-precision training (Micikevicius et al., 2018, for FP16; Micikevicius et al., 2022, for FP8 formats). Post-training methods include Reinforcement Learning from Human Feedback (RLHF), introduced for reinforcement-learning tasks by Christiano et al. (2017) and applied to language models by Ouyang et al. (2022), and Direct Preference Optimization (DPO; Rafailov et al., 2023), which are used to make pretrained next-token predictors follow instructions.

The following are speculative outlooks, not forecasts with formal records (see docs/EVIDENCE_FRAMEWORK.md section 2 and FCST-REG-001).

• Near-Term (0–3 Years): Speculative: wider deployment of sparse Mixture-of-Experts (MoE) architectures to reduce inference FLOPs; advancements in post-training quantization (4-bit and 2-bit weight representations).
• Medium-Term (3–10 Years): Speculative: possible shift toward energy-efficient neuromorphic hardware and continuous world models that integrate temporal physical dynamics for embodied robotic autonomy.
• Long-Term (10+ Years): Highly speculative: architectures that integrate symbolic reasoning with neural networks have been proposed, but whether they would enable verified reasoning or safety proofs is an open question.

• Foundational Concepts:
  • FOUND-001 Artificial Intelligence: Parent discipline.
  • TECH-ML-001 Machine Learning: Empirical learning principles and risk minimization.
• Related Methods & Technologies:
  • TECH-ARCH-001 The Transformer Architecture: The dominant attention-based neural architecture.
  • CAP-GENAI-001 Generative Artificial Intelligence: Generative modeling paradigms built on deep neural networks.
• Advanced Implications & Future Trajectories:
  • AGI-001 Artificial General Intelligence: Scaling pathways toward general capability.

A generative adversarial network (GAN) is a framework for training a generative model in which two neural networks are trained against each other: a generator that maps random noise to synthetic samples, and a discriminator that tries to tell synthetic samples from real training data. The generator is trained to make the discriminator err, so the generator's distribution is pushed toward the data distribution without an explicit likelihood being computed (Goodfellow et al., 2014).

Generative modelling aims to learn a distribution $p_{\text{data}}$ well enough to draw new samples from it. Many earlier approaches required an explicit, tractable density or approximate inference procedures (for example Markov chain sampling). Goodfellow et al. (2014) proposed an alternative: define a sampler directly as a network and train it using a second network's judgement as the learning signal. The paper notes that, when both networks are multilayer perceptrons, the system can be trained with backpropagation and needs no Markov chains or unrolled approximate inference networks (see [TECH-BACKPROP-001]).

GANs were the leading approach to high-fidelity image synthesis for several years and introduced an influential idea, learning from a trained critic rather than from a fixed loss. They have since been displaced for many large-scale image applications by diffusion models, whose authors report state-of-the-art synthesis results (Rombach et al., 2021; see [TECH-DIFF-001]), though GANs remain relevant for speed at generation time and for specific tasks. Their training is also a standard example of the difficulties of optimizing a game rather than a single loss.

Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville and Bengio (2014) introduced the framework. Radford, Metz and Chintala (2015) proposed a class of convolutional GANs (DCGANs) with architectural constraints that made training more reliable on image datasets, and reported evidence that the networks learn a hierarchy of representations (see [TECH-CNN-001]).

Training stability then became a main research theme. Arjovsky, Chintala and Bottou (2017) proposed the Wasserstein GAN (WGAN), whose authors report improved learning stability and meaningful learning curves, and offer a theoretical account in terms of distances between distributions. Gulrajani et al. (2017) reported that WGAN's weight clipping could lead to undesired behaviour and proposed a gradient penalty on the critic as an alternative. Karras, Laine and Aila (2018) proposed a style-based generator architecture that, as reported in the abstract, yields automatic separation of high-level attributes from stochastic variation and improved distribution-quality metrics on the datasets studied (StyleGAN).

From about 2020, diffusion models became prominent for image generation (Rombach et al., 2021), and latent-variable autoencoders are components of such systems (see [TECH-VAE-001]).

#4.1 The minimax game

Let $z \sim p_z$ be noise, $G_\theta(z)$ the generator and $D_\phi(x) \in (0,1)$ the discriminator's estimate that $x$ came from the training data. The original objective is the two-player game (Goodfellow et al., 2014)

The discriminator is trained to maximize $V$ (classify real versus fake correctly), and the generator to minimize it.

#4.2 Optimal discriminator and the divergence interpretation

For a fixed generator with sample distribution $p_g$, the optimal discriminator is

Substituting $D^*$ into $V$ yields

where JSD is the Jensen–Shannon divergence. The global minimum is reached when $p_g = p_{\text{data}}$, at which point $D^*(x) = 1/2$ everywhere. This result holds in the space of arbitrary functions $G$ and $D$ (Goodfellow et al., 2014), which is an idealization: real networks have finite capacity, and training is by stochastic gradient steps, so the guarantee does not carry over directly to practice.

#4.3 Training in practice

Training alternates gradient steps on $D$ and $G$ using minibatches. Early in training, when $D$ easily rejects generated samples, the term $\log(1 - D(G(z)))$ saturates and gives the generator weak gradients. The original paper suggests having the generator instead maximize $\log D(G(z))$, the non-saturating loss, which has the same fixed point and stronger early gradients.

#4.4 Wasserstein formulation

WGAN replaces the discriminator with a critic that estimates a Wasserstein distance between distributions, with a Lipschitz constraint on the critic (Arjovsky et al., 2017). The original method enforced the constraint by clipping weights. Gulrajani et al. (2017) instead penalized the norm of the critic's gradient with respect to its input and reported stable training across a wide variety of architectures.

#4.5 Conditional and style-based generation

A conditional GAN feeds a label or other conditioning signal to both networks, so that generation can be directed. Style-based generators inject a learned intermediate latent code at multiple layers of the generator and support scale-specific control of the synthesis (Karras et al., 2018). [VERIFICATION NEEDED: no primary source for conditional GANs was confirmed for this draft; the description is general.]

• Image synthesis. Face and object image generation is the best documented use (Radford et al., 2015; Karras et al., 2018).
• Representation learning. DCGAN authors reported that the discriminator's learned features were useful for other tasks (Radford et al., 2015).
• Data augmentation, image-to-image translation, super-resolution. These are commonly described applications. [VERIFICATION NEEDED: no primary sources for these applications were confirmed for this draft.]
• Synthetic media misuse. Realistic synthetic images and video raise concerns about deception and provenance; that topic belongs to the safety and governance volumes and is not analysed here (see [SAFE-IP-001] for copyright aspects and [GOV-LAW-001] for governance). [VERIFICATION NEEDED: no empirical source on prevalence of misuse was confirmed for this draft.]

Performance comparisons, model sizes and benchmark scores are omitted because they change quickly; see MODEL-001.

#Advantages

• Sharp samples in the studied image settings. Authors report high-quality image generation for GAN variants (Karras et al., 2018).
• Fast sampling. Generation requires one forward pass of the generator, in contrast to the many sequential evaluations typical of diffusion models (compare the cost remarks in Rombach et al., 2021).
• No explicit likelihood or inference network required (Goodfellow et al., 2014).

#Limitations

• Training instability. Gulrajani et al. (2017) state that GANs "suffer from training instability", and WGAN was proposed to improve stability (Arjovsky et al., 2017).
• Mode collapse. The generator may cover only part of the data distribution. Arjovsky et al. (2017) claim their approach removes problems such as mode collapse, but the extent to which this holds in all settings is not established by the sources reviewed here.
• No likelihood. A GAN does not provide a density estimate for evaluating samples or comparing models by likelihood.
• Evaluation difficulty. Judging sample quality and diversity requires auxiliary metrics, each with its own weaknesses. [VERIFICATION NEEDED: no source on evaluation metrics was confirmed.]

#Unresolved questions

1. Convergence of the game. Whether simultaneous gradient training of $G$ and $D$ converges in practice, and under what conditions, remains a research question; the idealized theory does not cover finite-capacity networks (see section 4.2).
2. GANs versus diffusion models. Which family is better depends on the criterion: sample quality, diversity, sampling speed, training stability and compute. Rombach et al. (2021) report state-of-the-art results for diffusion models, but a comprehensive controlled comparison was not confirmed for this draft.

As of 2026-10-15, GANs are a mature family whose core theory is understood in the idealized setting (Established, Goodfellow et al., 2014) and whose practical training is supported by a body of stabilization methods (Well Supported: Arjovsky et al., 2017; Gulrajani et al., 2017). For large-scale text-conditioned image generation, diffusion-based systems have become prominent (Rombach et al., 2021). The question of how GANs and other generative families compare is partly answered by benchmarks that change quickly (see MODEL-001).

This section describes plausible directions and does not register forecasts.

• Near term (0–3 years). Adversarial training is expected by some researchers to persist mainly as a component or fine-tuning signal within other generators rather than as a stand-alone approach; this is an observation about research direction, not a documented result. [VERIFICATION NEEDED: no source confirmed for this trend.]
• Medium term (3–10 years). The relative position of adversarial training in a landscape of diffusion, autoregressive and hybrid generators depends on sampling cost and quality comparisons that are not settled.
• Long term (10+ years). The adversarial idea of learning from a trained critic also appears outside image synthesis, for example in alignment techniques that use a learned judge (see [TECH-ALIGN-001]); this is an analogy and not an equivalence.

• Foundational Concepts:
  • TECH-NN-001 Artificial Neural Networks: the networks used as generator and discriminator.
  • TECH-BACKPROP-001 Backpropagation: gradients through the discriminator into the generator.
• Related Methods & Technologies:
  • TECH-VAE-001 Autoencoders and Variational Autoencoders: the likelihood-based latent-variable alternative.
  • TECH-DIFF-001 Diffusion Models: the family that has displaced GANs in many image applications.
  • TECH-CNN-001 Convolutional Neural Networks: the backbone of DCGAN-style generators.
  • TECH-ALIGN-001 Pretraining, Fine-Tuning and RLHF: learned-judge training in language models.
• Advanced Implications & Future Trajectories:
  • CAP-GENAI-001 Generative Artificial Intelligence: the capability landscape of generative systems.
  • SAFE-IP-001 Copyright and Intellectual Property: legal questions raised by generated media.

A graph neural network (GNN) is a neural network whose input is a graph (a set of nodes, optionally with edges, node features and edge features) and whose computation respects the graph's structure, so that its outputs do not depend on an arbitrary ordering of the nodes. Most GNNs build node representations by repeatedly aggregating information from each node's neighbours, an operation usually called message passing.

Many data sets are relational rather than grid-like or sequential: molecules are atoms joined by bonds, social and citation networks are people or documents joined by links, and meshes used in physical simulation are points joined by edges. Convolutional networks assume a regular grid and recurrent networks assume an ordered sequence; neither fits irregular, variable-size relational data directly. GNNs supply the corresponding architecture family, and are one of the principal examples of building a structural symmetry (here, permutation of nodes) into a model as an inductive bias (Battaglia et al., 2018; Bronstein et al., 2021).

Within the encyclopedia, GNNs matter for three reasons. They are the standard neural approach to learning on molecules and other structured scientific data (see APP-SCI-001 AI in Science). They provide a common formal language in which convolutional, recurrent and attention-based layers can be described as special cases of operations on graphs (Bronstein et al., 2021). And they are a testbed for questions about expressive power that are hard to pose for other architectures.

Scarselli et al. (2009) introduced a "graph neural network model" in which each node's state is computed by iterating a contraction-style update over its neighbours until a fixed point is reached, and then fed to an output function. Earlier work on recursive neural networks over trees and directed acyclic graphs is the stated lineage of that model.

The modern wave began in 2016–2017. Kipf and Welling (2017) proposed the graph convolutional network (GCN), a simple layer derived as a first-order approximation of spectral graph convolutions, for semi-supervised node classification. Hamilton et al. (2017) introduced GraphSAGE, which learns aggregation functions over sampled neighbourhoods and can therefore produce embeddings for nodes not seen during training (the "inductive" setting). Gilmer et al. (2017) unified several existing models under a message-passing neural network (MPNN) framework and applied it to molecular property prediction. Veličković et al. (2018) added learned attention weights over neighbours (graph attention networks, GAT). Battaglia et al. (2018), a 27-author position paper, generalised these into a "graph network" block that also updates edge and global attributes.

From 2018 the theoretical analysis of expressive power (Xu et al., 2019) and of failure modes such as over-smoothing (Li et al., 2018) and over-squashing (Alon & Yahav, 2021) became a distinct research thread. Bronstein et al. (2021) placed GNNs inside a broader "geometric deep learning" programme organised around symmetry groups.

#4.1 Graph data and permutation symmetry

A graph is $G=(V,E)$ with node features $x_v$ and optionally edge features $e_{uv}$. Let $A$ be the adjacency matrix. A function $f$ on node features is permutation equivariant if relabelling the nodes by a permutation matrix $P$ permutes the outputs in the same way, $f(PX, PAP^{\top}) = P f(X,A)$. A graph-level output must be permutation invariant. Message-passing layers satisfy these properties by construction because they combine neighbours with symmetric (order-independent) operations.

#4.2 Message passing

In the framework of Gilmer et al. (2017), layer $k$ computes, for each node $v$ with neighbourhood $\mathcal{N}(v)$:

where $\bigoplus$ is a permutation-invariant aggregator (sum, mean or max) and $\psi,\phi$ are learned functions, typically small neural networks. After $K$ layers, $h_v^{(K)}$ depends on the $K$-hop neighbourhood of $v$. For graph-level prediction a readout such as $h_G = \rho\big(\{h_v^{(K)}: v\in V\}\big)$ is applied with another invariant pooling function. Parameters are shared across nodes, so the same model applies to graphs of any size.

#4.3 Representative layers

The GCN layer of Kipf and Welling (2017) is

with $\tilde D$ the degree matrix of $\tilde A$ and $W^{(l)}$ a learned weight matrix; its cost is linear in the number of edges. GraphSAGE replaces the fixed normalisation with a learned aggregator over a sampled neighbour subset, which bounds the per-node cost (Hamilton et al., 2017). GAT computes, for each edge, a normalised attention coefficient $\alpha_{vu}=\mathrm{softmax}_u\, e(h_v,h_u)$ and uses it to weight neighbour messages (Veličković et al., 2018).

#4.4 Expressive power

Xu et al. (2019) show that standard message-passing GNNs are at most as discriminative as the one-dimensional Weisfeiler–Leman (1-WL) graph isomorphism test, and that, with injective update and aggregation (the Graph Isomorphism Network, GIN, using sum aggregation), they can reach that bound. Consequently, some non-isomorphic graphs cannot be distinguished by these models regardless of training. This is a mathematical result about a model class, not an empirical tendency.

#4.5 Depth-related failure modes

Li et al. (2018) analyse the GCN as a form of Laplacian smoothing and argue that stacking many layers makes node features of a connected component converge toward indistinguishable values (over-smoothing). Alon and Yahav (2021) identify over-squashing: when information from an exponentially growing receptive field must pass through fixed-size vectors along a few paths, long-range dependencies are compressed and lost. Both are active areas of mitigation research (rewiring, residual connections, global nodes), and neither is fully resolved.

• By task level: node classification, link prediction, and graph classification or regression.
• By layer family: convolutional (GCN-style, fixed neighbour weights), attentional (GAT, learned neighbour weights), and general message-passing (MPNN, with edge-conditioned messages). Battaglia et al. (2018) give the most general "graph network" block, which also carries edge and global state.
• By setting: transductive (all nodes available at training, as in the GCN experiments) versus inductive (new nodes or graphs at test time, as in GraphSAGE).
• By scale strategy: full-graph training versus neighbour sampling (GraphSAGE) or mini-batching on subgraphs for graphs with billions of edges.
• Graph transformers: models that apply self-attention over all nodes, with graph structure injected through positional or structural encodings. Bronstein et al. (2021) discuss self-attention in the transformer as a GAT-style operation on a fully connected graph, which makes the relation to TECH-ARCH-001 Transformer Architecture precise.
• Geometric GNNs: layers that are also equivariant to rotations and translations of 3-D coordinates, used for atoms and meshes; they are one instance of the symmetry-based programme described by Bronstein et al. (2021).

Deployed. PinSage, a GraphSAGE-style model, was built and deployed at Pinterest on a graph the authors report as roughly 3 billion nodes and 18 billion edges, to generate item embeddings for recommendation (Ying et al., 2018); the figures are as reported in 2018 and describe that system only.

Published research results. Gilmer et al. (2017) applied MPNNs to molecular property prediction, the quantum-chemistry setting that motivated the framework. Lam et al. (2023) describe GraphCast, which the authors report predicts hundreds of weather variables over 10 days at 0.25-degree global resolution in under one minute and, in their evaluation, outperforms the most accurate operational deterministic system on 90% of 1,380 verification targets; the architecture is graph-based, operating on a mesh representation of the globe. These results are single-group reports (published in Science), and operational use of machine-learned forecasts is a separate matter from the benchmark comparison.

Benchmarks. Citation-network node classification (Kipf & Welling, 2017) remains a common teaching example but is small by current standards.

Other widely cited uses include traffic, knowledge-graph completion and combinatorial optimisation; these are not individually sourced here.

Advantages. Parameter sharing across nodes and permutation symmetry give sample-efficient handling of relational data of variable size. Per-layer cost for sparse graphs is linear in edges (Kipf & Welling, 2017).

Limitations. (1) Expressive power of standard message passing is bounded by 1-WL (Xu et al., 2019). (2) Over-smoothing and over-squashing limit depth and long-range information flow (Li et al., 2018; Alon & Yahav, 2021). (3) Neighbourhood size grows exponentially with depth, making full-batch training on very large graphs costly and motivating sampling (Hamilton et al., 2017). (4) Results depend heavily on how the graph is constructed; when the graph is itself an approximation (for instance, a mesh for a physical field), construction choices matter as much as the layer.

Unresolved. Whether richer message passing, higher-order methods or graph transformers offer the best accuracy-cost trade-off, and how GNNs compare with generic transformers given large data and compute, are open empirical questions; the literature does not show a single dominant design.

As of 2026-10-15, message passing, GCN/GAT/GraphSAGE-style layers and the 1-WL expressiveness bound are standard material in the literature (Well Supported to Established, depending on the specific claim; see section 12). Large-scale deployment is documented for recommendation (Ying et al., 2018) and published for weather forecasting and molecular modelling (Lam et al., 2023; Gilmer et al., 2017). This article does not tabulate current benchmark leaderboards for graph tasks; those change frequently and are not tracked in a stable article. See MODEL-001 for model records where applicable.

Observed trends. Work has moved from small citation graphs toward scientific domains with physical symmetries and toward hybrids with attention. The theoretical programme of Bronstein et al. (2021) frames this as design of architectures around the symmetry of the problem.

Near term (0–3 years). Continued use in scientific modelling and recommendation is the extrapolation of the existing record; no quantitative forecast is registered here.

Medium and long term. Whether graph-specific layers remain a distinct architecture family or are absorbed into general attention-based models is uncertain. This article asserts no forecast; accordingly no FCST- record is attached.

• Foundational Concepts:
  • TECH-NN-001 Artificial Neural Networks: Layers, parameters and training that GNNs build on.
  • TECH-ML-001 Machine Learning: Supervised, semi-supervised and inductive learning settings.
• Related Methods & Technologies:
  • TECH-CNN-001 Convolutional Neural Networks: The grid-structured special case of local aggregation.
  • TECH-RNN-001 Recurrent Networks and LSTM: The sequence-structured counterpart.
  • TECH-ARCH-001 Transformer Architecture: Self-attention as aggregation over a fully connected graph.
  • TECH-DL-001 Deep Learning: General training and optimisation context.
• Applications:
  • APP-SCI-001 AI in Science: Molecules, materials and weather as graph problems.

A Mixture-of-Experts (MoE) model is a neural network in which part of the computation is split among several parallel sub-networks ("experts"), and a learned routing function (the "gate" or "router") selects which experts process each input. In the sparse variant used in modern large language models, only a small subset of experts is active per token, so the total parameter count can grow much faster than the compute spent per token.

In a dense network every parameter participates in every forward pass, so parameters and per-token compute rise together. Sparse MoE decouples the two: capacity (total parameters) and cost (active parameters) become separate quantities. This matters because empirical scaling studies relate model quality to both parameters and training compute (see [TECH-SCALE-001]), and MoE offers a way to add parameters without a proportional increase in floating-point operations per token.

The decoupling has costs. All experts must be stored in accelerator memory even though few are used per token; routing introduces load-balancing and stability problems; and distributed training requires heavy all-to-all communication. MoE is therefore best understood as a trade of memory and engineering complexity for lower compute per token, not as a free increase in capability.

Among public frontier-model disclosures, several developers state that their models use MoE layers. Others do not describe their architecture. This article restricts claims about specific systems to what the developers have published and marks the remainder as undisclosed (Section 8).

The idea predates deep learning. Jacobs, Jordan, Nowlan and Hinton (1991) proposed an "adaptive mixture of local experts": several expert networks and a gating network trained jointly, so that experts specialise on different regions of the input space. In that formulation every expert was evaluated and outputs were combined by the gate's soft weights, so there was no compute saving.

Shazeer et al. (2017) introduced the sparsely-gated MoE layer, with up to thousands of feed-forward experts and a trainable gate that selects a sparse combination per example; they applied it between stacked LSTM layers, with up to 137 billion parameters in the MoE layer, for language modelling and machine translation. GShard (Lepikhin et al., 2020) moved the idea into Transformers and added compiler-level automatic sharding, scaling a multilingual translation model beyond 600 billion parameters. Switch Transformers (Fedus, Zoph & Shazeer, 2022; arXiv 2021) simplified routing to a single expert per token and reported up to 7x pre-training speed-ups over dense T5 baselines at matched compute. ST-MoE (Zoph et al., 2022) concentrated on training instability and transfer to downstream tasks. GLaM (Du et al., 2022) reported a 1.2-trillion-parameter sparse model whose training used about one third of the energy of GPT-3 according to its authors.

From 2024, open-weight MoE models made the architecture inspectable: Mixtral 8x7B (Jiang et al., 2024), DeepSeekMoE (Dai et al., 2024) and DeepSeek-V3 (DeepSeek-AI, 2024). Gemini 1.5 Pro is described by its developer as a sparse MoE Transformer (Gemini Team, 2024).

#4.1 The sparse MoE layer

In a Transformer (Vaswani et al., 2017; see [TECH-ARCH-001]) the position-wise feed-forward network (FFN) in a block is replaced by $N$ expert FFNs $E_1,\dots,E_N$ and a router. For a token representation $x \in \mathbb{R}^d$ the router computes scores $s = x W_r$ with $W_r \in \mathbb{R}^{d \times N}$ and keeps the $k$ largest:

where $\mathcal{T}(x)$ is the set of the top-$k$ indices. Normalising only over the selected experts is one common choice; other designs apply the softmax over all $N$ scores before selection. With $k \ll N$ the per-token compute is roughly $k$ expert FFNs rather than $N$. Attention layers are normally dense and shared.

#4.2 Load balancing

A router trained by gradient descent tends to favour a few experts, which then receive more updates and become more favoured ("routing collapse"). Shazeer et al. (2017) added auxiliary losses on expert importance and load. The Switch Transformer uses a loss of the form

where $f_i$ is the fraction of tokens dispatched to expert $i$, $P_i$ is the mean router probability assigned to expert $i$, and $\alpha$ is a small coefficient (Fedus et al., 2022). The loss is minimised by uniform dispatch. Auxiliary losses inject gradients unrelated to the language-modelling objective; Wang et al. (2024) proposed a bias-based, loss-free alternative in which a per-expert bias is added to routing scores before top-$k$ selection and updated according to recent load, and DeepSeek-V3 reports adopting an auxiliary-loss-free strategy (DeepSeek-AI, 2024).

#4.3 Capacity and token dropping

For efficient batched execution on accelerators each expert is given a fixed buffer ("capacity"), commonly set as a multiple (the capacity factor) of the even-split share of tokens. Tokens routed to a full expert are dropped from that layer and pass through the residual connection unchanged (GShard; Switch). Capacity limits make shapes static but add a quality-versus-efficiency hyperparameter.

#4.4 Routing variants

• Token choice (top-1 in Switch, top-2 in GShard and Mixtral): each token picks experts.
• Expert choice (Zhou et al., 2022): each expert picks its top tokens, giving perfect balance and a variable number of experts per token; the authors report more than 2x faster training convergence than Switch top-1 and GShard top-2 at matched compute. Because an expert's selection depends on other tokens in the batch, the scheme is awkward for autoregressive decoding.
• Fine-grained and shared experts (DeepSeekMoE): split experts into $mN$ smaller ones and activate $mK$, and reserve $K_s$ always-active shared experts for common knowledge (Dai et al., 2024).

#4.5 Stabilisation

ST-MoE (Zoph et al., 2022) studies instabilities and transfer behaviour, and Switch Transformers show training sparse models in bfloat16 with selective precision measures. Details are in those papers rather than reproduced here.

Mixtral 8x7B has 8 feed-forward experts per layer with 2 selected per token, giving access to 47 billion parameters while using 13 billion active parameters per token (Jiang et al., 2024). DeepSeek-V3 reports 671 billion total and 37 billion activated parameters per token, pre-trained on 14.8 trillion tokens (DeepSeek-AI, 2024). These are figures from the respective reports as of their publication dates; current model tables belong in the Living Reference (MODEL-001).

• Open-weight language models. Mixtral 8x7B was released with open weights under the Apache 2.0 licence; DeepSeek-V3 checkpoints were also released publicly (primary reports above). Both can be inspected and fine-tuned, which makes them the main empirical basis for MoE research outside industrial labs.
• Machine translation. GShard's multilingual translation model was an early demonstration of MoE at scale.
• Closed models. Google's Gemini 1.5 report states that Gemini 1.5 Pro is a sparse MoE Transformer; expert count, routing design and parameter counts are undisclosed. OpenAI's GPT-4 technical report states it contains no further details about architecture, including model size, so whether GPT-4 uses MoE is undisclosed by the developer. Public figures for either system that do not come from the developer are not repeated here.

#Advantages

• Lower training and inference FLOPs per token for a given total parameter count; the Switch, GLaM and DeepSeekMoE papers each report efficiency gains over dense baselines at matched quality or compute, measured on their own setups.
• Scaling-law analyses indicate a systematic advantage. Clark et al. (2022) derive scaling laws in which parameter count and per-token compute are separate axes for routed models. Krajewski et al. (2024) report that MoE models outperform dense Transformers across the budgets they studied and that the gap widens with scale, and that setting expert size equal to the dense FFN size is rarely optimal. Both are single-group studies with fitted laws and rest on the compute regimes tested.

#Limitations

• Memory. All experts must be resident. Throughput at low batch sizes is bounded by memory bandwidth for the experts actually touched, and larger batches touch more experts, eroding the sparsity advantage on the inference side (see [TECH-INFER-001]).
• Communication. Expert parallelism requires all-to-all token exchange between devices; this shapes cluster design (see [TECH-DIST-001], [TECH-HW-001]).
• Training stability and balance. Router collapse, token dropping and loss spikes require the mitigations above.
• Fine-tuning. Early sparse models transferred less reliably than dense ones; ST-MoE was written specifically to address this.

#Unresolved questions

1. Expert specialisation. Intuition suggests experts specialise by topic. In Mixtral's analysis on The Pile the authors "do not observe obvious patterns in the assignment of experts based on the topic", with specialisation appearing more syntactic (e.g. consistent expert use for certain tokens such as code indentation) and with some positional locality across consecutive tokens (Jiang et al., 2024). This is a finding about one model; how general it is remains open.
2. Best balancing method. Auxiliary-loss and loss-free approaches have been compared mainly at small scale (Wang et al. test up to 3 billion parameters and 200 billion tokens); evidence at frontier scale is limited to developer reports.
3. Interpretability. Whether experts are meaningful units for analysis is unsettled (see [SAFE-INTERP-001]).

As of 2026-10-15, sparse MoE is a mainstream design for large language models in published open-weight work, and at least one closed developer (Google, for Gemini 1.5 Pro) states it uses MoE. The architectural components described in Section 4 (top-$k$ routing, capacity limits, load balancing, fine-grained plus shared experts) are well documented in primary papers. Claims about which proprietary models use MoE, how many experts they have, or their active-parameter counts are undisclosed unless a developer report says otherwise, and this encyclopedia does not infer them. Current model details should be read from MODEL-001.

The comparative claim "MoE is more compute-efficient than dense at the same quality" is supported by multiple papers from different groups using different setups, but the margin depends on the budget, the comparison baseline (equal total parameters, equal active parameters, or equal training FLOPs), and the serving constraints. It is rated Well Supported with those qualifications.

Observed trends, not forecasts:

• Finer-grained experts with shared experts (DeepSeekMoE), and loss-free balancing, have appeared in recent open reports.
• Research on scaling laws for MoE (Clark et al.; Krajewski et al.) is moving toward treating granularity, expert count and active parameters as joint design variables.
• Serving MoE efficiently is an active systems problem (see [TECH-INFER-001]).

No quantitative forecast is registered in this article. Whether MoE remains dominant depends on hardware memory and interconnect characteristics, and on whether alternatives to the dense FFN (for example hybrid or recurrent state-space designs, see [TECH-ARCH-001]) reduce its advantage; both are open.

• Foundational Concepts:
  • TECH-ARCH-001 The Transformer Architecture: The dense FFN block that MoE replaces.
  • TECH-DL-001 Deep Learning: General background on training deep networks.
• Related Methods & Technologies:
  • TECH-LLM-001 Large Language Models: The main setting where sparse MoE is deployed.
  • TECH-SCALE-001 Scaling Laws: Dense and routed scaling relationships.
  • TECH-HW-001 AI Hardware: Memory and interconnect constraints on expert parallelism.
  • TECH-DIST-001 Distributed Training: All-to-all communication and expert parallelism.
  • TECH-INFER-001 Inference, Quantization and Efficiency: Serving cost of sparse models.
• Advanced Implications & Future Trajectories:
  • CAP-OPEN-001 Open-Weight versus Closed Models: Disclosure differences across MoE releases.
  • SAFE-INTERP-001 Interpretability and Mechanistic Interpretability: Open question of expert interpretability.

An artificial neural network (ANN) is a parameterized function built by composing many simple units, each of which computes a weighted sum of its inputs and passes the result through a nonlinear activation function. The weights are set by fitting the network to data, almost always by gradient-based optimization of a loss function. Neural networks are the model class underlying deep learning and, through it, most current large-scale AI systems.

Neural networks address a specific problem in machine learning (see [TECH-ML-001]): learning a mapping from inputs to outputs when the useful intermediate features are not known in advance. A linear model can only combine the raw inputs it is given. A neural network with at least one hidden layer learns its own intermediate representations, and deeper networks can stack them into a hierarchy.

Three properties explain the model class's role in modern AI. First, it is expressive: with a suitable nonlinearity, even shallow networks can approximate broad classes of functions arbitrarily well (Cybenko, 1989; Hornik et al., 1989). Second, it is trainable at scale: the whole model is differentiable, so one algorithm, backpropagation (see [TECH-BACKPROP-001]), supplies gradients for every parameter. Third, it is modular: the same units can be arranged with different connectivity patterns to encode assumptions about the data, producing convolutional networks, recurrent networks, attention-based networks and others.

This article describes the shared foundation. Specific families are treated in their own entries, and the dominant current family is covered in [TECH-ARCH-001].

Origins. McCulloch and Pitts (1943) proposed a logical model of the neuron as a threshold unit and showed that networks of such units could in principle compute logical propositions. This was a formal model; it did not include learning. Rosenblatt (1958) introduced the perceptron, a single-layer threshold classifier with an error-driven learning rule, and presented it as a probabilistic model for information storage and organization in the brain.

Limits of single-layer models and the multilayer revival. Single-layer perceptrons can only separate classes that are linearly separable, a limitation that Minsky and Papert analyzed in their 1969 monograph Perceptrons. [VERIFICATION NEEDED: bibliographic details of Minsky and Papert (1969) were not checked against a publisher or library record for this draft, so the book is named in the text but not listed in section 11.] The period of reduced funding and interest that followed is discussed in the history volume (see [HIST-004]). Interest returned in the 1980s with the popularization of multilayer networks trained by backpropagation (Rumelhart, Hinton and Williams, 1986), covered in [HIST-008] and [TECH-BACKPROP-001].

Specialized architectures. Fukushima (1980) described the neocognitron, an early hierarchical network inspired by the visual cortex studies of Hubel and Wiesel (1962). LeCun et al. (1989) trained a convolutional network with backpropagation to recognize handwritten zip code digits. Hochreiter and Schmidhuber (1997) introduced long short-term memory for sequences. These lines are described in [TECH-CNN-001] and [TECH-RNN-001].

Deep learning period. Surveys of the field (Schmidhuber, 2015; LeCun, Bengio and Hinton, 2015) place the shift to deep networks in the late 2000s and early 2010s, driven by larger datasets, GPU computation, and improved training practice. The AlexNet result on the ImageNet benchmark (Krizhevsky, Sutskever and Hinton, 2017, journal version of the 2012 paper) is usually treated as the visible turning point. See [TECH-DL-001] and [HIST-005].

#4.1 The unit and the layer

A single unit with input vector $x \in \mathbb{R}^n$, weight vector $w \in \mathbb{R}^n$, and bias $b \in \mathbb{R}$ computes

where $\phi$ is an activation function. A layer applies many such units in parallel to the same input, which is compactly written with a weight matrix $W$:

A feedforward network (multilayer perceptron, MLP) with $L$ layers is the composition

with parameters $\theta = \{W_\ell, b_\ell\}_{\ell=1}^{L}$. Layers between the input and the output are called hidden layers.

#4.2 Why the nonlinearity matters

If every $\phi_\ell$ were the identity, the composition would collapse to a single affine map $x \mapsto W' x + b'$, and depth would add nothing. The nonlinearity is what lets depth add expressive power. Common choices include the logistic sigmoid, the hyperbolic tangent, the rectified linear unit $\mathrm{ReLU}(z) = \max(0, z)$ and smooth variants such as GELU; for the output layer, a softmax converts scores to class probabilities.

#4.3 What a network can represent

The universal approximation theorems state that a feedforward network with one hidden layer and a suitable nonconstant activation can approximate any continuous function on a compact set to arbitrary accuracy, provided the hidden layer is wide enough (Cybenko, 1989, for sigmoidal activations; Hornik, Stinchcombe and White, 1989, for more general squashing functions). Pinkus (1999) surveys the approximation theory of this model class, including the conditions on the activation. These are existence results. They do not say how many units are needed in practice, whether gradient descent will find the approximating weights, or how well the fitted function will generalize to new data.

#4.4 Training

Given a dataset $\{(x_i, y_i)\}_{i=1}^{m}$ and a loss $\ell$, training seeks parameters that reduce the empirical risk

Typical losses are squared error for regression and cross-entropy for classification. Parameters are updated iteratively, for example by gradient descent,

where $\eta$ is the learning rate; in practice the gradient is estimated on small random subsets of the data (minibatches), and adaptive variants of the update rule are common. Computing $\nabla_\theta \mathcal{L}$ for networks with millions or billions of parameters is made feasible by backpropagation. See [TECH-BACKPROP-001] for the algorithm and [FOUND-010] for optimization background.

#4.5 Practical training components

Several techniques are routinely combined with the basic scheme. Initialization schemes set the scale of the starting weights so that signals neither vanish nor explode with depth. Normalization layers, such as batch normalization (Ioffe and Szegedy, 2015), stabilize and often speed training. Residual connections (He et al., 2015) add a layer's input to its output and made much deeper networks trainable. Regularization, including dropout (Hinton et al., 2012), reduces overfitting. Each of these is an engineering device whose benefits are demonstrated empirically; theoretical accounts of why they work are in several cases still disputed.

Neural networks are usually classified by how units are connected.

A second axis is the learning setting: supervised, self-supervised, unsupervised or reinforcement learning (see [TECH-RL-001]). The same architecture can be used under several settings.

The applications below are deployed or widely reproduced research results, not hypothetical capabilities.

• Image recognition. Convolutional networks became the standard approach to image classification after Krizhevsky et al. (2017) and were extended by residual networks (He et al., 2015).
• Sequence modeling and translation. Recurrent encoder-decoder networks produced the first neural systems competitive in machine translation (Sutskever, Vinyals and Le, 2014), later superseded in most settings by attention-based models.
• Language and multimodal models. Large language models are neural networks of the transformer family (see [TECH-LLM-001]).
• Scientific prediction. Neural networks are used in protein structure prediction and other scientific tasks (see [APP-SCI-001]).
• Control. Deep reinforcement learning uses neural networks as value and policy functions; Mnih et al. (2015) trained a convolutional network to play Atari games from pixels.

Specific deployment scale, accuracy and cost figures change quickly and are tracked in the Living Reference collections (MODEL-001, FACT-001) rather than here.

#Demonstrated advantages

• Representation learning. Features are learned from data rather than hand-designed, which is the main reason neural networks displaced hand-engineered pipelines in perception and language tasks (LeCun, Bengio and Hinton, 2015).
• Generality. One training recipe applies across modalities and tasks.
• Scalability. Training parallelizes well on accelerator hardware (see [TECH-HW-001]).

#Limitations

• Data and compute requirements. Strong performance generally requires large datasets and substantial computation.
• Opacity. Learned representations are hard to interpret; the field of interpretability studies this (see [SAFE-INTERP-001]).
• Brittleness and distribution shift. Performance can degrade sharply on inputs unlike the training data, and networks can be sensitive to small adversarial perturbations. [VERIFICATION NEEDED: no primary source on adversarial examples was confirmed for this draft; the statement is made without citation.]
• Optimization difficulty. The loss surface is non-convex, and training can fail through vanishing or exploding gradients (Bengio, Simard and Frasconi, 1994).

#Unresolved questions

1. Generalization. Zhang et al. (2016) showed that standard networks can fit random labels, which challenges explanations of generalization based only on model capacity. Belkin et al. (2019) describe a "double descent" pattern in which test error can fall again beyond the point of fitting the training data. Why large networks generalize as well as they do is not settled.
2. Role of architecture versus scale. How much of recent progress is due to architectural choices rather than data and compute is debated, and the answer differs by task.
3. Relationship to biological neurons. Artificial units are loose abstractions. Claims that network training or structure resembles biological learning are contested and are not needed to justify the engineering use of the models.

As of 2026-10-15, neural networks are the dominant model class for perception, language and generative tasks. The theory of approximation is mature and is Established. The theory of optimization and generalization for large networks is incomplete: there are partial explanations and informative empirical phenomena (Zhang et al., 2017; Belkin et al., 2019), but no accepted general account. Training practice (initialization, normalization, residual connections, adaptive optimizers) is largely empirical and well replicated, which supports the rating "Well Supported" for the effectiveness of individual techniques but not necessarily for the stated reasons they work.

This section describes observed trends and plausible directions. It does not register forecasts with probabilities.

• Near term (0–3 years). Continued dominance of transformer-style networks, with active work on efficiency, including sparsity and lower-precision arithmetic (see [TECH-INFER-001]).
• Medium term (3–10 years). Open questions include whether alternatives to dense attention, such as state-space and hybrid designs, will match transformers at scale (see [TECH-RNN-001]), and whether training can become substantially more data-efficient.
• Long term (10+ years). Whether neural networks alone are a sufficient substrate for general intelligence is a central question of the AGI debate (see [AGI-001], [AGI-PATH-001]); the question is open and is treated in Volume V.

• Foundational Concepts:
  • TECH-ML-001 Machine Learning: learning from data and empirical risk minimization.
  • FOUND-010 Optimization and Gradient Descent: the optimization methods used to train networks.
  • HIST-008 Backpropagation and the Connectionist Revival: historical account of the 1980s revival.
• Related Methods & Technologies:
  • TECH-DL-001 Deep Learning: networks with many layers and the surrounding practice.
  • TECH-BACKPROP-001 Backpropagation: the gradient computation algorithm.
  • TECH-CNN-001 Convolutional Neural Networks: networks with spatial weight sharing.
  • TECH-RNN-001 Recurrent Networks and LSTM: networks for sequences.
  • TECH-ATTN-001 Attention Mechanisms: content-based weighting.
  • TECH-GAN-001 Generative Adversarial Networks: adversarial generative training.
  • TECH-VAE-001 Autoencoders and Variational Autoencoders: latent-variable generative models.
  • TECH-GNN-001 Graph Neural Networks: networks for graph-structured data.
  • TECH-RL-001 Reinforcement Learning: networks as policies and value functions.
  • TECH-HW-001 AI Hardware & Accelerator Systems: the hardware that makes training practical.
• Advanced Implications & Future Trajectories:
  • TECH-ARCH-001 The Transformer Architecture: the dominant current family.
  • SAFE-INTERP-001 Interpretability and Mechanistic Interpretability: understanding what networks compute.
  • HIST-005 The Deep Learning Breakthrough: the historical transition to deep networks.

A recurrent neural network (RNN) processes a sequence one element at a time while carrying a hidden state from each step to the next, so that its output at a given step can depend on everything seen earlier in the sequence. Long short-term memory (LSTM) is a gated recurrent design that adds an explicitly controlled memory cell, introduced to make learning over long time lags feasible.

Many inputs are sequences of variable length: text, speech, sensor streams, time series. A feedforward network (see [TECH-NN-001]) needs a fixed-size input and treats positions independently. A recurrent network instead applies the same function at every time step and passes a state forward, which gives it two useful properties: weight sharing across time, and a memory that is, in principle, unbounded in the length of context it can use.

In practice, plain RNNs struggle to learn dependencies across many steps because the gradients involved shrink or grow exponentially (Bengio, Simard and Frasconi, 1994; see [TECH-BACKPROP-001]). Gated architectures, LSTM and later the GRU, mitigated this and were the standard tools for sequence tasks through the mid-2010s. They were then largely replaced in language modelling and translation by attention-based architectures (Vaswani et al., 2017; see [TECH-ARCH-001]), but recurrent ideas remain relevant, including in current efficient sequence models.

Early recurrent models. Elman (1990) described a simple recurrent network in which the previous hidden state is fed back as context, and showed that such a network could discover structure in temporal sequences. Training used backpropagation, extended through time (see [TECH-BACKPROP-001]).

The long-lag problem. Bengio, Simard and Frasconi (1994) analyzed why gradient-based learning of long-term dependencies is difficult. This result is the main motivation for the gated designs that followed.

LSTM. Hochreiter and Schmidhuber (1997) introduced the LSTM, whose central device is a memory cell with a self-connection of fixed weight (the "constant error carousel") guarded by multiplicative gates, so that error signals can flow across many steps without decaying. Gers, Schmidhuber and Cummins (2000) added a forget gate, allowing the cell to reset its content, and this form is the one commonly used in practice. Schuster and Paliwal (1997) proposed bidirectional recurrent networks, which read the sequence in both directions when the whole input is available.

Neural machine translation. Sutskever, Vinyals and Le (2014) used LSTM encoder and decoder networks for sequence-to-sequence translation. Cho et al. (2014) proposed an RNN encoder-decoder with a new hidden unit with reset and update gates, now known as the GRU. Bahdanau, Cho and Bengio (2014) added attention to this encoder-decoder, relieving the need to compress a whole source sentence into one vector (see [TECH-ATTN-001]).

Displacement and revival. Vaswani et al. (2017) showed that a model built only on attention, without recurrence, achieved strong translation results while being more parallelizable. Subsequent research has revisited recurrence in forms that are efficient to train and to run, including linear-attention formulations that can be written as RNNs (Katharopoulos et al., 2020) and selective state space models (Gu and Dao, 2023).

#4.1 The basic recurrence

For an input sequence $x_1, \dots, x_T$, a simple RNN maintains a hidden state $h_t$:

with initial state $h_0$ (often zero), a nonlinearity $\phi$ (such as $\tanh$), and the same parameters $W_{hh}, W_{xh}, W_{hy}$ at every step. The loss is a sum of per-step losses or a loss on the final output.

#4.2 Training and the gradient problem

Training uses backpropagation through time (BPTT): unroll the recurrence for $T$ steps and apply backpropagation to the resulting deep feedforward graph with tied weights. The gradient of a loss at step $T$ with respect to an earlier hidden state involves a product of Jacobians,

where $a_k$ is the pre-activation at step $k$. When the relevant singular values are consistently below one, this product decays exponentially with the lag $T - t$; when above one, it can grow without bound (Bengio et al., 1994; Pascanu, Mikolov and Bengio, 2012). Gradient clipping addresses exploding gradients, while the vanishing case motivates architectural change.

#4.3 LSTM

An LSTM keeps a cell state $c_t$ in addition to the hidden state $h_t$. With the forget gate of Gers et al. (2000), one common formulation is

where $\sigma$ is the logistic sigmoid, $[\cdot,\cdot]$ denotes concatenation, and $\odot$ is elementwise multiplication. The forget gate $f_t$ controls how much of the old cell content is kept, the input gate $i_t$ controls how much new content is written, and the output gate $o_t$ controls how much of the cell is exposed. The cell update is additive, so the derivative of $c_t$ with respect to $c_{t-1}$ is the gate $f_t$ (ignoring indirect paths through the gates). Where $f_t$ is near one, gradient can travel through many steps without repeated squashing, which is the intuition behind the constant error carousel (Hochreiter and Schmidhuber, 1997). Exact formulations vary across papers and software libraries (for example, whether the gates also receive the cell state), so a specific implementation should be consulted when reproducing results.

#4.4 GRU

The gated recurrent unit of Cho et al. (2014) merges the cell and hidden state and uses two gates, an update gate and a reset gate. It has fewer parameters than an LSTM of the same width. Reports comparing the two on specific tasks do not support a general ranking. [VERIFICATION NEEDED: no comparative study was confirmed for this draft; the statement reflects the absence of a confirmed source rather than a documented finding.]

#4.5 Sequence-to-sequence and bidirectional models

In an encoder-decoder configuration, one RNN reads the input sequence into a state, and a second RNN generates the output sequence from that state (Sutskever et al., 2014; Cho et al., 2014). A single fixed-length vector can become a bottleneck for long inputs, which motivated attention (Bahdanau et al., 2014). Bidirectional RNNs run one network forward and one backward over the input and combine their states (Schuster and Paliwal, 1997); they are not suitable for step-by-step generation where the future is unavailable.

• Simple (Elman) RNN. One hidden layer with feedback (Elman, 1990).
• LSTM. Gated cell with input, forget and output gates (Hochreiter and Schmidhuber, 1997; Gers et al., 2000).
• GRU. Two-gate simplified variant (Cho et al., 2014).
• Bidirectional RNN. Reads both directions (Schuster and Paliwal, 1997).
• Stacked (deep) RNN. Several recurrent layers, each taking the previous layer's states as input.
• Encoder-decoder RNN. For sequence-to-sequence mapping (Sutskever et al., 2014).
• Linear-attention and state-space recurrences. Modern constant-state sequence models. Katharopoulos et al. (2020) show that causal linear attention can be written as an RNN, and Gu and Dao (2023) propose a selective state space architecture with linear scaling in sequence length.

• Machine translation. LSTM-based sequence-to-sequence systems were an early neural approach competitive with phrase-based statistical systems (Sutskever et al., 2014; Cho et al., 2014 used an RNN encoder-decoder to score phrase pairs in a statistical translation system). Attention-based RNNs then improved results (Bahdanau et al., 2014).
• Language modelling and generation. Recurrent language models predicted the next token from the hidden state; transformer-based models have since largely taken over this role (see [TECH-LLM-001]).
• Speech and time series. Recurrent networks were widely applied to sequential signals. [VERIFICATION NEEDED: no specific deployment source was confirmed for speech or time-series applications.]
• Reinforcement learning. Recurrent policies handle partially observable environments by carrying a state (see [TECH-RL-001]).
• Efficient sequence models. State-space and linear-attention recurrences target long contexts with constant-size state (Gu and Dao, 2023; Katharopoulos et al., 2020).

Which production systems currently use recurrent components is not tracked here; see the Living Reference collection MODEL-001.

#Advantages

• Constant-size state. Inference with a recurrent model needs memory that does not grow with the processed length, since the state has fixed size.
• Natural handling of variable-length input and streaming data.
• Parameter sharing across time.

#Limitations

• Sequential computation. Step $t$ depends on step $t-1$, so training cannot be parallelized over time, which is a main reason for the shift to attention-based models (Vaswani et al., 2017).
• Long-range dependency. Gating mitigates but does not remove the difficulty of carrying information over very long spans, and a fixed-size state is a bounded memory.
• Bottleneck in encoder-decoder use. Compressing an input into one vector limits fidelity for long inputs (Bahdanau et al., 2014).
• Training sensitivity. Exploding gradients require clipping or careful design (Pascanu et al., 2012).

#Unresolved questions

1. Can fixed-state recurrent models match attention at scale? Selective state-space and linear-attention models are active candidates. Gu and Dao (2023) report favourable results for their model in the settings they test, but independent comparisons across training regimes are not covered by the sources confirmed for this draft.
2. How much context can a fixed-size state faithfully retain? The trade-off between state size and recall is an active research topic. [VERIFICATION NEEDED: no source on state capacity limits was confirmed.]

As of 2026-10-15, attention-based transformers dominate large-scale language and translation systems, and LSTMs and GRUs are no longer the default for those tasks. The mathematics of gated recurrence and of gradient decay is Established. That LSTMs improved long-lag learning over simple RNNs on the test problems studied is Well Supported by the original experiments and later usage (Hochreiter and Schmidhuber, 1997). The competitiveness of recurrent-style sequence models at frontier scale is Preliminary to Contested; it depends on results of individual groups (Gu and Dao, 2023) that await systematic independent comparison.

This section describes plausible directions and does not register forecasts.

• Near term (0–3 years). Hybrid models combining attention layers with recurrent or state-space layers, motivated by inference cost for long contexts, are an active direction.
• Medium term (3–10 years). Whether constant-state models close the quality gap with attention on recall-heavy tasks is an empirical question.
• Long term (10+ years). Persistent, updateable memory for agents (see [AGI-CL-001]) may revisit recurrent state as one mechanism among several; this is speculative.

• Foundational Concepts:
  • TECH-NN-001 Artificial Neural Networks: units, layers and training.
  • TECH-BACKPROP-001 Backpropagation: backpropagation through time and gradient decay.
• Related Methods & Technologies:
  • TECH-ATTN-001 Attention Mechanisms: attention introduced on top of RNN encoder-decoders.
  • TECH-CNN-001 Convolutional Neural Networks: the other structured family; convolutions as an alternative for sequences.
  • TECH-RL-001 Reinforcement Learning: recurrent policies for partial observability.
  • TECH-DL-001 Deep Learning: broader context.
• Advanced Implications & Future Trajectories:
  • TECH-ARCH-001 The Transformer Architecture: the architecture that displaced recurrence in language modelling.
  • TECH-LLM-001 Large Language Models: current use of autoregressive sequence models.
  • AGI-CL-001 Continual Learning and Persistent Memory: memory beyond a context window.

An autoencoder is a neural network trained to reproduce its input after passing it through a constrained intermediate representation, so that the representation (the code or latent) captures the structure needed for reconstruction. A variational autoencoder (VAE) is a probabilistic version in which the encoder outputs a distribution over latent variables and the model is trained by maximizing a lower bound on the data likelihood, which makes it a generative model from which new samples can be drawn.

Autoencoders address representation learning without labels: the training signal is the input itself. A network with an encoder $f$ and a decoder $g$ is trained so that $g(f(x)) \approx x$, with a bottleneck or other constraint preventing a trivial identity mapping. The learned code can be used for compression, denoising and as features for other tasks. Hinton and Salakhutdinov (2006) reported that deep autoencoders trained to reduce high-dimensional data to low-dimensional codes worked better than principal component analysis on the data they studied.

The plain autoencoder is not a generative model, because nothing guarantees that arbitrary points in code space decode to plausible data. The VAE of Kingma and Welling (2013) addressed this by combining deep neural networks with variational inference over continuous latent variables, with an encoder acting as an approximate posterior. Rezende, Mohamed and Wierstra (2014) independently developed a closely related approach that they call stochastic backpropagation. The VAE's significance is twofold. It gives a principled training objective for deep latent-variable models, and its encoder-decoder structure has become a component of larger systems; for example, latent diffusion models apply diffusion in the latent space of a pretrained autoencoder (Rombach et al., 2021; see [TECH-DIFF-001]).

Classical and deep autoencoders. Autoencoders were studied as nonlinear generalizations of dimensionality reduction. Hinton and Salakhutdinov (2006) described how to train deep autoencoders effectively and reported that they outperformed principal component analysis on their tasks. Vincent, Larochelle, Bengio and Manzagol (2008) proposed denoising autoencoders, which learn to reconstruct a clean input from a corrupted version, as a way to learn robust features.

Variational autoencoders. Kingma and Welling (2013) introduced a stochastic variational inference and learning algorithm that scales to large datasets, based on a reparameterization of the variational lower bound and an approximate inference ("recognition") model. Rezende, Mohamed and Wierstra (2014) presented stochastic backpropagation, a rule for backpropagation through stochastic variables, together with a recognition model that acts as a stochastic encoder. Kingma and Welling (2019) later wrote an introductory treatment of the framework.

Extensions. Bowman et al. (2015) built an RNN-based VAE for sentences (see [TECH-RNN-001]) and reported negative results on its use as a language model. Van den Oord, Vinyals and Kavukcuoglu (2017) proposed the VQ-VAE, in which the encoder outputs discrete codes and the prior is learned; they state that this avoids "posterior collapse", in which the latents are ignored when paired with a powerful autoregressive decoder. Rombach et al. (2021) used pretrained autoencoders as the representation in which diffusion models operate.

#4.1 The deterministic autoencoder

With encoder $z = f_\phi(x)$ and decoder $\hat x = g_\theta(z)$, training minimizes a reconstruction loss such as

or a cross-entropy loss for binary data. Constraints that prevent trivial copying include a code dimension smaller than the input (an undercomplete autoencoder), regularization on the code, or input corruption as in the denoising autoencoder (Vincent et al., 2008). With linear encoder and decoder and squared error, the optimal subspace is the same as that of principal component analysis. [VERIFICATION NEEDED: this is a standard textbook result; no source was checked for this draft.]

#4.2 The latent-variable model

A VAE posits a generative process: draw a latent $z \sim p(z)$ (usually a standard Gaussian $\mathcal{N}(0, I)$), then draw $x \sim p_\theta(x \mid z)$, where the decoder network parameterizes $p_\theta(x \mid z)$. The data likelihood

is intractable for flexible decoders, and so is the true posterior $p_\theta(z \mid x)$. The VAE introduces an encoder $q_\phi(z \mid x)$, typically a Gaussian with mean $\mu_\phi(x)$ and diagonal covariance $\sigma^2_\phi(x)$ produced by a network, as an approximation to the posterior (Kingma and Welling, 2013).

#4.3 The evidence lower bound

For any $q_\phi$, the log-likelihood decomposes as

Because the last term is non-negative, the ELBO is a lower bound on $\log p_\theta(x)$, and maximizing it with respect to both $\theta$ and $\phi$ improves the model and tightens the approximation. The first term rewards accurate reconstruction from latents sampled from the encoder; the second term, the KL divergence, regularizes the encoder's distribution toward the prior. For a Gaussian encoder and standard normal prior the KL term has a closed form:

#4.4 The reparameterization trick

The expectation over $z \sim q_\phi(z \mid x)$ depends on $\phi$ through the sampling distribution, which blocks ordinary backpropagation. Writing the sample as a deterministic function of the parameters and an independent noise variable,

moves the randomness into $\varepsilon$, so gradients flow through $\mu_\phi$ and $\sigma_\phi$ and the lower bound can be optimized by standard stochastic gradient methods (Kingma and Welling, 2013; Rezende et al., 2014; see [TECH-BACKPROP-001]).

#4.5 Generation and inference

To generate, sample $z \sim p(z)$ and decode. To embed data, pass $x$ through the encoder. Interpolation between codes of two inputs produces intermediate decoded outputs; Bowman et al. (2015) used this to study the latent space of sentences.

#4.6 Posterior collapse

If the decoder is powerful enough to model the data without using the latent, the optimization can push $q_\phi(z\mid x)$ toward the prior, making the code uninformative; this is called posterior collapse. It is reported as a difficulty for VAEs with autoregressive decoders (Bowman et al., 2015; van den Oord et al., 2017).