Backpropagation

Backpropagation is an algorithm that computes the gradient of a scalar loss with respect to every parameter of a neural network in a single backward sweep, by applying the chain rule of calculus layer by layer from the output toward the input and reusing intermediate results. It is a special case of reverse-mode automatic differentiation. Backpropagation computes gradients only; the parameter update is performed by a separate optimization rule such as gradient descent.

Training a neural network (see [TECH-NN-001]) means minimizing a loss $\mathcal{L}(\theta)$ over a very large parameter vector $\theta$. First-order methods need $\nabla_\theta \mathcal{L}$ at every step. Computing each partial derivative separately, for example by perturbing one parameter at a time, would cost one extra forward evaluation per parameter. For a network with $P$ parameters, that is $P$ additional passes per step, which is impractical for modern values of $P$.

Backpropagation removes this factor. Because the loss is a scalar and the network is a composition of differentiable operations, a single forward pass followed by a single backward pass yields all $P$ partial derivatives, at a cost of the same order as the forward pass (Baydin et al., 2018). This property, not any particular network design, is what made gradient-based training of deep models routine. Nearly all systems discussed in this encyclopedia that are trained by gradient descent rely on it, including large language models (see [TECH-LLM-001]).

The distinction between backpropagation and the optimizer is a frequent source of confusion. Backpropagation answers "in which direction does each parameter change the loss?". The optimizer answers "how far and in what manner do we move?" (see [FOUND-010]).

The chain rule itself is classical. The history of its use for efficiently differentiating long compositions is a matter of documented priority disputes, and this article states only what the cited sources support.

Reverse-mode differentiation. Baydin et al. (2018) survey automatic differentiation (AD) and describe backpropagation as a special case of reverse-mode AD, noting that the machine learning and AD communities developed related ideas largely independently. Reverse-mode AD for general computational graphs was developed in the numerical analysis literature before its widespread use in neural networks (Baydin et al., 2018). Baydin et al. state that Linnainmaa's 1970 master's thesis and his 1976 BIT paper ("Taylor expansion of the accumulated rounding error", BIT 16(2), 146-160, doi:10.1007/BF01931367, confirmed via Crossref) are "often cited as the first published description of the reverse mode"; this is a priority attribution reported by a survey, and the 1970 thesis itself was not examined.

Application to neural networks. Werbos (1982) described the use of backward sensitivity calculations for nonlinear systems, and later accounts treat this as an early proposal to train multilayer networks this way. Rumelhart, Hinton and Williams (1986) demonstrated in Nature that multilayer networks trained by this procedure learn useful internal representations, and this paper is the usual point of reference for the connectionist revival described in [HIST-008]. LeCun et al. (1989) applied the algorithm to a convolutional network for handwritten digit recognition (see [TECH-CNN-001]).

Difficulties in depth and time. Bengio, Simard and Frasconi (1994) analyzed why gradient-based learning of long-term dependencies is difficult, identifying exponentially vanishing or growing gradient contributions. This motivated gated recurrent units such as the LSTM (Hochreiter and Schmidhuber, 1997; see [TECH-RNN-001]) and, later, residual connections (He et al., 2015).

Software. From the 2010s, deep learning frameworks implement reverse-mode AD generically, so that practitioners define the forward computation and obtain gradients automatically. Baydin et al. (2018) describe the "dynamic computational graph" and "differentiable programming" approaches in this setting. Schmidhuber (2015) provides a broader historical survey of the field.

#4.1 Setup

Consider a feedforward network with layers $\ell = 1, \dots, L$. Let $a^{0} = x$ be the input. For each layer,

and let the loss for a single example be $\mathcal{L} = c\big(a^{L}, y\big)$ (the cost function $c$ is written with a different symbol from the layer index $\ell$). The goal is $\partial \mathcal{L} / \partial W^{\ell}$ and $\partial \mathcal{L} / \partial b^{\ell}$ for every $\ell$. (Batches are handled by averaging; the derivation below is for one example.)

#4.2 The error signal

Define the error signal (sensitivity) of layer $\ell$ as the gradient of the loss with respect to the pre-activation:

By the chain rule, with $\odot$ denoting elementwise multiplication and $\phi'$ the elementwise derivative:

The second equation is the backward recursion: the error at layer $\ell$ is the error at layer $\ell+1$ propagated through the transposed weights and then gated by the local activation derivative. Once $\delta^{\ell}$ is known, the parameter gradients follow directly, since $z^{\ell}$ depends on $W^{\ell}$ and $b^{\ell}$ linearly:

#4.3 The algorithm

1. Forward pass. Compute and store $z^{\ell}$ and $a^{\ell}$ for all layers, then evaluate the loss.
2. Output error. Compute $\delta^{L}$.
3. Backward pass. For $\ell = L-1$ down to $1$, compute $\delta^{\ell}$ from $\delta^{\ell+1}$.
4. Gradients. Form $\partial\mathcal{L}/\partial W^{\ell}$ and $\partial\mathcal{L}/\partial b^{\ell}$ for all $\ell$.
5. Update (outside backpropagation). Apply an optimizer step, for example $\theta \leftarrow \theta - \eta \nabla_\theta \mathcal{L}$.

The stored forward quantities are needed in the backward pass, so memory grows with depth and batch size. The activation-memory cost is a principal constraint in large-model training (see [TECH-DIST-001]).

#4.4 Computational graph view and reverse-mode AD

Backpropagation generalizes to any computation expressed as a directed acyclic graph of differentiable operations. For each node $v$ with output $u_v$, define the adjoint $\bar u_v = \partial \mathcal{L} / \partial u_v$. If $u_v$ feeds into several downstream nodes $w \in \mathrm{succ}(v)$, then

Adjoints are computed in reverse topological order, starting from $\bar{\mathcal{L}} = 1$. Each operation only needs to supply a rule for multiplying an incoming adjoint by its local Jacobian (a vector-Jacobian product); the full Jacobian is never formed. This is reverse-mode AD (Baydin et al., 2018).

Why reverse rather than forward mode. Forward-mode AD propagates derivatives with respect to one input direction per sweep. For a function from $\mathbb{R}^{P}$ to a scalar, obtaining the full gradient takes $P$ forward-mode sweeps but one reverse-mode sweep. Since neural network losses are scalar functions of many parameters, reverse mode is the efficient choice. Forward mode is preferable when a function has few inputs and many outputs (Baydin et al., 2018).

#4.5 Gradient flow through depth and time

Unrolling the recursion shows that the error signal at layer $\ell$ contains a product of per-layer Jacobians,

where $D_k$ holds the activation derivatives of layer $k$ on its diagonal. If the norms of these factors are consistently below one, gradients shrink exponentially with depth (vanishing); if above one, they grow (exploding). The same product structure arises in recurrent networks across time steps, which is the case analyzed by Bengio et al. (1994) and, for clipping and regularization remedies, by Pascanu, Mikolov and Bengio (2012). Common mitigations include non-saturating activations, careful initialization, normalization (Ioffe and Szegedy, 2015), gated architectures, residual connections (He et al., 2015), and gradient clipping.

#4.6 Backpropagation through time

For a recurrent network unrolled over $T$ steps with shared weights, backpropagation applied to the unrolled graph is called backpropagation through time (BPTT). Gradients for shared weights are summed over steps. Because memory and compute scale with $T$, truncated BPTT limits the backward window. See [TECH-RNN-001].

#4.7 Checkpointing and the memory-compute trade

Activation storage can be reduced by saving only a subset of activations and recomputing the rest during the backward pass. This trades additional forward computation for lower memory. The technique is widely used in large-model training, and its details belong to [TECH-DIST-001].

• Reverse-mode AD (standard backpropagation). One backward sweep per scalar output; memory proportional to stored intermediates.
• Forward-mode AD. One sweep per input direction. Efficient for few inputs and used for Jacobian-vector products.
• Backpropagation through time and truncated variants. For recurrent models.
• Second-order information. Hessian-vector products can be computed with AD-based techniques and underlie some optimization methods (Baydin et al., 2018). Full second-order methods are rarely used at large scale.
• Local learning and backprop alternatives. Several proposals compute weight updates without a global backward pass, including forward-only and local-learning rules, motivated by hardware efficiency or biological plausibility. No specific alternative-algorithm papers are named here.
• Stochastic estimators for non-differentiable steps. Where a computation includes sampling, gradients require either the reparameterization trick (see [TECH-VAE-001]) or score-function estimators.

• Supervised training of classifiers and regressors. The algorithm in section 4.3 is the standard training loop of deep networks (LeCun, Bengio and Hinton, 2015).
• Convolutional networks. The convolution operation is linear, so its backward pass is also a convolution-like operation, and weight sharing means gradients are summed over positions (LeCun et al., 1989; see [TECH-CNN-001]).
• Sequence models. BPTT trains LSTM and GRU networks ([TECH-RNN-001]); attention-based networks are trained by backpropagating through the attention weights ([TECH-ATTN-001]).
• Generative models. Variational autoencoders rely on a differentiable sampling formulation so that gradients pass through a stochastic node ([TECH-VAE-001]); adversarial training backpropagates through a discriminator into a generator ([TECH-GAN-001]).
• Deep learning frameworks. Automatic differentiation libraries apply reverse-mode AD to arbitrary programs built from differentiable primitives (Baydin et al., 2018).

#Advantages

• Efficiency. One backward sweep produces all parameter gradients at a cost of the same order as the forward pass (Baydin et al., 2018).
• Exactness. Backpropagation yields the exact gradient of the computed function (up to floating-point error), unlike finite-difference estimates.
• Modularity. Any differentiable operation that supplies a vector-Jacobian product can be inserted into a network.

#Limitations

• Memory. Stored activations scale with depth and batch size.
• Gradient pathologies. Vanishing and exploding gradients (Bengio et al., 1994; Pascanu et al., 2012).
• Local information only. Gradients describe the loss locally. They do not by themselves guarantee convergence to a good solution on a non-convex loss.
• Sequential dependency. The backward pass cannot begin until the forward pass has finished, which constrains some forms of parallelism (see [TECH-DIST-001]).
• Differentiability requirement. Discrete or non-differentiable operations need relaxations or estimators.

#Unresolved questions

1. Biological plausibility. Whether the brain implements anything equivalent to backpropagation, given requirements such as symmetric forward and backward weights, is an open question in neuroscience and machine learning; this is a description of an open question, not a cited finding.
2. Why gradient descent with backpropagation finds solutions that generalize. See [TECH-NN-001], section 7.
3. Efficient alternatives at scale. Whether training methods that avoid a global backward pass can match backpropagation-based training at frontier scale is undetermined.

The mathematics of backpropagation as exact reverse-mode differentiation is Established and is covered in standard texts and in the AD literature (Baydin et al., 2018). Its practical use is universal in gradient-trained deep networks. Open research concerns the surrounding system: how to hold activation memory in check, how to stabilize gradient flow in very deep or long-context models, and whether credit assignment can be done more cheaply or more locally. As of 2026-10-19, no alternative has been confirmed in the sources reviewed for this draft to replace backpropagation in frontier-scale training. This is a statement about the sources reviewed, not a survey of all current research.

This section describes plausible directions and does not register forecasts.

• Near term (0–3 years). Continued engineering of memory-saving backward passes, kernel fusion and recomputation strategies, tied closely to accelerator characteristics (see [TECH-HW-001]).
• Medium term (3–10 years). Research on local or forward-only learning rules may be driven by hardware energy constraints. Whether any such method becomes competitive at scale is an open empirical question.
• Long term (10+ years). Whether continual or lifelong learning systems (see [AGI-CL-001]) will continue to use backpropagation over stored data, or move to online credit-assignment mechanisms, is unresolved.

• Foundational Concepts:
  • TECH-NN-001 Artificial Neural Networks: the model class whose parameters are differentiated.
  • FOUND-010 Optimization and Gradient Descent: the update rules that consume backpropagated gradients.
  • TECH-ML-001 Machine Learning: the empirical risk minimization setting.
• Related Methods & Technologies:
  • TECH-DL-001 Deep Learning: the broader practice built on this algorithm.
  • TECH-RNN-001 Recurrent Networks and LSTM: backpropagation through time and gradient pathologies.
  • TECH-CNN-001 Convolutional Neural Networks: early application of backpropagation to a structured network.
  • TECH-VAE-001 Autoencoders and Variational Autoencoders: gradient estimation through stochastic nodes.
  • TECH-DIST-001 Distributed Training: memory and parallelism constraints of the backward pass.
  • TECH-HW-001 AI Hardware & Accelerator Systems: hardware efficiency of forward and backward passes.
• Advanced Implications & Future Trajectories:
  • HIST-008 Backpropagation and the Connectionist Revival: historical account.
  • TECH-ARCH-001 The Transformer Architecture: a model family trained by backpropagation at very large scale.
  • AGI-CL-001 Continual Learning and Persistent Memory: open problems in learning without forgetting.