---
id: TECH-BACKPROP-001
title: "Backpropagation"
alternative_names:
  - "Backprop"
  - "Reverse-Mode Automatic Differentiation (as applied to neural networks)"
  - "Error Backpropagation"
category: "Technical → Neural Networks"
volume: III
chapter: 18
entry_type: "Technical Article"
level: 4
status: "Draft"
last_reviewed: 2026-10-15
version: "0.1.0"
prerequisites:
  - "TECH-NN-001"
  - "FOUND-010"
  - "Multivariable Calculus"
  - "Linear Algebra"
related_entries:
  - "TECH-DL-001"
  - "TECH-RNN-001"
  - "TECH-ARCH-001"
  - "HIST-008"
  - "TECH-HW-001"
  - "SAFE-INTERP-001"
---

# Backpropagation

## 1. Concise Definition
**Backpropagation** is an algorithm that computes the gradient of a scalar loss with respect to every parameter of a neural network in a single backward sweep, by applying the chain rule of calculus layer by layer from the output toward the input and reusing intermediate results. It is a special case of reverse-mode automatic differentiation. Backpropagation computes gradients only; the parameter update is performed by a separate optimization rule such as gradient descent.

## 2. Overview & Significance
Training a neural network (see [TECH-NN-001]) means minimizing a loss $\mathcal{L}(\theta)$ over a very large parameter vector $\theta$. First-order methods need $\nabla_\theta \mathcal{L}$ at every step. Computing each partial derivative separately, for example by perturbing one parameter at a time, would cost one extra forward evaluation per parameter. For a network with $P$ parameters, that is $P$ additional passes per step, which is impractical for modern values of $P$.

Backpropagation removes this factor. Because the loss is a scalar and the network is a composition of differentiable operations, a single forward pass followed by a single backward pass yields all $P$ partial derivatives, at a cost of the same order as the forward pass (Baydin et al., 2018). This property, not any particular network design, is what made gradient-based training of deep models routine. Nearly all systems discussed in this encyclopedia that are trained by gradient descent rely on it, including large language models (see [TECH-LLM-001]).

The distinction between backpropagation and the optimizer is a frequent source of confusion. Backpropagation answers "in which direction does each parameter change the loss?". The optimizer answers "how far and in what manner do we move?" (see [FOUND-010]).

## 3. Historical Development
The chain rule itself is classical. The history of its use for efficiently differentiating long compositions is a matter of documented priority disputes, and this article states only what the cited sources support.

**Reverse-mode differentiation.** Baydin et al. (2018) survey automatic differentiation (AD) and describe backpropagation as a special case of reverse-mode AD, noting that the machine learning and AD communities developed related ideas largely independently. Reverse-mode AD for general computational graphs was developed in the numerical analysis literature before its widespread use in neural networks (Baydin et al., 2018). [VERIFICATION NEEDED: the commonly cited attribution of the first reverse-mode formulation to Linnainmaa (1970) was not confirmed against a primary or publisher source for this draft.]

**Application to neural networks.** Werbos (1982; year [VERIFICATION NEEDED]: Crossref record carries no date) described the use of backward sensitivity calculations for nonlinear systems, and later accounts treat this as an early proposal to train multilayer networks this way. [VERIFICATION NEEDED: the exact content of Werbos's earlier 1974 thesis, which is sometimes cited for the neural network application, was not checked.] Rumelhart, Hinton and Williams (1986) demonstrated in *Nature* that multilayer networks trained by this procedure learn useful internal representations, and this paper is the usual point of reference for the connectionist revival described in [HIST-008]. LeCun et al. (1989) applied the algorithm to a convolutional network for handwritten digit recognition (see [TECH-CNN-001]).

**Difficulties in depth and time.** Bengio, Simard and Frasconi (1994) analyzed why gradient-based learning of long-term dependencies is difficult, identifying exponentially vanishing or growing gradient contributions. This motivated gated recurrent units such as the LSTM (Hochreiter and Schmidhuber, 1997; see [TECH-RNN-001]) and, later, residual connections (He et al., 2015).

**Software.** From the 2010s, deep learning frameworks implement reverse-mode AD generically, so that practitioners define the forward computation and obtain gradients automatically. Baydin et al. (2018) describe the "dynamic computational graph" and "differentiable programming" approaches in this setting. Schmidhuber (2015) provides a broader historical survey of the field.

## 4. Technical / Conceptual Explanation

### 4.1 Setup
Consider a feedforward network with layers $\ell = 1, \dots, L$. Let $a^{0} = x$ be the input. For each layer,

$$ z^{\ell} = W^{\ell} a^{\ell-1} + b^{\ell}, \qquad a^{\ell} = \phi_\ell(z^{\ell}), $$

and let the loss for a single example be $\mathcal{L} = \ell\big(a^{L}, y\big)$. The goal is $\partial \mathcal{L} / \partial W^{\ell}$ and $\partial \mathcal{L} / \partial b^{\ell}$ for every $\ell$. (Batches are handled by averaging; the derivation below is for one example.)

### 4.2 The error signal
Define the **error signal** (sensitivity) of layer $\ell$ as the gradient of the loss with respect to the pre-activation:

$$ \delta^{\ell} \;=\; \frac{\partial \mathcal{L}}{\partial z^{\ell}} \in \mathbb{R}^{n_\ell}. $$

By the chain rule, with $\odot$ denoting elementwise multiplication and $\phi'$ the elementwise derivative:

$$ \delta^{L} = \nabla_{a^{L}} \mathcal{L} \;\odot\; \phi_L'(z^{L}), $$

$$ \delta^{\ell} = \Big( (W^{\ell+1})^\top \delta^{\ell+1} \Big) \odot \phi_\ell'(z^{\ell}), \qquad \ell = L-1, \dots, 1. $$

The second equation is the backward recursion: the error at layer $\ell$ is the error at layer $\ell+1$ propagated through the transposed weights and then gated by the local activation derivative. Once $\delta^{\ell}$ is known, the parameter gradients follow directly, since $z^{\ell}$ depends on $W^{\ell}$ and $b^{\ell}$ linearly:

$$ \frac{\partial \mathcal{L}}{\partial W^{\ell}} = \delta^{\ell} \,(a^{\ell-1})^\top, \qquad \frac{\partial \mathcal{L}}{\partial b^{\ell}} = \delta^{\ell}. $$

### 4.3 The algorithm
1. **Forward pass.** Compute and store $z^{\ell}$ and $a^{\ell}$ for all layers, then evaluate the loss.
2. **Output error.** Compute $\delta^{L}$.
3. **Backward pass.** For $\ell = L-1$ down to $1$, compute $\delta^{\ell}$ from $\delta^{\ell+1}$.
4. **Gradients.** Form $\partial\mathcal{L}/\partial W^{\ell}$ and $\partial\mathcal{L}/\partial b^{\ell}$ for all $\ell$.
5. **Update (outside backpropagation).** Apply an optimizer step, for example $\theta \leftarrow \theta - \eta \nabla_\theta \mathcal{L}$.

The stored forward quantities are needed in the backward pass, so memory grows with depth and batch size. The activation-memory cost is a principal constraint in large-model training (see [TECH-DIST-001]).

### 4.4 Computational graph view and reverse-mode AD
Backpropagation generalizes to any computation expressed as a directed acyclic graph of differentiable operations. For each node $v$ with output $u_v$, define the **adjoint** $\bar u_v = \partial \mathcal{L} / \partial u_v$. If $u_v$ feeds into several downstream nodes $w \in \mathrm{succ}(v)$, then

$$ \bar u_v = \sum_{w \in \mathrm{succ}(v)} \bar u_w \,\frac{\partial u_w}{\partial u_v}. $$

Adjoints are computed in reverse topological order, starting from $\bar{\mathcal{L}} = 1$. Each operation only needs to supply a rule for multiplying an incoming adjoint by its local Jacobian (a vector-Jacobian product); the full Jacobian is never formed. This is reverse-mode AD (Baydin et al., 2018).

**Why reverse rather than forward mode.** Forward-mode AD propagates derivatives with respect to one input direction per sweep. For a function from $\mathbb{R}^{P}$ to a scalar, obtaining the full gradient takes $P$ forward-mode sweeps but one reverse-mode sweep. Since neural network losses are scalar functions of many parameters, reverse mode is the efficient choice. Forward mode is preferable when a function has few inputs and many outputs (Baydin et al., 2018).

### 4.5 Gradient flow through depth and time
Unrolling the recursion shows that the error signal at layer $\ell$ contains a product of per-layer Jacobians,

$$ \delta^{\ell} = D_\ell \,(W^{\ell+1})^\top D_{\ell+1}\,(W^{\ell+2})^\top \cdots D_{L-1}\,(W^{L})^\top\, \delta^{L}, \qquad D_k = \mathrm{diag}\big(\phi_k'(z^{k})\big), $$

where $D_k$ holds the activation derivatives of layer $k$ on its diagonal. If the norms of these factors are consistently below one, gradients shrink exponentially with depth (**vanishing**); if above one, they grow (**exploding**). The same product structure arises in recurrent networks across time steps, which is the case analyzed by Bengio et al. (1994) and, for clipping and regularization remedies, by Pascanu, Mikolov and Bengio (2012). Common mitigations include non-saturating activations, careful initialization, normalization (Ioffe and Szegedy, 2015), gated architectures, residual connections (He et al., 2015), and gradient clipping.

### 4.6 Backpropagation through time
For a recurrent network unrolled over $T$ steps with shared weights, backpropagation applied to the unrolled graph is called **backpropagation through time** (BPTT). Gradients for shared weights are summed over steps. Because memory and compute scale with $T$, truncated BPTT limits the backward window. See [TECH-RNN-001].

### 4.7 Checkpointing and the memory-compute trade
Activation storage can be reduced by saving only a subset of activations and recomputing the rest during the backward pass. This trades additional forward computation for lower memory. The technique is widely used in large-model training, and its details belong to [TECH-DIST-001].

## 5. Types, Components & Classifications
- **Reverse-mode AD (standard backpropagation).** One backward sweep per scalar output; memory proportional to stored intermediates.
- **Forward-mode AD.** One sweep per input direction. Efficient for few inputs and used for Jacobian-vector products.
- **Backpropagation through time and truncated variants.** For recurrent models.
- **Second-order information.** Hessian-vector products can be computed with AD-based techniques and underlie some optimization methods (Baydin et al., 2018). Full second-order methods are rarely used at large scale.
- **Local learning and backprop alternatives.** Several proposals compute weight updates without a global backward pass, including forward-only and local-learning rules, motivated by hardware efficiency or biological plausibility. [VERIFICATION NEEDED: no specific alternative-algorithm papers were confirmed for this draft, so none are named.]
- **Stochastic estimators for non-differentiable steps.** Where a computation includes sampling, gradients require either the reparameterization trick (see [TECH-VAE-001]) or score-function estimators.

## 6. Applications & Concrete Examples
- **Supervised training of classifiers and regressors.** The algorithm in section 4.3 is the standard training loop of deep networks (LeCun, Bengio and Hinton, 2015).
- **Convolutional networks.** The convolution operation is linear, so its backward pass is also a convolution-like operation, and weight sharing means gradients are summed over positions (LeCun et al., 1989; see [TECH-CNN-001]).
- **Sequence models.** BPTT trains LSTM and GRU networks ([TECH-RNN-001]); attention-based networks are trained by backpropagating through the attention weights ([TECH-ATTN-001]).
- **Generative models.** Variational autoencoders rely on a differentiable sampling formulation so that gradients pass through a stochastic node ([TECH-VAE-001]); adversarial training backpropagates through a discriminator into a generator ([TECH-GAN-001]).
- **Deep learning frameworks.** Automatic differentiation libraries apply reverse-mode AD to arbitrary programs built from differentiable primitives (Baydin et al., 2018).

## 7. Advantages, Limitations & Unresolved Questions

### Advantages
- **Efficiency.** One backward sweep produces all parameter gradients at a cost of the same order as the forward pass (Baydin et al., 2018).
- **Exactness.** Backpropagation yields the exact gradient of the computed function (up to floating-point error), unlike finite-difference estimates.
- **Modularity.** Any differentiable operation that supplies a vector-Jacobian product can be inserted into a network.

### Limitations
- **Memory.** Stored activations scale with depth and batch size.
- **Gradient pathologies.** Vanishing and exploding gradients (Bengio et al., 1994; Pascanu et al., 2012).
- **Local information only.** Gradients describe the loss locally. They do not by themselves guarantee convergence to a good solution on a non-convex loss.
- **Sequential dependency.** The backward pass cannot begin until the forward pass has finished, which constrains some forms of parallelism (see [TECH-DIST-001]).
- **Differentiability requirement.** Discrete or non-differentiable operations need relaxations or estimators.

### Unresolved questions
1. **Biological plausibility.** Whether the brain implements anything equivalent to backpropagation, given requirements such as symmetric forward and backward weights, is an open question in neuroscience and machine learning. [VERIFICATION NEEDED: no specific primary source on this debate was confirmed; the statement is a description of an open question, not a cited finding.]
2. **Why gradient descent with backpropagation finds solutions that generalize.** See [TECH-NN-001], section 7.
3. **Efficient alternatives at scale.** Whether training methods that avoid a global backward pass can match backpropagation-based training at frontier scale is undetermined.

## 8. Current State of Knowledge
The mathematics of backpropagation as exact reverse-mode differentiation is Established and is covered in standard texts and in the AD literature (Baydin et al., 2018). Its practical use is universal in gradient-trained deep networks. Open research concerns the surrounding system: how to hold activation memory in check, how to stabilize gradient flow in very deep or long-context models, and whether credit assignment can be done more cheaply or more locally. As of 2026-10-15, no alternative has been confirmed in the sources reviewed for this draft to replace backpropagation in frontier-scale training. This is a statement about the sources reviewed, not a survey of all current research.

## 9. Future Trajectory & Strategic Outlook
This section describes plausible directions and does not register forecasts.

- **Near term (0–3 years).** Continued engineering of memory-saving backward passes, kernel fusion and recomputation strategies, tied closely to accelerator characteristics (see [TECH-HW-001]).
- **Medium term (3–10 years).** Research on local or forward-only learning rules may be driven by hardware energy constraints. Whether any such method becomes competitive at scale is an open empirical question.
- **Long term (10+ years).** Whether continual or lifelong learning systems (see [AGI-CL-001]) will continue to use backpropagation over stored data, or move to online credit-assignment mechanisms, is unresolved.

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[TECH-NN-001] Artificial Neural Networks](./TECH-NN-001_artificial_neural_networks.md): the model class whose parameters are differentiated.
  - [[FOUND-010] Optimization and Gradient Descent](../vol01_foundations/FOUND-010_optimization_and_gradient_descent.md): the update rules that consume backpropagated gradients.
  - [[TECH-ML-001] Machine Learning](./TECH-ML-001_machine_learning.md): the empirical risk minimization setting.
- **Related Methods & Technologies**:
  - [[TECH-DL-001] Deep Learning](./TECH-DL-001_deep_learning.md): the broader practice built on this algorithm.
  - [[TECH-RNN-001] Recurrent Networks and LSTM](./TECH-RNN-001_recurrent_networks_and_lstm.md): backpropagation through time and gradient pathologies.
  - [[TECH-CNN-001] Convolutional Neural Networks](./TECH-CNN-001_convolutional_neural_networks.md): early application of backpropagation to a structured network.
  - [[TECH-VAE-001] Autoencoders and Variational Autoencoders](./TECH-VAE-001_autoencoders_and_variational_autoencoders.md): gradient estimation through stochastic nodes.
  - [[TECH-DIST-001] Distributed Training](./TECH-DIST-001_distributed_training.md): memory and parallelism constraints of the backward pass.
  - [[TECH-HW-001] AI Hardware & Accelerator Systems](./TECH-HW-001_ai_hardware.md): hardware efficiency of forward and backward passes.
- **Advanced Implications & Future Trajectories**:
  - [[HIST-008] Backpropagation and the Connectionist Revival](../vol02_history/HIST-008_backpropagation_and_the_connectionist_revival.md): historical account.
  - [[TECH-ARCH-001] The Transformer Architecture](./TECH-ARCH-001_transformer_architecture.md): a model family trained by backpropagation at very large scale.
  - [[AGI-CL-001] Continual Learning and Persistent Memory](../vol05_agi/AGI-CL-001_continual_learning_and_persistent_memory.md): open problems in learning without forgetting.

## 11. References & Foundational Literature
1. Werbos, P. J. ([VERIFICATION NEEDED: year; commonly given as 1982, but the Crossref record has no date]). Applications of advances in nonlinear sensitivity analysis. In *System Modeling and Optimization*, Lecture Notes in Control and Information Sciences, pp. 762–770. Springer. https://doi.org/10.1007/BFb0006203. [Supports an early description of using backward sensitivity calculations for nonlinear models. Title, author, series and pages confirmed via Crossref.]
2. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. *Nature*, 323(6088), 533–536. https://doi.org/10.1038/323533a0. [Supports the demonstration that backpropagation-trained multilayer networks learn internal representations.]
3. LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989). Backpropagation applied to handwritten zip code recognition. *Neural Computation*, 1(4), 541–551. https://doi.org/10.1162/neco.1989.1.4.541. [Supports application to a convolutional network.]
4. Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. *IEEE Transactions on Neural Networks*, 5(2), 157–166. https://doi.org/10.1109/72.279181. [Supports the analysis of vanishing and exploding gradients.]
5. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. *Neural Computation*, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735. [Supports gated memory cells as a response to gradient decay.]
6. Pascanu, R., Mikolov, T., & Bengio, Y. (2012). On the difficulty of training recurrent neural networks. arXiv:1211.5063. https://arxiv.org/abs/1211.5063. [Supports analysis of vanishing and exploding gradients and gradient-clipping style remedies in recurrent networks.]
7. Schmidhuber, J. (2015). Deep learning in neural networks: An overview. *Neural Networks*, 61, 85–117. https://doi.org/10.1016/j.neunet.2014.09.003. [Supports the broader historical survey of the field.]
8. LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. *Nature*, 521(7553), 436–444. https://doi.org/10.1038/nature14539. [Supports the role of backpropagation in training deep networks.]
9. Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167. https://arxiv.org/abs/1502.03167. [Supports normalization as a training-stabilization technique.]
10. He, K., Zhang, X., Ren, S., & Sun, J. (2015). Deep residual learning for image recognition. arXiv:1512.03385. https://arxiv.org/abs/1512.03385. [Supports residual connections for training very deep networks.]
11. Baydin, A. G., Pearlmutter, B. A., Radul, A. A., & Siskind, J. M. (2018). Automatic differentiation in machine learning: a survey. arXiv:1502.05767 (first submitted 2015; the 2018 date is the arXiv online date for the latest revision). https://arxiv.org/abs/1502.05767. [Supports the characterization of backpropagation as reverse-mode automatic differentiation and the forward versus reverse comparison. The arXiv abstract page lists four authors; the journal venue was not confirmed for this draft.]

Note: references 2–5, 7 and 8 were confirmed against Crossref (title, authors, venue, year); reference 1 was confirmed against Crossref except for the publication year, which the record lacks; references 6 and 9–11 were confirmed against arXiv abstract pages (title, authors, date).

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Layerwise gradient recursion (sections 4.1–4.3) | Established (mathematical derivation) | Rumelhart et al. (1986); Baydin et al. (2018) | High |
| Backpropagation as a case of reverse-mode AD | Established | Baydin et al. (2018) | High |
| Gradient cost of the same order as a forward pass | Established | Baydin et al. (2018) | High |
| Reverse mode preferred for scalar loss with many parameters | Established | Baydin et al. (2018) | High |
| Vanishing and exploding gradients in depth and time | Established | Bengio et al. (1994); Pascanu et al. (2012) | High |
| Residual and normalization remedies | Well Supported | He et al. (2015); Ioffe & Szegedy (2015) | Moderate to High |
| Priority of invention of reverse-mode AD for networks | Contested (historiographic) | Schmidhuber (2015); Werbos (1982) | Low; attribution unverified in this draft |
| Biological plausibility of backpropagation | Contested | No source confirmed in this draft | Open |
| Competitive backprop-free training at frontier scale | Hypothetical | None confirmed | Open |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.