---
id: TECH-NN-001
title: "Artificial Neural Networks"
alternative_names:
  - "Neural Networks"
  - "Connectionist Models"
  - "Multilayer Perceptrons (as the canonical feedforward case)"
category: "Technical → Neural Networks"
volume: III
chapter: 18
entry_type: "Standard Article"
level: 3
status: "Draft"
last_reviewed: 2026-10-15
version: "0.1.0"
prerequisites:
  - "TECH-ML-001"
related_entries:
  - "TECH-DL-001"
  - "TECH-ARCH-001"
  - "TECH-BACKPROP-001"
  - "TECH-CNN-001"
  - "TECH-RNN-001"
  - "TECH-GAN-001"
  - "TECH-VAE-001"
  - "FOUND-010"
---

# Artificial Neural Networks

## 1. Concise Definition
An **artificial neural network** (ANN) is a parameterized function built by composing many simple units, each of which computes a weighted sum of its inputs and passes the result through a nonlinear activation function. The weights are set by fitting the network to data, almost always by gradient-based optimization of a loss function. Neural networks are the model class underlying deep learning and, through it, most current large-scale AI systems.

## 2. Overview & Significance
Neural networks address a specific problem in machine learning (see [TECH-ML-001]): learning a mapping from inputs to outputs when the useful intermediate features are not known in advance. A linear model can only combine the raw inputs it is given. A neural network with at least one hidden layer learns its own intermediate representations, and deeper networks can stack them into a hierarchy.

Three properties explain the model class's role in modern AI. First, it is expressive: with a suitable nonlinearity, even shallow networks can approximate broad classes of functions arbitrarily well (Cybenko, 1989; Hornik et al., 1989). Second, it is trainable at scale: the whole model is differentiable, so one algorithm, backpropagation (see [TECH-BACKPROP-001]), supplies gradients for every parameter. Third, it is modular: the same units can be arranged with different connectivity patterns to encode assumptions about the data, producing convolutional networks, recurrent networks, attention-based networks and others.

This article describes the shared foundation. Specific families are treated in their own entries, and the dominant current family is covered in [TECH-ARCH-001].

## 3. Historical Development
**Origins.** McCulloch and Pitts (1943) proposed a logical model of the neuron as a threshold unit and showed that networks of such units could in principle compute logical propositions. This was a formal model; it did not include learning. Rosenblatt (1958) introduced the **perceptron**, a single-layer threshold classifier with an error-driven learning rule, and presented it as a probabilistic model for information storage and organization in the brain.

**Limits of single-layer models and the multilayer revival.** Single-layer perceptrons can only separate classes that are linearly separable, a limitation that Minsky and Papert analyzed in their 1969 monograph *Perceptrons*. [VERIFICATION NEEDED: bibliographic details of Minsky and Papert (1969) were not checked against a publisher or library record for this draft, so the book is named in the text but not listed in section 11.] The period of reduced funding and interest that followed is discussed in the history volume (see [HIST-004]). Interest returned in the 1980s with the popularization of multilayer networks trained by backpropagation (Rumelhart, Hinton and Williams, 1986), covered in [HIST-008] and [TECH-BACKPROP-001].

**Specialized architectures.** Fukushima (1980) described the neocognitron, an early hierarchical network inspired by the visual cortex studies of Hubel and Wiesel (1962). LeCun et al. (1989) trained a convolutional network with backpropagation to recognize handwritten zip code digits. Hochreiter and Schmidhuber (1997) introduced long short-term memory for sequences. These lines are described in [TECH-CNN-001] and [TECH-RNN-001].

**Deep learning period.** Surveys of the field (Schmidhuber, 2015; LeCun, Bengio and Hinton, 2015) place the shift to deep networks in the late 2000s and early 2010s, driven by larger datasets, GPU computation, and improved training practice. The AlexNet result on the ImageNet benchmark (Krizhevsky, Sutskever and Hinton, 2017, journal version of the 2012 paper) is usually treated as the visible turning point. See [TECH-DL-001] and [HIST-005].

## 4. Technical / Conceptual Explanation

### 4.1 The unit and the layer
A single unit with input vector $x \in \mathbb{R}^n$, weight vector $w \in \mathbb{R}^n$, and bias $b \in \mathbb{R}$ computes

$$ y = \phi(w^\top x + b), $$

where $\phi$ is an activation function. A **layer** applies many such units in parallel to the same input, which is compactly written with a weight matrix $W$:

$$ h = \phi(Wx + b). $$

A **feedforward network** (multilayer perceptron, MLP) with $L$ layers is the composition

$$ f_\theta(x) = \phi_L\big(W_L\, \phi_{L-1}( \cdots \phi_1(W_1 x + b_1) \cdots) + b_L\big), $$

with parameters $\theta = \{W_\ell, b_\ell\}_{\ell=1}^{L}$. Layers between the input and the output are called **hidden layers**.

### 4.2 Why the nonlinearity matters
If every $\phi_\ell$ were the identity, the composition would collapse to a single affine map $x \mapsto W' x + b'$, and depth would add nothing. The nonlinearity is what lets depth add expressive power. Common choices include the logistic sigmoid, the hyperbolic tangent, the rectified linear unit $\mathrm{ReLU}(z) = \max(0, z)$ and smooth variants such as GELU; for the output layer, a softmax converts scores to class probabilities.

### 4.3 What a network can represent
The **universal approximation theorems** state that a feedforward network with one hidden layer and a suitable nonconstant activation can approximate any continuous function on a compact set to arbitrary accuracy, provided the hidden layer is wide enough (Cybenko, 1989, for sigmoidal activations; Hornik, Stinchcombe and White, 1989, for more general squashing functions). Pinkus (1999) surveys the approximation theory of this model class, including the conditions on the activation. These are existence results. They do not say how many units are needed in practice, whether gradient descent will find the approximating weights, or how well the fitted function will generalize to new data.

### 4.4 Training
Given a dataset $\{(x_i, y_i)\}_{i=1}^{m}$ and a loss $\ell$, training seeks parameters that reduce the empirical risk

$$ \mathcal{L}(\theta) = \frac{1}{m}\sum_{i=1}^{m} \ell\big(f_\theta(x_i), y_i\big). $$

Typical losses are squared error for regression and cross-entropy for classification. Parameters are updated iteratively, for example by gradient descent,

$$ \theta \leftarrow \theta - \eta\, \nabla_\theta \mathcal{L}(\theta), $$

where $\eta$ is the learning rate; in practice the gradient is estimated on small random subsets of the data (minibatches), and adaptive variants of the update rule are common. Computing $\nabla_\theta \mathcal{L}$ for networks with millions or billions of parameters is made feasible by backpropagation. See [TECH-BACKPROP-001] for the algorithm and [FOUND-010] for optimization background.

### 4.5 Practical training components
Several techniques are routinely combined with the basic scheme. **Initialization** schemes set the scale of the starting weights so that signals neither vanish nor explode with depth. **Normalization** layers, such as batch normalization (Ioffe and Szegedy, 2015), stabilize and often speed training. **Residual connections** (He et al., 2015) add a layer's input to its output and made much deeper networks trainable. **Regularization**, including dropout (Hinton et al., 2012), reduces overfitting. Each of these is an engineering device whose benefits are demonstrated empirically; theoretical accounts of why they work are in several cases still disputed.

## 5. Types, Components & Classifications
Neural networks are usually classified by how units are connected.

| Family | Connectivity assumption | Typical data | Entry |
| :--- | :--- | :--- | :--- |
| Multilayer perceptron | Dense connections between consecutive layers; no structural prior | Tabular or fixed-size vectors | This article |
| Convolutional network | Local connectivity with weights shared across positions | Images, audio, grids | [TECH-CNN-001] |
| Recurrent network (RNN, LSTM, GRU) | Hidden state carried across time steps | Sequences | [TECH-RNN-001] |
| Attention-based network | Content-dependent weighting of other positions | Sequences, sets, patches | [TECH-ATTN-001], [TECH-ARCH-001] |
| Autoencoder and variational autoencoder | Encoder-decoder with a bottleneck or latent variable | Representation learning, generation | [TECH-VAE-001] |
| Generative adversarial network | Two networks trained against each other | Image and other generation | [TECH-GAN-001] |
| Graph neural network | Message passing along graph edges | Molecules, networks | [TECH-GNN-001] |

A second axis is the learning setting: supervised, self-supervised, unsupervised or reinforcement learning (see [TECH-RL-001]). The same architecture can be used under several settings.

## 6. Applications & Concrete Examples
The applications below are deployed or widely reproduced research results, not hypothetical capabilities.

- **Image recognition.** Convolutional networks became the standard approach to image classification after Krizhevsky et al. (2017) and were extended by residual networks (He et al., 2015).
- **Sequence modeling and translation.** Recurrent encoder-decoder networks produced the first neural systems competitive in machine translation (Sutskever, Vinyals and Le, 2014), later superseded in most settings by attention-based models.
- **Language and multimodal models.** Large language models are neural networks of the transformer family (see [TECH-LLM-001]).
- **Scientific prediction.** Neural networks are used in protein structure prediction and other scientific tasks (see [APP-SCI-001]).
- **Control.** Deep reinforcement learning uses neural networks as value and policy functions; Mnih et al. (2015) trained a convolutional network to play Atari games from pixels.

Specific deployment scale, accuracy and cost figures change quickly and are tracked in the Living Reference collections (`MODEL-001`, `FACT-001`) rather than here.

## 7. Advantages, Limitations & Unresolved Questions

### Demonstrated advantages
- **Representation learning.** Features are learned from data rather than hand-designed, which is the main reason neural networks displaced hand-engineered pipelines in perception and language tasks (LeCun, Bengio and Hinton, 2015).
- **Generality.** One training recipe applies across modalities and tasks.
- **Scalability.** Training parallelizes well on accelerator hardware (see [TECH-HW-001]).

### Limitations
- **Data and compute requirements.** Strong performance generally requires large datasets and substantial computation.
- **Opacity.** Learned representations are hard to interpret; the field of interpretability studies this (see [SAFE-INTERP-001]).
- **Brittleness and distribution shift.** Performance can degrade sharply on inputs unlike the training data, and networks can be sensitive to small adversarial perturbations. [VERIFICATION NEEDED: no primary source on adversarial examples was confirmed for this draft; the statement is made without citation.]
- **Optimization difficulty.** The loss surface is non-convex, and training can fail through vanishing or exploding gradients (Bengio, Simard and Frasconi, 1994).

### Unresolved questions
1. **Generalization.** Zhang et al. (2016) showed that standard networks can fit random labels, which challenges explanations of generalization based only on model capacity. Belkin et al. (2019) describe a "double descent" pattern in which test error can fall again beyond the point of fitting the training data. Why large networks generalize as well as they do is not settled.
2. **Role of architecture versus scale.** How much of recent progress is due to architectural choices rather than data and compute is debated, and the answer differs by task.
3. **Relationship to biological neurons.** Artificial units are loose abstractions. Claims that network training or structure resembles biological learning are contested and are not needed to justify the engineering use of the models.

## 8. Current State of Knowledge
As of 2026-10-15, neural networks are the dominant model class for perception, language and generative tasks. The theory of approximation is mature and is Established. The theory of optimization and generalization for large networks is incomplete: there are partial explanations and informative empirical phenomena (Zhang et al., 2017; Belkin et al., 2019), but no accepted general account. Training practice (initialization, normalization, residual connections, adaptive optimizers) is largely empirical and well replicated, which supports the rating "Well Supported" for the effectiveness of individual techniques but not necessarily for the stated reasons they work.

## 9. Future Trajectory & Strategic Outlook
This section describes observed trends and plausible directions. It does not register forecasts with probabilities.

- **Near term (0–3 years).** Continued dominance of transformer-style networks, with active work on efficiency, including sparsity and lower-precision arithmetic (see [TECH-INFER-001]).
- **Medium term (3–10 years).** Open questions include whether alternatives to dense attention, such as state-space and hybrid designs, will match transformers at scale (see [TECH-RNN-001]), and whether training can become substantially more data-efficient.
- **Long term (10+ years).** Whether neural networks alone are a sufficient substrate for general intelligence is a central question of the AGI debate (see [AGI-001], [AGI-PATH-001]); the question is open and is treated in Volume V.

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[TECH-ML-001] Machine Learning](./TECH-ML-001_machine_learning.md): learning from data and empirical risk minimization.
  - [[FOUND-010] Optimization and Gradient Descent](../vol01_foundations/FOUND-010_optimization_and_gradient_descent.md): the optimization methods used to train networks.
  - [[HIST-008] Backpropagation and the Connectionist Revival](../vol02_history/HIST-008_backpropagation_and_the_connectionist_revival.md): historical account of the 1980s revival.
- **Related Methods & Technologies**:
  - [[TECH-DL-001] Deep Learning](./TECH-DL-001_deep_learning.md): networks with many layers and the surrounding practice.
  - [[TECH-BACKPROP-001] Backpropagation](./TECH-BACKPROP-001_backpropagation.md): the gradient computation algorithm.
  - [[TECH-CNN-001] Convolutional Neural Networks](./TECH-CNN-001_convolutional_neural_networks.md): networks with spatial weight sharing.
  - [[TECH-RNN-001] Recurrent Networks and LSTM](./TECH-RNN-001_recurrent_networks_and_lstm.md): networks for sequences.
  - [[TECH-ATTN-001] Attention Mechanisms](./TECH-ATTN-001_attention_mechanisms.md): content-based weighting.
  - [[TECH-GAN-001] Generative Adversarial Networks](./TECH-GAN-001_generative_adversarial_networks.md): adversarial generative training.
  - [[TECH-VAE-001] Autoencoders and Variational Autoencoders](./TECH-VAE-001_autoencoders_and_variational_autoencoders.md): latent-variable generative models.
  - [[TECH-GNN-001] Graph Neural Networks](./TECH-GNN-001_graph_neural_networks.md): networks for graph-structured data.
  - [[TECH-RL-001] Reinforcement Learning](./TECH-RL-001_reinforcement_learning.md): networks as policies and value functions.
  - [[TECH-HW-001] AI Hardware & Accelerator Systems](./TECH-HW-001_ai_hardware.md): the hardware that makes training practical.
- **Advanced Implications & Future Trajectories**:
  - [[TECH-ARCH-001] The Transformer Architecture](./TECH-ARCH-001_transformer_architecture.md): the dominant current family.
  - [[SAFE-INTERP-001] Interpretability and Mechanistic Interpretability](../vol08_safety_governance/SAFE-INTERP-001_interpretability_and_mechanistic_interpretability.md): understanding what networks compute.
  - [[HIST-005] The Deep Learning Breakthrough](../vol02_history/HIST-005_deep_learning_breakthrough.md): the historical transition to deep networks.

## 11. References & Foundational Literature
1. McCulloch, W. S., & Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. *The Bulletin of Mathematical Biophysics*, 5(4), 115–133. https://doi.org/10.1007/BF02478259. [Supports the threshold-unit model of the neuron and its logical interpretation.]
2. Rosenblatt, F. (1958). The perceptron: A probabilistic model for information storage and organization in the brain. *Psychological Review*, 65(6), 386–408. https://doi.org/10.1037/h0042519. [Supports the perceptron and its error-driven learning rule.]
3. Hubel, D. H., & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. *The Journal of Physiology*, 160, 106–154. https://doi.org/10.1113/jphysiol.1962.sp006837. [Neurophysiological background cited as inspiration for hierarchical visual models.]
4. Fukushima, K. (1980). Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. *Biological Cybernetics*, 36(4), 193–202. https://doi.org/10.1007/BF00344251. [Supports the early hierarchical visual network.]
5. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. *Nature*, 323(6088), 533–536. https://doi.org/10.1038/323533a0. [Supports training multilayer networks with backpropagation.]
6. Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. *Mathematics of Control, Signals, and Systems*, 2(4), 303–314. https://doi.org/10.1007/BF02551274. [Supports universal approximation for one-hidden-layer sigmoidal networks.]
7. Hornik, K., Stinchcombe, M., & White, H. (1989). Multilayer feedforward networks are universal approximators. *Neural Networks*, 2(5), 359–366. https://doi.org/10.1016/0893-6080(89)90020-8. [Supports universal approximation for general squashing activations.]
8. LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989). Backpropagation applied to handwritten zip code recognition. *Neural Computation*, 1(4), 541–551. https://doi.org/10.1162/neco.1989.1.4.541. [Supports the early application of a convolutional network trained with backpropagation.]
9. Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. *IEEE Transactions on Neural Networks*, 5(2), 157–166. https://doi.org/10.1109/72.279181. [Supports the vanishing and exploding gradient difficulty.]
10. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. *Neural Computation*, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735. [Supports the introduction of LSTM.]
11. Pinkus, A. (1999). Approximation theory of the MLP model in neural networks. *Acta Numerica*, 8, 143–195. https://doi.org/10.1017/S0962492900002919. [Supports the survey of approximation results for multilayer perceptrons.]
12. Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012). Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580. https://arxiv.org/abs/1207.0580. [Supports dropout as a regularizer.]
13. Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. arXiv:1409.3215. https://arxiv.org/abs/1409.3215. [Supports recurrent encoder-decoder translation.]
14. Schmidhuber, J. (2015). Deep learning in neural networks: An overview. *Neural Networks*, 61, 85–117. https://doi.org/10.1016/j.neunet.2014.09.003. [Supports the historical survey of the field.]
15. LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. *Nature*, 521(7553), 436–444. https://doi.org/10.1038/nature14539. [Supports the account of representation learning and the deep learning period.]
16. Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167. https://arxiv.org/abs/1502.03167. [Supports batch normalization.]
17. Mnih, V., Kavukcuoglu, K., Silver, D., et al. (2015). Human-level control through deep reinforcement learning. *Nature*, 518(7540), 529–533. https://doi.org/10.1038/nature14236. [Supports deep networks as value functions for Atari play.]
18. He, K., Zhang, X., Ren, S., & Sun, J. (2015). Deep residual learning for image recognition. arXiv:1512.03385. https://arxiv.org/abs/1512.03385. [Supports residual connections.]
19. Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016). Understanding deep learning requires rethinking generalization. arXiv:1611.03530. https://arxiv.org/abs/1611.03530. [Supports the observation that networks can fit random labels.]
20. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017). ImageNet classification with deep convolutional neural networks. *Communications of the ACM*, 60(6), 84–90. https://doi.org/10.1145/3065386. [Supports the ImageNet convolutional network result; journal version of the 2012 conference paper.]
21. Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias–variance trade-off. *Proceedings of the National Academy of Sciences*, 116, 15849–15854. https://doi.org/10.1073/pnas.1903070116. [Supports the double descent description.]

Note: the in-text bracketed identifiers such as [TECH-BACKPROP-001] refer to encyclopedia entries listed in section 10, not to bibliography items. References 1–11, 14, 15, 17, 20 and 21 were confirmed against Crossref; references 12, 13, 16, 18 and 19 were confirmed against arXiv abstract pages (title, authors, date).

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Unit, layer and feedforward network definitions | Established | Rumelhart et al. (1986); LeCun et al. (2015) | High |
| Universal approximation by one-hidden-layer networks | Established (theorem) | Cybenko (1989); Hornik et al. (1989); Pinkus (1999) | High |
| Gradient-based training of the empirical risk | Established | Rumelhart et al. (1986) | High |
| Effectiveness of residual connections, normalization, dropout | Well Supported | He et al. (2015); Ioffe & Szegedy (2015); Hinton et al. (2012) | Moderate to High (mechanistic explanations vary) |
| Networks can fit random labels | Well Supported | Zhang et al. (2016) | High |
| Double descent as a general account of generalization | Contested | Belkin et al. (2019) | Moderate; scope of the phenomenon is debated |
| Vanishing and exploding gradients hinder long-range learning | Established | Bengio et al. (1994) | High |
| Neural networks as a sufficient route to general intelligence | Contested | See Volume V | Open |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.