---
id: TECH-RNN-001
title: "Recurrent Networks and LSTM"
alternative_names:
  - "Recurrent Neural Networks"
  - "RNN"
  - "Long Short-Term Memory"
  - "LSTM"
  - "Gated Recurrent Unit (GRU)"
category: "Technical → Neural Networks"
volume: III
chapter: 18
entry_type: "Standard Article"
level: 3
status: "Draft"
last_reviewed: 2026-10-15
version: "0.1.0"
prerequisites:
  - "TECH-NN-001"
related_entries:
  - "TECH-BACKPROP-001"
  - "TECH-ATTN-001"
  - "TECH-ARCH-001"
  - "TECH-CNN-001"
  - "TECH-LLM-001"
---

# Recurrent Networks and LSTM

## 1. Concise Definition
A **recurrent neural network** (RNN) processes a sequence one element at a time while carrying a **hidden state** from each step to the next, so that its output at a given step can depend on everything seen earlier in the sequence. **Long short-term memory** (LSTM) is a gated recurrent design that adds an explicitly controlled memory cell, introduced to make learning over long time lags feasible.

## 2. Overview & Significance
Many inputs are sequences of variable length: text, speech, sensor streams, time series. A feedforward network (see [TECH-NN-001]) needs a fixed-size input and treats positions independently. A recurrent network instead applies the same function at every time step and passes a state forward, which gives it two useful properties: weight sharing across time, and a memory that is, in principle, unbounded in the length of context it can use.

In practice, plain RNNs struggle to learn dependencies across many steps because the gradients involved shrink or grow exponentially (Bengio, Simard and Frasconi, 1994; see [TECH-BACKPROP-001]). Gated architectures, LSTM and later the GRU, mitigated this and were the standard tools for sequence tasks through the mid-2010s. They were then largely replaced in language modelling and translation by attention-based architectures (Vaswani et al., 2017; see [TECH-ARCH-001]), but recurrent ideas remain relevant, including in current efficient sequence models.

## 3. Historical Development
**Early recurrent models.** Elman (1990) described a simple recurrent network in which the previous hidden state is fed back as context, and showed that such a network could discover structure in temporal sequences. Training used backpropagation, extended through time (see [TECH-BACKPROP-001]).

**The long-lag problem.** Bengio, Simard and Frasconi (1994) analyzed why gradient-based learning of long-term dependencies is difficult. This result is the main motivation for the gated designs that followed.

**LSTM.** Hochreiter and Schmidhuber (1997) introduced the LSTM, whose central device is a memory cell with a self-connection of fixed weight (the "constant error carousel") guarded by multiplicative gates, so that error signals can flow across many steps without decaying. Gers, Schmidhuber and Cummins (2000) added a **forget gate**, allowing the cell to reset its content, and this form is the one commonly used in practice. Schuster and Paliwal (1997) proposed bidirectional recurrent networks, which read the sequence in both directions when the whole input is available.

**Neural machine translation.** Sutskever, Vinyals and Le (2014) used LSTM encoder and decoder networks for sequence-to-sequence translation. Cho et al. (2014) proposed an RNN encoder-decoder with a new hidden unit with reset and update gates, now known as the GRU. Bahdanau, Cho and Bengio (2014) added attention to this encoder-decoder, relieving the need to compress a whole source sentence into one vector (see [TECH-ATTN-001]).

**Displacement and revival.** Vaswani et al. (2017) showed that a model built only on attention, without recurrence, achieved strong translation results while being more parallelizable. Subsequent research has revisited recurrence in forms that are efficient to train and to run, including linear-attention formulations that can be written as RNNs (Katharopoulos et al., 2020) and selective state space models (Gu and Dao, 2023).

## 4. Technical / Conceptual Explanation

### 4.1 The basic recurrence
For an input sequence $x_1, \dots, x_T$, a simple RNN maintains a hidden state $h_t$:

$$ h_t = \phi\big(W_{hh}\, h_{t-1} + W_{xh}\, x_t + b_h\big), \qquad y_t = g\big(W_{hy}\, h_t + b_y\big), $$

with initial state $h_0$ (often zero), a nonlinearity $\phi$ (such as $\tanh$), and the **same** parameters $W_{hh}, W_{xh}, W_{hy}$ at every step. The loss is a sum of per-step losses or a loss on the final output.

### 4.2 Training and the gradient problem
Training uses backpropagation through time (BPTT): unroll the recurrence for $T$ steps and apply backpropagation to the resulting deep feedforward graph with tied weights. The gradient of a loss at step $T$ with respect to an earlier hidden state involves a product of Jacobians,

$$ \frac{\partial h_T}{\partial h_t} = \prod_{k=t+1}^{T} \frac{\partial h_k}{\partial h_{k-1}} = \prod_{k=t+1}^{T} \mathrm{diag}\big(\phi'(a_k)\big)\, W_{hh}, $$

where $a_k$ is the pre-activation at step $k$. When the relevant singular values are consistently below one, this product decays exponentially with the lag $T - t$; when above one, it can grow without bound (Bengio et al., 1994; Pascanu, Mikolov and Bengio, 2012). Gradient clipping addresses exploding gradients, while the vanishing case motivates architectural change.

### 4.3 LSTM
An LSTM keeps a cell state $c_t$ in addition to the hidden state $h_t$. With the forget gate of Gers et al. (2000), one common formulation is

$$ \begin{aligned} f_t &= \sigma(W_f [h_{t-1}, x_t] + b_f), \\ i_t &= \sigma(W_i [h_{t-1}, x_t] + b_i), \\ o_t &= \sigma(W_o [h_{t-1}, x_t] + b_o), \\ \tilde c_t &= \tanh(W_c [h_{t-1}, x_t] + b_c), \\ c_t &= f_t \odot c_{t-1} + i_t \odot \tilde c_t, \\ h_t &= o_t \odot \tanh(c_t), \end{aligned} $$

where $\sigma$ is the logistic sigmoid, $[\cdot,\cdot]$ denotes concatenation, and $\odot$ is elementwise multiplication. The forget gate $f_t$ controls how much of the old cell content is kept, the input gate $i_t$ controls how much new content is written, and the output gate $o_t$ controls how much of the cell is exposed. The cell update is additive, so the derivative of $c_t$ with respect to $c_{t-1}$ is the gate $f_t$ (ignoring indirect paths through the gates). Where $f_t$ is near one, gradient can travel through many steps without repeated squashing, which is the intuition behind the constant error carousel (Hochreiter and Schmidhuber, 1997). Exact formulations vary across papers and software libraries (for example, whether the gates also receive the cell state), so a specific implementation should be consulted when reproducing results.

### 4.4 GRU
The gated recurrent unit of Cho et al. (2014) merges the cell and hidden state and uses two gates, an update gate and a reset gate. It has fewer parameters than an LSTM of the same width. Reports comparing the two on specific tasks do not support a general ranking. [VERIFICATION NEEDED: no comparative study was confirmed for this draft; the statement reflects the absence of a confirmed source rather than a documented finding.]

### 4.5 Sequence-to-sequence and bidirectional models
In an encoder-decoder configuration, one RNN reads the input sequence into a state, and a second RNN generates the output sequence from that state (Sutskever et al., 2014; Cho et al., 2014). A single fixed-length vector can become a bottleneck for long inputs, which motivated attention (Bahdanau et al., 2014). Bidirectional RNNs run one network forward and one backward over the input and combine their states (Schuster and Paliwal, 1997); they are not suitable for step-by-step generation where the future is unavailable.

## 5. Types, Components & Classifications
- **Simple (Elman) RNN.** One hidden layer with feedback (Elman, 1990).
- **LSTM.** Gated cell with input, forget and output gates (Hochreiter and Schmidhuber, 1997; Gers et al., 2000).
- **GRU.** Two-gate simplified variant (Cho et al., 2014).
- **Bidirectional RNN.** Reads both directions (Schuster and Paliwal, 1997).
- **Stacked (deep) RNN.** Several recurrent layers, each taking the previous layer's states as input.
- **Encoder-decoder RNN.** For sequence-to-sequence mapping (Sutskever et al., 2014).
- **Linear-attention and state-space recurrences.** Modern constant-state sequence models. Katharopoulos et al. (2020) show that causal linear attention can be written as an RNN, and Gu and Dao (2023) propose a selective state space architecture with linear scaling in sequence length.

## 6. Applications & Concrete Examples
- **Machine translation.** LSTM-based sequence-to-sequence systems were an early neural approach competitive with phrase-based statistical systems (Sutskever et al., 2014; Cho et al., 2014 used an RNN encoder-decoder to score phrase pairs in a statistical translation system). Attention-based RNNs then improved results (Bahdanau et al., 2014).
- **Language modelling and generation.** Recurrent language models predicted the next token from the hidden state; transformer-based models have since largely taken over this role (see [TECH-LLM-001]).
- **Speech and time series.** Recurrent networks were widely applied to sequential signals. [VERIFICATION NEEDED: no specific deployment source was confirmed for speech or time-series applications.]
- **Reinforcement learning.** Recurrent policies handle partially observable environments by carrying a state (see [TECH-RL-001]).
- **Efficient sequence models.** State-space and linear-attention recurrences target long contexts with constant-size state (Gu and Dao, 2023; Katharopoulos et al., 2020).

Which production systems currently use recurrent components is not tracked here; see the Living Reference collection `MODEL-001`.

## 7. Advantages, Limitations & Unresolved Questions

### Advantages
- **Constant-size state.** Inference with a recurrent model needs memory that does not grow with the processed length, since the state has fixed size.
- **Natural handling of variable-length input** and streaming data.
- **Parameter sharing across time.**

### Limitations
- **Sequential computation.** Step $t$ depends on step $t-1$, so training cannot be parallelized over time, which is a main reason for the shift to attention-based models (Vaswani et al., 2017).
- **Long-range dependency.** Gating mitigates but does not remove the difficulty of carrying information over very long spans, and a fixed-size state is a bounded memory.
- **Bottleneck in encoder-decoder use.** Compressing an input into one vector limits fidelity for long inputs (Bahdanau et al., 2014).
- **Training sensitivity.** Exploding gradients require clipping or careful design (Pascanu et al., 2012).

### Unresolved questions
1. **Can fixed-state recurrent models match attention at scale?** Selective state-space and linear-attention models are active candidates. Gu and Dao (2023) report favourable results for their model in the settings they test, but independent comparisons across training regimes are not covered by the sources confirmed for this draft.
2. **How much context can a fixed-size state faithfully retain?** The trade-off between state size and recall is an active research topic. [VERIFICATION NEEDED: no source on state capacity limits was confirmed.]

## 8. Current State of Knowledge
As of 2026-10-15, attention-based transformers dominate large-scale language and translation systems, and LSTMs and GRUs are no longer the default for those tasks. The mathematics of gated recurrence and of gradient decay is Established. That LSTMs improved long-lag learning over simple RNNs on the test problems studied is Well Supported by the original experiments and later usage (Hochreiter and Schmidhuber, 1997). The competitiveness of recurrent-style sequence models at frontier scale is Preliminary to Contested; it depends on results of individual groups (Gu and Dao, 2023) that await systematic independent comparison.

## 9. Future Trajectory & Strategic Outlook
This section describes plausible directions and does not register forecasts.

- **Near term (0–3 years).** Hybrid models combining attention layers with recurrent or state-space layers, motivated by inference cost for long contexts, are an active direction.
- **Medium term (3–10 years).** Whether constant-state models close the quality gap with attention on recall-heavy tasks is an empirical question.
- **Long term (10+ years).** Persistent, updateable memory for agents (see [AGI-CL-001]) may revisit recurrent state as one mechanism among several; this is speculative.

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[TECH-NN-001] Artificial Neural Networks](./TECH-NN-001_artificial_neural_networks.md): units, layers and training.
  - [[TECH-BACKPROP-001] Backpropagation](./TECH-BACKPROP-001_backpropagation.md): backpropagation through time and gradient decay.
- **Related Methods & Technologies**:
  - [[TECH-ATTN-001] Attention Mechanisms](./TECH-ATTN-001_attention_mechanisms.md): attention introduced on top of RNN encoder-decoders.
  - [[TECH-CNN-001] Convolutional Neural Networks](./TECH-CNN-001_convolutional_neural_networks.md): the other structured family; convolutions as an alternative for sequences.
  - [[TECH-RL-001] Reinforcement Learning](./TECH-RL-001_reinforcement_learning.md): recurrent policies for partial observability.
  - [[TECH-DL-001] Deep Learning](./TECH-DL-001_deep_learning.md): broader context.
- **Advanced Implications & Future Trajectories**:
  - [[TECH-ARCH-001] The Transformer Architecture](./TECH-ARCH-001_transformer_architecture.md): the architecture that displaced recurrence in language modelling.
  - [[TECH-LLM-001] Large Language Models](./TECH-LLM-001_large_language_models.md): current use of autoregressive sequence models.
  - [[AGI-CL-001] Continual Learning and Persistent Memory](../vol05_agi/AGI-CL-001_continual_learning_and_persistent_memory.md): memory beyond a context window.

## 11. References & Foundational Literature
1. Elman, J. L. (1990). Finding structure in time. *Cognitive Science*, 14(2), 179–211. https://doi.org/10.1207/s15516709cog1402_1. [Supports the simple recurrent network and its discovery of temporal structure.]
2. Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. *IEEE Transactions on Neural Networks*, 5(2), 157–166. https://doi.org/10.1109/72.279181. [Supports the analysis of vanishing and exploding gradients over long lags.]
3. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. *Neural Computation*, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735. [Supports the LSTM and the constant error carousel.]
4. Schuster, M., & Paliwal, K. K. (1997). Bidirectional recurrent neural networks. *IEEE Transactions on Signal Processing*, 45(11), 2673–2681. https://doi.org/10.1109/78.650093. [Supports bidirectional recurrent networks.]
5. Gers, F. A., Schmidhuber, J., & Cummins, F. (2000). Learning to forget: Continual prediction with LSTM. *Neural Computation*, 12(10), 2451–2471. https://doi.org/10.1162/089976600300015015. [Supports the forget gate.]
6. Pascanu, R., Mikolov, T., & Bengio, Y. (2012). On the difficulty of training recurrent neural networks. arXiv:1211.5063. https://arxiv.org/abs/1211.5063. [Supports the analysis of gradient problems and clipping in recurrent networks.]
7. Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv:1406.1078. https://arxiv.org/abs/1406.1078. [Supports the RNN encoder-decoder and the reset/update-gate hidden unit (GRU).]
8. Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. arXiv:1409.3215. https://arxiv.org/abs/1409.3215. [Supports LSTM encoder-decoder translation.]
9. Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv:1409.0473. https://arxiv.org/abs/1409.0473. [Supports attention added to an RNN encoder-decoder.]
10. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. arXiv:1706.03762. https://arxiv.org/abs/1706.03762. [Supports an attention-only model without recurrence.]
11. Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F. (2020). Transformers are RNNs: Fast autoregressive transformers with linear attention. arXiv:2006.16236. https://arxiv.org/abs/2006.16236. [Supports the formulation of causal linear attention as an RNN.]
12. Gu, A., & Dao, T. (2023). Mamba: Linear-time sequence modeling with selective state spaces. arXiv:2312.00752. https://arxiv.org/abs/2312.00752. [Supports the selective state-space architecture with linear scaling in sequence length.]

Note: references 1–5 were confirmed against Crossref; references 6–12 were confirmed against arXiv abstract pages (title, authors, date). Descriptions of the contents of the Cho et al., Sutskever et al., Katharopoulos et al. and Gu and Dao papers are limited to what their titles and abstracts state.

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Recurrence equations and BPTT | Established | Elman (1990); Bengio et al. (1994) | High |
| Gradient decay/growth over long lags | Established | Bengio et al. (1994); Pascanu et al. (2012) | High |
| LSTM gating equations | Established (definition; variants exist) | Hochreiter & Schmidhuber (1997); Gers et al. (2000) | High |
| LSTM improves long-lag learning over simple RNNs | Well Supported | Hochreiter & Schmidhuber (1997) | Moderate to High |
| GRU introduced for RNN encoder-decoder | Established (historical) | Cho et al. (2014) | High |
| Attention-only models replaced recurrence in most large-scale language and translation systems | Well Supported | Vaswani et al. (2017) | Moderate (adoption claim not tied to a survey) |
| Linear attention can be formulated as an RNN | Established (derivation) | Katharopoulos et al. (2020) | High |
| Selective state-space models are competitive with transformers at scale | Preliminary / Contested | Gu & Dao (2023) | Moderate; awaits independent comparison |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.