Recurrent Networks and LSTM

A recurrent neural network (RNN) processes a sequence one element at a time while carrying a hidden state from each step to the next, so that its output at a given step can depend on everything seen earlier in the sequence. Long short-term memory (LSTM) is a gated recurrent design that adds an explicitly controlled memory cell, introduced to make learning over long time lags feasible.

Many inputs are sequences of variable length: text, speech, sensor streams, time series. A feedforward network (see [TECH-NN-001]) needs a fixed-size input and treats positions independently. A recurrent network instead applies the same function at every time step and passes a state forward, which gives it two useful properties: weight sharing across time, and a memory that is, in principle, unbounded in the length of context it can use.

In practice, plain RNNs struggle to learn dependencies across many steps because the gradients involved shrink or grow exponentially (Bengio, Simard and Frasconi, 1994; see [TECH-BACKPROP-001]). Gated architectures, LSTM and later the GRU, mitigated this and were the standard tools for sequence tasks through the mid-2010s. They were then largely replaced in language modelling and translation by attention-based architectures (Vaswani et al., 2017; see [TECH-ARCH-001]), but recurrent ideas remain relevant, including in current efficient sequence models.

Early recurrent models. Elman (1990) described a simple recurrent network in which the previous hidden state is fed back as context, and showed that such a network could discover structure in temporal sequences. Recurrent networks in general are commonly trained by backpropagation through time, in which the recurrence is unrolled over the sequence (see [TECH-BACKPROP-001]); the training procedure of Elman's original network is not described here.

The long-lag problem. Bengio, Simard and Frasconi (1994) analyzed why gradient-based learning of long-term dependencies is difficult. This result is the main motivation for the gated designs that followed.

LSTM. Hochreiter and Schmidhuber (1997) introduced the LSTM, whose central device is a memory cell with a self-connection of fixed weight (the "constant error carousel") guarded by multiplicative gates, so that error signals can flow across many steps without decaying. Gers, Schmidhuber and Cummins (2000) added a forget gate, allowing the cell to reset its content, and this form is the one commonly used in practice. Schuster and Paliwal (1997) proposed bidirectional recurrent networks, which read the sequence in both directions when the whole input is available.

Neural machine translation. Sutskever, Vinyals and Le (2014) used LSTM encoder and decoder networks for sequence-to-sequence translation. Cho et al. (2014) proposed an RNN encoder-decoder with a new hidden unit with reset and update gates, now known as the GRU. Bahdanau, Cho and Bengio (2014) added attention to this encoder-decoder, relieving the need to compress a whole source sentence into one vector (see [TECH-ATTN-001]).

Displacement and revival. Vaswani et al. (2017) showed that a model built only on attention, without recurrence, achieved strong translation results while being more parallelizable. Subsequent research has revisited recurrence in forms that are efficient to train and to run, including linear-attention formulations that can be written as RNNs (Katharopoulos et al., 2020) and selective state space models (Gu and Dao, 2023).

#4.1 The basic recurrence

For an input sequence $x_1, \dots, x_T$, a simple RNN maintains a hidden state $h_t$:

with initial state $h_0$ (often zero), a nonlinearity $\phi$ (such as $\tanh$), and the same parameters $W_{hh}, W_{xh}, W_{hy}$ at every step. The loss is a sum of per-step losses or a loss on the final output.

#4.2 Training and the gradient problem

Training uses backpropagation through time (BPTT): unroll the recurrence for $T$ steps and apply backpropagation to the resulting deep feedforward graph with tied weights. The gradient of a loss at step $T$ with respect to an earlier hidden state involves a product of Jacobians,

where $a_k$ is the pre-activation at step $k$. When the norms of the Jacobian factors are consistently below one (for example, the largest singular value of $W_{hh}$ is small enough that the product is contracting), this product decays exponentially with the lag $T - t$; when the factors are large enough, it can grow without bound (Bengio et al., 1994; Pascanu, Mikolov and Bengio, 2012). Gradient clipping addresses exploding gradients, while the vanishing case motivates architectural change.

#4.3 LSTM

An LSTM keeps a cell state $c_t$ in addition to the hidden state $h_t$. With the forget gate of Gers et al. (2000), one common formulation is

where $\sigma$ is the logistic sigmoid, $[\cdot,\cdot]$ denotes concatenation, and $\odot$ is elementwise multiplication. The forget gate $f_t$ controls how much of the old cell content is kept, the input gate $i_t$ controls how much new content is written, and the output gate $o_t$ controls how much of the cell is exposed. The cell update is additive, so the derivative of $c_t$ with respect to $c_{t-1}$ is the gate $f_t$ (ignoring indirect paths through the gates). Where $f_t$ is near one, gradient can travel through many steps without repeated squashing, which is the intuition behind the constant error carousel (Hochreiter and Schmidhuber, 1997). Exact formulations vary across papers and software libraries (for example, whether the gates also receive the cell state), so a specific implementation should be consulted when reproducing results.

#4.4 GRU

The gated recurrent unit of Cho et al. (2014) merges the cell and hidden state and uses two gates, an update gate and a reset gate. It has fewer parameters than an LSTM of the same width. Chung et al. (2014) compared gated units on polyphonic music and raw speech signal modelling and report that the GRU is comparable to the LSTM, and that both outperform plain tanh units; this is limited to the tasks they tested and does not establish a general ranking.

#4.5 Sequence-to-sequence and bidirectional models

In an encoder-decoder configuration, one RNN reads the input sequence into a state, and a second RNN generates the output sequence from that state (Sutskever et al., 2014; Cho et al., 2014). A single fixed-length vector can become a bottleneck for long inputs, which motivated attention (Bahdanau et al., 2014). Bidirectional RNNs run one network forward and one backward over the input and combine their states (Schuster and Paliwal, 1997); they are not suitable for step-by-step generation where the future is unavailable.

• Simple (Elman) RNN. One hidden layer with feedback (Elman, 1990).
• LSTM. Gated cell with input, forget and output gates (Hochreiter and Schmidhuber, 1997; Gers et al., 2000).
• GRU. Two-gate simplified variant (Cho et al., 2014).
• Bidirectional RNN. Reads both directions (Schuster and Paliwal, 1997).
• Stacked (deep) RNN. Several recurrent layers, each taking the previous layer's states as input.
• Encoder-decoder RNN. For sequence-to-sequence mapping (Sutskever et al., 2014).
• Linear-attention and state-space recurrences. Modern constant-state sequence models. Katharopoulos et al. (2020) show that causal linear attention can be written as an RNN, and Gu and Dao (2023) propose a selective state space architecture with linear scaling in sequence length.

• Machine translation. LSTM-based sequence-to-sequence systems were an early neural approach competitive with phrase-based statistical systems (Sutskever et al., 2014; Cho et al., 2014 used an RNN encoder-decoder to score phrase pairs in a statistical translation system). Attention-based RNNs then improved results (Bahdanau et al., 2014).
• Language modelling and generation. Recurrent language models predicted the next token from the hidden state; transformer-based models have since largely taken over this role (see [TECH-LLM-001]).
• Speech. Deep LSTM networks trained end-to-end reported a 17.7% test-set phoneme error rate on the TIMIT benchmark, which the authors described as the best score known to them at the time of the 2013 preprint (Graves, Mohamed and Hinton, 2013).
• Reinforcement learning. Recurrent policies handle partially observable environments by carrying a state (see [TECH-RL-001]).
• Efficient sequence models. State-space and linear-attention recurrences target long contexts with constant-size state (Gu and Dao, 2023; Katharopoulos et al., 2020).

Which production systems currently use recurrent components is not tracked here; see the Living Reference collection MODEL-001.

#Advantages

• Constant-size state. Inference with a recurrent model needs memory that does not grow with the processed length, since the state has fixed size.
• Natural handling of variable-length input and streaming data.
• Parameter sharing across time.

#Limitations

• Sequential computation. Step $t$ depends on step $t-1$, so training cannot be parallelized over time, which is a main reason for the shift to attention-based models (Vaswani et al., 2017).
• Long-range dependency. Gating mitigates but does not remove the difficulty of carrying information over very long spans, and a fixed-size state is a bounded memory.
• Bottleneck in encoder-decoder use. Compressing an input into one vector limits fidelity for long inputs (Bahdanau et al., 2014).
• Training sensitivity. Exploding gradients require clipping or careful design (Pascanu et al., 2012).

#Unresolved questions

1. Can fixed-state recurrent models match attention at scale? Selective state-space and linear-attention models are active candidates. Gu and Dao (2023) report favourable results for their model in the settings they test, but independent comparisons across training regimes are not covered by the sources confirmed for this draft.
2. How much context can a fixed-size state faithfully retain? The trade-off between state size and recall is not quantified in this article.

As of 2026-10-19, attention-based transformers dominate large-scale language and translation systems, and LSTMs and GRUs are no longer the default for those tasks. The mathematics of gated recurrence and of gradient decay is Established. That LSTMs improved long-lag learning over simple RNNs on the test problems studied is Well Supported by the original experiments and later usage (Hochreiter and Schmidhuber, 1997). The competitiveness of recurrent-style sequence models at frontier scale is Preliminary to Contested; it depends on results of individual groups (Gu and Dao, 2023) that await systematic independent comparison.

This section describes plausible directions and does not register forecasts.

• Near term (0–3 years). Hybrid models combining attention layers with recurrent or state-space layers, motivated by inference cost for long contexts, are an active direction.
• Medium term (3–10 years). Whether constant-state models close the quality gap with attention on recall-heavy tasks is an empirical question.
• Long term (10+ years). Persistent, updateable memory for agents (see [AGI-CL-001]) may revisit recurrent state as one mechanism among several; this is speculative.

• Foundational Concepts:
  • TECH-NN-001 Artificial Neural Networks: units, layers and training.
  • TECH-BACKPROP-001 Backpropagation: backpropagation through time and gradient decay.
• Related Methods & Technologies:
  • TECH-ATTN-001 Attention Mechanisms: attention introduced on top of RNN encoder-decoders.
  • TECH-CNN-001 Convolutional Neural Networks: the other structured family; convolutions as an alternative for sequences.
  • TECH-RL-001 Reinforcement Learning: recurrent policies for partial observability.
  • TECH-DL-001 Deep Learning: broader context.
• Advanced Implications & Future Trajectories:
  • TECH-ARCH-001 The Transformer Architecture: the architecture that displaced recurrence in language modelling.
  • TECH-LLM-001 Large Language Models: current use of autoregressive sequence models.
  • AGI-CL-001 Continual Learning and Persistent Memory: memory beyond a context window.