In short
A transformer is a neural network that reads a sequence of tokens all at once and uses self-attention to let every position weigh information from every other position, instead of stepping through the sequence one item at a time like a recurrent network. It was introduced by Vaswani et al. in the 2017 paper "Attention Is All You Need," which showed it could beat the best translation systems while training faster because its computation parallelizes well. It became the basis of later large language models such as BERT and GPT-3.
This guide stays close to what the original paper says. For the full technical article, with equations, see The Transformer architecture; for terms, see the glossary.
#Quick reference
| Concept | What it means | Evidence grade |
|---|---|---|
| Transformer | Encoder-decoder network "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely" (Vaswani et al. 2017) | Established |
| Token | A unit of text (word or word piece) that the model reads and predicts | Established |
| Embedding | A learned vector of numbers representing each token | Established |
| Self-attention | Each position computes a weighted mix of all positions in the same sequence | Established |
| Original results | 28.4 BLEU English-German, 41.8 BLEU English-French (WMT 2014) | Established |
| Why it works so well at scale | Partly understood; an active research area | Preliminary |
#What is a transformer?
A transformer is a type of neural network for sequences, such as sentences. The original 2017 design, by eight authors at Google Brain, Google Research and the University of Toronto, was built for machine translation. It has two halves:
- an encoder, which reads the input sentence and builds a representation of it, and
- a decoder, which generates the output sentence one token at a time, using the encoder's representation and the tokens it has already produced.
Each half is a stack of identical layers; the paper used 6 in each. Every layer has a self-attention step followed by a small feed-forward network applied to each position separately, with residual connections and layer normalization around both. Later systems often keep only one half: BERT (Devlin et al. 2018) uses the encoder, while GPT-style language models generate text left to right like the decoder and are usually described as decoder-only. See Large language models.
#What are tokens and embeddings?
A model cannot read letters directly. Text is first split into tokens, chunks that are often whole words or pieces of words. The 2017 paper used byte-pair encoding with a shared vocabulary of about 37,000 tokens for English-German, and a 32,000 word-piece vocabulary for English-French.
Each token is then turned into an embedding: a list of numbers (512 in the base model) that the network learns during training, so that tokens used in similar ways end up with similar vectors. See Tokenization and embeddings.
Because self-attention on its own ignores word order, the model adds positional encodings to the embeddings. The paper used fixed sine and cosine patterns of different frequencies. It also tried learned position embeddings and "found that the two versions produced nearly identical results"; it chose the sinusoidal version because it "may allow the model to extrapolate" to longer sequences than seen in training.
#What is self-attention?
Attention, in the paper's words, maps "a query and a set of key-value pairs to an output." A simple way to picture it: every token asks a question (its query), every token offers a label (its key) and some content (its value). The model compares each query with all keys, turns the match scores into weights that add up to one, and outputs a weighted blend of the values. Self-attention means the queries, keys and values all come from the same sequence, so each word's new representation is built from the words around it that matter most to it.
Two details from the paper:
- Scaling. The match scores are divided by the square root of the key size. The authors suspected that, without this, large dot products push the softmax "into regions where it has extremely small gradients," which slows learning.
- Multiple heads. The model runs 8 attention operations in parallel, each on a smaller slice (64 dimensions). This "allows the model to jointly attend to information from different representation subspaces at different positions." Loosely, different heads can track different kinds of relationship.
In the decoder, attention is masked so that a position cannot look at later positions. That keeps generation honest: when predicting the next token, the model sees only what came before. See Attention mechanisms.
#What is the difference between training and inference?
Training sets the model's parameters (its adjustable numbers). The model sees many examples where the right answer is known, predicts each next token, and is nudged to make the correct token more likely. Thanks to the mask, a whole target sentence can be processed in one pass, with every position predicted at the same time. The 2017 base model (about 65 million parameters) trained on roughly 4.5 million English-German sentence pairs for 100,000 steps, about 12 hours on 8 NVIDIA P100 GPUs. The "big" model (about 213 million parameters) trained for 3.5 days.
Inference is using the trained model. The parameters are fixed, and the decoder generates output token by token, each time feeding its own previous outputs back in. The paper used beam search, which keeps several candidate translations open and picks the best. Modern chat models generate the same way, one token at a time, which is one reason fluent output can still be wrong; see Why AI hallucinates. Making inference cheaper is its own field, covered in Inference, quantization and efficiency.
#Why did transformers replace recurrent networks?
Before 2017, the leading translation systems were recurrent neural networks (RNNs), especially long short-term memory networks (LSTMs; Hochreiter and Schmidhuber 1997), arranged as an encoder-decoder (Sutskever, Vinyals and Le 2014) and often with an attention mechanism added (Bahdanau, Cho and Bengio 2014). An RNN reads a sentence one step at a time, carrying a hidden state forward. The paper names the cost: this "inherently sequential nature precludes parallelization within training examples." See Recurrent networks and LSTM.
The paper gives three reasons for preferring self-attention:
- Parallel computation. A self-attention layer connects all positions with a constant number of sequential operations; a recurrent layer needs a number that grows with sentence length. That fits GPUs, which do many operations at once.
- Short paths between distant words. In a transformer, any two positions are one attention step apart, which the authors argue makes long-range dependencies easier to learn.
- Cost per layer. Self-attention is faster than a recurrent layer when the sequence is shorter than the representation size, which was usually true for sentences. The trade-off is that attention cost grows with the square of sequence length, a limit later work has tried to ease.
The results backed this up. The big model reached 28.4 BLEU on WMT 2014 English-German, improving on the previous best, including ensembles, by over 2 BLEU, and set a single-model record of 41.8 on English-French, at "a small fraction of the training costs" of earlier systems. The architecture then spread: BERT and GPT-3 (Brown et al. 2020, 175 billion parameters) built on it, and the scaling-law studies in AI scaling laws explained were fitted on transformer language models. The history is in The transformer and foundation model era.
#Frequently asked questions
#Who invented the transformer?
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin, in "Attention Is All You Need," posted to arXiv in June 2017 and published at NeurIPS 2017. The paper notes equal contribution and a random author order.
#What does "attention" mean in a transformer?
It is a weighted average: each position scores how relevant every other position is, then blends their information using those scores. Self-attention does this within a single sequence.
#Do transformers understand language?
The architecture specifies how numbers flow, not whether the result counts as understanding. That question is debated; see the glossary and the encyclopedia's philosophy volume.
#Is GPT a transformer?
Yes. The GPT-3 paper describes its model in terms of "the layers of the transformer," trained autoregressively to predict the next token. BERT, by contrast, uses only the encoder.
#What is the main weakness of the original transformer?
Self-attention compares every position with every other, so its cost grows with the square of sequence length. The paper itself suggested restricting attention to a local neighbourhood for very long inputs.
#Sources
- Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin: Attention Is All You Need (NeurIPS 2017 proceedings)
- Bahdanau, Cho, Bengio: Neural Machine Translation by Jointly Learning to Align and Translate
- Sutskever, Vinyals, Le: Sequence to Sequence Learning with Neural Networks
- Hochreiter, Schmidhuber: Long Short-Term Memory
- Devlin, Chang, Lee, Toutanova: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Brown, Mann, Ryder, et al.: Language Models are Few-Shot Learners
Status: draft, reviewed 26 October 2026 against the primary sources above. The architecture facts are stable; claims about why transformers scale so well remain an open research question.
