--- id: HIST-010 title: "The Transformer and Foundation Model Era, 2017-2022" alternative_names: - "The Pretraining Era" - "From Attention Is All You Need to ChatGPT" category: "History → Modern Deep Learning" volume: II chapter: 13 entry_type: "Historical Record" level: 3 status: "Draft" last_reviewed: 2026-10-15 version: "0.1.0" prerequisites: - "TECH-ARCH-001" - "HIST-005" related_entries: - "HIST-011" - "HIST-012" - "HIST-013" - "TECH-LLM-001" - "TECH-SCALE-001" - "TECH-DIFF-001" - "CAP-GENAI-001" --- # The Transformer and Foundation Model Era, 2017-2022 ## 1. Concise Definition The **Transformer and Foundation Model Era** is the period from the 2017 publication of the Transformer architecture to the public release of ChatGPT at the end of 2022. In it, large neural networks pretrained on broad data with self-supervised objectives became general-purpose starting points that could be adapted to many tasks; Bommasani et al. (2021) named such models **foundation models**. ## 2. Overview & Significance Before 2017, state-of-the-art systems in language, vision and speech were mostly built separately and trained for specific tasks. In the following five years a single architecture, the Transformer, and a single recipe, large-scale pretraining followed by adaptation, spread across language, images, protein structure and other fields. The period also produced the empirical study of scaling, the first widely used instruction-following chat models, and an early public debate about the risks and costs of ever larger models. This article gives the chronology and the principal sources. The architecture itself is treated in [[TECH-ARCH-001] The Transformer Architecture](../vol03_technical/TECH-ARCH-001_transformer_architecture.md), language models in [[TECH-LLM-001] Large Language Models](../vol03_technical/TECH-LLM-001_large_language_models.md), and events after 2022 in [[HIST-011] The Contemporary AI Era, 2022 onward (living record)](./HIST-011_the_contemporary_ai_era_2022_onward_living_record.md). Fast-changing figures are deliberately omitted; see [[MODEL-001] Model Registry](../reference/factbook/MODEL-001_model_registry.md). ## 3. Historical Development ### 3.1 The architecture (2017) Vaswani et al. (2017) proposed the Transformer, a sequence-transduction model "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely." They reported 28.4 BLEU on the WMT 2014 English-to-German task and 41.8 BLEU on English-to-French, with training costs lower than the best prior models they compared against. The design's main practical advantage was parallelism across sequence positions, which suited the GPU and accelerator hardware of the time. ### 3.2 Pretraining for language (2018-2019) Peters et al. (2018) showed that contextual word representations, derived from the internal states of a pretrained bidirectional language model (ELMo), improved a range of language tasks. Devlin et al. (2018) introduced BERT, a Transformer encoder pretrained with a masked-language-modelling objective and then fine-tuned on each task; they reported state-of-the-art results on a range of benchmarks. Raffel et al. (2019) proposed the T5 framework that casts every language task as text-to-text and systematically compared pretraining choices. The dominant pattern in this phase was *pretrain, then fine-tune on a labelled task*. ### 3.3 Scaling and few-shot prompting (2020) Kaplan et al. (2020) measured how language-model loss varies with model size, dataset size and training compute, and reported smooth power-law trends over several orders of magnitude. Brown et al. (2020) trained GPT-3, an autoregressive model with 175 billion parameters, and showed that it could perform many tasks from a few examples placed in its prompt, without gradient updates ("few-shot" or in-context learning), sometimes competitively with fine-tuned systems. These results shifted attention from task-specific fine-tuning towards prompting a single large model. ### 3.4 Compute-optimal training and the largest dense models (2022) Hoffmann et al. (2022) trained over 400 models and concluded that, for a fixed compute budget, parameter count and training tokens should be scaled in equal proportion; their model Chinchilla (70 billion parameters trained on about four times as much data as the 280-billion-parameter Gopher) outperformed several larger models on their evaluations. Chowdhery et al. (2022) reported PaLM, a 540-billion-parameter dense Transformer language model. Together, these papers made the trade-off between model size and data a central question for system design; see [[TECH-SCALE-001] Scaling Laws](../vol03_technical/TECH-SCALE-001_scaling_laws.md). ### 3.5 Prompting for reasoning and the emergence debate (2022) Wei et al. (2022a) reported that prompting large models to produce intermediate reasoning steps ("chain of thought") improved performance on arithmetic, commonsense and symbolic reasoning tasks. Wei et al. (2022b) described "emergent abilities", defined as abilities not present in smaller models but present in larger ones, which cannot be predicted by extrapolating smaller models. Schaeffer, Miranda, and Koyejo (2023) argued that apparent emergence can result from the researcher's choice of a nonlinear or discontinuous metric rather than from sharp changes in model behaviour. Whether emergent abilities are real discontinuities or measurement artefacts is contested. ### 3.6 Alignment through human feedback Ouyang et al. (2022) fine-tuned GPT-3 first on labeler-written demonstrations and then using reinforcement learning from human feedback (RLHF) on ranked outputs, producing InstructGPT. In their human evaluations, outputs of the 1.3-billion-parameter InstructGPT were preferred to those of the 175-billion-parameter GPT-3. On 2022-11-30 OpenAI introduced ChatGPT as a free research preview, describing it as a sibling model to InstructGPT trained for dialogue (OpenAI, 2022). Technical details of ChatGPT were not published in a paper at the time; this article makes no claims about them beyond OpenAI's blog statement. ### 3.7 Beyond text - **Vision**: Dosovitskiy et al. (2020) showed that a pure Transformer applied to sequences of image patches (ViT) could match or exceed convolutional networks on image classification when pretrained on large data. Radford et al. (2021) trained CLIP on 400 million image-text pairs to enable zero-shot transfer to many vision tasks. - **Image generation**: Ramesh et al. (2021) described a Transformer for text-to-image generation. Ho, Jain, and Abbeel (2020) introduced denoising diffusion probabilistic models, and Rombach et al. (2022) applied diffusion in a learned latent space, the basis of widely used open image generators; see [[TECH-DIFF-001] Diffusion Models](../vol03_technical/TECH-DIFF-001_diffusion_models.md). - **Science**: Jumper et al. (2021) reported AlphaFold, which predicted protein structures with accuracy competitive with experimental accuracy in a majority of cases, as reported by the authors, at the 14th Critical Assessment of protein Structure Prediction (CASP14). See [[APP-SCI-001] AI in Science](../vol07_applications/APP-SCI-001_ai_in_science.md). ### 3.8 Critique and risk Bender, Gebru, McMillan-Major, and Shmitchell (2021) raised concerns about the environmental and financial costs, training-data documentation, and social risks of ever larger language models. This article cites it as a primary statement of these concerns, not as a verdict on them. ## 4. Technical / Conceptual Explanation ### 4.1 Self-attention Given input vectors stacked as a matrix $X$, a Transformer layer computes queries, keys and values $Q=XW_Q$, $K=XW_K$, $V=XW_V$ and $$\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\!\Big(\frac{QK^\top}{\sqrt{d_k}}\Big)V,$$ so every position can attend directly to every other position (Vaswani et al., 2017). ### 4.2 Pretraining objectives - *Masked language modelling* (BERT): predict randomly hidden tokens from bidirectional context (Devlin et al., 2018). - *Autoregressive modelling* (GPT-3): predict the next token, $p(x)=\prod_t p(x_t\mid x_{