Language and Generative Models

The standard training pipeline for a modern language model has two broad phases. Pretraining fits a model to predict the next token on a very large text corpus. Post-training then adapts the pretrained model to follow instructions and match human preferences, typically through supervised fine-tuning (SFT) on demonstrations followed by preference-based optimisation such as reinforcement learning from human feedback (RLHF), in which a reward model learned from human comparisons is used to steer the model.

A pretrained language model is a general-purpose text predictor. It continues documents rather than answering requests, and it reproduces the distribution of its training text, including falsehoods and toxic content. Post-training is the stage that turns such a model into an assistant. Ouyang et al. (2022) reported that outputs from a 1.3-billion-parameter model trained with human feedback were preferred by labelers to those of the 175-billion-parameter GPT-3 on their prompt distribution, illustrating that the choice of training objective after pretraining can matter more for perceived helpfulness than a 100-fold difference in parameters.

This article covers the technical methods. The broader question of whether such methods suffice to make advanced systems safe is treated in [SAFE-ALIGN-001] and [SAFE-OVERSIGHT-001]; the use of reinforcement learning to train long-form reasoning is treated in [TECH-REASON-001]. Here, as elsewhere, details of closed systems are reported only to the extent that developers have disclosed them, and the remainder is marked "undisclosed".

Transfer learning for language predates the current pipeline: ULMFiT (Howard & Ruder, 2018) showed that fine-tuning a pretrained language model could improve text classification, and Gururangan et al. (2020) showed that a second phase of in-domain pretraining improved downstream performance across the four domains and eight tasks they studied. GPT-3 (Brown et al., 2020) demonstrated few-shot prompting of a pretrained model without gradient updates.

Instruction tuning appeared in FLAN (Wei et al., 2022; arXiv 2021), which fine-tuned a 137-billion-parameter pretrained model on over 60 tasks phrased as instructions and reported better zero-shot results on held-out tasks, and in T0 (Sanh et al., 2022), which reported that explicit multitask prompted training gave strong zero-shot performance, often outperforming models up to 16 times larger.

Learning from human preferences originated in reinforcement learning, where Christiano et al. (2017) trained agents from human comparisons of trajectory segments, using feedback on less than one percent of interactions in their tasks. Ziegler et al. (2019) applied the idea to language models, Stiennon et al. (2020) to summarisation, and Ouyang et al. (2022) to broad instruction following (InstructGPT). Bai et al. (2022a) applied preference modelling and RLHF to helpful and harmless assistants and reported a roughly linear relation between reward and the square root of the KL divergence from the initial policy. Bai et al. (2022b) proposed Constitutional AI, in which AI-generated preferences guided by written principles replace human labels for harmlessness. Rafailov et al. (2023) introduced Direct Preference Optimization (DPO), which removes the explicit reward model and RL loop. From 2024, reinforcement learning against automatically checkable rewards became prominent in open recipes (Shao et al., 2024; Lambert et al., 2024).

#4.1 Pretraining

Given a corpus tokenised into sequences $t_1,\dots,t_n$ (see [TECH-TOK-001]), the model with parameters $\theta$ minimises

Data composition and quality, and the scaling relationships between parameters, data and compute, are covered in [TECH-DATA-001] and [TECH-SCALE-001]. Pretraining accounts for the bulk of training compute for most documented systems, and it is where most factual knowledge and general capability are acquired. For GPT-4, its developer's report states that the model's capabilities on the exams it tested "appear to stem primarily from the pre-training process and are not significantly affected by RLHF" (OpenAI et al., 2023). That is a statement about those exams, not about every behaviour.

#4.2 Supervised fine-tuning (SFT)

SFT continues training with the same next-token loss, but on pairs $(x, y)$ of prompts and desired responses, usually with the loss applied to the response tokens only:

Demonstrations may be written by labelers (Ouyang et al., 2022), collected as templated task collections (Wei et al., 2022; Sanh et al., 2022), or curated. Zhou et al. (2023) trained a 65B-parameter model on only 1,000 curated prompt-response pairs with no reinforcement learning (LIMA) and reported that in a controlled human study its responses were equivalent or preferred to GPT-4 in 43% of cases; they suggest that most knowledge is learned in pretraining and that alignment mainly teaches format and style. This "superficial alignment" view is a hypothesis supported by their evaluation, not a settled result.

Because full fine-tuning updates all parameters, parameter-efficient methods such as LoRA (Hu et al., 2022), which trains low-rank updates to weight matrices, are widely used for adaptation.

#4.3 Reward modelling

Human labelers rank or compare model outputs. For a prompt $x$ with preferred response $y_w$ and less-preferred response $y_l$, a reward model $r_\phi(x,y)$ is trained under a Bradley–Terry model of pairwise preference:

where $\sigma$ is the logistic function. The reward model is a proxy: it is trained on a finite sample of judgements from a particular group of labelers.

#4.4 RL optimisation

The policy $\pi_\theta$ (initialised from the SFT model, $\pi_{\text{ref}}$) is then optimised to maximise the proxy reward while staying close to the reference:

The KL penalty (coefficient $\beta$) limits drift. The optimiser in the best-documented pipelines is Proximal Policy Optimization (PPO; Schulman et al., 2017), a policy-gradient method described in [TECH-RL-001]. Ouyang et al. (2022) followed three steps: SFT on demonstrations, reward-model training on rankings, and PPO against the reward model. Llama 2 documented an RLHF stage combining rejection sampling and PPO on top of SFT (Touvron et al., 2023).

#4.5 Direct preference optimisation

Rafailov et al. (2023) show that the KL-regularised objective above has a closed-form optimal policy, so the reward can be re-parameterised through the policy itself, giving a classification-style loss on preference pairs:

DPO trains on a fixed preference dataset without sampling during training, which makes it simpler and more stable to run than PPO-based RLHF, at the cost of the on-policy exploration that RL provides.

#4.6 AI feedback and verifiable rewards

• RLAIF / Constitutional AI. In the supervised phase the model critiques and revises its own responses according to a list of principles; in the RL phase a preference model is trained from AI comparisons rather than human labels for harmlessness (Bai et al., 2022b).
• Reinforcement learning with verifiable rewards (RLVR). For tasks where correctness can be checked automatically (mathematics answers, code tests), the reward is computed by a program rather than a learned model. Tulu 3 describes RLVR as a post-training step alongside SFT and DPO (Lambert et al., 2024), and DeepSeekMath introduced Group Relative Policy Optimization (GRPO), a PPO variant that reduces memory use (Shao et al., 2024). The role of this technique in reasoning models is covered in [TECH-REASON-001].

• InstructGPT. SFT, reward modelling and PPO on GPT-3; 1.3B variant preferred over 175B GPT-3 by labelers on the authors' prompts, with reported improvements in truthfulness (Ouyang et al., 2022).
• Anthropic assistants. Published research on HH-RLHF and Constitutional AI (Bai et al., 2022a, 2022b). The training recipes of Anthropic's current deployed models are undisclosed beyond what the company has published.
• GPT-4. OpenAI's report states the model was pre-trained to predict the next token and then fine-tuned with RLHF, and that it contains no further details about training method, dataset construction, architecture or compute; these are undisclosed (OpenAI et al., 2023).
• Open recipes. Llama 2 and Tulu 3 publish extensive post-training descriptions (Touvron et al., 2023; Lambert et al., 2024), making them the main public basis for study of the full pipeline.

#Advantages

Large improvements in perceived helpfulness and instruction following at low cost relative to pretraining; compatibility with specialised skills (Bai et al., 2022a); a framework in which new preferences, including safety policies, can be expressed through data.

#Limitations

Casper et al. (2023) survey open problems and fundamental limitations of RLHF, organised around problems with human feedback, the reward model and the policy, and propose disclosure and auditing standards. Specific documented issues:

• Reward overoptimisation. Optimising too hard against an imperfect reward model degrades the true objective, in line with Goodhart's law. In a synthetic setup with a fixed gold reward model, Gao et al. (2023) find different functional forms for RL and best-of-$n$ sampling, with coefficients that scale smoothly with reward-model parameters.
• Sycophancy. Sharma et al. (2024) report that five AI assistants consistently exhibited sycophancy on four free-form tasks, that human and preference-model judgements sometimes prefer convincingly written sycophantic responses to correct ones, and that optimising against preference models sometimes sacrifices truthfulness. They infer that human preference judgements likely play a role, not that RLHF is the sole cause.
• Labeler limits. Preferences reflect the labelers, instructions and prompt distribution used; they are weak for tasks the labelers cannot evaluate (see [SAFE-OVERSIGHT-001]).
• Training complexity. PPO-based RLHF requires several models (policy, reference, reward, and often a value model), which motivated simpler objectives such as DPO.

#Unresolved questions

1. Relative benefit of online RL versus offline preference optimisation; the evidence comes from varied setups and no single comparison is decisive.
2. How much of post-training changes capability versus style. The LIMA result is a single human-evaluation study of one 1,000-example fine-tuning set and does not settle whether larger post-training data or RL add capability beyond what pretraining supplies.
3. Whether alignment training that works for current assistants scales to systems more capable than their overseers (see [SAFE-ALIGN-001]).
4. Hallucination and calibration effects of post-training (see [CAP-LIMITS-001]).

As of 2026-10-15, the pretrain, SFT and preference/RL pipeline is the documented standard for assistant models, with variation in the specific preference-optimisation algorithm and growing use of verifiable-reward RL in open recipes. For open-weight models, the pipeline is documented in some detail. For closed models the stages that developers have named (for example RLHF in the GPT-4 report) are known, but datasets, reward-model design, hyperparameters, proportions of compute and later changes are undisclosed. Claims that a specific current closed model was trained with a particular algorithm should be attributed to a developer statement or marked undisclosed. Model-specific facts are tracked in MODEL-001.

Observed trends: replacement or augmentation of human labels by AI feedback and programmatic checks; open, fully documented recipes (Lambert et al., 2024); growing attention to feedback quality, sycophancy and evaluation of reward models; and extension of RL from short-form preference tasks to long-form reasoning. Whether these methods continue to work as capabilities rise is an open question rather than a forecast made here; no forecast record is registered in this article.

• Foundational Concepts:
  • TECH-LLM-001 Large Language Models: The pretrained models being adapted.
  • TECH-RL-001 Reinforcement Learning: Policy gradients, PPO and reward design.
  • FOUND-010 Optimization and Gradient Descent: Optimisation background.
• Related Methods & Technologies:
  • TECH-SCALE-001 Scaling Laws: Pretraining scaling and reward overoptimisation scaling.
  • TECH-DATA-001 Training Data, Contamination and Provenance: Data used at each stage.
  • TECH-TOK-001 Tokenization and Embeddings: The units of the training loss.
  • TECH-REASON-001 Reasoning Models and Test-Time Compute: RL against verifiable rewards.
• Advanced Implications & Future Trajectories:
  • SAFE-ALIGN-001 AI Alignment: The wider alignment problem.
  • SAFE-OVERSIGHT-001 Scalable Oversight and Corrigibility: Supervision beyond labeler ability.
  • CAP-LIMITS-001 Hallucination, Reliability and Calibration: Reliability effects.

An attention mechanism is a differentiable operation that lets a neural network compute each output as a weighted combination of a set of input vectors, where the weights are themselves computed from the content of the inputs and a query. Attention replaces a fixed, position-based connection pattern with a learned, content-dependent one, and it is the central operation of the transformer architecture.

Attention was introduced to solve a concrete problem in recurrent sequence-to-sequence models (see [TECH-RNN-001]): the encoder had to compress an entire source sentence into one fixed-length vector from which the decoder generated the translation. Bahdanau, Cho and Bengio (2014) conjectured that this fixed-length vector was a bottleneck and proposed letting the decoder (soft-)search for the parts of the source relevant to each target word.

Two features explain the significance of attention. First, it gives any position direct access to any other, so the path length for information between distant positions does not grow with their separation. Second, when used without recurrence, it can be computed for all positions of a sequence at once, which fits parallel hardware. Vaswani et al. (2017) built the Transformer entirely on this operation, and the architecture is covered in [TECH-ARCH-001]. This entry describes the mechanism and its variants; it does not repeat the architectural account.

Alignment in translation. Bahdanau, Cho and Bengio (2014) added an attention (alignment) model to an RNN encoder-decoder, allowing the model to learn soft alignments between source and target words while translating. Luong, Pham and Manning (2015) examined two classes of attention for translation, a global approach that attends to all source words and a local approach that looks at a subset, and reported gains over non-attentional systems on WMT English-German tasks.

Beyond text. Xu et al. (2015) applied visual attention to image captioning, where the decoder attends to regions of an image while generating each word, showing that the idea was not specific to language.

Self-attention and the Transformer. Vaswani et al. (2017) proposed the Transformer, "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely", and reported superior translation quality with greater parallelizability and shorter training time on the tasks they tested. Attention in which queries, keys and values all come from the same sequence is called self-attention.

Efficiency and variants. Subsequent work addressed the cost of attention over long sequences, the memory cost of the key-value cache during generation, and positional treatment. Tay et al. (2020) survey a range of "efficient transformer" variants. Shazeer (2019) proposed multi-query attention to speed decoding, Ainslie et al. (2023) proposed grouped-query attention, and Dao et al. (2022) proposed an input/output-aware exact attention algorithm. Katharopoulos et al. (2020) formulated a linear-attention variant with an RNN form.

#4.1 General form: query, keys, values

Given a query $q$, a set of keys $k_1,\dots,k_n$ and values $v_1,\dots,v_n$, attention computes a weighted sum of the values:

The function $s$ is a score (compatibility) function; the softmax turns scores into non-negative weights that sum to one. All operations are differentiable, so the whole mechanism is trained end to end by backpropagation (see [TECH-BACKPROP-001]).

#4.2 Score functions

The additive (MLP) score of Bahdanau et al. (2014) feeds the decoder state and an encoder state through a small feedforward network. Luong et al. (2015) compared simpler multiplicative forms, including a dot product. The Transformer uses a scaled dot product (Vaswani et al., 2017):

where $d_k$ is the key dimension. The scaling prevents large dot products from pushing the softmax into regions of very small gradients.

#4.3 Matrix form and self-attention

For a sequence of $N$ tokens with representations $X \in \mathbb{R}^{N\times d}$, self-attention forms $Q = XW_Q$, $K = XW_K$, $V = XW_V$ and computes

The matrix $QK^\top \in \mathbb{R}^{N\times N}$ holds all pairwise scores, so time and memory in the naive algorithm grow quadratically with $N$ (Vaswani et al., 2017).

#4.4 Multi-head attention

Rather than one attention function, the Transformer runs $h$ of them in parallel on different learned linear projections and concatenates the results, which allows the model to attend to different kinds of relations at once (Vaswani et al., 2017). The head count and dimensions are architecture hyperparameters.

#4.5 Masking and cross-attention

In causal (masked) self-attention, scores to future positions are set to $-\infty$ before the softmax so each position depends only on earlier ones; this is how autoregressive generation is trained. In cross-attention, queries come from one sequence (such as the decoder) and keys and values from another (such as the encoder or an image encoder), as in the original translation setting and in image captioning (Xu et al., 2015).

#4.6 Positional information

Attention is permutation-equivariant: without extra information, reordering the inputs only reorders the outputs. Order must be injected, as in the sinusoidal encodings of Vaswani et al. (2017), or through learned or relative position schemes. The details of positional schemes are covered in [TECH-ARCH-001].

#4.7 Soft versus hard attention

The mechanisms above are soft: every input contributes with some weight, and gradients flow through the weights. Hard attention selects discrete locations and requires gradient estimators other than ordinary backpropagation; Xu et al. (2015) studied both a soft and a hard variant for captioning.

• Machine translation. The setting in which attention was introduced and where it gave reported gains over non-attentional models (Bahdanau et al., 2014; Luong et al., 2015).
• Image captioning. Attention over image regions (Xu et al., 2015).
• Large language models. Transformer language models are built from stacked self-attention layers (see [TECH-LLM-001]).
• Vision and multimodal models. Attention over image patches and across modalities (see [TECH-ARCH-001], [TECH-MM-001]; compare convolution in [TECH-CNN-001]).
• Retrieval-like use. Attention can be read as a soft lookup in a key-value memory, and this view links to retrieval augmentation (see [TECH-RAG-001]).

#Advantages

• Direct long-range access. Any position can interact with any other in one layer.
• Parallelism. With no recurrence, all positions can be processed in parallel during training (Vaswani et al., 2017).
• Flexibility. The same operation applies to text, images, audio and structured inputs.

#Limitations

• Quadratic cost. Standard self-attention scales quadratically in sequence length in time and memory (Vaswani et al., 2017; Tay et al., 2020).
• Generation-time memory. Autoregressive decoding stores keys and values for earlier tokens; Shazeer (2019) and Ainslie et al. (2023) target this cost.
• Weak built-in order and locality bias. Position information must be supplied, and the model must learn locality rather than receiving it by design.

#Unresolved questions

1. Are attention weights explanations? Jain and Wallace (2019) reported experiments in which attention weights were frequently uncorrelated with gradient-based measures of feature importance and in which very different attention distributions gave equivalent predictions, and concluded that attention weights largely do not provide meaningful explanations. Wiegreffe and Pinter (2019) argued that the claim depends on one's definition of explanation and on the experimental design, and proposed alternative tests. The question is Contested and is relevant to interpretability work (see [SAFE-INTERP-001]).
2. Do cheaper attention variants preserve quality? Efficient variants trade exactness or expressiveness for cost (Tay et al., 2020). Which trade-offs hold at very large scale is an empirical question. [VERIFICATION NEEDED: no comparative study at frontier scale was confirmed for this draft.]
3. Can attention-free or hybrid models match full attention? See [TECH-RNN-001], on state-space and linear-attention models.

As of 2026-10-15, dot-product multi-head self-attention in transformer-style networks is the standard component of the large language and multimodal models discussed in this encyclopedia. The definition and mathematics of attention are Established. That attention improved translation over fixed-vector encoder-decoders is Well Supported by the original papers' experiments (Bahdanau et al., 2014; Luong et al., 2015). Exact attention can now be computed with considerably less memory traffic through IO-aware kernels without changing results (Dao et al., 2022); this is an implementation-level improvement. The interpretability status of attention weights is Contested (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019). Specific context-window lengths and cost figures of deployed systems change quickly and are tracked in MODEL-001.

This section describes plausible directions and does not register forecasts.

• Near term (0–3 years). Continued refinement of memory-efficient and cache-efficient attention variants and their kernels (see [TECH-INFER-001]).
• Medium term (3–10 years). The relative roles of full attention, sparse or windowed attention and recurrent or state-space layers in long-context models remain to be settled by empirical comparison.
• Long term (10+ years). Whether attention as currently defined is a lasting component of general-purpose models, or one stage in the evolution of architectures, is an open question discussed in [TECH-ARCH-001].

• Foundational Concepts:
  • TECH-RNN-001 Recurrent Networks and LSTM: the encoder-decoder models in which attention first appeared.
  • TECH-NN-001 Artificial Neural Networks: background on layers and training.
  • TECH-BACKPROP-001 Backpropagation: how attention weights are trained.
• Related Methods & Technologies:
  • TECH-ARCH-001 The Transformer Architecture: the full architecture built on attention.
  • TECH-CNN-001 Convolutional Neural Networks: the competing local-structure operation.
  • TECH-LLM-001 Large Language Models: the main use of attention at scale.
  • TECH-MM-001 Multimodal Foundation Models: cross-modal attention.
  • TECH-RAG-001 Retrieval-Augmented Generation: external lookup related to attention-as-lookup.
  • TECH-INFER-001 Inference, Quantization and Efficiency: serving-time cost of attention.
• Advanced Implications & Future Trajectories:
  • SAFE-INTERP-001 Interpretability and Mechanistic Interpretability: whether attention patterns explain model behaviour.

AI code generation is the use of machine-learning models, principally large language models trained on source code and text, to produce, complete, edit, explain or repair computer programs from natural-language descriptions, partial code or repository context. It ranges from single-line autocomplete to autonomous agents that modify whole repositories.

Programs have properties that make them a favourable domain for machine learning: abundant public training text, a formal language with execution semantics, and an objective check (running tests) that can provide training and evaluation signals. Code generation was therefore among the earliest large-scale commercial uses of language models, and it is a frequent proxy for general reasoning ability. Because programming is also the activity through which AI systems themselves are built, code capability is relevant to discussions of automated AI research (see ASI-AIRD-001 Automated AI Research).

This article describes the technology, its evaluation methods and the quality of the evidence on productivity and security. The capability and benchmark landscape is treated in CAP-CODE-001 Software Engineering Capability and Evaluation, and deployment in APP-SW-001 AI in Software Engineering.

Program synthesis from specifications has a long symbolic history. The neural era for code accelerated when language models were trained on large code corpora. Austin et al. (2021) evaluated language models from 244 million to 137 billion parameters on two Python synthesis benchmarks (MBPP, with 974 tasks, and MathQA-Python) and reported log-linear improvement with model size, together with a gain of about 10 percentage points from fine-tuning on a held-out portion of MBPP. Chen et al. (2021) introduced Codex, a GPT model fine-tuned on public code, and the HumanEval benchmark; they reported 28.8% of HumanEval problems solved with a single sample, 11.4% for GPT-J and 0% for GPT-3, and 70.2% when 100 samples per problem were drawn.

Li et al. (2022) described AlphaCode, which sampled large numbers of candidate programs and filtered them by behaviour, achieving in simulated Codeforces contests an average ranking in the top 54.3% of contests with more than 5,000 participants. Open code models followed: StarCoder, a 15.5-billion-parameter model trained on permissively licensed code in The Stack with an opt-out process (Li et al., 2023), and Code Llama, a family built on Llama 2 with 7B to 70B parameters, infilling and long-context support (Rozière et al., 2023). From 2023 attention moved from function-level synthesis to repository-level tasks: SWE-bench collected 2,294 real GitHub issue-and-pull-request tasks from 12 Python repositories, on which the best model evaluated at the time resolved about 1.96% (Jimenez et al., 2024); Yang et al. (2024) reported that an agent-computer interface raised pass@1 on SWE-bench to 12.5% (the authors' 2024 claim; later results are tracked elsewhere, see section 8).

#4.1 Model formulation

A code model is an autoregressive language model (see TECH-LLM-001 Large Language Models) trained to maximise $\sum_t \log p_\theta(x_t \mid x_{<t})$ over sequences of code and text tokens. Code-specific features include: training on repository data, fill-in-the-middle objectives that allow infilling given prefix and suffix (named as a capability in Rozière et al., 2023), long context windows to hold multiple files, and instruction tuning to follow natural-language requests. Post-training with execution feedback (tests as reward) is used in reinforcement-learning pipelines, as described for reasoning models in TECH-REASON-001 Reasoning Models and Test-Time Compute; specifics of closed models are undisclosed.

#4.2 Evaluating by execution: pass@k

Functional correctness is measured by running unit tests. If $n$ samples are drawn for a problem and $c$ pass, the unbiased estimator of the probability that at least one of $k$ samples passes is

introduced with HumanEval (Chen et al., 2021). The measure rewards sampling many candidates and having a way to select among them; AlphaCode's pipeline made this explicit with large-scale sampling plus behavioural filtering (Li et al., 2022).

#4.3 Test quality and benchmark validity

Passing tests is only as meaningful as the tests. Liu et al. (2023) extended HumanEval's tests about 80-fold (HumanEval+) and reported that this lowered measured pass@k by up to roughly 19.3–28.9% across 26 models and changed some rankings. Benchmarks that are public and static are exposed to training-data contamination (see TECH-DATA-001 Training Data, Contamination and Provenance); LiveCodeBench responds with continuously collected contest problems (Jain et al., 2024).

#4.4 Repository-level and agentic generation

Real software work requires reading many files, running the code and iterating. SWE-bench frames tasks as: given a repository and an issue description, produce a patch that makes the repository's hidden tests pass (Jimenez et al., 2024). SWE-agent equips a model with a purpose-built interface for editing, navigation and test execution; the authors report that interface design affects agent performance (Yang et al., 2024). This is the code-specific instance of the agent loop discussed in TECH-AGENT-001 Autonomous AI Agents.

• Completion and autocomplete: next-token or fill-in-the-middle inside an editor.
• Instruction-to-code: chat or natural-language to function or script.
• Edit and repair: refactoring, bug fixing, test generation; HumanEvalFix is one repair benchmark mentioned by Yang et al. (2024).
• Agentic software engineering: models with tool access that navigate repositories, run commands and tests, and propose patches.
• Competitive programming systems: sample-and-filter pipelines (Li et al., 2022).
• Open versus closed models: open-weight code models (Rozière et al., 2023; Li et al., 2023) versus proprietary assistants whose training data and methods are undisclosed.

• Productivity studies (human outcomes). In a controlled experiment, developers with access to GitHub Copilot completed an HTTP-server task in JavaScript 55.8% faster than a control group (Peng et al., 2023). In contrast, a 2025 randomized controlled trial by METR with 16 experienced open-source developers on 246 tasks in mature projects found that, with early-2025 tools (mainly Cursor Pro with Claude 3.5/3.7 Sonnet), allowing AI increased completion time by 19%; developers had forecast about a 24% reduction and afterwards estimated a 20% reduction (Becker et al., 2025). The two results come from different tasks, populations and tool generations; neither establishes a general effect.
• Security. Pearce et al. (2022) prompted Copilot on 89 scenarios, produced 1,689 programs and reported about 40% were vulnerable; the study concerns a 2021 system and a selected set of scenarios (see SAFE-SEC-001 AI Security).
• Open model releases and training-data governance. StarCoder's training used permissively licensed repositories with an opt-out process and PII redaction (Li et al., 2023); licensing and attribution questions are discussed in SAFE-IP-001 Copyright and Intellectual Property.

Advantages. Faster completion of well-specified, self-contained tasks in some controlled settings (Peng et al., 2023); executable tests give a relatively objective evaluation channel compared with open-ended text.

Limitations. (1) Generated code can be wrong in ways that pass weak tests (Liu et al., 2023). (2) Insecure patterns can be generated (Pearce et al., 2022). (3) Struggles with long chains of operations were noted in early Codex evaluation (Chen et al., 2021). (4) Benchmark contamination and saturation complicate comparisons (Jain et al., 2024). (5) Productivity effects depend on developer experience, codebase maturity and task type, and measured effects conflict (Peng et al., 2023; Becker et al., 2025). (6) Generated code raises licence and provenance questions.

Unresolved. The net effect on software quality and maintenance cost over long horizons; how autonomy levels should be matched to verification effort; whether execution-feedback training generalises beyond tasks with tests.

As of 2026-10-15: that language models can generate functionally correct short programs is established (Chen et al., 2021; Austin et al., 2021). That test suites can overstate correctness is well supported (Liu et al., 2023). Productivity impact is Contested because randomized studies disagree (Peng et al., 2023; Becker et al., 2025). Benchmark scores for current models, particularly on SWE-bench and its variants and on LiveCodeBench, change rapidly and are not tabulated here; see MODEL-001 and the evaluation entries CAP-BENCH-001 AI Evaluation and CAP-CODE-001 Software Engineering Capability and Evaluation. The 2023–2024 scores cited above are historical data points.

Observed trends. The unit of evaluation has moved from functions (HumanEval, 2021) to repository-level issue resolution (SWE-bench, 2023) and to contamination-controlled sets (LiveCodeBench, 2024).

Forecast.

Medium and long term. Extrapolating autonomous software-engineering capability beyond this record requires assumptions not asserted here; see AGI-TIME-001 AGI Forecasting for treatment of forecasts about capability.

• Foundational Concepts:
  • TECH-LLM-001 Large Language Models: The underlying model class.
• Related Methods & Technologies:
  • TECH-AGENT-001 Autonomous AI Agents: Agent loops for repository tasks.
  • TECH-REASON-001 Reasoning Models and Test-Time Compute: Verifiable-reward training and sampling.
  • TECH-DATA-001 Training Data, Contamination and Provenance: Code corpora and benchmark leakage.
  • TECH-RL-001 Reinforcement Learning: Learning from execution feedback.
• Evaluation, Applications and Risk:
  • CAP-BENCH-001 AI Evaluation: Benchmark methodology.
  • CAP-CODE-001 Software Engineering Capability and Evaluation: Capability measurement.
  • APP-SW-001 AI in Software Engineering: Industry use.
  • SAFE-SEC-001 AI Security: Insecure code and prompt-injection risk.
  • SAFE-IP-001 Copyright and Intellectual Property: Licensing of training code and outputs.

A Large Language Model (LLM) is a high-capacity deep neural network—typically a decoder-only transformer—trained on massive corpora of text to approximate the joint probability distribution of natural language. By optimizing the fundamental objective of autoregressive next-token prediction, these models develop complex internal representations of syntax, semantics, and factual knowledge, which can be subsequently aligned to follow human instructions, perform multi-step reasoning, and execute zero-shot tasks across diverse linguistic and programmatic domains (Brown et al., 2020; Kaplan et al., 2020).

LLMs represent the dominant paradigm in contemporary artificial intelligence, acting as general-purpose cognitive engines. Rather than training specialized models for discrete tasks (e.g., a dedicated translation model, a dedicated summarization model), the field increasingly adopted a foundation model approach (Bommasani et al., 2021), although specialized models remain widely used. A sufficiently large language model, having compressed vast amounts of human knowledge during pretraining, can be coaxed into performing virtually any text-to-text mapping task.

The significance of LLMs lies in their emergent empirical behavior:

1. In-Context Learning: The ability to adapt to new tasks purely through prompt conditioning at inference time, without updating model weights.
2. Empirical Scaling Regularities: Test loss falls approximately as a power law in compute, parameter count, and dataset size across the ranges studied (Kaplan et al., 2020); the mapping from loss to specific downstream capabilities is less predictable.
3. Breadth: Because much written knowledge (code, mathematics, law, medicine, translation) can be serialized into discrete tokens, a single model family can be applied to a wide range of text-based tasks, with uneven reliability across them.

Language modeling originated with statistical n-gram models, which estimated the probability of a word given the $n-1$ preceding words based on raw frequency counts. These suffered fundamentally from the "curse of dimensionality" and data sparsity.

In 2003, Bengio and colleagues introduced the Neural Probabilistic Language Model (Bengio et al., 2003), using a feedforward neural network and continuous word embeddings to share statistical strength across semantically similar words. This was followed by Recurrent Neural Networks (RNNs) and LSTMs (Hochreiter & Schmidhuber, 1997), which could theoretically process unbounded context but struggled practically with vanishing gradients and sequential computational bottlenecks.

The inflection point occurred in 2017 with the introduction of the Transformer architecture (Vaswani et al., 2017). OpenAI's generative pre-trained transformer (GPT) series sequentially proved the efficacy of scaling:

• GPT-1 (2018): 117 million parameters (Radford et al., 2018); demonstrated the viability of unsupervised pretraining followed by supervised fine-tuning.
• GPT-2 (2019): 1.5 billion parameters; reported zero-shot task transfer (Radford et al., 2019).
• GPT-3 (2020): 175 billion parameters; reported few-shot in-context learning (Brown et al., 2020).

Simultaneously, the scaling laws formalized by Kaplan et al. (2020) and refined by Hoffmann et al. (2022) (the "Chinchilla" scaling laws) estimated compute-optimal ratios between parameter count and training tokens. Instruction tuning and Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022) were shown to make raw text predictors follow user instructions (InstructGPT). Deployed chat assistants use related post-training methods, but the details for closed products are largely undisclosed.

#4.1 Tokenization and Embeddings

Raw text is compressed into discrete subword units called tokens using algorithms like Byte-Pair Encoding (BPE) or WordPiece. A typical vocabulary $V$ contains 32,000 to 128,000 tokens. Each token index is mapped to a continuous embedding vector $\mathbf{e}_i \in \mathbb{R}^d$, enriched with positional encodings (such as Rotary Positional Embeddings, RoPE) to denote its position in the sequence.

#4.2 Autoregressive Pretraining

The fundamental training phase involves self-supervised learning on a massive corpus $\mathcal{D}$. The model is trained to minimize the negative log-likelihood (cross-entropy loss) of the next token $x_t$ given the preceding context $x_{<t}$:

To achieve compute-optimal training under a fixed FLOP budget $C$, Hoffmann et al. (2022) demonstrated that the model parameter count $N$ and the number of training tokens $D$ should be scaled approximately equally (fitted exponents of about 0.5, i.e. $N \propto C^{0.5}$, $D \propto C^{0.5}$ approximately), and argued that earlier large models were under-trained for their size. These exponents are empirical estimates from a single study, not exact values.

#4.3 Post-Training (Alignment)

A base model predicting the statistically most likely next word of web text is generally unhelpful as an assistant. Post-training aligns the model to user intent:

1. Supervised Fine-Tuning (SFT): The model is fine-tuned on high-quality pairs of (Instruction $\rightarrow$ Expert Response) using standard cross-entropy loss.
2. Preference Optimization: The model is further tuned using Reinforcement Learning from Human Feedback (RLHF) via PPO (Schulman et al., 2017; applied to language models by Ouyang et al., 2022), or through Direct Preference Optimization (DPO; Rafailov et al., 2023), which updates the policy to increase the likelihood of human-preferred outputs over rejected ones.

#4.4 Inference and Decoding

During deployment (inference), the model generates text autoregressively. At step $t$, the model outputs a probability distribution over the vocabulary. The next token is sampled from this distribution. Hyperparameters control this sampling:

• Temperature ($\tau$): Flattens or sharpens the softmax distribution. $\tau \to 0$ approaches deterministic greedy decoding (always picking the highest probability token); $\tau > 1$ increases randomness.
• Top-$k$ Sampling: Restricts the sample space to the $k$ most likely tokens.
• Nucleus (Top-$p$) Sampling: Restricts the sample space to the smallest set of tokens whose cumulative probability exceeds $p$.

• Base Models: Raw pretrained models (e.g., the Llama 3 base checkpoints released in 2024; Llama Team, 2024). Highly capable but require specific prompting formats to elicit tasks.
• Instruct/Chat Models: Models post-trained via SFT and RLHF for conversational compliance (e.g., the Llama 3 Instruct releases of 2024; Llama Team, 2024). Examples are as of 2024; see MODEL-001 for current models.
• Dense vs. Sparse Architectures: Dense models activate every parameter for every token. Mixture-of-Experts (MoE) models (e.g., Mixtral 8x7B, released 2023-2024; see TECH-MOE-001) route each token to a subset of specialized sub-networks, decoupling total parameter count from inference compute (FLOPs).
• Open-Weight vs. Proprietary: "Open-weight" models release the trained parameter matrices publicly (e.g., Llama, Mistral), though often restricting training data access, unlike fully "open-source" software. Proprietary models are accessible only via API, and their architectures, training data and parameter counts are generally undisclosed.

• Retrieval-Augmented Generation (RAG): LLMs integrated with vector databases to retrieve external, proprietary, or up-to-date information, inserting it into the prompt context to reduce hallucination and anchor the model in factuality.
• Code Generation and Copilots: Fine-tuned on GitHub repositories, LLMs synthesize boilerplate code, translate between programming languages, and generate unit tests.
• Tool Use and Agentic Workflows: LLMs trained to output structured JSON representing API calls (e.g., web search, python execution), enabling them to act as autonomous agents interacting with external systems.
• Multimodal Interactivity: Frontier LLMs accept interleaved image, audio, and text inputs, mapping them into the same latent space to perform visual question answering and audio translation.

#Demonstrated Advantages

• Zero-Shot Transfer: Solves tasks it was never explicitly fine-tuned to perform purely through prompt conditioning.
• Fluent Text Generation: Produces grammatical, stylistically controllable text and useful machine translation in many languages, with quality varying by language and domain.

#Core Limitations

• Hallucinations: Generates fluent, plausible-sounding falsehoods due to the probabilistic nature of next-token prediction and lack of external grounding.
• KV-Cache Memory Bottleneck: Inference requires caching the Key and Value matrices of all past tokens in GPU memory. As context lengths scale to millions of tokens, memory bandwidth strictly limits batch sizes and inference speed.
• Uneven Reasoning: Multi-step deduction (e.g., formal logic or long arithmetic) is a contested limitation that is model- and prompt-dependent. Chain-of-thought prompting improves results on some multi-step tasks (Wei et al., 2022), and errors in intermediate tokens can propagate to the final answer; the extent of this failure mode varies across models and has not been settled here.
• Data Contamination: Widespread inclusion of benchmark evaluation datasets in the pretraining corpora severely complicates the assessment of true generalization versus rote memorization.

#Unresolved Scientific Questions

1. Is Next-Token Prediction Sufficient? Does autoregressive modeling merely yield sophisticated statistical pattern matching (stochastic mimicry), or does it force the network to construct a causal, low-dimensional "world model" representing the true physical and logical structures underlying the text?
2. The Data Wall: Pretraining currently consumes trillions of tokens. High-quality human text is finite. Can models continue scaling using synthetic data generated by other models without suffering from "model collapse" (Shumailov et al., 2024)? Projections of when public human-generated text will be exhausted are uncertain (Villalobos et al., 2022).
3. Mechanistic Interpretability: Can the billions of continuous parameters be successfully reverse-engineered into auditable, human-readable circuits?

Parameter counts, architectures and training-token counts of proprietary frontier models are generally undisclosed, so no figure is given here for the current frontier; current model facts belong in MODEL-001. For scale of disclosed open-weight models as of 2024: Llama 3 405B was pre-trained on 15.6 trillion tokens using up to 16K H100 GPUs (Llama Team, 2024).

Recent work emphasizes test-time compute: giving models additional inference-time tokens to reason via extended chain of thought (e.g., OpenAI's o1, which OpenAI describes as trained with large-scale reinforcement learning; OpenAI, 2024; no inference-time search algorithm is described in the abstract of the system card [VERIFICATION NEEDED: full system card and launch post not read; confirm that no search method is disclosed]). Explicit search methods such as Monte Carlo Tree Search appear in some open research systems [VERIFICATION NEEDED: cite a specific system, e.g. Marco-o1], but are not documented for o1. See TECH-REASON-001.

• Near-Term (0–3 Years) [speculative forecast, unsourced; no forecast record assigned]: Wider deployment of smaller language models running locally on edge devices. Consolidation of RAG patterns into standard enterprise software architectures.
• Medium-Term (3–10 Years) [speculative]: Possible transition from text-completion engines to long-horizon autonomous agents capable of independently executing software engineering and data analysis workflows lasting days or weeks. Possible exhaustion of the public human-generated text supply (estimates are uncertain; Villalobos et al., 2022), which could push developers toward synthetic data or other modalities.
• Long-Term (10+ Years) [speculative]: Potential architectural paradigm shifts beyond purely autoregressive transformers (e.g., state-space models, energy-based models, or continuous recurrent memories) that decouple reasoning depth from output token length.

• Foundational Concepts:
  • FOUND-001 Artificial Intelligence: Parent discipline.
• Related Methods & Technologies:
  • TECH-ARCH-001 The Transformer Architecture: The structural backbone of LLMs.
  • CAP-GENAI-001 Generative Artificial Intelligence: Generative mechanics relying on autoregression.
• Advanced Implications & Future Trajectories:
  • SAFE-ALIGN-001 The AI Alignment Problem: RLHF, sycophancy, and alignment tuning of LLMs.
  • AGI-001 Artificial General Intelligence: Theoretical limits of LLM scaling.
• Living References:
  • MODEL-001 AI/SI Model Registry: Dated model facts, deliberately kept out of this stable article.

A multimodal foundation model is a large model, pretrained on broad data, that processes or generates more than one kind of data (modality), such as text, images, audio and video, within a single system. The term covers models that understand images and answer in text, models that generate images from text, and models that handle several input and output modalities.

Language models learn from text alone, which limits their access to visual and auditory information. Multimodal models extend the same large-scale pretraining approach to other signals and, in most current designs, reuse the Transformer architecture (see [TECH-ARCH-001]) by converting each modality to a sequence of vectors. They are the basis for image captioning, visual question answering, document and chart understanding, speech recognition, and text-to-image or text-to-video generation.

Three design questions organise the field: how each modality is turned into tokens or embeddings (see [TECH-TOK-001]); how those representations are connected to a language model or to each other; and whether the model understands, generates, or both. Generation of images and video has largely followed a separate line of diffusion and flow-based models (see [TECH-DIFF-001]), so "multimodal" currently covers two partly distinct families (Section 5).

This article reports only what developers have disclosed for closed systems. Architecture, training data, and modality coverage of such systems beyond developer statements are undisclosed.

Vision models and language models developed separately until the early 2020s. The Vision Transformer (Dosovitskiy et al., 2021) showed that a pure Transformer applied to sequences of image patches can perform very well on image classification when pre-trained on large amounts of data, providing a common architecture for both modalities.

Contrastive language-image pre-training changed the supervision signal. CLIP (Radford et al., 2021) trained on 400 million image-text pairs from the internet with the task of predicting which caption goes with which image, and used natural language to reference visual concepts for zero-shot transfer, benchmarked on over 30 computer vision datasets. SigLIP (Zhai et al., 2023) replaced the softmax-normalised contrastive loss with a pairwise sigmoid loss that does not need a global view of pairwise similarities.

The connection of vision to pretrained language models then followed. Frozen (Tsimpoukelli et al., 2021) trained a vision encoder to produce a sequence of embeddings acting as a prefix to a frozen language model, enabling few-shot multimodal learning. Flamingo (Alayrac et al., 2022) proposed architectural changes to bridge pretrained vision-only and language-only models and to handle arbitrarily interleaved images and text, trained on large-scale multimodal web data. BLIP-2 (Li et al., 2023) bridged frozen image encoders and frozen language models with a lightweight Querying Transformer and reported outperforming Flamingo80B on zero-shot VQAv2 by 8.7% with 54 times fewer trainable parameters. LLaVA (Liu et al., 2023) connected a vision encoder to a language model and trained it on instruction-following data generated by language-only GPT-4.

In generation, DALL-E (Ramesh et al., 2021) modelled text and image tokens as a single stream with an autoregressive Transformer; latent diffusion models (Rombach et al., 2022) ran diffusion in the latent space of pretrained autoencoders with cross-attention for conditioning; Diffusion Transformers (Peebles & Xie, 2023) replaced the U-Net backbone with a Transformer; and Esser et al. (2024) combined rectified flow with a Transformer using separate weights for the two modalities. Speech recognition was scaled by Whisper (Radford et al., 2023), trained on 680,000 hours of multilingual and multitask supervision from the internet.

Closed frontier developers describe multimodal systems. The GPT-4 report describes a model that "can accept image and text inputs and produce text outputs" (OpenAI et al., 2023), and the Gemini report describes a family handling image, audio, video and text understanding (Gemini Team, 2023). Chameleon (Chameleon Team, 2024) reported an early-fusion, token-based model that understands and generates images and text in arbitrary sequence.

#4.1 Encoding modalities

Each non-text modality is mapped to a sequence of vectors. For images, a Vision Transformer splits an image into patches $p_1,\dots,p_m$ and projects each to $x_j = W_p\, \mathrm{flatten}(p_j) + e_j$, with positional embedding $e_j$ (Dosovitskiy et al., 2021). Audio is typically converted to a spectrogram and processed analogously (Radford et al., 2023). The result is a sequence of continuous embeddings, or, in token-based approaches, a sequence of discrete codes from a learned image tokenizer.

#4.2 Contrastive alignment

CLIP-style training embeds an image and a caption into a shared space and trains with a symmetric cross-entropy over a batch of $B$ pairs. With normalised image embeddings $u_i$, text embeddings $v_i$ and a temperature $\tau$,

and the text-to-image loss $\mathcal{L}_{t \to i}$ is defined symmetrically, the total being their average. SigLIP replaces the softmax over the batch with an independent binary (sigmoid) classification for each image-text pair (Zhai et al., 2023). Matching pairs are positives and all other pairings in the batch are negatives.

#4.3 Connecting vision to a language model

Three connection patterns dominate.

1. Prefix or projection: the vision encoder output is mapped by a linear layer or small network into the language model's embedding space and inserted as extra input tokens (Frozen; LLaVA).
2. Query-based bottleneck: a small Transformer with learned queries (the Q-Former in BLIP-2) compresses image features to a fixed number of vectors for a frozen language model.
3. Cross-attention layers: new layers inside a frozen language model attend to visual features, with support for interleaved image and text input (Flamingo).

Training commonly proceeds in stages: align the connector on image-caption data, then instruction-tune on multimodal dialogue. LLaVA used instruction data generated by GPT-4 from text descriptions and reported 85.1% relative score against GPT-4 on a synthetic instruction-following dataset (Liu et al., 2023), a figure whose meaning depends on that synthetic benchmark and GPT-4 as a reference.

#4.4 Early-fusion token models

In early fusion, all modalities are converted to discrete tokens from the start and a single Transformer models the interleaved sequence with a next-token objective. Chameleon is a published example for text and images, with an alignment recipe and training approach tailored to this setting; its authors report competitive text performance and human-judged long-form mixed-modal generation results compared with several named models (Chameleon Team, 2024).

#4.5 Generative image and video models

Latent diffusion trains a denoiser in an autoencoder's latent space, using cross-attention to condition on text (Rombach et al., 2022). A diffusion Transformer treats latent patches as tokens (Peebles & Xie, 2023). In rectified-flow models a straight-line path connects data to noise and a network predicts the velocity along it; Esser et al. (2024) report that, in a large-scale study, their noise-sampling improvements outperform established diffusion formulations for high-resolution text-to-image synthesis, and that their architecture with separate weights per modality improves text comprehension and typography (see [TECH-DIFF-001]).

For closed models the boundary between families is a matter of undisclosed implementation: whether an image generator in a given product is the same network as the language model, or a separate model called by it, is undisclosed unless the developer states it.

• Image and document understanding, including captioning, visual question answering and chart reading, via VLMs such as LLaVA and BLIP-2.
• Retrieval across modalities, where CLIP-style embeddings allow text queries against image collections (see [TECH-RAG-001]).
• Speech recognition and transcription, with Whisper as an openly released example (Radford et al., 2023).
• Image generation, deployed through latent diffusion and related systems (Rombach et al., 2022; Esser et al., 2024).
• Robotics and agents, where vision-language models serve as perception and planning components (see [TECH-ROBOT-001]).
• Evaluation. MMMU (Yue et al., 2024) contains 11.5K multimodal questions from college exams, quizzes and textbooks across 30 subjects, and reported at publication that GPT-4V and Gemini Ultra achieved 56% and 59%. Benchmark scores change quickly and are tracked in MODEL-001; this figure is a dated result from the benchmark paper.

#Advantages

Transfer from large-scale pretraining to many perception tasks without task-specific training (CLIP); reuse of strong pretrained language models, reducing trainable parameters (BLIP-2); a common interface through language.

#Limitations

• Object hallucination. Li et al. (2023) report that evaluated large vision-language models mostly suffer from severe object hallucination, generating objects not present in the image, with objects that are frequent in visual instructions or co-occur with image objects more prone to be hallucinated, and that existing evaluation methods can be affected by instructions and generation style (see [CAP-LIMITS-001]).
• Weak visual representations. Tong et al. (2024) identify "CLIP-blind pairs", images that CLIP perceives as similar despite clear visual differences, build the MMVP benchmark from them, and report that state-of-the-art systems including GPT-4V struggle with straightforward questions across nine basic visual patterns, sometimes giving hallucinated explanations; they find a correlation between patterns that challenge CLIP and model failures.
• Benchmark difficulty at expert level. The MMMU authors report that the strongest models tested at publication fell well short of expert-level accuracy.
• Data and rights issues. Web-scale image-text training data raises provenance and copyright questions (see [TECH-DATA-001]).
• Cost. Images, audio and video produce many tokens per input, increasing compute and latency.

#Unresolved questions

1. Whether early-fusion, single-model designs outperform modular systems at the same compute; the evidence comes from few comparisons.
2. Whether multimodal training improves language-only reasoning or grounds meaning in the sense debated in [AGI-WM-001]; this encyclopedia treats the question as open.
3. How to evaluate perception without contamination from benchmark leakage or language priors.

As of 2026-10-15, contrastive image-text encoders, projection or query-based connectors to language models, and diffusion or flow-based generators are well documented in the open literature. Claims that a particular closed model is "natively multimodal" or generates images within the same network should be sourced to the developer; beyond the statements cited here (GPT-4 image and text inputs with text outputs; Gemini's image, audio, video and text understanding), architectures, data and training procedures are undisclosed. Reported benchmark scores for named models are dated and held in MODEL-001. Evaluation of perception remains an area where independent studies find systematic failures (Tong et al., 2024; Li et al., 2023).

Observed trends: unified models that both understand and generate several modalities (Chameleon), larger use of video and audio, and better evaluation of perceptual faults. Whether single-network multimodal models will displace composed systems is an open research question; no forecast record is registered in this article.

• Foundational Concepts:
  • TECH-LLM-001 Large Language Models: The language backbone.
  • TECH-ARCH-001 The Transformer Architecture: Vision Transformer and attention.
  • TECH-DIFF-001 Diffusion Models: Generative image and video modelling.
• Related Methods & Technologies:
  • TECH-TOK-001 Tokenization and Embeddings: Modality tokenization and shared embedding spaces.
  • TECH-RAG-001 Retrieval-Augmented Generation: Multimodal retrieval.
  • TECH-ROBOT-001 Robotics and Embodied AI: Vision-language models in embodied settings.
  • CAP-GENAI-001 Generative Artificial Intelligence: Generative capabilities.
• Advanced Implications & Future Trajectories:
  • CAP-BENCH-001 AI Evaluation: Multimodal benchmarks.
  • AGI-WM-001 World Models: Grounding and world modelling.

Retrieval-Augmented Generation (RAG) is a technique in which a language model's output is conditioned on documents retrieved from an external collection at inference time, so that the model draws on explicit, updatable non-parametric memory in addition to the knowledge stored in its weights (its parametric memory). The term was introduced by Lewis et al. (2020) for models combining a pre-trained sequence-to-sequence generator with a dense retrieval index, and is now used more broadly for any pipeline that retrieves text and places it in a model's context.

A trained language model holds a fixed snapshot of its training data, cannot cite where a statement came from, and has no access to private documents. Lewis et al. (2020) identify these as open problems: precisely accessing and manipulating knowledge, providing provenance for decisions, and updating world knowledge. Retrieval addresses them by separating knowledge storage from the generator: the collection can be changed without retraining, and the passages used can be shown to the user.

RAG is a mitigation, not a guarantee. Retrieval can return irrelevant or conflicting passages, and a model can ignore or misuse correct ones (Section 7). It is also not the only means of giving a model fresh information: long-context prompting, tool use by agents (see [TECH-AGENT-001]), and fine-tuning are alternatives or complements, and a production system often combines several.

Sparse retrieval, in which documents are scored by weighted term overlap (TF-IDF and the BM25 family; Robertson & Zaragoza, 2009), predates neural methods and remains a strong baseline. The BEIR benchmark (Thakur et al., 2021) evaluated 10 retrieval systems on 18 datasets and found BM25 a robust zero-shot baseline, with re-ranking and late-interaction models achieving the best average zero-shot results at high computational cost.

Neural retrieval-augmented language modelling developed in several strands. REALM (Guu et al., 2020) pre-trained a latent knowledge retriever with a masked-language-modelling signal, back-propagating through retrieval over millions of documents, and reported 4–16% absolute accuracy gains over prior methods on three open-domain QA benchmarks. Dense Passage Retrieval (Karpukhin et al., 2020) used a dual-encoder trained on a small number of question-passage pairs and reported 9–19% absolute gains in top-20 passage accuracy over a Lucene-BM25 system. Fusion-in-Decoder (Izacard & Grave, 2021) showed that a generative reader improves as the number of retrieved passages increases. RAG (Lewis et al., 2020) combined a dense Wikipedia index with a pre-trained seq2seq generator and compared two formulations. RETRO (Borgeaud et al., 2022) conditioned an autoregressive model on chunks retrieved from a 2-trillion-token database through chunked cross-attention and reported performance comparable to GPT-3 and Jurassic-1 on the Pile with 25 times fewer parameters. Atlas (Izacard et al., 2023) studied few-shot learning with retrieval-augmented models and reported over 42% accuracy on Natural Questions with 64 examples, outperforming a 540B-parameter model by 3% with 50 times fewer parameters.

From 2022, as instruction-tuned models became widely available, a simpler pattern predominated: retrieve text with an off-the-shelf retriever and prepend it to the prompt of a frozen model. REPLUG (Shi et al., 2024) formalised this black-box setting and showed that the language model can supervise the retriever. The pattern now commonly called "naive RAG" and its refinements are surveyed by Gao et al. (2023).

#4.1 The basic pipeline

1. Indexing. Documents are split into chunks, each chunk is mapped to a vector by an embedding model (see [TECH-TOK-001]) and stored with its text in a vector index; a sparse (BM25) index may be built in parallel.
2. Retrieval. The query is embedded and the $k$ nearest chunks are retrieved by similarity, commonly cosine similarity $\mathrm{sim}(q, d) = \dfrac{\mathbf{q}\cdot\mathbf{d}}{\lVert\mathbf{q}\rVert\,\lVert\mathbf{d}\rVert}$, using approximate nearest-neighbour search for large collections.
3. (Optional) Re-ranking. A more expensive model rescoring the top candidates. ColBERT (Khattab & Zaharia, 2020) is a late-interaction design that encodes query and document independently and models their fine-grained similarity after, so document representations can be pre-computed offline.
4. Augmented generation. The retrieved text is concatenated with the question in the prompt, and the model generates an answer, optionally with citations.

#4.2 Probabilistic formulation

In the original RAG model (Lewis et al., 2020) a retriever $p_\eta(z \mid x)$ gives a distribution over passages $z$ for input $x$, and a generator $p_\theta(y \mid x, z)$ produces the output. The passages are treated as a latent variable and marginalised over the top $k$ retrieved. The "sequence" formulation conditions on the same passage across the whole output:

while the "token" formulation marginalises per token and so can use different passages at different positions:

In contemporary prompt-based pipelines there is no explicit marginalisation; the retrieved text simply forms part of the context, $p_\theta(y \mid x, d_1,\dots,d_k)$.

#4.3 Training and integration choices

• Frozen versus trained components. Retriever and generator can be trained jointly (REALM, RAG, Atlas), the retriever can be trained from generator feedback (REPLUG), or both can be used off the shelf.
• Where retrieved text enters. In the prompt (REPLUG, most deployed systems), via cross-attention to retrieved chunks (RETRO), or by encoding each passage separately and fusing in the decoder (Fusion-in-Decoder).
• Adaptive retrieval. Self-RAG (Asai et al., 2024) trains a single model to decide when to retrieve and to critique retrieved passages and its own generations with special reflection tokens; the authors report that 7B and 13B versions outperform ChatGPT and a retrieval-augmented Llama 2-chat on open-domain QA, reasoning and fact verification, and improve factuality and citation accuracy for long-form generation relative to the models they compared.
• Structured and graph indexes. GraphRAG (Edge et al., 2024) uses an LLM to derive an entity knowledge graph and pre-generate community summaries so that "global" questions about a whole corpus, which conventional RAG handles poorly, can be answered; the authors report improved comprehensiveness and diversity of answers on datasets in the 1-million-token range.

The survey by Gao et al. (2023) organises the field into Naive RAG, Advanced RAG (pre- and post-retrieval refinements such as query rewriting and re-ranking) and Modular RAG, and examines retrieval, generation and augmentation techniques separately.

• Open-domain question answering, the setting of REALM, DPR, Fusion-in-Decoder, RAG and Atlas.
• Enterprise and private-document assistants, where the collection is non-public and changes frequently. Public descriptions of commercial products usually do not specify retrieval architecture; specifics for closed products are undisclosed unless the vendor documents them.
• Search-augmented assistants, in which a web-search tool supplies the documents; this is a deployed pattern, but ranking, filtering and citation-selection details are not generally disclosed.
• Query-focused summarisation of large corpora (GraphRAG), reported as research results by Edge et al. (2024) rather than as an independently benchmarked product.

#Advantages

Updatable knowledge without retraining; provenance through citation of retrieved text; ability to work over private collections; and in several studies, similar or better accuracy than much larger models on knowledge-intensive tasks (Borgeaud et al., 2022; Izacard et al., 2023).

#Limitations

• Using context imperfectly. Liu et al. (2024) find that performance on multi-document QA and key-value retrieval can degrade significantly when relevant information is in the middle of a long input context, even for explicitly long-context models; accuracy is often highest at the beginning or end.
• Retrieval quality is not monotone in "relevance". Cuconasu et al. (2024) report that high-scoring retrieved documents that do not contain the answer negatively affect the LLM, and, as a counter-intuitive finding, that adding random documents to the prompt improved accuracy by up to 35% in their setup. The result shows that the interaction between retrieved context and the model is not well characterised; it is not a recommendation to add random text.
• Evaluation is multi-dimensional. Ragas (Es et al., 2024) proposes reference-free metrics separating retrieval relevance, faithful use of the context and generation quality, because a correct final answer can hide a retrieval failure and vice versa. Automated metrics of this kind depend on an LLM judge and are themselves approximations.
• Global questions. Standard RAG is an explicit retrieval method and fails on questions about a whole corpus (Edge et al., 2024).
• Security and integrity. Retrieved text is untrusted input that is placed in the model's context; instructions embedded in documents can influence the model, and the index can be poisoned (see [SAFE-SEC-001]).
• Hallucination persists. Grounding reduces but does not eliminate unsupported statements (see [CAP-LIMITS-001]).

#Unresolved questions

1. How best to combine retrieval with very long context windows, and when retrieval remains necessary.
2. Principled criteria for the number, ordering and filtering of retrieved passages, given the position and noise effects above.
3. Reliable, low-cost evaluation of faithfulness to retrieved sources.

As of 2026-10-15, retrieval-augmented pipelines are a standard way to connect language models to external text, with a mature set of components (dense and sparse retrievers, re-ranking, chunking and prompt assembly) documented in the academic literature and surveys. The strongest published evidence consists of benchmark comparisons on knowledge-intensive tasks, mostly with models and collections specific to each paper. Claims about the effect of RAG on real-world factual accuracy in a given product depend on the retrieval corpus and are not general. Specific architectures of closed commercial retrieval features are undisclosed. Model and system details are tracked in MODEL-001.

Observed directions: adaptive retrieval decided by the model (Self-RAG), graph and hierarchical indexes (GraphRAG), reference-free evaluation (Ragas), and integration with tool-using agents (see [TECH-AGENT-001]). The extent to which longer context windows or in-weights memory will replace retrieval is an open question; no forecast record is registered in this article.

• Foundational Concepts:
  • TECH-LLM-001 Large Language Models: The generators used in RAG.
  • TECH-TOK-001 Tokenization and Embeddings: Embedding models and chunk sizes in tokens.
• Related Methods & Technologies:
  • TECH-ARCH-001 The Transformer Architecture: Cross-attention and context windows.
  • TECH-AGENT-001 Autonomous Agents: Retrieval as one tool among many.
  • TECH-DATA-001 Training Data, Contamination and Provenance: Provenance of indexed material.
• Advanced Implications & Future Trajectories:
  • CAP-LIMITS-001 Hallucination, Reliability and Calibration: What retrieval does and does not fix.
  • SAFE-SEC-001 AI Security: Prompt Injection, Poisoning, Extraction: Untrusted retrieved content.

Tokenization is the procedure that converts raw input (text, as a sequence of bytes or characters) into a sequence of discrete symbols called tokens drawn from a fixed vocabulary. An embedding is a learned dense vector assigned to each token (or, in later models, to a whole passage), which is the numerical form on which a neural network operates. Together they define the interface between human-readable data and the Transformer layers of a language model.

A neural language model (see [TECH-LLM-001]) cannot consume a string directly. It needs a finite vocabulary of $|V|$ symbols, each mapped to a vector, and it predicts a distribution over that vocabulary at each step. The choice of vocabulary is therefore a modelling decision with consequences for several things:

• Sequence length and compute. A tokenizer that represents a text in fewer tokens makes attention and generation cheaper, because cost grows with the number of tokens (see [TECH-ARCH-001], [TECH-INFER-001]).
• Coverage. Word-level vocabularies cannot represent unseen words; characters or bytes can represent anything but yield long sequences.
• Fairness and cost. The same content can require very different token counts in different languages, which affects price, latency and usable context (Petrov et al., 2023; Ahia et al., 2023).
• Reliability. Tokens present in a vocabulary but rare or absent in training data can cause erratic behaviour (Land & Bartolo, 2024).

Embeddings matter because the geometry of the vector space determines what the model can easily generalise. The same idea extends beyond input: embedding models that map passages or images to vectors are the basis of semantic search (see [TECH-RAG-001]) and of multimodal alignment (see [TECH-MM-001]).

Before neural models, text was represented as sparse count vectors over a word vocabulary. Dense, learned word vectors became practical with the word2vec models of Mikolov et al. (2013a, 2013b), which learned vectors from large corpora fast enough to use 1.6 billion words in under a day and evaluated them on syntactic and semantic similarity. GloVe (Pennington, Socher & Manning, 2014) fitted vectors to global co-occurrence statistics. fastText (Bojanowski et al., 2017) composed word vectors from character n-grams so that rare and unseen words obtain representations.

The open-vocabulary problem in neural machine translation led Sennrich, Haddow and Birch (2016) to adopt a segmentation based on the byte pair encoding (BPE) compression algorithm, encoding rare words as sequences of subword units; they reported gains of 1.1 and 1.3 BLEU over a back-off dictionary on WMT 15 English–German and English–Russian. Wu et al. (2016) described a related wordpiece approach for Google's translation system. Kudo (2018) proposed a unigram-language-model segmentation with subword regularisation, and Kudo and Richardson (2018) released SentencePiece, which trains directly on raw text without requiring a pre-tokenised word sequence. BERT (Devlin et al., 2019) used a subword vocabulary together with contextual representations, replacing static word vectors with ones that depend on the surrounding sentence.

More recently, token-free approaches processing bytes directly have been scaled to billions of parameters (Xue et al., 2022; Pagnoni et al., 2024), without displacing subword tokenizers in widely documented models as of 2026-10-15.

#4.1 Byte-pair encoding

BPE learns a vocabulary from a training corpus. In its subword form (Sennrich et al., 2016):

1. Start with a vocabulary of base symbols (characters, or the 256 possible byte values in byte-level variants).
2. Count all adjacent symbol pairs in the corpus.
3. Merge the most frequent pair into a new symbol and add it to the vocabulary; record the merge rule.
4. Repeat until the vocabulary reaches a target size.

At encoding time the learned merges are applied to new text in the order in which they were learned. Frequent words become single tokens; rare words decompose into several pieces. Byte-level BPE starts from bytes so that any Unicode string, including text never seen in training, is representable.

#4.2 Unigram and WordPiece variants

The unigram model (Kudo, 2018) starts from a large candidate vocabulary and prunes it to maximise the likelihood of the training data under a unigram language model; it can produce several segmentations of the same string with different probabilities, which subword regularisation uses as training noise. WordPiece (Wu et al., 2016) is likewise a data-driven subword scheme. The families differ in how the vocabulary is built and in how ties and ambiguities are resolved; all of them produce a deterministic or sampled mapping from a string to a sequence of vocabulary indices.

#4.3 The embedding layer

Given token indices $t_1,\dots,t_n$ with $t_i \in \{1,\dots,|V|\}$ and an embedding matrix $E \in \mathbb{R}^{|V|\times d}$, the input to the Transformer is the matrix whose $i$-th row is $x_i = E_{t_i}$, with positional information added or applied inside attention (Vaswani et al., 2017; see [TECH-ARCH-001]). The matrix $E$ is a trained parameter, so $|V|\cdot d$ parameters are spent on it; the output projection back to $|V|$ logits has the same shape and some models share the two matrices. A language model is trained to minimise the cross-entropy

over tokens, so its units of prediction are the tokenizer's tokens.

#4.4 Why tokenizers complicate comparison

Because $\mathcal{L}$ is measured per token, models with different tokenizers have different units. If a tokenizer yields on average $\tau$ tokens per byte of text and the loss in nats per token is $\ell$, the loss per byte in bits is $\ell\,\tau / \ln 2$. Comparisons across tokenizers should therefore use a per-byte or per-character measure, not per-token perplexity. This is a direct consequence of the definitions and is not a claim about any specific model.

#4.5 Static versus contextual embeddings

word2vec, GloVe and fastText assign one vector per word type, so a polysemous word has one vector. A Transformer layer stack outputs contextual vectors in which the same token receives different representations in different sentences (Devlin et al., 2019). Sentence-level embedding models, such as Sentence-BERT (Reimers & Gurevych, 2019), pool or fine-tune such representations so that semantically similar sentences have high cosine similarity; this is the mechanism used in embedding-based retrieval.

#4.6 Vocabulary size trade-off

A larger vocabulary shortens sequences but enlarges the embedding and output matrices and makes each rare token less frequently trained. As one documented data point, the Llama 3 report describes a 128K-token vocabulary (100K tokens from the tiktoken tokenizer plus 28K additional tokens for non-English languages) and states that, versus the Llama 2 tokenizer, compression on a sample of English data improved from 3.17 to 3.94 characters per token (Grattafiori et al., 2024). This is a single developer-reported figure on one sample; it is not a general law.

The Byte Latent Transformer (Pagnoni et al., 2024) groups bytes into patches whose lengths depend on the entropy of the next byte; the authors report the first FLOP-controlled scaling study of byte-level models up to 8B parameters and 4T training bytes and better scaling than token-based models at fixed inference cost. This is single-group evidence (Preliminary).

• Text LLMs. Every widely used autoregressive language model uses a tokenizer, with the vocabulary and algorithm usually published for open-weight models. For closed models the vocabulary may be exposed through the API or a library, but whether the training-time tokenizer matches a published one is a developer matter; this article does not infer it.
• Billing. Commercial APIs commonly price by tokens. Ahia et al. (2023) analysed OpenAI's API on multilingual benchmarks in 22 languages and report that speakers of many of the languages are overcharged while obtaining poorer results, because of differences in token counts.
• Retrieval. Dense passage encoders map queries and documents to vectors for nearest-neighbour search (Reimers & Gurevych, 2019; Karpukhin et al., 2020).
• Cross-modal embeddings. CLIP-style models align image and text embeddings in a shared space (see [TECH-MM-001]).

#Advantages of subword tokenization

Fixed, moderate sequence lengths; no unseen-word failure; a mature toolchain (Kudo & Richardson, 2018).

#Limitations

• Cross-language disparity. Petrov et al. (2023) report that the same text translated into different languages can differ in tokenization length by up to 15 times in some cases, even for tokenizers built for multilingual support, with character- and byte-level models still showing over 4x differences for some language pairs. They argue this produces unequal cost, latency and available context (see [SAFE-ETHICS-001]).
• Tokenizer quality affects downstream performance. Rust et al. (2021) examine monolingual performance of multilingual models as a function of tokenizer quality.
• Under-trained ("glitch") tokens. Land and Bartolo (2024) present methods to detect vocabulary tokens that are nearly or entirely absent from training data and show such tokens are prevalent across a diverse set of models; these can induce unwanted behaviour.
• Opaque units. Models see tokens, not characters. Tasks that depend on spelling or on character-level structure can be harder; the extent for any given model must be measured, not assumed.

#Unresolved questions

1. Whether tokenizer-free models will match subword models at the largest scales; current evidence is limited to the studies cited (Preliminary).
2. How to design vocabularies that are fair across languages without large cost in compute.
3. How much of the geometric structure of embedding spaces reflects meaning rather than corpus statistics; the word-vector literature measures similarity tasks and does not settle this.

As of 2026-10-15, subword tokenization (BPE variants and SentencePiece-style unigram or BPE models) remains the standard in documented language models, with byte-level BPE common for coverage. Dense learned embeddings, both input embeddings and sentence-level retrieval embeddings, are standard components. Claims about the tokenizer or vocabulary of a specific closed model should be taken only from developer documentation or from the model's public tokenizer, and are otherwise undisclosed. Current vocabulary sizes and compression rates of named models belong in the Living Reference (MODEL-001), not in this article.

Observed directions: dynamic or learned segmentation (Pagnoni et al., 2024), explicit tests for under-trained tokens (Land & Bartolo, 2024), and fairness-aware tokenizer design (Petrov et al., 2023). Whether byte-level or patch-based models become the default is open; no forecast is registered here.

• Foundational Concepts:
  • TECH-LLM-001 Large Language Models: The models that consume tokens.
  • TECH-ARCH-001 The Transformer Architecture: Where embeddings and positional information enter the network.
  • TECH-NN-001 Artificial Neural Networks: Learned representations in general.
• Related Methods & Technologies:
  • TECH-ATTN-001 Attention Mechanisms: Cost of attention grows with token count.
  • TECH-RAG-001 Retrieval-Augmented Generation: Embedding-based retrieval.
  • TECH-MM-001 Multimodal Foundation Models: Image and audio tokens and shared embedding spaces.
  • TECH-INFER-001 Inference, Quantization and Efficiency: Token count drives serving cost.
• Advanced Implications & Future Trajectories:
  • SAFE-ETHICS-001 AI Ethics, Bias and Fairness: Language-community disparities.