--- id: TECH-RAG-001 title: "Retrieval-Augmented Generation" alternative_names: - "RAG" - "Retrieval-Augmented Language Models" - "Grounded Generation" category: "Technical → Language Models" volume: III chapter: 19 entry_type: "Technical Article" level: 3 status: "Draft" last_reviewed: 2026-10-15 version: "0.1.0" prerequisites: - "TECH-LLM-001" - "TECH-TOK-001" related_entries: - "TECH-ARCH-001" - "TECH-AGENT-001" - "CAP-LIMITS-001" - "SAFE-SEC-001" - "TECH-DATA-001" --- # Retrieval-Augmented Generation ## 1. Concise Definition **Retrieval-Augmented Generation (RAG)** is a technique in which a language model's output is conditioned on documents retrieved from an external collection at inference time, so that the model draws on explicit, updatable non-parametric memory in addition to the knowledge stored in its weights (its parametric memory). The term was introduced by Lewis et al. (2020) for models combining a pre-trained sequence-to-sequence generator with a dense retrieval index, and is now used more broadly for any pipeline that retrieves text and places it in a model's context. ## 2. Overview & Significance A trained language model holds a fixed snapshot of its training data, cannot cite where a statement came from, and has no access to private documents. Lewis et al. (2020) identify these as open problems: precisely accessing and manipulating knowledge, providing provenance for decisions, and updating world knowledge. Retrieval addresses them by separating knowledge storage from the generator: the collection can be changed without retraining, and the passages used can be shown to the user. RAG is a mitigation, not a guarantee. Retrieval can return irrelevant or conflicting passages, and a model can ignore or misuse correct ones (Section 7). It is also not the only means of giving a model fresh information: long-context prompting, tool use by agents (see [TECH-AGENT-001]), and fine-tuning are alternatives or complements, and a production system often combines several. ## 3. Historical Development Sparse retrieval, in which documents are scored by weighted term overlap (TF-IDF and the BM25 family; Robertson & Zaragoza, 2009), predates neural methods and remains a strong baseline. The BEIR benchmark (Thakur et al., 2021) evaluated 10 retrieval systems on 18 datasets and found BM25 a robust zero-shot baseline, with re-ranking and late-interaction models achieving the best average zero-shot results at high computational cost. Neural retrieval-augmented language modelling developed in several strands. REALM (Guu et al., 2020) pre-trained a latent knowledge retriever with a masked-language-modelling signal, back-propagating through retrieval over millions of documents, and reported 4–16% absolute accuracy gains over prior methods on three open-domain QA benchmarks. Dense Passage Retrieval (Karpukhin et al., 2020) used a dual-encoder trained on a small number of question-passage pairs and reported 9–19% absolute gains in top-20 passage accuracy over a Lucene-BM25 system. Fusion-in-Decoder (Izacard & Grave, 2021) showed that a generative reader improves as the number of retrieved passages increases. RAG (Lewis et al., 2020) combined a dense Wikipedia index with a pre-trained seq2seq generator and compared two formulations. RETRO (Borgeaud et al., 2022) conditioned an autoregressive model on chunks retrieved from a 2-trillion-token database through chunked cross-attention and reported performance comparable to GPT-3 and Jurassic-1 on the Pile with 25 times fewer parameters. Atlas (Izacard et al., 2023) studied few-shot learning with retrieval-augmented models and reported over 42% accuracy on Natural Questions with 64 examples, outperforming a 540B-parameter model by 3% with 50 times fewer parameters. From 2022, as instruction-tuned models became widely available, a simpler pattern predominated: retrieve text with an off-the-shelf retriever and prepend it to the prompt of a frozen model. REPLUG (Shi et al., 2024) formalised this black-box setting and showed that the language model can supervise the retriever. The pattern now commonly called "naive RAG" and its refinements are surveyed by Gao et al. (2023). ## 4. Technical / Conceptual Explanation ### 4.1 The basic pipeline 1. **Indexing.** Documents are split into chunks, each chunk is mapped to a vector by an embedding model (see [TECH-TOK-001]) and stored with its text in a vector index; a sparse (BM25) index may be built in parallel. 2. **Retrieval.** The query is embedded and the $k$ nearest chunks are retrieved by similarity, commonly cosine similarity $\mathrm{sim}(q, d) = \dfrac{\mathbf{q}\cdot\mathbf{d}}{\lVert\mathbf{q}\rVert\,\lVert\mathbf{d}\rVert}$, using approximate nearest-neighbour search for large collections. 3. **(Optional) Re-ranking.** A more expensive model rescoring the top candidates. ColBERT (Khattab & Zaharia, 2020) is a late-interaction design that encodes query and document independently and models their fine-grained similarity after, so document representations can be pre-computed offline. 4. **Augmented generation.** The retrieved text is concatenated with the question in the prompt, and the model generates an answer, optionally with citations. ### 4.2 Probabilistic formulation In the original RAG model (Lewis et al., 2020) a retriever $p_\eta(z \mid x)$ gives a distribution over passages $z$ for input $x$, and a generator $p_\theta(y \mid x, z)$ produces the output. The passages are treated as a latent variable and marginalised over the top $k$ retrieved. The "sequence" formulation conditions on the same passage across the whole output: $$ p(y \mid x) \approx \sum_{z \in \text{top-}k\,p_\eta(\cdot\mid x)} p_\eta(z \mid x)\, \prod_{i} p_\theta(y_i \mid x, z, y_{