Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) is a technique in which a language model's output is conditioned on documents retrieved from an external collection at inference time, so that the model draws on explicit, updatable non-parametric memory in addition to the knowledge stored in its weights (its parametric memory). The term was introduced by Lewis et al. (2020) for models combining a pre-trained sequence-to-sequence generator with a dense retrieval index, and is now used more broadly for any pipeline that retrieves text and places it in a model's context.

A trained language model holds a fixed snapshot of its training data, cannot cite where a statement came from, and has no access to private documents. Lewis et al. (2020) identify these as open problems: precisely accessing and manipulating knowledge, providing provenance for decisions, and updating world knowledge. Retrieval addresses them by separating knowledge storage from the generator: the collection can be changed without retraining, and the passages used can be shown to the user.

RAG is a mitigation, not a guarantee. Retrieval can return irrelevant or conflicting passages, and a model can ignore or misuse correct ones (Section 7). It is also not the only means of giving a model fresh information: long-context prompting, tool use by agents (see [TECH-AGENT-001]), and fine-tuning are alternatives or complements, and a production system often combines several.

Sparse retrieval, in which documents are scored by weighted term overlap (TF-IDF and the BM25 family; Robertson & Zaragoza, 2009), predates neural methods and remains a strong baseline. The BEIR benchmark (Thakur et al., 2021) evaluated 10 retrieval systems on 18 datasets and found BM25 a robust zero-shot baseline, with re-ranking and late-interaction models achieving the best average zero-shot results at high computational cost.

Neural retrieval-augmented language modelling developed in several strands. REALM (Guu et al., 2020) pre-trained a latent knowledge retriever with a masked-language-modelling signal, back-propagating through retrieval over millions of documents, and reported 4–16% absolute accuracy gains over prior methods on three open-domain QA benchmarks. Dense Passage Retrieval (Karpukhin et al., 2020) used a dual-encoder trained on a small number of question-passage pairs and reported 9–19% absolute gains in top-20 passage accuracy over a Lucene-BM25 system. Fusion-in-Decoder (Izacard & Grave, 2021) showed that a generative reader improves as the number of retrieved passages increases. RAG (Lewis et al., 2020) combined a dense Wikipedia index with a pre-trained seq2seq generator and compared two formulations. RETRO (Borgeaud et al., 2022) conditioned an autoregressive model on chunks retrieved from a 2-trillion-token database through chunked cross-attention and reported performance comparable to GPT-3 and Jurassic-1 on the Pile with 25 times fewer parameters. Atlas (Izacard et al., 2023) studied few-shot learning with retrieval-augmented models and reported over 42% accuracy on Natural Questions with 64 examples, outperforming a 540B-parameter model by 3% with 50 times fewer parameters.

As instruction-tuned models became widely available, a simpler pattern became common in practice (a characterisation of the field by this encyclopedia, not a measured adoption statistic): retrieve text with an off-the-shelf retriever and prepend it to the prompt of a frozen model. REPLUG (Shi et al., 2024) formalised this black-box setting and showed that the language model can supervise the retriever. The pattern now commonly called "naive RAG" and its refinements are surveyed by Gao et al. (2023).

#4.1 The basic pipeline

1. Indexing. Documents are split into chunks, each chunk is mapped to a vector by an embedding model (see [TECH-TOK-001]) and stored with its text in a vector index; a sparse (BM25) index may be built in parallel.
2. Retrieval. The query is embedded and the $k$ nearest chunks are retrieved by similarity, commonly cosine similarity $\mathrm{sim}(q, d) = \dfrac{\mathbf{q}\cdot\mathbf{d}}{\lVert\mathbf{q}\rVert\,\lVert\mathbf{d}\rVert}$, using approximate nearest-neighbour search for large collections.
3. (Optional) Re-ranking. A more expensive model rescores the top candidates. ColBERT (Khattab & Zaharia, 2020) is a late-interaction design that encodes query and document independently and models their fine-grained similarity after, so document representations can be pre-computed offline.
4. Augmented generation. The retrieved text is concatenated with the question in the prompt, and the model generates an answer, optionally with citations.

#4.2 Probabilistic formulation

In the original RAG model (Lewis et al., 2020) a retriever $p_\eta(z \mid x)$ gives a distribution over passages $z$ for input $x$, and a generator $p_\theta(y \mid x, z)$ produces the output. The passages are treated as a latent variable and marginalised over the top $k$ retrieved. The "sequence" formulation conditions on the same passage across the whole output:

while the "token" formulation marginalises per token and so can use different passages at different positions:

In contemporary prompt-based pipelines there is no explicit marginalisation; the retrieved text simply forms part of the context, $p_\theta(y \mid x, d_1,\dots,d_k)$.

#4.3 Training and integration choices

• Frozen versus trained components. Retriever and generator can be trained jointly (REALM, RAG, Atlas), the retriever can be trained from generator feedback (REPLUG), or both can be used off the shelf.
• Where retrieved text enters. In the prompt (REPLUG, most deployed systems), via cross-attention to retrieved chunks (RETRO), or by encoding each passage separately and fusing in the decoder (Fusion-in-Decoder).
• Adaptive retrieval. Self-RAG (Asai et al., 2024) trains a single model to decide when to retrieve and to critique retrieved passages and its own generations with special reflection tokens; the authors report that 7B and 13B versions outperform ChatGPT and a retrieval-augmented Llama 2-chat on open-domain QA, reasoning and fact verification, and improve factuality and citation accuracy for long-form generation relative to the models they compared.
• Structured and graph indexes. GraphRAG (Edge et al., 2024) uses an LLM to derive an entity knowledge graph and pre-generate community summaries so that "global" questions about a whole corpus, which conventional RAG handles poorly, can be answered; the authors report improved comprehensiveness and diversity of answers on datasets in the 1-million-token range.

The survey by Gao et al. (2023) organises the field into Naive RAG, Advanced RAG (pre- and post-retrieval refinements such as query rewriting and re-ranking) and Modular RAG, and examines retrieval, generation and augmentation techniques separately.

• Open-domain question answering, the setting of REALM, DPR, Fusion-in-Decoder, RAG and Atlas.
• Enterprise and private-document assistants, where the collection is non-public and changes frequently. Public descriptions of commercial products usually do not specify retrieval architecture; specifics for closed products are undisclosed unless the vendor documents them.
• Search-augmented assistants, in which a web-search tool supplies the documents; this is a deployed pattern, but ranking, filtering and citation-selection details are not generally disclosed.
• Query-focused summarisation of large corpora (GraphRAG), reported as research results by Edge et al. (2024) rather than as an independently benchmarked product.

#Advantages

Updatable knowledge without retraining; provenance through citation of retrieved text; ability to work over private collections; and in several studies, similar or better accuracy than much larger models on knowledge-intensive tasks (Borgeaud et al., 2022; Izacard et al., 2023).

#Limitations

• Using context imperfectly. Liu et al. (2024) find that performance on multi-document QA and key-value retrieval can degrade significantly when relevant information is in the middle of a long input context, even for explicitly long-context models; accuracy is often highest at the beginning or end.
• Retrieval quality is not monotone in "relevance". Cuconasu et al. (2024) report that high-scoring retrieved documents that do not contain the answer negatively affect the LLM, and, as a counter-intuitive finding, that adding random documents to the prompt improved accuracy by up to 35% in their setup. The result shows that the interaction between retrieved context and the model is not well characterised; it is not a recommendation to add random text.
• Evaluation is multi-dimensional. Ragas (Es et al., 2024) proposes reference-free metrics separating retrieval relevance, faithful use of the context and generation quality, because a correct final answer can hide a retrieval failure and vice versa. Automated metrics of this kind depend on an LLM judge and are themselves approximations.
• Global questions. Standard RAG is an explicit retrieval method and fails on questions about a whole corpus (Edge et al., 2024).
• Security and integrity. Retrieved text is untrusted input that is placed in the model's context; instructions embedded in documents can influence the model, and the index can be poisoned (see [SAFE-SEC-001]).
• Hallucination persists. Grounding reduces but does not eliminate unsupported statements (see [CAP-LIMITS-001]).

#Unresolved questions

1. How best to combine retrieval with very long context windows, and when retrieval remains necessary.
2. Principled criteria for the number, ordering and filtering of retrieved passages, given the position and noise effects above.
3. Reliable, low-cost evaluation of faithfulness to retrieved sources.

As of 2026-10-19, retrieval-augmented pipelines are a standard way to connect language models to external text, with a mature set of components (dense and sparse retrievers, re-ranking, chunking and prompt assembly) documented in the academic literature and surveys. The strongest published evidence consists of benchmark comparisons on knowledge-intensive tasks, mostly with models and collections specific to each paper. Claims about the effect of RAG on real-world factual accuracy in a given product depend on the retrieval corpus and are not general. Specific architectures of closed commercial retrieval features are undisclosed. Model and system details are tracked in MODEL-001.

Observed directions: adaptive retrieval decided by the model (Self-RAG), graph and hierarchical indexes (GraphRAG), reference-free evaluation (Ragas), and integration with tool-using agents (see [TECH-AGENT-001]). The extent to which longer context windows or in-weights memory will replace retrieval is an open question; no forecast record is registered in this article.

• Foundational Concepts:
  • TECH-LLM-001 Large Language Models: The generators used in RAG.
  • TECH-TOK-001 Tokenization and Embeddings: Embedding models and chunk sizes in tokens.
• Related Methods & Technologies:
  • TECH-ARCH-001 The Transformer Architecture: Cross-attention and context windows.
  • TECH-AGENT-001 Autonomous Agents: Retrieval as one tool among many.
  • TECH-DATA-001 Training Data, Contamination and Provenance: Provenance of indexed material.
• Advanced Implications & Future Trajectories:
  • CAP-LIMITS-001 Hallucination, Reliability and Calibration: What retrieval does and does not fix.
  • SAFE-SEC-001 AI Security: Prompt Injection, Poisoning, Extraction: Untrusted retrieved content.