---
id: TECH-LLM-001
title: "Large Language Models"
alternative_names:
  - "LLMs"
  - "Foundation Language Models"
  - "Autoregressive Language Models"
category: "Technical → Language Models"
volume: III
chapter: 19
entry_type: "Technical Reference"
level: 4
status: "Draft"
last_reviewed: 2026-10-15
version: "0.2.0"
prerequisites:
  - "TECH-DL-001"
  - "TECH-ARCH-001"
  - "Probability Theory"
related_entries:
  - "CAP-GENAI-001"
  - "SAFE-ALIGN-001"
  - "TECH-ML-001"
---

# Large Language Models

## 1. Concise Definition
A **Large Language Model (LLM)** is a high-capacity deep neural network—typically a decoder-only transformer—trained on massive corpora of text to approximate the joint probability distribution of natural language. By optimizing the fundamental objective of autoregressive next-token prediction, these models develop complex internal representations of syntax, semantics, and factual knowledge, which can be subsequently aligned to follow human instructions, perform multi-step reasoning, and execute zero-shot tasks across diverse linguistic and programmatic domains (Brown et al., 2020; Kaplan et al., 2020).

## 2. Overview & Significance
LLMs represent the dominant paradigm in contemporary artificial intelligence, acting as general-purpose cognitive engines. Rather than training specialized models for discrete tasks (e.g., a dedicated translation model, a dedicated summarization model), the field increasingly adopted a **foundation model** approach (Bommasani et al., 2021), although specialized models remain widely used. A sufficiently large language model, having compressed vast amounts of human knowledge during pretraining, can be coaxed into performing virtually any text-to-text mapping task.

The significance of LLMs lies in their emergent empirical behavior:
1. **In-Context Learning**: The ability to adapt to new tasks purely through prompt conditioning at inference time, without updating model weights.
2. **Empirical Scaling Regularities**: Test loss falls approximately as a power law in compute, parameter count, and dataset size across the ranges studied (Kaplan et al., 2020); the mapping from loss to specific downstream capabilities is less predictable.
3. **Breadth**: Because much written knowledge (code, mathematics, law, medicine, translation) can be serialized into discrete tokens, a single model family can be applied to a wide range of text-based tasks, with uneven reliability across them.

## 3. Historical Development
Language modeling originated with **statistical n-gram models**, which estimated the probability of a word given the $n-1$ preceding words based on raw frequency counts. These suffered fundamentally from the "curse of dimensionality" and data sparsity.

In 2003, Bengio and colleagues introduced the **Neural Probabilistic Language Model** (Bengio et al., 2003), using a feedforward neural network and continuous word embeddings to share statistical strength across semantically similar words. This was followed by **Recurrent Neural Networks (RNNs)** and LSTMs (Hochreiter & Schmidhuber, 1997), which could theoretically process unbounded context but struggled practically with vanishing gradients and sequential computational bottlenecks.

The inflection point occurred in 2017 with the introduction of the **Transformer** architecture (Vaswani et al., 2017). OpenAI's generative pre-trained transformer (GPT) series sequentially proved the efficacy of scaling:
- **GPT-1 (2018)**: 117 million parameters (Radford et al., 2018); demonstrated the viability of unsupervised pretraining followed by supervised fine-tuning.
- **GPT-2 (2019)**: 1.5 billion parameters; reported zero-shot task transfer (Radford et al., 2019).
- **GPT-3 (2020)**: 175 billion parameters; reported few-shot in-context learning (Brown et al., 2020).

Simultaneously, the scaling laws formalized by Kaplan et al. (2020) and refined by Hoffmann et al. (2022) (the "Chinchilla" scaling laws) estimated compute-optimal ratios between parameter count and training tokens. Instruction tuning and Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022) were shown to make raw text predictors follow user instructions (InstructGPT). Deployed chat assistants use related post-training methods, but the details for closed products are largely undisclosed.

## 4. Technical / Conceptual Explanation

### 4.1 Tokenization and Embeddings
Raw text is compressed into discrete subword units called **tokens** using algorithms like Byte-Pair Encoding (BPE) or WordPiece. A typical vocabulary $V$ contains 32,000 to 128,000 tokens. Each token index is mapped to a continuous embedding vector $\mathbf{e}_i \in \mathbb{R}^d$, enriched with positional encodings (such as Rotary Positional Embeddings, RoPE) to denote its position in the sequence.

### 4.2 Autoregressive Pretraining
The fundamental training phase involves self-supervised learning on a massive corpus $\mathcal{D}$. The model is trained to minimize the negative log-likelihood (cross-entropy loss) of the next token $x_t$ given the preceding context $x_{<t}$:

$$\mathcal{L}_{\text{pretrain}}(\theta) = -\mathbb{E}_{x \sim \mathcal{D}} \left[ \sum_{t=1}^{T} \log P_\theta(x_t \mid x_1, \dots, x_{t-1}) \right]$$

To achieve compute-optimal training under a fixed FLOP budget $C$, Hoffmann et al. (2022) demonstrated that the model parameter count $N$ and the number of training tokens $D$ should be scaled approximately equally (fitted exponents of about 0.5, i.e. $N \propto C^{0.5}$, $D \propto C^{0.5}$ approximately), and argued that earlier large models were under-trained for their size. These exponents are empirical estimates from a single study, not exact values.

### 4.3 Post-Training (Alignment)
A base model predicting the statistically most likely next word of web text is generally unhelpful as an assistant. **Post-training** aligns the model to user intent:
1. **Supervised Fine-Tuning (SFT)**: The model is fine-tuned on high-quality pairs of (Instruction $\rightarrow$ Expert Response) using standard cross-entropy loss.
2. **Preference Optimization**: The model is further tuned using Reinforcement Learning from Human Feedback (RLHF) via PPO (Schulman et al., 2017; applied to language models by Ouyang et al., 2022), or through Direct Preference Optimization (DPO; Rafailov et al., 2023), which updates the policy to increase the likelihood of human-preferred outputs over rejected ones.

### 4.4 Inference and Decoding
During deployment (inference), the model generates text autoregressively. At step $t$, the model outputs a probability distribution over the vocabulary. The next token is sampled from this distribution. Hyperparameters control this sampling:
- **Temperature ($\tau$)**: Flattens or sharpens the softmax distribution. $\tau \to 0$ approaches deterministic greedy decoding (always picking the highest probability token); $\tau > 1$ increases randomness.
- **Top-$k$ Sampling**: Restricts the sample space to the $k$ most likely tokens.
- **Nucleus (Top-$p$) Sampling**: Restricts the sample space to the smallest set of tokens whose cumulative probability exceeds $p$.

## 5. Types, Components & Classifications
- **Base Models**: Raw pretrained models (e.g., the Llama 3 base checkpoints released in 2024; Llama Team, 2024). Highly capable but require specific prompting formats to elicit tasks.
- **Instruct/Chat Models**: Models post-trained via SFT and RLHF for conversational compliance (e.g., the Llama 3 Instruct releases of 2024; Llama Team, 2024). Examples are as of 2024; see [MODEL-001](../reference/factbook/MODEL-001_model_registry.md) for current models.
- **Dense vs. Sparse Architectures**: Dense models activate every parameter for every token. **Mixture-of-Experts (MoE)** models (e.g., Mixtral 8x7B, released 2023-2024; see [TECH-MOE-001](./TECH-MOE-001_mixture_of_experts_models.md)) route each token to a subset of specialized sub-networks, decoupling total parameter count from inference compute (FLOPs).
- **Open-Weight vs. Proprietary**: "Open-weight" models release the trained parameter matrices publicly (e.g., Llama, Mistral), though often restricting training data access, unlike fully "open-source" software. Proprietary models are accessible only via API, and their architectures, training data and parameter counts are generally undisclosed.

## 6. Applications & Concrete Examples
- **Retrieval-Augmented Generation (RAG)**: LLMs integrated with vector databases to retrieve external, proprietary, or up-to-date information, inserting it into the prompt context to reduce hallucination and anchor the model in factuality.
- **Code Generation and Copilots**: Fine-tuned on GitHub repositories, LLMs synthesize boilerplate code, translate between programming languages, and generate unit tests.
- **Tool Use and Agentic Workflows**: LLMs trained to output structured JSON representing API calls (e.g., web search, python execution), enabling them to act as autonomous agents interacting with external systems.
- **Multimodal Interactivity**: Frontier LLMs accept interleaved image, audio, and text inputs, mapping them into the same latent space to perform visual question answering and audio translation.

## 7. Advantages, Limitations & Unresolved Questions

### Demonstrated Advantages
- **Zero-Shot Transfer**: Solves tasks it was never explicitly fine-tuned to perform purely through prompt conditioning.
- **Fluent Text Generation**: Produces grammatical, stylistically controllable text and useful machine translation in many languages, with quality varying by language and domain.

### Core Limitations
- **Hallucinations**: Generates fluent, plausible-sounding falsehoods due to the probabilistic nature of next-token prediction and lack of external grounding.
- **KV-Cache Memory Bottleneck**: Inference requires caching the Key and Value matrices of all past tokens in GPU memory. As context lengths scale to millions of tokens, memory bandwidth strictly limits batch sizes and inference speed.
- **Uneven Reasoning**: Multi-step deduction (e.g., formal logic or long arithmetic) is a contested limitation that is model- and prompt-dependent. Chain-of-thought prompting improves results on some multi-step tasks (Wei et al., 2022), and errors in intermediate tokens can propagate to the final answer; the extent of this failure mode varies across models and has not been settled here.
- **Data Contamination**: Widespread inclusion of benchmark evaluation datasets in the pretraining corpora severely complicates the assessment of true generalization versus rote memorization.

### Unresolved Scientific Questions
1. *Is Next-Token Prediction Sufficient?* Does autoregressive modeling merely yield sophisticated statistical pattern matching (stochastic mimicry), or does it force the network to construct a causal, low-dimensional "world model" representing the true physical and logical structures underlying the text?
2. *The Data Wall*: Pretraining currently consumes trillions of tokens. High-quality human text is finite. Can models continue scaling using synthetic data generated by other models without suffering from "model collapse" (Shumailov et al., 2024)? Projections of when public human-generated text will be exhausted are uncertain (Villalobos et al., 2022).
3. *Mechanistic Interpretability*: Can the billions of continuous parameters be successfully reverse-engineered into auditable, human-readable circuits?

## 8. Current State of Knowledge
Parameter counts, architectures and training-token counts of proprietary frontier models are generally undisclosed, so no figure is given here for the current frontier; current model facts belong in [MODEL-001](../reference/factbook/MODEL-001_model_registry.md). For scale of disclosed open-weight models as of 2024: Llama 3 405B was pre-trained on 15.6 trillion tokens using up to 16K H100 GPUs (Llama Team, 2024).

Recent work emphasizes **test-time compute**: giving models additional inference-time tokens to reason via extended chain of thought (e.g., OpenAI's o1, which OpenAI describes as trained with large-scale reinforcement learning; OpenAI, 2024; no inference-time search algorithm is described in the abstract of the system card [VERIFICATION NEEDED: full system card and launch post not read; confirm that no search method is disclosed]). Explicit search methods such as Monte Carlo Tree Search appear in some open research systems [VERIFICATION NEEDED: cite a specific system, e.g. Marco-o1], but are not documented for o1. See [TECH-REASON-001](./TECH-REASON-001_reasoning_models_and_test_time_compute.md).

## 9. Future Trajectory & Strategic Outlook
- **Near-Term (0–3 Years)** [speculative forecast, unsourced; no forecast record assigned]: Wider deployment of smaller language models running locally on edge devices. Consolidation of RAG patterns into standard enterprise software architectures.
- **Medium-Term (3–10 Years)** [speculative]: Possible transition from text-completion engines to long-horizon autonomous agents capable of independently executing software engineering and data analysis workflows lasting days or weeks. Possible exhaustion of the public human-generated text supply (estimates are uncertain; Villalobos et al., 2022), which could push developers toward synthetic data or other modalities.
- **Long-Term (10+ Years)** [speculative]: Potential architectural paradigm shifts beyond purely autoregressive transformers (e.g., state-space models, energy-based models, or continuous recurrent memories) that decouple reasoning depth from output token length.

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[FOUND-001] Artificial Intelligence](../vol01_foundations/FOUND-001_artificial_intelligence.md): Parent discipline.
- **Related Methods & Technologies**:
  - [[TECH-ARCH-001] The Transformer Architecture](./TECH-ARCH-001_transformer_architecture.md): The structural backbone of LLMs.
  - [[CAP-GENAI-001] Generative Artificial Intelligence](../vol04_capabilities/CAP-GENAI-001_generative_ai.md): Generative mechanics relying on autoregression.
- **Advanced Implications & Future Trajectories**:
  - [[SAFE-ALIGN-001] The AI Alignment Problem](../vol08_safety_governance/SAFE-ALIGN-001_ai_alignment.md): RLHF, sycophancy, and alignment tuning of LLMs.
  - [[AGI-001] Artificial General Intelligence](../vol05_agi/AGI-001_artificial_general_intelligence.md): Theoretical limits of LLM scaling.
- **Living References**:
  - [[MODEL-001] AI/SI Model Registry](../reference/factbook/MODEL-001_model_registry.md): Dated model facts, deliberately kept out of this stable article.

## 11. References & Foundational Literature
1. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). *Attention Is All You Need*. NeurIPS 2017 [venue not independently confirmed]. arXiv:1706.03762. [Introduces the Transformer architecture.]
2. Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). *Improving Language Understanding by Generative Pre-Training*. OpenAI technical report. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf. [GPT-1: generative pretraining followed by discriminative fine-tuning; the 117M parameter figure is from this report.]
3. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). *Language Models are Unsupervised Multitask Learners*. OpenAI technical report. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf. [GPT-2: reports zero-shot task transfer.]
4. Brown, T. B., et al. (2020). *Language Models are Few-Shot Learners*. NeurIPS 2020 [venue not independently confirmed]. arXiv:2005.14165 (31 authors; first author listed). [GPT-3: 175B parameters and few-shot in-context learning.]
5. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020). *Scaling Laws for Neural Language Models*. arXiv:2001.08361. [Power-law dependence of test loss on compute, parameters and data.]
6. Hoffmann, J., et al. (2022). *Training Compute-Optimal Large Language Models*. arXiv:2203.15556 (22 authors; first author listed). [Chinchilla: estimates of compute-optimal parameter and token scaling; preprint.]
7. Ouyang, L., et al. (2022). *Training language models to follow instructions with human feedback*. NeurIPS 2022 [venue not independently confirmed]. arXiv:2203.02155 (20 authors; first author listed). [InstructGPT: RLHF with PPO for instruction following.]
8. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022). *Chain-of-Thought Prompting Elicits Reasoning in Large Language Models*. NeurIPS 2022 [venue not independently confirmed]. arXiv:2201.11903. [Chain-of-thought prompting improves results on several multi-step reasoning benchmarks.]
9. Bommasani, R., et al. (2021). *On the Opportunities and Risks of Foundation Models*. arXiv:2108.07258 (114 authors; first author listed). [Defines and surveys the foundation-model paradigm.]
10. Hochreiter, S., & Schmidhuber, J. (1997). *Long Short-Term Memory*. Neural Computation, 9(8), 1735-1780. https://doi.org/10.1162/neco.1997.9.8.1735. [Introduces the LSTM.]
11. Bengio, Y., Ducharme, R., Vincent, P., & Jauvin, C. (2003). *A Neural Probabilistic Language Model*. Journal of Machine Learning Research, 3, 1137-1155. https://jmlr.org/papers/v3/bengio03a.html. [Feedforward neural language model with learned word embeddings.]
12. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). *Direct Preference Optimization: Your Language Model Is Secretly a Reward Model*. arXiv:2305.18290. [Introduces DPO.]
13. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). *Proximal Policy Optimization Algorithms*. arXiv:1707.06347. [Introduces PPO.]
14. Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H., & Wattenberg, M. (2023). *Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task*. ICLR 2023. arXiv:2210.13382. [Evidence that a sequence model trained on Othello moves develops an internal board representation; one side of the world-model debate.]
15. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). *On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?* Proceedings of FAccT 2021, 610-623. https://doi.org/10.1145/3442188.3445922. [Source of the "stochastic parrot" critique.]
16. Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., & Gal, Y. (2024). *AI models collapse when trained on recursively generated data*. Nature, 631, 755-759. https://doi.org/10.1038/s41586-024-07566-y. [Reports tail loss when models are recursively trained on their own outputs in the studied setups.]
17. Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., & Hobbhahn, M. (2022). *Will we run out of data? Limits of LLM scaling based on human-generated data*. arXiv:2211.04325. [Projects the stock of human-generated public text relative to training demand; estimates are uncertain.]
18. Llama Team, AI @ Meta (2024). *The Llama 3 Herd of Models*. arXiv:2407.21783. [Reports Llama 3 405B pre-training on 15.6T tokens with up to 16K H100 GPUs.]
19. OpenAI (2024). *OpenAI o1 System Card*. arXiv:2412.16720. [Primary OpenAI description of o1; states training with large-scale reinforcement learning and chain of thought. Only the abstract was read in this revision.]

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Compute-Optimal Scaling Laws (approx. equal scaling of $N$ and $D$) | Well Supported (Preliminary; single-lab preprint, fitted exponents are estimates) | Hoffmann et al. (2022); Kaplan et al. (2020) | Moderate |
| Next-Token Cross-Entropy Pretraining as the standard objective | Established (description of method) | Brown et al. (2020); Radford et al. (2019) | High |
| Emergence of In-Context Learning | Well Supported | Brown et al. (2020), Wei et al. (2022) | Moderate to High |
| Internal Representations as "World Models" | Contested | Li et al. (2023) vs. Bender et al. (2021) | Active Debate |
| Model collapse under recursive training on own outputs | Well Supported (conditions-specific: replacing rather than accumulating real data) | Shumailov et al. (2024) | Moderate |

## 13. Revision History
- **v1.0.0** (2026-10-15): Initial draft; citations not independently verified.
- **v0.2.0** (2026-10-15): Corrected following independent fact-check audit (docs/audits/TECH-LLM-001_audit.md); 12 findings applied, 2 left marked for verification.