---
id: TECH-SCALE-001
title: "Scaling Laws"
alternative_names:
  - "Neural Scaling Laws"
  - "Compute-Optimal Scaling"
  - "Chinchilla Scaling"
category: "Technical → Training and Scaling"
volume: III
chapter: 13
entry_type: "Technical Article"
level: 4
status: "Draft"
last_reviewed: 2026-10-15
version: "0.1.0"
prerequisites:
  - "TECH-LLM-001"
  - "TECH-ARCH-001"
  - "FOUND-008"
related_entries:
  - "TECH-MOE-001"
  - "TECH-REASON-001"
  - "TECH-HW-001"
  - "TECH-DATA-001"
  - "ECON-COMP-001"
  - "CAP-BENCH-001"
  - "AGI-PATH-001"
---

# Scaling Laws

## 1. Concise Definition
**Scaling laws** are empirical relationships, usually power laws, that describe how the performance of a trained machine learning model, typically its held-out cross-entropy loss, changes as the number of parameters $N$, the amount of training data $D$ and the training compute $C$ are increased. Within the regimes where they have been fitted, they let practitioners predict the loss of a large run from smaller runs and choose how to divide a fixed compute budget between model size and data.

## 2. Overview & Significance
Scaling laws turned model building from a search over architectures into a resource-allocation problem. If loss is a smooth, predictable function of $(N, D, C)$, a laboratory can estimate the benefit of a more expensive run before paying for it, and can decide whether additional parameters or additional tokens give more improvement for the same cost. They are the quantitative basis of the "scaling hypothesis" that features in discussions of future capability (see [AGI-PATH-001]) and of the compute economics of training (see [ECON-COMP-001]).

Two cautions apply throughout this article. First, the established laws describe **pretraining loss** on a given data distribution. They do not by themselves predict which downstream abilities appear, how they are measured, or whether systems are safe or useful; the relation between loss and benchmark performance is contested (Section 7). Second, scaling laws are fits to finite experiments. Extrapolation beyond the fitted range is a forecast, not a finding, and several published fits have been revised or disputed (Section 8).

## 3. Historical Development
Power-law behaviour of error in training-set size was characterised across machine translation, language modelling, image classification and speech recognition by Hestness et al. (2017), who also observed that model improvements shifted the error curve without changing its exponent in their tests and that model size scaled sublinearly with data size.

Kaplan et al. (2020) fitted the cross-entropy loss of Transformer language models as power laws in non-embedding parameters, dataset size and compute across more than seven orders of magnitude in some trends, concluding that larger models are significantly more sample-efficient and that compute-efficient training should use very large models on relatively modest data and stop well before convergence. Henighan et al. (2020) found a "power-law plus constant" form for autoregressive Transformers in image, video, text-to-image and mathematical-problem domains, with the compute-optimal model size following a power law whose exponents were nearly universal across those domains.

Hoffmann et al. (2022) trained over 400 models from 70 million to over 16 billion parameters on 5 to 500 billion tokens and concluded that model size and training tokens should be scaled in equal proportion. They tested this by training Chinchilla (70B parameters, 4 times more data) with the compute budget of Gopher (280B parameters) and reported that it outperformed Gopher, GPT-3, Jurassic-1 and Megatron-Turing NLG on a wide range of downstream evaluations. Their finding contradicted the Kaplan allocation, under which a 10-fold compute increase would suggest a 5.5-fold larger model but only 1.8 times more tokens.

Later work revisited the discrepancy and extended the framework to inference cost, repeated data, sparse models, and test-time compute (Sections 4 and 5).

## 4. Technical / Conceptual Explanation

### 4.1 Single-variable power laws
Kaplan et al. (2020) report, for language models trained to convergence on sufficiently large data, a loss $L(N) = (N_c/N)^{\alpha_N}$ with $\alpha_N \sim 0.076$, and for large models trained with early stopping on limited data $L(D) = (D_c/D)^{\alpha_D}$ with $\alpha_D \sim 0.095$. For compute-limited training with an optimally sized model, $L(C_{\min}) = (C_c^{\min}/C_{\min})^{\alpha_C^{\min}}$ with $\alpha_C^{\min} \sim 0.050$. These values are the exponents fitted in that paper for its tokenizer, architecture and data. They are not universal constants (see Section 8, and [TECH-TOK-001] for why loss values depend on the tokenizer).

### 4.2 Compute accounting
The standard estimate of training cost of a dense decoder-only Transformer is
$$ C \approx 6\,N\,D $$
floating-point operations, where the factor 6 covers the forward pass (about 2 operations per parameter per token) and the backward pass (about twice the forward cost) (Kaplan et al., 2020; Hoffmann et al., 2022). The approximation excludes context-dependent attention terms, and Hoffmann et al. report that their more detailed FLOP counts for models between 73 million and 6.8 billion parameters were within a few percent to about 10 percent of $6ND$.

### 4.3 The parametric form and the compute-optimal frontier
Hoffmann et al. fit a parametric loss
$$ \hat{L}(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} $$
where $E$ is an irreducible term attributed to the entropy of the data distribution and the two other terms capture the finite-model and finite-data penalties. Minimising $\hat{L}$ subject to $6ND = C$ yields power laws $N_{\mathrm{opt}} \propto C^{a}$ and $D_{\mathrm{opt}} \propto C^{b}$. Across their three estimation approaches they report $(a, b)$ of $(0.50, 0.50)$, $(0.49, 0.51)$ and $(0.46, 0.54)$, that is, parameters and tokens both grow roughly as the square root of compute. A frequently quoted rule of thumb of about 20 training tokens per parameter is a rounded reading of those results rather than a statement in the paper's abstract, and should not be treated as a precise constant.

### 4.4 Why the two laws disagreed
Besiroglu et al. (2024) attempted to replicate Hoffmann et al.'s third estimation procedure from data reconstructed from the paper's plots and report that the published estimates were inconsistent with the paper's first two methods and had implausibly narrow confidence intervals; their own refit agreed with the first two methods. Porian et al. (2024) attribute the discrepancy between the Kaplan and Hoffmann laws to three factors, namely last-layer computational cost, warmup duration and scale-dependent optimiser tuning; with these corrected they obtain agreement with the Hoffmann law, and they derive scaling laws for optimal learning rate and batch size. These replication studies are from groups independent of the original authors, but each rests on specific reproduction choices.

### 4.5 Beyond a single training run
- **Inference-aware scaling.** Sardana et al. (2024) modify the Chinchilla analysis to account for the lifetime inference demand of a deployed model and find that when demand is large (on the order of 1 billion requests) it is optimal to train smaller models for longer than Chinchilla-optimal; they report continued improvement up to about 10,000 tokens per parameter in their experiments.
- **Repeated data.** Muennighoff et al. (2023) find, for up to 900 billion training tokens and 9 billion parameters, that training for up to 4 epochs on repeated data gives negligible change in loss relative to unique data, after which the value of additional compute decays towards zero, and propose a scaling law accounting for the decreasing value of repeated tokens.
- **Sparse and routed models.** Clark et al. (2022) derive laws in which parameter count and per-token compute are independent axes for routing networks, and Krajewski et al. (2024) add expert granularity as a variable (see [TECH-MOE-001]).
- **Test-time compute.** Brown et al. (2024) show that the fraction of problems solved by any of repeatedly sampled outputs ("coverage") grows with the number of samples over four orders of magnitude, and Snell et al. (2024) study how to allocate inference compute per prompt; this adds a third axis to the training-time laws (see [TECH-REASON-001]).
- **Other domains.** Henighan et al. (2020) for multimodal and math settings; Gao et al. (2023) study how the gold-standard reward of a policy optimised against a proxy reward model changes with proxy size and optimisation, in a synthetic setup (see [TECH-ALIGN-001]).

## 5. Types, Components & Classifications

| Family | Quantity predicted | Key variables | Representative source |
| :--- | :--- | :--- | :--- |
| Loss-vs-scale (dense) | Held-out cross-entropy | $N$, $D$, $C$ | Kaplan et al. (2020) |
| Compute-optimal allocation | Best $N$, $D$ for given $C$ | $N_{opt}$, $D_{opt}$ | Hoffmann et al. (2022) |
| Inference-adjusted | Best $N$, $D$ given lifetime serving | Expected requests | Sardana et al. (2024) |
| Data-constrained | Loss with repeated data | Epochs, unique tokens | Muennighoff et al. (2023) |
| Sparse / routed | Loss with expert count | Active vs. total parameters, granularity | Clark et al. (2022); Krajewski et al. (2024) |
| Inference-time | Coverage or accuracy vs. samples or search | Samples, search budget | Brown et al. (2024); Snell et al. (2024) |
| Downstream / emergent | Task metrics vs. scale | Metric choice | Wei et al. (2022); Schaeffer et al. (2023) |

## 6. Applications & Concrete Examples
- **Planning large runs.** Chinchilla is the best-documented example of a model sized from a fitted law and then tested against a larger model trained with the same compute (Hoffmann et al., 2022).
- **Deployment-cost-aware design.** Training smaller models on more data than the Hoffmann optimum is the practice that Sardana et al. (2024) analyse; the Llama 3 report states that its scaling laws suggested the 405B-parameter flagship (trained on 15.6 trillion text tokens) was approximately compute-optimal for its training budget, while its smaller models were trained for much longer than compute-optimal and performed better than compute-optimal models at the same inference budget (Grattafiori et al., 2024). Details of closed developers' scaling studies are **undisclosed** unless published.
- **Hardware and investment decisions.** Because compute is the shared currency of the laws, they are used in arguments about chip demand and training cost (see [TECH-HW-001], [ECON-COMP-001]). Those arguments rely on assumptions beyond the laws themselves.

## 7. Advantages, Limitations & Unresolved Questions

### Advantages
Scaling laws are predictive within range, cheap to test at small scale, and have been reproduced in qualitative form across several modalities and by several groups (Hestness et al., 2017; Kaplan et al., 2020; Henighan et al., 2020).

### Limitations
1. **Loss is not capability.** A smooth decline in cross-entropy does not specify what a user can do with the model.
2. **Sensitivity to recipe.** The Kaplan/Hoffmann disagreement shows that exponents depend on implementation details such as warmup, optimiser tuning and the counting of embedding parameters (Porian et al., 2024).
3. **Data limits.** The laws assume sufficient unique data. Muennighoff et al. (2023) quantify diminishing returns from repetition, and the supply of high-quality text is the subject of [TECH-DATA-001].
4. **Extrapolation risk.** Fits cover specific ranges of $N$, $D$ and $C$; Sardana et al. (2024) note that laws fitted at ordinary token-to-parameter ratios mis-estimate the effect of tokens at extreme ratios.

### Unresolved questions
- **Emergent abilities (Contested).** Wei et al. (2022) define emergent abilities as those not present in smaller models but present in larger ones, so they "cannot be predicted simply by extrapolating the performance of smaller models". Schaeffer et al. (2023) argue that apparent emergence depends on nonlinear or discontinuous metrics: with linear or continuous metrics performance changes smoothly and predictably, which they support by tests on the InstructGPT/GPT-3 family and BIG-Bench meta-analysis. The dispute concerns how to measure and forecast task performance, and is not resolved by either paper alone.
- **Whether any floor is reached.** The irreducible term $E$ and the point at which pretraining loss stops translating into capability are not known for frontier-scale systems.
- **Interaction with post-training and test-time compute.** How the pretraining laws combine with reinforcement learning and inference-time search is still being mapped (see [TECH-REASON-001]).

## 8. Current State of Knowledge
As of 2026-10-15 the following are Well Supported: pretraining loss of Transformer language models follows smooth power-law-like trends in parameters, data and compute over many orders of magnitude; compute-optimal training requires scaling data with parameters rather than parameters alone; and the optimal ratio depends on whether inference cost is counted. The numerical exponents, the precise optimal ratio, and the extent to which different groups' fits agree are **Contested** or recipe-dependent (Besiroglu et al., 2024; Porian et al., 2024). Claims that scaling laws remain valid at the scale of the largest current closed models cannot be checked, because their training data, compute and loss curves are **undisclosed** unless published by the developer. Reports of current training compute for named models belong in `MODEL-001`, not here.

## 9. Future Trajectory & Strategic Outlook
Observed trends: laws are being extended to cover data repetition, sparsity, inference demand, and test-time compute, shifting the question from "how large a model" to "how to divide compute among pretraining, post-training and inference". Near-term (0–3 years) work will likely refine fits and publish more replications; medium and long-term outcomes depend on data supply, hardware and algorithmic change, which this article does not forecast beyond the record below.

**Forecast record**

| Field | Content |
| :--- | :--- |
| **Forecast ID** | `FCST-SCALE-001` |
| **Forecast Statement** | Held-out pretraining loss of compute-optimally trained dense decoder-only Transformers will continue to follow a power law plus a constant in training compute, with fit quality comparable to that reported by Hoffmann et al. (2022), over at least two further orders of magnitude of training compute beyond the largest run in that paper. |
| **Target Event** | Publication of loss measurements at those compute levels |
| **Operational Definition** | A published study or developer report with at least three training runs spanning the new compute range, on a fixed data distribution and tokenizer, shows the fitted law, extrapolated from the smaller runs, predicting final loss to within 2 percent of the measured value. |
| **Time Horizon** | Near-term (0–3 years), to 2029-10-15 |
| **Assumptions** | Sufficient unique high-quality data or well-behaved synthetic or repeated data; no change of architecture that alters exponents; stable optimisation recipes; the results are published. |
| **Confidence Level** | Medium (editorial estimate, about 60%) |
| **Alternative Scenarios** | Data exhaustion bends the curve; developers do not publish loss curves so the forecast cannot be resolved; new training regimes make pretraining-loss laws less informative. |
| **Review Schedule** | Annual review each October; first review 2027-10-15 |
| **Status** | `Active` |

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[TECH-LLM-001] Large Language Models](./TECH-LLM-001_large_language_models.md): The models whose loss is being modelled.
  - [[TECH-ARCH-001] The Transformer Architecture](./TECH-ARCH-001_transformer_architecture.md): Architecture the laws were fitted on.
  - [[FOUND-008] Probability, Statistics and Bayesian Inference](../vol01_foundations/FOUND-008_probability_statistics_and_bayesian_inference.md): Cross-entropy and fitting background.
- **Related Methods & Technologies**:
  - [[TECH-MOE-001] Mixture-of-Experts Models](./TECH-MOE-001_mixture_of_experts_models.md): Scaling with active versus total parameters.
  - [[TECH-REASON-001] Reasoning Models and Test-Time Compute](./TECH-REASON-001_reasoning_models_and_test_time_compute.md): Scaling inference compute.
  - [[TECH-ALIGN-001] Pretraining, Fine-Tuning and RLHF](./TECH-ALIGN-001_pretraining_fine_tuning_and_rlhf.md): Reward-model overoptimisation scaling.
  - [[TECH-DATA-001] Training Data, Contamination and Provenance](./TECH-DATA-001_training_data_contamination_and_provenance.md): Data supply limits.
  - [[TECH-HW-001] AI Hardware](./TECH-HW-001_ai_hardware.md): Compute supply.
  - [[CAP-BENCH-001] AI Evaluation](../vol04_capabilities/CAP-BENCH-001_ai_evaluation.md): Metric choice and emergent-ability disputes.
- **Advanced Implications & Future Trajectories**:
  - [[ECON-COMP-001] Compute Economics](../vol09_economy/ECON-COMP-001_compute_economics.md): Cost implications.
  - [[AGI-PATH-001] Pathways to AGI](../vol05_agi/AGI-PATH-001_pathways_to_agi.md): The scaling hypothesis in AGI discussions.

## 11. References & Foundational Literature
1. Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., & Zhou, Y. (2017). *Deep Learning Scaling is Predictable, Empirically*. arXiv:1712.00409. [Power-law generalisation error in training-set size across four domains].
2. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020). *Scaling Laws for Neural Language Models*. arXiv:2001.08361. [Power laws in $N$, $D$, $C$ with fitted exponents; $C \approx 6NBS$].
3. Henighan, T., Kaplan, J., Katz, M., et al. (2020). *Scaling Laws for Autoregressive Generative Modeling*. arXiv:2010.14701. [Power-law-plus-constant laws across image, video, multimodal and math domains].
4. Hoffmann, J., Borgeaud, S., Mensch, A., et al. (2022). *Training Compute-Optimal Large Language Models*. NeurIPS 2022. arXiv:2203.15556. [Equal scaling of parameters and tokens; Chinchilla 70B versus Gopher 280B].
5. Besiroglu, T., Erdil, E., Barnett, M., & You, J. (2024). *Chinchilla Scaling: A replication attempt*. arXiv:2404.10102. [Replication of the third estimation approach and critique of reported intervals].
6. Porian, T., Wortsman, M., Jitsev, J., Schmidt, L., & Carmon, Y. (2024). *Resolving Discrepancies in Compute-Optimal Scaling of Language Models*. NeurIPS 2024. arXiv:2406.19146. [Three factors explaining the Kaplan-Hoffmann discrepancy].
7. Sardana, N., Portes, J., Doubov, S., & Frankle, J. (2024). *Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws*. arXiv:2401.00448. [Inference-adjusted optimum; experiments up to 10,000 tokens per parameter].
8. Muennighoff, N., Rush, A. M., Barak, B., Le Scao, T., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., & Raffel, C. (2023). *Scaling Data-Constrained Language Models*. NeurIPS 2023. arXiv:2305.16264. [Up to 4 epochs of repetition with negligible loss change; law for repeated tokens].
9. Clark, A., de las Casas, D., Guy, A., et al. (2022). *Unified Scaling Laws for Routed Language Models*. ICML 2022. arXiv:2202.01169. [Scaling laws for routing networks].
10. Krajewski, J., Ludziejewski, J., Adamczewski, K., et al. (2024). *Scaling Laws for Fine-Grained Mixture of Experts*. arXiv:2402.07871. [Granularity as a scaling variable in MoE].
11. Wei, J., Tay, Y., Bommasani, R., et al. (2022). *Emergent Abilities of Large Language Models*. Transactions on Machine Learning Research. arXiv:2206.07682. [Definition and examples of emergent abilities].
12. Schaeffer, R., Miranda, B., & Koyejo, S. (2023). *Are Emergent Abilities of Large Language Models a Mirage?* NeurIPS 2023. arXiv:2304.15004. [Metric-choice explanation of apparent emergence].
13. Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., & Mirhoseini, A. (2024). *Large Language Monkeys: Scaling Inference Compute with Repeated Sampling*. arXiv:2407.21787. [Coverage scaling with sample count over four orders of magnitude].
14. Snell, C., Lee, J., Xu, K., & Kumar, A. (2024). *Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters*. arXiv:2408.03314. [Compute-optimal allocation of test-time compute per prompt].
15. Grattafiori, A., Dubey, A., Jauhri, A., et al. (2024). *The Llama 3 Herd of Models*. arXiv:2407.21783. [Developer-reported use of scaling laws and of over-training smaller models relative to compute-optimal].
16. Gao, L., Schulman, J., & Hilton, J. (2023). *Scaling Laws for Reward Model Overoptimization*. ICML 2023. arXiv:2210.10760. [Gold reward versus proxy optimisation in a synthetic setup].

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Pretraining loss follows smooth power-law trends in $N$, $D$, $C$ | Well Supported | Hestness et al. (2017); Kaplan et al. (2020); Henighan et al. (2020) | High within fitted ranges |
| $C \approx 6ND$ compute estimate | Established | Kaplan et al. (2020); Hoffmann et al. (2022) | High |
| Equal scaling of parameters and tokens is compute-optimal | Well Supported | Hoffmann et al. (2022); Porian et al. (2024) | Moderate-High |
| Specific exponents and constants (e.g. $\alpha_N \sim 0.076$) | Contested (recipe-dependent) | Kaplan et al. (2020); Besiroglu et al. (2024); Porian et al. (2024) | Moderate |
| Inference demand shifts optimum to smaller, longer-trained models | Well Supported | Sardana et al. (2024) | Moderate |
| Up to 4 epochs of repetition is nearly as good as unique data | Preliminary | Muennighoff et al. (2023) | Moderate; single study at the tested scale |
| Emergent abilities as unpredictable jumps | Contested | Wei et al. (2022); Schaeffer et al. (2023) | Active debate |
| Developer-reported use of scaling laws and over-training of small models (Llama 3) | Preliminary (developer report) | Grattafiori et al. (2024) | Moderate |
| Laws hold at scale of largest closed models | Undisclosed / unverifiable | n/a | Not assessable |
| Loss law continues two more orders of magnitude | Forecast | `FCST-SCALE-001` | Medium (about 60%) |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.