AI Scaling Laws Explained

The encyclopedia's full treatment is Scaling laws. Terms such as loss, token and parameter are defined in the glossary.

#Quick reference

#What is a neural scaling law?

A scaling law is a curve fitted to many training runs that predicts a model's performance from its size and resources. The performance measure is almost always cross-entropy loss: how surprised the model is, on average, by the next token in held-out text (a token is a word or word fragment). Lower is better.

The three inputs are the number of parameters N (the adjustable weights in the network), the dataset size D in tokens, and the training compute C, counted in floating-point operations. For dense Transformers, a common estimate is that training costs about 6 operations per parameter per token, so C ≈ 6ND (Kaplan et al. state it as 6NBS, batch size times steps).

The practical value is prediction: fit the curve on small runs, estimate the loss of an expensive run before paying for it, and decide how to split a budget between model size and data. See Large language models for the systems these laws were fitted on.

#What did Kaplan et al. (2020) find?

Jared Kaplan and colleagues at OpenAI trained Transformer language models on the WebText2 dataset, ranging from 768 to 1.5 billion non-embedding parameters and from 22 million to 23 billion tokens. Their abstract reports that loss "scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude."

Their main findings:

• Scale matters, shape barely does. Within a wide range, depth and width had minimal effect compared with total size.
• Fitted exponents. Loss fell with parameters at an exponent of about 0.076, with data at about 0.095, and with optimally allocated compute at about 0.050. The paper labels these values tokenization-dependent.
• Allocation. Optimal model size grew roughly as compute to the power 0.73, so "as the computational budget C increases, it should be spent primarily on larger models." Compute-efficient training meant very large models stopped well before convergence.

The authors also warned that the trends "must eventually level off" because natural language has non-zero entropy, and that their compute and data laws contradict each other at scales several orders of magnitude beyond those studied.

#What did Chinchilla (Hoffmann et al. 2022) change?

Jordan Hoffmann and colleagues at DeepMind trained over 400 models, from 70 million to over 16 billion parameters, on 5 to 500 billion tokens. They concluded that "current large language models are significantly undertrained" and that "for every doubling of model size the number of training tokens should also be doubled."

This contradicted the Kaplan allocation. Hoffmann et al. summarize the older advice as: a 10x increase in compute should make the model 5.5x larger but use only 1.8x more tokens. Their three estimation methods instead gave exponents of 0.50, 0.49 and 0.46 for model size, with data taking the rest, meaning both grow roughly as the square root of compute.

They tested the claim directly. Chinchilla used the same compute budget as Gopher (280 billion parameters) but had 70 billion parameters and was trained on 1.4 trillion tokens, four times more data. It outperformed Gopher, GPT-3 (175B), Jurassic-1 (178B) and Megatron-Turing NLG (530B) on a large range of downstream tasks, and reached 67.5% average accuracy on the MMLU benchmark. The often-quoted rule of "about 20 tokens per parameter" is simply 1.4 trillion divided by 70 billion; it is a rounded reading, not a constant stated in the abstract.

Later work explains much of the disagreement. Porian et al. (2024) reproduced the Kaplan law and traced the gap to three factors: last-layer compute cost, warmup duration, and scale-dependent optimizer tuning. With those corrected they obtained "excellent agreement" with the Chinchilla law. Besiroglu et al. (2024) found problems with one of Chinchilla's three fitting methods, including implausibly narrow confidence intervals, though their refit agreed with the other two methods.

#Are scaling laws real laws?

No. They are empirical regularities, not laws of physics. Three limits matter.

1. They predict loss, not abilities. A smooth fall in next-token loss does not say which tasks a model can do. Wei et al. (2022) described "emergent abilities" that are absent in smaller models and present in larger ones. Schaeffer et al. (2023) argued that much apparent emergence comes from the researcher's choice of metric, and disappears with continuous metrics. The dispute is unresolved. See AI evaluation.
2. The numbers depend on the recipe. Exponents shift with tokenizer, data, architecture and optimizer settings, as the Kaplan-Chinchilla episode showed.
3. Extrapolation is a forecast. A fit only describes the range that was measured. Beyond it, any prediction needs to be stated and tracked as a forecast.

#Will scaling laws keep holding?

Most of the debate concerns inputs rather than the curves themselves.

Data supply. Villalobos et al. (first posted 2022, revised June 2024) projected that, if trends continue, models will be trained on datasets roughly equal to the available stock of public human text between 2026 and 2032. Muennighoff et al. (2023) found that up to about 4 epochs of repeated data gave negligible change in loss compared with unique data, but beyond that the value of added compute decays towards zero. See Training data, contamination and provenance.

Inference cost. Chinchilla optimizes training compute only. Sardana et al. (2024) showed that when a model will serve many requests (on the order of 1 billion), it pays to train a smaller model for longer than Chinchilla-optimal, and they saw quality keep improving up to 10,000 tokens per parameter.

New axes. Snell et al. (2024) studied spending compute at inference time instead of on a bigger model, and reported that on some problems a smaller model with optimally allocated test-time compute outperformed a 14x larger model in a FLOPs-matched comparison. See Reasoning models and test-time compute.

Unverifiable frontier. As of 26 October 2026, the largest closed models' training compute, data and loss curves are mostly undisclosed, so whether the published laws still hold at that scale cannot be checked from public evidence. Arguments that scaling alone leads to AGI are a separate, Hypothetical claim covered in Pathways to AGI and in Expert surveys on AGI timelines.

#Frequently asked questions

#What is the Chinchilla scaling law?

It is the finding by Hoffmann et al. (2022) that, for a fixed training compute budget, model parameters and training tokens should be scaled in equal proportion. Their 70-billion-parameter Chinchilla, trained on 1.4 trillion tokens, beat the 280-billion-parameter Gopher at the same compute.

#Why did Kaplan and Chinchilla disagree?

Kaplan et al. recommended spending most extra compute on model size; Hoffmann et al. found data should grow as fast as size. Porian et al. (2024) attribute the gap mainly to last-layer compute accounting, warmup length and optimizer tuning.

#Do scaling laws predict what a model can do?

Not directly. They predict pretraining loss on a given data distribution, and the link from loss to specific abilities is contested.

#Are we running out of training data?

Villalobos et al. projected that, if trends continue, training datasets will reach roughly the size of the public stock of human text between 2026 and 2032. Repetition, synthetic data and other data sources are proposed responses whose effect at frontier scale is not yet established.

#Is "20 tokens per parameter" a rule?

It is a rounded reading of Chinchilla's own configuration (1.4 trillion tokens for 70 billion parameters). It depends on the recipe and ignores inference cost, which pushes the best ratio higher.

#Sources

• Kaplan, McCandlish, Henighan, Brown, Chess, Child, Gray, Radford, Wu, Amodei: Scaling Laws for Neural Language Models
• Hoffmann, Borgeaud, Mensch, et al.: Training Compute-Optimal Large Language Models
• Porian, Wortsman, Jitsev, Schmidt, Carmon: Resolving Discrepancies in Compute-Optimal Scaling of Language Models
• Besiroglu, Erdil, Barnett, You: Chinchilla Scaling: A replication attempt
• Wei, Tay, Bommasani, et al.: Emergent Abilities of Large Language Models
• Schaeffer, Miranda, Koyejo: Are Emergent Abilities of Large Language Models a Mirage?
• Villalobos, Ho, Sevilla, Besiroglu, Heim, Hobbhahn: Will we run out of data? Limits of LLM scaling based on human-generated data
• Muennighoff, Rush, Barak, et al.: Scaling Data-Constrained Language Models
• Sardana, Portes, Doubov, Frankle: Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws
• Snell, Lee, Xu, Kumar: Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

Status: draft, reviewed 26 October 2026 against the primary sources above. Most sources are preprints or conference papers; frontier labs rarely publish current scaling data, so claims about today's largest models remain unverifiable.