---
id: FOUND-009
title: "Information Theory"
alternative_names:
  - "Shannon Information Theory"
  - "Entropy and Coding Theory"
category: "Foundations → Mathematics"
volume: I
chapter: 3
entry_type: "Standard Article"
level: 3
status: "Draft"
last_reviewed: 2026-10-15
version: "0.1.0"
prerequisites:
  - "FOUND-008"
related_entries:
  - "FOUND-007"
  - "FOUND-010"
  - "TECH-LLM-001"
  - "TECH-VAE-001"
  - "BIO-SHANNON-001"
---

# Information Theory

## 1. Concise Definition
**Information theory** is the mathematical study of the quantification, storage and communication of information. Founded by Claude Shannon in 1948, it defines the **entropy** of a probability distribution as the average uncertainty (or minimum average description length) of its outcomes, and provides the limits of lossless compression and of reliable communication over noisy channels. In machine learning its quantities, entropy, cross-entropy, KL divergence and mutual information, serve as loss functions, similarity measures and analytic tools.

## 2. Overview & Significance
Information theory supplies precise answers to two engineering questions: how compactly can data from a source be represented, and how fast can data be sent reliably through a noisy channel? Shannon's (1948) results showed that both have sharp limits determined by the statistics of the source and the channel, and that these limits are achievable in principle.

For AI the significance is largely through three routes.
1. **Training objectives.** Cross-entropy, the standard loss for classification and language modelling, is an information-theoretic quantity ([[FOUND-008] Probability, Statistics and Bayesian Inference](FOUND-008_probability_statistics_and_bayesian_inference.md)).
2. **Prediction and compression.** A model that predicts data well can be used to compress it, and vice versa. This equivalence supports a view of learning as finding regularities that shorten descriptions.
3. **Theories of representation.** Concepts such as the information bottleneck propose that good representations retain information relevant to a task while discarding the rest.

Shannon's theory defines information in terms of statistical structure and ignores meaning. It is therefore a theory of the quantity of information, which is distinct from questions of semantics and understanding (see [[FOUND-014] Consciousness and Machine Understanding](FOUND-014_consciousness_and_machine_understanding.md)).

## 3. Historical Development
Shannon's "A Mathematical Theory of Communication" (1948), published in the *Bell System Technical Journal*, introduced entropy as a measure of the information produced by a source, the notion of channel capacity, and the coding theorems that relate them. Shannon was also a participant in the 1956 Dartmouth workshop and wrote on computer chess ([[BIO-SHANNON-001] Claude Shannon](../vol10_people_organizations/BIO-SHANNON-001_claude_shannon.md)).

Subsequent milestones include Huffman's (1952) method for constructing minimum-redundancy prefix codes, Kullback and Leibler's (1951) measure of divergence between distributions, and Jaynes's (1957) principle of maximum entropy, which proposed choosing the least-committal distribution consistent with known constraints and linked statistical mechanics with inference. Solomonoff (1964) formulated a theory of inductive inference based on algorithmic descriptions of data, an early root of **algorithmic information theory**, in which complexity is measured by the length of the shortest program that generates an object.

Textbook syntheses are provided by Cover and Thomas (2006) and MacKay (2003); the latter explicitly links information theory with inference and learning.

In the deep learning era, Tishby and Zaslavsky (2015) proposed analysing deep networks through the information bottleneck principle. A critical study by Saxe et al. (2019) challenged key empirical claims associated with that proposal (Section 7). More recently, Delétang et al. (2023) revisited the prediction-compression link for large language models.

## 4. Technical / Conceptual Explanation

### 4.1 Entropy
For a discrete random variable $X$ with distribution $p$, the **Shannon entropy** is
$$H(X) = -\sum_x p(x)\log_2 p(x),$$
measured in bits when the logarithm has base 2 (nats for the natural logarithm). It is zero for a deterministic variable and maximal, $\log_2 |\mathcal{X}|$, for the uniform distribution. **Joint** and **conditional** entropies are defined analogously, with the chain rule $H(X,Y) = H(X) + H(Y \mid X)$.

### 4.2 Source coding
Shannon's source coding theorem states that the entropy is the minimum average number of bits per symbol needed to encode i.i.d. symbols from the source without loss, in the limit of long messages (Shannon, 1948; Cover & Thomas, 2006). Practical codes approach this bound; Huffman (1952) codes are optimal among symbol-by-symbol prefix codes. An ideal code assigns a symbol of probability $p$ a length of about $-\log_2 p$ bits.

### 4.3 Cross-entropy and KL divergence
If data are drawn from $p$ but encoded using a code optimized for a model $q$, the average code length is the **cross-entropy**
$$H(p, q) = -\sum_x p(x)\log q(x) = H(p) + D_{\mathrm{KL}}(p \,\|\, q),$$
where the **Kullback-Leibler divergence** (Kullback & Leibler, 1951) is
$$D_{\mathrm{KL}}(p \,\|\, q) = \sum_x p(x)\log\frac{p(x)}{q(x)} \;\ge\; 0,$$
with equality only when $p = q$. KL divergence is not symmetric and is not a metric. Minimizing cross-entropy between the empirical data distribution and a model is therefore equivalent to maximum likelihood estimation, and the excess over the data entropy $H(p)$ is the KL divergence.

For a language model, the average negative log-probability per token on held-out text is its cross-entropy, and **perplexity** is its exponential, $\mathrm{PPL} = 2^{H(p,q)}$ for entropy in bits, which can be read as an effective branching factor.

### 4.4 Mutual information
$$I(X;Y) = D_{\mathrm{KL}}\bigl(p(x,y)\,\|\,p(x)p(y)\bigr) = H(X) - H(X \mid Y)$$
measures how much knowing one variable reduces uncertainty about the other. It is zero if and only if $X$ and $Y$ are independent. Mutual information is used in feature selection, representation learning objectives, and analyses of what a representation preserves.

### 4.5 Channel capacity
For a noisy channel with input $X$ and output $Y$, the **capacity** is $C = \max_{p(x)} I(X;Y)$. Shannon's noisy-channel coding theorem states that reliable communication is possible at any rate below $C$ and impossible above it (Shannon, 1948). The theorem is existential: it does not provide a practical code.

### 4.6 Maximum entropy
Among all distributions satisfying given expected-value constraints, the maximum-entropy distribution is the one with the greatest uncertainty, and it takes an exponential-family form (Jaynes, 1957). Many familiar models, including logistic regression and the Gaussian under fixed mean and variance, have maximum-entropy derivations.

### 4.7 Algorithmic information
The **Kolmogorov complexity** of a string is the length of the shortest program producing it on a fixed universal machine. It is uncomputable and defined up to an additive constant, but gives a conceptual basis for "simplicity" in learning. Solomonoff (1964) used such descriptions to define a universal prior for prediction. Practical systems use computable proxies such as compression ratios.

## 5. Types, Components & Classifications
- **Quantities**: entropy, conditional entropy, cross-entropy, KL divergence, mutual information.
- **Coding**: lossless (Huffman and other entropy coding) and lossy (rate-distortion theory).
- **Discrete versus differential entropy**: the continuous analogue (differential entropy) can be negative and is not invariant under change of variables, so interpretation differs from the discrete case.
- **Classical versus algorithmic information theory**: the first is defined over distributions, the second over individual objects.
- **Information-theoretic principles in learning**: maximum entropy (Jaynes, 1957), minimum description length, the information bottleneck (Tishby & Zaslavsky, 2015).

## 6. Applications & Concrete Examples
- **Training losses.** Cross-entropy for classification and next-token prediction; KL terms in the variational autoencoder objective ([[TECH-VAE-001] Autoencoders and Variational Autoencoders](../vol03_technical/TECH-VAE-001_autoencoders_and_variational_autoencoders.md)); KL penalties in reinforcement-learning-based fine-tuning.
- **Data compression.** Entropy coding is a deployed component of file, image and video formats; Huffman coding is a standard example (Huffman, 1952).
- **Language modelling as compression.** Delétang et al. (2023) argue that prediction and compression are equivalent via arithmetic coding, and evaluate large language models as general-purpose compressors on text, image and audio data; their results are laboratory evaluations of compression ratio and do not by themselves show that compression performance measures understanding.
- **Evaluation.** Perplexity is a standard intrinsic metric for language models, but it depends on tokenization and dataset and is not directly comparable across models with different vocabularies.
- **Communications.** Capacity-approaching error-correcting codes in wireless and storage systems rest on Shannon's theorems (Cover & Thomas, 2006).

## 7. Advantages, Limitations & Unresolved Questions
**Strengths.** Rigorous, general, and mathematically tractable; provides tight bounds for compression and communication; unifies several learning objectives.

**Limitations.**
- Shannon information is defined relative to a probability model. Its value depends on that model, and it is blind to semantic content.
- Mutual information is hard to estimate reliably in high dimensions from finite samples.
- Kolmogorov complexity and Solomonoff induction are uncomputable; they serve as idealizations (Solomonoff, 1964).
- **Information bottleneck in deep learning.** Tishby and Zaslavsky (2015) proposed that deep networks can be analysed as trading compression of the input against retention of information about the label. Saxe et al. (2019) reported that some of the claimed compression phases depend on the activation function and measurement method, and did not observe a general link between compression and generalization. The status of the proposal as a general theory of deep learning is therefore **Contested**.

**Open questions.** Whether information-theoretic quantities explain generalization in large networks; how to measure information in learned representations robustly; and the relation between compression and intelligence (an argument discussed in [[FOUND-002] Definitions of Intelligence](FOUND-002_definitions_of_intelligence.md)).

## 8. Current State of Knowledge
As of 2026-10-15:
- Shannon's coding theorems, the properties of entropy, KL divergence and mutual information, and the cross-entropy/maximum-likelihood equivalence are **Established** and standard in textbooks (Cover & Thomas, 2006; MacKay, 2003).
- The statement that good prediction implies good compression, via arithmetic coding, is a mathematical equivalence; its use to evaluate trained language models is documented in a preprint-stage and conference literature (Delétang et al., 2023, an arXiv preprint). [VERIFICATION NEEDED: final venue and publication status of Delétang et al. (2023)]
- The information bottleneck as an explanation of deep learning dynamics is **Contested** (Tishby & Zaslavsky, 2015; Saxe et al., 2019).

## 9. Future Trajectory & Strategic Outlook
No registered forecast is made. Observed trends: continued use of information-theoretic objectives in generative modelling; research on compression-based evaluation of language models; and efforts to relate representation quality to measurable information. Speculation that a compression principle alone could characterize general intelligence is a hypothesis, not a result, and is discussed with its objections in the definitional literature ([[FOUND-002] Definitions of Intelligence](FOUND-002_definitions_of_intelligence.md)).

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[FOUND-008] Probability, Statistics and Bayesian Inference](FOUND-008_probability_statistics_and_bayesian_inference.md): distributions, likelihood and Bayesian objectives.
- **Related Methods & Technologies**:
  - [[FOUND-007] Learning, Reasoning and Generalization](FOUND-007_learning_reasoning_and_generalization.md): generalization and inductive bias.
  - [[FOUND-010] Optimization and Gradient Descent](FOUND-010_optimization_and_gradient_descent.md): minimizing information-theoretic losses.
  - [[TECH-LLM-001] Large Language Models](../vol03_technical/TECH-LLM-001_large_language_models.md): cross-entropy training and perplexity.
  - [[TECH-VAE-001] Autoencoders and Variational Autoencoders](../vol03_technical/TECH-VAE-001_autoencoders_and_variational_autoencoders.md): KL terms in latent-variable models.
- **Advanced Implications & Future Trajectories**:
  - [[BIO-SHANNON-001] Claude Shannon](../vol10_people_organizations/BIO-SHANNON-001_claude_shannon.md): the founder of the field.
  - [[FOUND-014] Consciousness and Machine Understanding](FOUND-014_consciousness_and_machine_understanding.md): information versus meaning.

## 11. References & Foundational Literature
1. Shannon, C. E. (1948). A mathematical theory of communication. *Bell System Technical Journal*, 27(3), 379-423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x. [Founding paper: entropy, source coding and channel capacity. Checked against Crossref (pagination listed is the first part).]
2. Kullback, S., & Leibler, R. A. (1951). On information and sufficiency. *The Annals of Mathematical Statistics*, 22(1), 79-86. https://doi.org/10.1214/aoms/1177729694. [Introduces the divergence now called KL divergence. Checked against Crossref.]
3. Huffman, D. A. (1952). A method for the construction of minimum-redundancy codes. *Proceedings of the IRE*, 40(9), 1098-1101. https://doi.org/10.1109/JRPROC.1952.273898. [Optimal prefix codes. Checked against Crossref.]
4. Jaynes, E. T. (1957). Information theory and statistical mechanics. *Physical Review*, 106(4), 620-630. https://doi.org/10.1103/PhysRev.106.620. [Maximum-entropy principle. Checked against Crossref.]
5. Solomonoff, R. J. (1964). A formal theory of inductive inference. Part I. *Information and Control*, 7(1), 1-22. https://doi.org/10.1016/S0019-9958(64)90223-2. [Algorithmic-information basis for prediction. Checked against Crossref.]
6. Cover, T. M., & Thomas, J. A. (2006). *Elements of Information Theory* (2nd ed.). Wiley-Interscience. ISBN 978-0471241959. [Standard textbook for entropy, coding theorems and capacity. ISBN checked against Open Library.]
7. MacKay, D. J. C. (2003). *Information Theory, Inference, and Learning Algorithms*. Cambridge University Press. ISBN 978-0521642989. [Textbook linking information theory to inference and learning. Checked against Open Library.]
8. Tishby, N., & Zaslavsky, N. (2015). Deep learning and the information bottleneck principle. arXiv:1503.02406 (preprint). [Proposes analysing deep networks via the information bottleneck. Checked against arXiv page.]
9. Saxe, A. M., Bansal, Y., Dapello, J., Advani, M., Kolchinsky, A., Tracey, B. D., & Cox, D. D. (2019). On the information bottleneck theory of deep learning. *Journal of Statistical Mechanics: Theory and Experiment*, 2019(12). https://doi.org/10.1088/1742-5468/ab3985. [Critical examination of empirical claims of the information bottleneck account. Authors, journal, volume and issue checked against Crossref.]
10. Delétang, G., et al. (2023). Language modeling is compression. arXiv:2309.10668 (preprint). [Evaluates large language models as compressors and discusses prediction-compression equivalence. Twelve authors confirmed on arXiv; later venue not confirmed here.]

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Entropy, source-coding and channel-coding theorems | Established | Shannon (1948); Cover & Thomas (2006) | High |
| Cross-entropy decomposes into entropy plus KL; minimizing it equals MLE | Established | Cover & Thomas (2006); Kullback & Leibler (1951) | High |
| Maximum-entropy principle as inference rule | Established (as mathematics); Contested (as universal principle) | Jaynes (1957) | Moderate |
| Prediction and compression are equivalent via arithmetic coding | Established | Cover & Thomas (2006); Delétang et al. (2023) | High for equivalence |
| LLMs are strong general-purpose compressors | Preliminary | Delétang et al. (2023) | Moderate; preprint-stage in this article |
| Information bottleneck explains deep-learning training | Contested | Tishby & Zaslavsky (2015); Saxe et al. (2019) | Low to Moderate |
| Kolmogorov complexity as basis for universal induction | Hypothetical (idealization; uncomputable) | Solomonoff (1964) | Moderate |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.