Information Theory

Information theory is the mathematical study of the quantification, storage and communication of information. Founded by Claude Shannon in 1948, it defines the entropy of a probability distribution as the average uncertainty (or minimum average description length) of its outcomes, and provides the limits of lossless compression and of reliable communication over noisy channels. In machine learning its quantities, entropy, cross-entropy, KL divergence and mutual information, serve as loss functions, similarity measures and analytic tools.

Information theory supplies precise answers to two engineering questions: how compactly can data from a source be represented, and how fast can data be sent reliably through a noisy channel? Shannon's (1948) results showed that both have sharp limits determined by the statistics of the source and the channel, and that these limits are achievable in principle.

For AI the significance is largely through three routes.

1. Training objectives. Cross-entropy, the standard loss for classification and language modelling, is an information-theoretic quantity (FOUND-008 Probability, Statistics and Bayesian Inference).
2. Prediction and compression. A model that predicts data well can be used to compress it, and vice versa. This equivalence supports a view of learning as finding regularities that shorten descriptions.
3. Theories of representation. Concepts such as the information bottleneck propose that good representations retain information relevant to a task while discarding the rest.

Shannon's theory defines information in terms of statistical structure and ignores meaning. It is therefore a theory of the quantity of information, which is distinct from questions of semantics and understanding (see FOUND-014 Consciousness and Machine Understanding).

Shannon's "A Mathematical Theory of Communication" (1948), published in the Bell System Technical Journal, introduced entropy as a measure of the information produced by a source, the notion of channel capacity, and the coding theorems that relate them. Shannon was also one of the four signatories of the 1955 proposal for the Dartmouth summer research project (with McCarthy, Minsky and Rochester) and wrote on computer chess (BIO-SHANNON-001 Claude Shannon).

Subsequent milestones include Huffman's (1952) method for constructing minimum-redundancy prefix codes, Kullback and Leibler's (1951) measure of divergence between distributions, and Jaynes's (1957) principle of maximum entropy, which proposed choosing the least-committal distribution consistent with known constraints and linked statistical mechanics with inference. Solomonoff (1964) formulated a theory of inductive inference based on algorithmic descriptions of data, an early root of algorithmic information theory, in which complexity is measured by the length of the shortest program that generates an object.

Textbook syntheses are provided by Cover and Thomas (2006) and MacKay (2003); the latter explicitly links information theory with inference and learning.

In the deep learning era, Tishby and Zaslavsky (2015) proposed analysing deep networks through the information bottleneck principle. A critical study by Saxe et al. (2019) challenged key empirical claims associated with that proposal (Section 7). More recently, Delétang et al. (2024) revisited the prediction-compression link for large language models.

#4.1 Entropy

For a discrete random variable $X$ with distribution $p$, the Shannon entropy is

measured in bits when the logarithm has base 2 (nats for the natural logarithm). It is zero for a deterministic variable and maximal, $\log_2 |\mathcal{X}|$, for the uniform distribution. Joint and conditional entropies are defined analogously, with the chain rule $H(X,Y) = H(X) + H(Y \mid X)$.

#4.2 Source coding

Shannon's source coding theorem states that the entropy is the minimum average number of bits per symbol needed to encode i.i.d. symbols from the source without loss, in the limit of long messages (Shannon, 1948; Cover & Thomas, 2006). Practical codes approach this bound; Huffman (1952) codes are optimal among symbol-by-symbol prefix codes. An ideal code assigns a symbol of probability $p$ a length of about $-\log_2 p$ bits.

#4.3 Cross-entropy and KL divergence

If data are drawn from $p$ but encoded using a code optimized for a model $q$, the average code length is the cross-entropy

where the Kullback-Leibler divergence (Kullback & Leibler, 1951) is

with equality only when $p = q$. KL divergence is not symmetric and is not a metric. Minimizing cross-entropy between the empirical data distribution and a model is therefore equivalent to maximum likelihood estimation, and the excess over the data entropy $H(p)$ is the KL divergence.

For a language model, the average negative log-probability per token on held-out text is its cross-entropy, and perplexity is its exponential, $\mathrm{PPL} = 2^{H(p,q)}$ for entropy in bits, which can be read as an effective branching factor.

#4.4 Mutual information

measures how much knowing one variable reduces uncertainty about the other. It is zero if and only if $X$ and $Y$ are independent. Mutual information is used in feature selection, representation learning objectives, and analyses of what a representation preserves.

#4.5 Channel capacity

For a noisy channel with input $X$ and output $Y$, the capacity is $C = \max_{p(x)} I(X;Y)$. Shannon's noisy-channel coding theorem states that reliable communication is possible at any rate below $C$ and impossible above it (Shannon, 1948). The theorem is existential: it does not provide a practical code.

#4.6 Maximum entropy

Among all distributions satisfying given expected-value constraints, the maximum-entropy distribution is the one with the greatest uncertainty, and it takes an exponential-family form (Jaynes, 1957). Many familiar models, including logistic regression and the Gaussian under fixed mean and variance, have maximum-entropy derivations.

#4.7 Algorithmic information

The Kolmogorov complexity of a string is the length of the shortest program producing it on a fixed universal machine. It is uncomputable and defined up to an additive constant, but gives a conceptual basis for "simplicity" in learning. Solomonoff (1964) used such descriptions to define a universal prior for prediction. Practical systems use computable proxies such as compression ratios.

• Quantities: entropy, conditional entropy, cross-entropy, KL divergence, mutual information.
• Coding: lossless (Huffman and other entropy coding) and lossy (rate-distortion theory).
• Discrete versus differential entropy: the continuous analogue (differential entropy) can be negative and is not invariant under change of variables, so interpretation differs from the discrete case.
• Classical versus algorithmic information theory: the first is defined over distributions, the second over individual objects.
• Information-theoretic principles in learning: maximum entropy (Jaynes, 1957), minimum description length, the information bottleneck (Tishby & Zaslavsky, 2015).

• Training losses. Cross-entropy for classification and next-token prediction; KL terms in the variational autoencoder objective (TECH-VAE-001 Autoencoders and Variational Autoencoders); KL penalties in reinforcement-learning-based fine-tuning.
• Data compression. Entropy coding is a deployed component of file, image and video formats; Huffman coding is a standard example (Huffman, 1952).
• Language modelling as compression. Delétang et al. (2024) argue that prediction and compression are equivalent via arithmetic coding, and evaluate large language models as general-purpose compressors on text, image and audio data; their results are laboratory evaluations of compression ratio and do not by themselves show that compression performance measures understanding.
• Evaluation. Perplexity is a standard intrinsic metric for language models, but it depends on tokenization and dataset and is not directly comparable across models with different vocabularies.
• Communications. Error-correcting codes in wireless and storage systems are designed against the limits set by Shannon's channel-coding theorem (Shannon, 1948; Cover & Thomas, 2006 for the theory); this article does not cite a source on specific deployed codes.

Strengths. Rigorous, general, and mathematically tractable; provides tight bounds for compression and communication; unifies several learning objectives.

Limitations.

• Shannon information is defined relative to a probability model. Its value depends on that model, and it is blind to semantic content.
• Mutual information is hard to estimate reliably in high dimensions from finite samples.
• Kolmogorov complexity and Solomonoff induction are uncomputable; they serve as idealizations (Solomonoff, 1964).
• Information bottleneck in deep learning. Tishby and Zaslavsky (2015) proposed that deep networks can be analysed as trading compression of the input against retention of information about the label. Saxe et al. (2019) reported that some of the claimed compression phases depend on the activation function and measurement method, and did not observe a general link between compression and generalization. The status of the proposal as a general theory of deep learning is therefore Contested.

Open questions. Whether information-theoretic quantities explain generalization in large networks; how to measure information in learned representations robustly; and the relation between compression and intelligence (an argument discussed in FOUND-002 Definitions of Intelligence).

As of 2026-10-15:

• Shannon's coding theorems, the properties of entropy, KL divergence and mutual information, and the cross-entropy/maximum-likelihood equivalence are Established and standard in textbooks (Cover & Thomas, 2006; MacKay, 2003).
• The statement that good prediction implies good compression, via arithmetic coding, is a mathematical equivalence; its use to evaluate trained language models is documented in a conference paper (Delétang et al., 2024, ICLR 2024; first posted as an arXiv preprint in 2023).
• The information bottleneck as an explanation of deep learning dynamics is Contested (Tishby & Zaslavsky, 2015; Saxe et al., 2019).

No registered forecast is made. Observed trends: continued use of information-theoretic objectives in generative modelling; research on compression-based evaluation of language models; and efforts to relate representation quality to measurable information. Speculation that a compression principle alone could characterize general intelligence is a hypothesis, not a result, and is discussed with its objections in the definitional literature (FOUND-002 Definitions of Intelligence).

• Foundational Concepts:
  • FOUND-008 Probability, Statistics and Bayesian Inference: distributions, likelihood and Bayesian objectives.
• Related Methods & Technologies:
  • FOUND-007 Learning, Reasoning and Generalization: generalization and inductive bias.
  • FOUND-010 Optimization and Gradient Descent: minimizing information-theoretic losses.
  • TECH-LLM-001 Large Language Models: cross-entropy training and perplexity.
  • TECH-VAE-001 Autoencoders and Variational Autoencoders: KL terms in latent-variable models.
• Advanced Implications & Future Trajectories:
  • BIO-SHANNON-001 Claude Shannon: the founder of the field.
  • FOUND-014 Consciousness and Machine Understanding: information versus meaning.