---
id: TECH-DIFF-001
title: "Diffusion Models"
alternative_names:
  - "Denoising Diffusion Probabilistic Models"
  - "DDPM"
  - "Latent Diffusion Models"
  - "Score-Based Generative Models"
category: "Technical → Generative Models"
volume: III
chapter: 18
entry_type: "Technical Reference"
level: 3
status: "Draft"
last_reviewed: 2026-10-15
version: "0.2.0"
prerequisites:
  - "TECH-DL-001"
  - "Probability Theory"
related_entries:
  - "CAP-GENAI-001"
  - "TECH-ARCH-001"
---

# Diffusion Models

## 1. Concise Definition
**Diffusion Models** are a class of deep generative models that synthesize high-fidelity data by learning to reverse a gradual, stochastic noising process. Mathematically, they progressively corrupt training data with Gaussian noise through a forward Markov chain, and then train a neural network (typically a U-Net or Transformer) to estimate and subtract that noise—effectively learning the score (the gradient of the log-probability density) of the data distribution to iteratively denoise pure random noise into coherent images, audio, or continuous data structures (Ho, Jain, & Abbeel, 2020).

## 2. Overview & Significance
The introduction of diffusion models revolutionized the field of generative artificial intelligence, particularly in visual and continuous modalities. Before diffusion models, Generative Adversarial Networks (GANs) were the leading approach to image synthesis; Dhariwal and Nichol (2021) reported that diffusion models could surpass GANs on image-synthesis benchmarks. GANs generate in a single forward pass, but their adversarial minimax training is widely reported to be unstable and prone to mode collapse (failing to represent the full diversity of the dataset); see TECH-GAN-001.

The significance of the diffusion paradigm includes:
1. **Training Stability & Diversity**: Diffusion models optimize a simplified, reweighted variational lower bound (Ho et al., 2020), avoiding adversarial training dynamics.
2. **Compositional Conditioning**: Through cross-attention mechanisms and classifier-free guidance, diffusion models show improved adherence to compositional natural language prompts, with known failures on spatial relations and counting (see Section 7).
3. **Continuous Domains**: While autoregressive Transformers dominate discrete sequence modeling (text), diffusion models are widely used for continuous data such as images, video and protein backbones (see Section 6).

## 3. Historical Development
The theoretical groundwork for diffusion models was laid in non-equilibrium thermodynamics. In 2015, Sohl-Dickstein et al. introduced diffusion probabilistic models for machine learning, drawing on non-equilibrium thermodynamics (Sohl-Dickstein et al., 2015).

Separately, Yang Song and Stefano Ermon (2019) developed **Score-Based Generative Modeling**, which trained networks via score matching with Langevin dynamics. In 2020, Ho, Jain, and Abbeel introduced **Denoising Diffusion Probabilistic Models (DDPM)**, establishing an explicit connection between diffusion models and denoising score matching and reporting an FID of 3.17 on CIFAR-10 (Ho et al., 2020). Song et al. (2021) later unified score-based models and diffusion models through stochastic differential equations.

In late 2021, OpenAI's GLIDE demonstrated text-conditioned diffusion and compared CLIP guidance with **Classifier-Free Guidance** (Ho & Salimans, 2022), which human evaluators preferred (Nichol et al., 2021); DALL-E 2 followed in 2022 (Ramesh et al., 2022). Running diffusion directly in high-resolution pixel space is computationally expensive (Rombach et al., 2022).

The critical architectural breakthrough occurred in 2022 with **Latent Diffusion Models (LDM)** (Rombach et al., 2022). By compressing images into a lower-dimensional latent space using an autoencoder and running the diffusion process within that space, LDMs reduced compute requirements relative to pixel-space diffusion (Rombach et al., 2022). The open-weight **Stable Diffusion** model is based on this approach.

## 4. Technical / Conceptual Explanation

### 4.1 The Forward Process (Noising)
Given an uncorrupted data sample $x_0 \sim q(x)$, the forward process adds Gaussian noise in $T$ discrete timesteps according to a predefined variance schedule $\beta_1, \dots, \beta_T$:
$$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{1 - \beta_t} x_{t-1}, \beta_t \mathbf{I})$$
Because the noise is Gaussian, any arbitrary timestep $t$ can be sampled directly from $x_0$ without simulating the intermediate steps, using $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$:
$$q(x_t \mid x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1 - \bar{\alpha}_t) \mathbf{I})$$
As $T \to \infty$, $x_T$ becomes isotropic Gaussian noise $\mathcal{N}(0, \mathbf{I})$.

### 4.2 The Reverse Process (Denoising)
The goal is to learn the reverse distribution $p_\theta(x_{t-1} \mid x_t)$ to denoise the data. This is approximated by a neural network (often a U-Net) parameterized by $\theta$. Ho et al. (2020) reported that, instead of predicting the mean of the distribution, training the network $\epsilon_\theta(x_t, t)$ to predict the noise vector $\epsilon$ added to $x_0$ gave good sample quality; the resulting loss is a simplified, reweighted variational bound rather than the exact likelihood bound:
$$\mathcal{L}(\theta) = \mathbb{E}_{t, x_0, \epsilon} \left[ \| \epsilon - \epsilon_\theta(\sqrt{\bar{\alpha}_t} x_0 + \sqrt{1 - \bar{\alpha}_t} \epsilon, t) \|^2 \right]$$

### 4.3 Conditioning and Classifier-Free Guidance (CFG)
To generate an image from text, the network must be conditioned on a text embedding $c$ (typically extracted from a CLIP text encoder). The network becomes $\epsilon_\theta(x_t, t, c)$.
**Classifier-Free Guidance** balances image fidelity and prompt adherence. During training, the conditioning vector $c$ is randomly dropped out (replaced with a null token $\emptyset$). During inference, the model predicts both the conditioned and unconditioned noise, and the update is extrapolated away from the unconditional prediction by a guidance scale $w$:
$$\hat{\epsilon} = \epsilon_\theta(x_t, t, \emptyset) + w \left( \epsilon_\theta(x_t, t, c) - \epsilon_\theta(x_t, t, \emptyset) \right)$$
Higher values of $w$ force the image to adhere more strictly to the prompt, often at the cost of diversity. This $w$ equals $1 + w$ in the convention of Ho and Salimans (2022), whose sampler uses $(1+w)\epsilon_\theta(z_t, c) - w\,\epsilon_\theta(z_t)$.

## 5. Types, Components & Classifications
1. **Pixel-Space Diffusion**: (e.g., the original DDPM; Ho et al., 2020). Operates directly on pixel values, which is computationally expensive at high resolution (Rombach et al., 2022).
2. **Latent Diffusion**: (e.g., Stable Diffusion). Operates in a compressed latent space. In Stable Diffusion v1, an autoencoder with downsampling factor 8 maps $512 \times 512$ images to $64 \times 64$ latents, where diffusion occurs, and a decoder reconstructs the final image (CompVis, 2022; Rombach et al., 2022).
3. **Transformer-Based Diffusion Backbones**: DiT replaces the convolutional U-Net backbone with a Transformer operating on latent patches (Peebles & Xie, 2023). Stable Diffusion 3 uses a transformer-based backbone (Esser et al., 2024), and OpenAI's Sora report describes a transformer operating on spacetime patches of latent codes (OpenAI, 2024).

## 6. Applications & Concrete Examples
- **Text-to-Image Generation**: Stable Diffusion (Rombach et al., 2022), GLIDE, DALL-E 2 and Imagen (Saharia et al., 2022) are diffusion-based text-to-image systems with published descriptions; the architectures of some commercial systems (for example Midjourney) are undisclosed.
- **Video Generation**: OpenAI's technical report (OpenAI, 2024) states that Sora is a text-conditional diffusion model with a transformer operating on spacetime patches of video and image latent codes; this is the developer's own description, and the report states that model and implementation details are not included.
- **Biomolecular Design**: RFdiffusion generates protein backbones by retraining the RoseTTAFold network on denoising tasks, with experimental validation reported for several design tasks (Watson et al., 2023).

## 7. Advantages, Limitations & Unresolved Questions

### Core Advantages
- **Distributional Coverage**: Diffusion models are widely described as less prone to mode collapse than GANs [VERIFICATION NEEDED: a source comparing sample diversity or recall of diffusion models and GANs].
- **Compositionality**: Can combine disjoint concepts (e.g., "an astronaut riding a horse in the style of Van Gogh") in conditioned generation, although failures occur (see below).

### Fundamental Limitations
- **Inference Latency**: GANs generate in a single forward pass, whereas autoregressive models require one pass per token. Diffusion models require iterative denoising, from about 1000 steps in the original DDPM setup (Ho et al., 2020) to tens of steps with accelerated samplers, making inference computationally intensive.
- **Compositional Semantic Failures**: Despite improvements, diffusion models can struggle with precise spatial relations ("a red cube *under* a blue sphere"), counting, exact text rendering and complex anatomy. [VERIFICATION NEEDED: a systematic source for spatial-relation, counting and text-rendering failures of diffusion models]
- **Memorization and Provenance**: Diffusion models can memorize and regenerate individual training images; Carlini et al. (2023) extracted over a thousand training examples from state-of-the-art models, so training-data provenance matters for privacy and copyright. The legal status of training on copyrighted data is not assessed in this article.

### Unresolved Scientific Questions
1. *Consistency vs. Iteration*: Can distillation methods (like Consistency Models; Song et al., 2023) reduce inference to one or a few steps without losing sample quality relative to many-step diffusion?
2. *Unifying Discrete and Continuous*: Can diffusion processes be natively adapted to discrete token sequences (natural language) to effectively compete with autoregressive LLMs?

## 8. Current State of Knowledge
Diffusion is a widely used paradigm for image, video and other continuous-data generation. Some recent systems use Transformer backbones in place of convolutional U-Nets; Peebles and Xie (2023) report that sample quality (FID) improves consistently as the forward-pass compute (Gflops) of DiT models increases. Inference-acceleration techniques, such as Consistency Models (Song et al., 2023) and Latent Consistency Models (Luo et al., 2023), aim to reduce the number of sampling steps.

## 9. Future Trajectory & Strategic Outlook
The following are speculative outlooks, not forecasts with formal records (see `docs/EVIDENCE_FRAMEWORK.md` section 2 and FCST-REG-001).
- **Near-Term (0–3 Years)**: Speculative: faster sampling through distillation methods, and wider use of diffusion in video and 3D generation.
- **Medium-Term (3–10 Years)**: Speculative: video generative models may be explored as simulators for robot planning; this is an open research question.
- **Long-Term (10+ Years)**: Highly speculative: autoregressive and diffusion methods may converge into a unified paradigm; no evidence establishes this outcome.

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[TECH-ML-001] Machine Learning](./TECH-ML-001_machine_learning.md): General principles of loss objectives.
- **Related Methods & Technologies**:
  - [[CAP-GENAI-001] Generative Artificial Intelligence](../vol04_capabilities/CAP-GENAI-001_generative_ai.md): The overarching category of generative systems.
  - [[TECH-GAN-001] Generative Adversarial Networks](./TECH-GAN-001_generative_adversarial_networks.md): The adversarial alternative to diffusion.
  - [[TECH-DL-001] Deep Learning](./TECH-DL-001_deep_learning.md): Optimization and representations underlying the U-Net and Transformer backbones.
  - [[TECH-ARCH-001] The Transformer Architecture](./TECH-ARCH-001_transformer_architecture.md): The architecture displacing U-Nets in modern DiT models.

## 11. References & Foundational Literature
1. Ho, J., Jain, A., & Abbeel, P. (2020). *Denoising Diffusion Probabilistic Models*. NeurIPS 2020. https://arxiv.org/abs/2006.11239. [Introduces DDPM, the noise-prediction loss, T = 1000 steps and the FID of 3.17 on CIFAR-10].
2. Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., & Ganguli, S. (2015). *Deep Unsupervised Learning using Nonequilibrium Thermodynamics*. ICML 2015. https://arxiv.org/abs/1503.03585. [Introduces diffusion probabilistic models for machine learning].
3. Song, Y., & Ermon, S. (2019). *Generative Modeling by Estimating Gradients of the Data Distribution*. NeurIPS 2019. https://arxiv.org/abs/1907.05600. [Score-based generative modeling].
4. Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2021). *Score-Based Generative Modeling through Stochastic Differential Equations*. ICLR 2021. https://arxiv.org/abs/2011.13456. [Unifies score-based and diffusion models via SDEs].
5. Dhariwal, P., & Nichol, A. (2021). *Diffusion Models Beat GANs on Image Synthesis*. https://arxiv.org/abs/2105.05233. [Reports diffusion models outperforming GANs on image-synthesis benchmarks].
6. Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., & Chen, M. (2021). *GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models*. https://arxiv.org/abs/2112.10741 (posted 2021-12-20). [Text-conditional diffusion comparing CLIP guidance and classifier-free guidance].
7. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022). *Hierarchical Text-Conditional Image Generation with CLIP Latents*. https://arxiv.org/abs/2204.06125 (posted 2022-04-13). [DALL-E 2].
8. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). *High-Resolution Image Synthesis with Latent Diffusion Models*. CVPR 2022. https://arxiv.org/abs/2112.10752. [Latent diffusion models].
9. CompVis (2022). *Stable Diffusion* (code repository README). https://github.com/CompVis/stable-diffusion. [States that Stable Diffusion v1 uses a downsampling-factor-8 autoencoder, an 860M-parameter UNet and a frozen CLIP ViT-L/14 text encoder].
10. Ho, J., & Salimans, T. (2022). *Classifier-Free Diffusion Guidance*. NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications (short version); arXiv:2207.12598 (posted 2022-07-26). https://arxiv.org/abs/2207.12598. [Classifier-free guidance].
11. Peebles, W., & Xie, S. (2023). *Scalable Diffusion Models with Transformers*. ICCV 2023. https://arxiv.org/abs/2212.09748. [The Diffusion Transformer (DiT)].
12. Esser, P., Kulal, S., Blattmann, A., et al. (2024). *Scaling Rectified Flow Transformers for High-Resolution Image Synthesis*. https://arxiv.org/abs/2403.03206 (17 authors). [Stable Diffusion 3 transformer-based backbone].
13. OpenAI (2024). *Video generation models as world simulators* (technical report, 2024-02-15). https://openai.com/index/video-generation-models-as-world-simulators/. [Developer description of Sora; the page returned HTTP 403 to automated fetch, and its content was checked through search-result excerpts only].
14. Saharia, C., Chan, W., Saxena, S., et al. (2022). *Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding*. https://arxiv.org/abs/2205.11487 (14 authors). [Imagen, a diffusion-based text-to-image system].
15. Watson, J. L., Juergens, D., Bennett, N. R., et al. (2023). *De novo design of protein structure and function with RFdiffusion*. Nature, 620(7976), 1089–1100. https://doi.org/10.1038/s41586-023-06415-8. [RFdiffusion protein backbone generation].
16. Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramèr, F., Balle, B., Ippolito, D., & Wallace, E. (2023). *Extracting Training Data from Diffusion Models*. USENIX Security Symposium 2023. https://arxiv.org/abs/2301.13188. [Measured extraction of over a thousand training examples from diffusion models].
17. Song, Y., Dhariwal, P., Chen, M., & Sutskever, I. (2023). *Consistency Models*. ICML 2023. https://arxiv.org/abs/2303.01469. [One-step generation by distillation or standalone training].
18. Luo, S., Tan, Y., Huang, L., Li, J., & Zhao, H. (2023). *Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference*. https://arxiv.org/abs/2310.04378. [Few-step latent diffusion sampling].

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| DDPM Training Objective (simplified, reweighted variational bound) | Well Supported (Empirical + Derivation) | Ho et al. (2020) | High |
| Latent Compression for Compute Reduction | Established (Empirical Validation) | Rombach et al. (2022) | High |
| Effectiveness of Classifier-Free Guidance | Well Supported | Ho & Salimans (2022) | High |
| Training-Data Memorization in Diffusion Models | Well Supported (empirical measurement) | Carlini et al. (2023) | High |
| Connection between Score Matching and DDPMs | Established (Derivation) | Ho et al. (2020); Song et al. (2021) | High |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft; citations not independently fact-checked.
- **v0.2.0** (2026-10-15): Corrected following independent fact-check audit (docs/audits/TECH-DIFF-001_audit.md); 11 findings applied, 2 left marked for verification.