---
id: TECH-VAE-001
title: "Autoencoders and Variational Autoencoders"
alternative_names:
  - "Autoencoder"
  - "Variational Autoencoder"
  - "VAE"
  - "Latent-Variable Generative Models (deep)"
category: "Technical → Generative Models"
volume: III
chapter: 18
entry_type: "Standard Article"
level: 3
status: "Draft"
last_reviewed: 2026-10-15
version: "0.1.0"
prerequisites:
  - "TECH-NN-001"
related_entries:
  - "TECH-GAN-001"
  - "TECH-DIFF-001"
  - "TECH-BACKPROP-001"
  - "TECH-CNN-001"
  - "TECH-MM-001"
  - "CAP-GENAI-001"
---

# Autoencoders and Variational Autoencoders

## 1. Concise Definition
An **autoencoder** is a neural network trained to reproduce its input after passing it through a constrained intermediate representation, so that the representation (the **code** or **latent**) captures the structure needed for reconstruction. A **variational autoencoder** (VAE) is a probabilistic version in which the encoder outputs a distribution over latent variables and the model is trained by maximizing a lower bound on the data likelihood, which makes it a generative model from which new samples can be drawn.

## 2. Overview & Significance
Autoencoders address representation learning without labels: the training signal is the input itself. A network with an encoder $f$ and a decoder $g$ is trained so that $g(f(x)) \approx x$, with a bottleneck or other constraint preventing a trivial identity mapping. The learned code can be used for compression, denoising and as features for other tasks. Hinton and Salakhutdinov (2006) reported that deep autoencoders trained to reduce high-dimensional data to low-dimensional codes worked better than principal component analysis on the data they studied.

The plain autoencoder is not a generative model, because nothing guarantees that arbitrary points in code space decode to plausible data. The VAE of Kingma and Welling (2013) addressed this by combining deep neural networks with variational inference over continuous latent variables, with an encoder acting as an approximate posterior. Rezende, Mohamed and Wierstra (2014) independently developed a closely related approach that they call stochastic backpropagation. The VAE's significance is twofold. It gives a principled training objective for deep latent-variable models, and its encoder-decoder structure has become a component of larger systems; for example, latent diffusion models apply diffusion in the latent space of a pretrained autoencoder (Rombach et al., 2021; see [TECH-DIFF-001]).

## 3. Historical Development
**Classical and deep autoencoders.** Autoencoders were studied as nonlinear generalizations of dimensionality reduction. Hinton and Salakhutdinov (2006) described how to train deep autoencoders effectively and reported that they outperformed principal component analysis on their tasks. Vincent, Larochelle, Bengio and Manzagol (2008) proposed **denoising autoencoders**, which learn to reconstruct a clean input from a corrupted version, as a way to learn robust features.

**Variational autoencoders.** Kingma and Welling (2013) introduced a stochastic variational inference and learning algorithm that scales to large datasets, based on a reparameterization of the variational lower bound and an approximate inference ("recognition") model. Rezende, Mohamed and Wierstra (2014) presented stochastic backpropagation, a rule for backpropagation through stochastic variables, together with a recognition model that acts as a stochastic encoder. Kingma and Welling (2019) later wrote an introductory treatment of the framework.

**Extensions.** Bowman et al. (2015) built an RNN-based VAE for sentences (see [TECH-RNN-001]) and reported negative results on its use as a language model. Van den Oord, Vinyals and Kavukcuoglu (2017) proposed the **VQ-VAE**, in which the encoder outputs discrete codes and the prior is learned; they state that this avoids "posterior collapse", in which the latents are ignored when paired with a powerful autoregressive decoder. Rombach et al. (2021) used pretrained autoencoders as the representation in which diffusion models operate.

## 4. Technical / Conceptual Explanation

### 4.1 The deterministic autoencoder
With encoder $z = f_\phi(x)$ and decoder $\hat x = g_\theta(z)$, training minimizes a reconstruction loss such as

$$ \mathcal{L}_{\text{AE}} = \frac{1}{m}\sum_{i=1}^{m} \big\| x_i - g_\theta\big(f_\phi(x_i)\big) \big\|^2 , $$

or a cross-entropy loss for binary data. Constraints that prevent trivial copying include a code dimension smaller than the input (an **undercomplete** autoencoder), regularization on the code, or input corruption as in the denoising autoencoder (Vincent et al., 2008). With linear encoder and decoder and squared error, the optimal subspace is the same as that of principal component analysis. [VERIFICATION NEEDED: this is a standard textbook result; no source was checked for this draft.]

### 4.2 The latent-variable model
A VAE posits a generative process: draw a latent $z \sim p(z)$ (usually a standard Gaussian $\mathcal{N}(0, I)$), then draw $x \sim p_\theta(x \mid z)$, where the decoder network parameterizes $p_\theta(x \mid z)$. The data likelihood

$$ p_\theta(x) = \int p_\theta(x \mid z)\, p(z)\, dz $$

is intractable for flexible decoders, and so is the true posterior $p_\theta(z \mid x)$. The VAE introduces an **encoder** $q_\phi(z \mid x)$, typically a Gaussian with mean $\mu_\phi(x)$ and diagonal covariance $\sigma^2_\phi(x)$ produced by a network, as an approximation to the posterior (Kingma and Welling, 2013).

### 4.3 The evidence lower bound
For any $q_\phi$, the log-likelihood decomposes as

$$ \log p_\theta(x) = \underbrace{\mathbb{E}_{q_\phi(z\mid x)}\big[\log p_\theta(x \mid z)\big] - D_{\mathrm{KL}}\big(q_\phi(z \mid x)\,\|\,p(z)\big)}_{\text{ELBO}(\theta,\phi;x)} + D_{\mathrm{KL}}\big(q_\phi(z \mid x)\,\|\,p_\theta(z \mid x)\big). $$

Because the last term is non-negative, the ELBO is a lower bound on $\log p_\theta(x)$, and maximizing it with respect to both $\theta$ and $\phi$ improves the model and tightens the approximation. The first term rewards accurate reconstruction from latents sampled from the encoder; the second term, the KL divergence, regularizes the encoder's distribution toward the prior. For a Gaussian encoder and standard normal prior the KL term has a closed form:

$$ D_{\mathrm{KL}}\big(\mathcal{N}(\mu,\operatorname{diag}(\sigma^2))\,\|\,\mathcal{N}(0,I)\big) = \tfrac{1}{2}\sum_{j=1}^{d}\big(\mu_j^2 + \sigma_j^2 - \log \sigma_j^2 - 1\big). $$

### 4.4 The reparameterization trick
The expectation over $z \sim q_\phi(z \mid x)$ depends on $\phi$ through the sampling distribution, which blocks ordinary backpropagation. Writing the sample as a deterministic function of the parameters and an independent noise variable,

$$ z = \mu_\phi(x) + \sigma_\phi(x)\odot\varepsilon, \qquad \varepsilon \sim \mathcal{N}(0, I), $$

moves the randomness into $\varepsilon$, so gradients flow through $\mu_\phi$ and $\sigma_\phi$ and the lower bound can be optimized by standard stochastic gradient methods (Kingma and Welling, 2013; Rezende et al., 2014; see [TECH-BACKPROP-001]).

### 4.5 Generation and inference
To generate, sample $z \sim p(z)$ and decode. To embed data, pass $x$ through the encoder. Interpolation between codes of two inputs produces intermediate decoded outputs; Bowman et al. (2015) used this to study the latent space of sentences.

### 4.6 Posterior collapse
If the decoder is powerful enough to model the data without using the latent, the optimization can push $q_\phi(z\mid x)$ toward the prior, making the code uninformative; this is called posterior collapse. It is reported as a difficulty for VAEs with autoregressive decoders (Bowman et al., 2015; van den Oord et al., 2017).

## 5. Types, Components & Classifications
| Model | Key idea | Source |
| :--- | :--- | :--- |
| Undercomplete autoencoder | Bottleneck smaller than input | Hinton & Salakhutdinov (2006) |
| Denoising autoencoder | Reconstruct clean input from corrupted input | Vincent et al. (2008) |
| VAE | Probabilistic encoder, ELBO, reparameterization | Kingma & Welling (2013); Rezende et al. (2014) |
| Sequence VAE | RNN encoder and decoder over sentences | Bowman et al. (2015) |
| VQ-VAE | Discrete codes, learned prior | van den Oord et al. (2017) |
| Autoencoder used as a diffusion latent space | Diffusion in a pretrained autoencoder's latent space | Rombach et al. (2021) |

Other regularized variants (sparse, contractive, and disentanglement-oriented autoencoders) exist. [VERIFICATION NEEDED: none were checked against sources for this draft, so none are described.]

## 6. Applications & Concrete Examples
- **Dimensionality reduction and visualization.** Deep autoencoders were reported to work better than principal component analysis on the data studied (Hinton and Salakhutdinov, 2006); Rezende et al. (2014) report use for high-dimensional data visualization.
- **Denoising and feature learning.** Denoising autoencoders learn robust features (Vincent et al., 2008).
- **Missing data imputation.** Rezende et al. (2014) report accurate imputations of missing data in their experiments.
- **Image, video and speech generation.** VQ-VAE with an autoregressive prior is reported to generate high-quality images, video and speech (van den Oord et al., 2017).
- **Latent spaces for large generative systems.** Autoencoders compress images before diffusion modelling (Rombach et al., 2021), and the idea extends to multimodal systems (see [TECH-MM-001]).
- **Convolutional implementations.** Image VAEs commonly use convolutional encoders and decoders (see [TECH-CNN-001]).

## 7. Advantages, Limitations & Unresolved Questions

### Advantages
- **Stable, likelihood-based training.** The ELBO gives a single objective optimized by gradient descent, in contrast with the adversarial game of [TECH-GAN-001].
- **Inference model.** The encoder gives a fast mapping from data to latent variables (Kingma and Welling, 2013).
- **Flexible latent structure.** Continuous or discrete latents, and a learned prior, are possible (van den Oord et al., 2017).

### Limitations
- **Approximation gap.** The ELBO is only a lower bound; the tightness depends on how well $q_\phi$ matches the true posterior.
- **Posterior collapse** with strong decoders (Bowman et al., 2015; van den Oord et al., 2017).
- **Sample quality in some settings.** VAE samples are often described as blurrier than those of adversarial models for images. [VERIFICATION NEEDED: no source for this comparison was confirmed in this draft.]
- **Gaussian assumptions.** Simple factorized Gaussian encoders and fixed priors may limit expressiveness.

### Unresolved questions
1. **Meaningful and disentangled latent representations.** Whether latent coordinates correspond to human-interpretable factors is a research question. [VERIFICATION NEEDED: no disentanglement paper was confirmed for this draft.]
2. **Best role of autoencoders in large systems.** They are widely used as compressors for diffusion models (Rombach et al., 2021), but what design of latent space gives the best downstream quality for different modalities is open.
3. **Avoiding collapse while keeping informative latents** in sequence models (Bowman et al., 2015).

## 8. Current State of Knowledge
As of 2026-10-15, the ELBO derivation and the reparameterization trick are Established mathematical results, and are standard textbook material (Kingma and Welling, 2013, 2019). Autoencoders remain widely used as learned compressors within larger generative pipelines (Rombach et al., 2021), while VAEs as stand-alone generators are less prominent for high-fidelity image synthesis than diffusion-based approaches. This is a statement about the sources reviewed here, not a measured survey of usage. The success of discrete-latent autoencoders is Well Supported by the VQ-VAE authors' reports on images, video and speech, though independent comparison across tasks was not examined for this draft.

## 9. Future Trajectory & Strategic Outlook
This section describes plausible directions and does not register forecasts.

- **Near term (0–3 years).** Continued use of autoencoders as tokenizers and compressors for images, audio and video that feed larger generative or multimodal models.
- **Medium term (3–10 years).** Research on how latent representations should be structured for downstream generation and for reasoning in latent space is active, and no consensus has been confirmed.
- **Long term (10+ years).** Learned latent models as components of world models (see [AGI-WM-001]) is a hypothesis, not an established route.

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[TECH-NN-001] Artificial Neural Networks](./TECH-NN-001_artificial_neural_networks.md): encoder and decoder networks.
  - [[TECH-BACKPROP-001] Backpropagation](./TECH-BACKPROP-001_backpropagation.md): gradients through stochastic nodes via reparameterization.
  - [[TECH-ML-001] Machine Learning](./TECH-ML-001_machine_learning.md): unsupervised and self-supervised learning context.
- **Related Methods & Technologies**:
  - [[TECH-GAN-001] Generative Adversarial Networks](./TECH-GAN-001_generative_adversarial_networks.md): the adversarial alternative.
  - [[TECH-DIFF-001] Diffusion Models](./TECH-DIFF-001_diffusion_models.md): models that operate in an autoencoder's latent space.
  - [[TECH-CNN-001] Convolutional Neural Networks](./TECH-CNN-001_convolutional_neural_networks.md): common backbone of image encoders and decoders.
  - [[TECH-RNN-001] Recurrent Networks and LSTM](./TECH-RNN-001_recurrent_networks_and_lstm.md): sequence VAEs.
  - [[TECH-TOK-001] Tokenization and Embeddings](./TECH-TOK-001_tokenization_and_embeddings.md): discrete representations of inputs.
- **Advanced Implications & Future Trajectories**:
  - [[TECH-MM-001] Multimodal Foundation Models](./TECH-MM-001_multimodal_foundation_models.md): latent codes across modalities.
  - [[AGI-WM-001] World Models](../vol05_agi/AGI-WM-001_world_models.md): learned latent dynamics models.
  - [[CAP-GENAI-001] Generative Artificial Intelligence](../vol04_capabilities/CAP-GENAI-001_generative_ai.md): the capability landscape of generative systems.

## 11. References & Foundational Literature
1. Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks. *Science*, 313(5786), 504–507. https://doi.org/10.1126/science.1127647. [Supports training deep autoencoders and the reported advantage over principal component analysis on the studied data.]
2. Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008). Extracting and composing robust features with denoising autoencoders. In *Proceedings of the 25th International Conference on Machine Learning (ICML '08)*, pp. 1096–1103. https://doi.org/10.1145/1390156.1390294. [Supports the denoising autoencoder.]
3. Kingma, D. P., & Welling, M. (2013). Auto-encoding variational Bayes. arXiv:1312.6114. https://arxiv.org/abs/1312.6114. [Supports the reparameterized variational lower bound and the recognition model.]
4. Rezende, D. J., Mohamed, S., & Wierstra, D. (2014). Stochastic backpropagation and approximate inference in deep generative models. arXiv:1401.4082. https://arxiv.org/abs/1401.4082. [Supports stochastic backpropagation, the recognition model and reported imputation and visualization uses.]
5. Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A. M., Jozefowicz, R., & Bengio, S. (2015). Generating sentences from a continuous space. arXiv:1511.06349. https://arxiv.org/abs/1511.06349. [Supports the RNN-based sentence VAE, latent interpolation and negative results on language modelling.]
6. van den Oord, A., Vinyals, O., & Kavukcuoglu, K. (2017). Neural discrete representation learning. arXiv:1711.00937. https://arxiv.org/abs/1711.00937. [Supports VQ-VAE and the authors' statement that it avoids posterior collapse.]
7. Kingma, D. P., & Welling, M. (2019). An introduction to variational autoencoders. arXiv:1906.02691. https://arxiv.org/abs/1906.02691. [Supports the introductory treatment of the framework. Only title, authors and date were checked; content not read for this draft.]
8. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021). High-resolution image synthesis with latent diffusion models. arXiv:2112.10752. https://arxiv.org/abs/2112.10752. [Supports diffusion in the latent space of pretrained autoencoders.]

Note: references 1 and 2 were confirmed against Crossref; references 3–8 were confirmed against arXiv abstract pages (title, authors, date). In-text years follow the first-submission year of the arXiv version. The statement in section 8 that the ELBO and reparameterization are textbook material rests on Kingma and Welling (2013, 2019) only; no textbook was checked.

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| ELBO decomposition and Gaussian KL formula | Established (mathematical result) | Kingma & Welling (2013) | High |
| Reparameterization trick enables gradient-based training | Established | Kingma & Welling (2013); Rezende et al. (2014) | High |
| Deep autoencoders outperform PCA on the studied tasks | Well Supported (task-specific) | Hinton & Salakhutdinov (2006) | Moderate |
| Denoising autoencoders learn robust features | Well Supported | Vincent et al. (2008) | Moderate |
| Posterior collapse with strong decoders | Well Supported | Bowman et al. (2015); van den Oord et al. (2017) | Moderate |
| VQ-VAE avoids posterior collapse | Preliminary (author claim) | van den Oord et al. (2017) | Moderate |
| Autoencoders as latent spaces for diffusion | Well Supported | Rombach et al. (2021) | Moderate to High |
| Learned latents as basis for world models | Hypothetical | None confirmed | Open |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.