Autoencoders and Variational Autoencoders

An autoencoder is a neural network trained to reproduce its input after passing it through a constrained intermediate representation, so that the representation (the code or latent) captures the structure needed for reconstruction. A variational autoencoder (VAE) is a probabilistic version in which the encoder outputs a distribution over latent variables and the model is trained by maximizing a lower bound on the data likelihood, which makes it a generative model from which new samples can be drawn.

Autoencoders address representation learning without labels: the training signal is the input itself. A network with an encoder $f$ and a decoder $g$ is trained so that $g(f(x)) \approx x$, with a bottleneck or other constraint preventing a trivial identity mapping. The learned code can be used for compression, denoising and as features for other tasks. Hinton and Salakhutdinov (2006) reported that deep autoencoders trained to reduce high-dimensional data to low-dimensional codes worked better than principal component analysis on the data they studied.

The plain autoencoder is not a generative model, because nothing guarantees that arbitrary points in code space decode to plausible data. The VAE of Kingma and Welling (2013) addressed this by combining deep neural networks with variational inference over continuous latent variables, with an encoder acting as an approximate posterior. Rezende, Mohamed and Wierstra (2014) independently developed a closely related approach that they call stochastic backpropagation. The VAE's significance is twofold. It gives a principled training objective for deep latent-variable models, and its encoder-decoder structure has become a component of larger systems; for example, latent diffusion models apply diffusion in the latent space of a pretrained autoencoder (Rombach et al., 2021; see [TECH-DIFF-001]).

Classical and deep autoencoders. Autoencoders were studied as nonlinear generalizations of dimensionality reduction. Hinton and Salakhutdinov (2006) described how to train deep autoencoders effectively and reported that they outperformed principal component analysis on their tasks. Vincent, Larochelle, Bengio and Manzagol (2008) proposed denoising autoencoders, which learn to reconstruct a clean input from a corrupted version, as a way to learn robust features.

Variational autoencoders. Kingma and Welling (2013) introduced a stochastic variational inference and learning algorithm that scales to large datasets, based on a reparameterization of the variational lower bound and an approximate inference ("recognition") model. Rezende, Mohamed and Wierstra (2014) presented stochastic backpropagation, a rule for backpropagation through stochastic variables, together with a recognition model that acts as a stochastic encoder. Kingma and Welling (2019) later wrote an introductory treatment of the framework.

Extensions. Bowman et al. (2015) built an RNN-based VAE for sentences (see [TECH-RNN-001]) and reported negative results on its use as a language model. Van den Oord, Vinyals and Kavukcuoglu (2017) proposed the VQ-VAE, in which the encoder outputs discrete codes and the prior is learned; they state that this avoids "posterior collapse", in which the latents are ignored when paired with a powerful autoregressive decoder. Rombach et al. (2021) used pretrained autoencoders as the representation in which diffusion models operate.

#4.1 The deterministic autoencoder

With encoder $z = f_\phi(x)$ and decoder $\hat x = g_\theta(z)$, training minimizes a reconstruction loss such as

or a cross-entropy loss for binary data. Constraints that prevent trivial copying include a code dimension smaller than the input (an undercomplete autoencoder), regularization on the code, or input corruption as in the denoising autoencoder (Vincent et al., 2008). With linear encoder and decoder and squared error, the optimal subspace is the same as that of principal component analysis. Baldi and Hornik (1989) analyse this linear setting in relation to principal component analysis; their text was not read for this article, so the equivalence is stated here as the standard result rather than as a quoted finding.

#4.2 The latent-variable model

A VAE posits a generative process: draw a latent $z \sim p(z)$ (usually a standard Gaussian $\mathcal{N}(0, I)$), then draw $x \sim p_\theta(x \mid z)$, where the decoder network parameterizes $p_\theta(x \mid z)$. The data likelihood

is intractable for flexible decoders, and so is the true posterior $p_\theta(z \mid x)$. The VAE introduces an encoder $q_\phi(z \mid x)$, typically a Gaussian with mean $\mu_\phi(x)$ and diagonal covariance $\sigma^2_\phi(x)$ produced by a network, as an approximation to the posterior (Kingma and Welling, 2013).

#4.3 The evidence lower bound

For any $q_\phi$, the log-likelihood decomposes as

Because the last term is non-negative, the ELBO is a lower bound on $\log p_\theta(x)$, and maximizing it with respect to both $\theta$ and $\phi$ improves the model and tightens the approximation. The first term rewards accurate reconstruction from latents sampled from the encoder; the second term, the KL divergence, regularizes the encoder's distribution toward the prior. For a Gaussian encoder and standard normal prior the KL term has a closed form:

#4.4 The reparameterization trick

The expectation over $z \sim q_\phi(z \mid x)$ depends on $\phi$ through the sampling distribution, which blocks ordinary backpropagation. Writing the sample as a deterministic function of the parameters and an independent noise variable,

moves the randomness into $\varepsilon$, so gradients flow through $\mu_\phi$ and $\sigma_\phi$ and the lower bound can be optimized by standard stochastic gradient methods (Kingma and Welling, 2013; Rezende et al., 2014; see [TECH-BACKPROP-001]).

#4.5 Generation and inference

To generate, sample $z \sim p(z)$ and decode. To embed data, pass $x$ through the encoder. Interpolation between codes of two inputs produces intermediate decoded outputs; Bowman et al. (2015) used this to study the latent space of sentences.

#4.6 Posterior collapse

If the decoder is powerful enough to model the data without using the latent, the optimization can push $q_\phi(z\mid x)$ toward the prior, making the code uninformative; this is called posterior collapse. It is reported as a difficulty for VAEs with autoregressive decoders (van den Oord et al., 2017, whose abstract names posterior collapse explicitly; Bowman et al. (2015) describe a difficult learning problem for their sentence VAE, but their abstract does not use the term).

Other regularized variants exist but are not described here.

• Dimensionality reduction and visualization. Deep autoencoders were reported to work better than principal component analysis on the data studied (Hinton and Salakhutdinov, 2006); Rezende et al. (2014) report use for high-dimensional data visualization.
• Denoising and feature learning. Denoising autoencoders learn robust features (Vincent et al., 2008).
• Missing data imputation. Rezende et al. (2014) report accurate imputations of missing data in their experiments.
• Image, video and speech generation. VQ-VAE with an autoregressive prior is reported to generate high-quality images, video and speech (van den Oord et al., 2017).
• Latent spaces for large generative systems. Autoencoders compress images before diffusion modelling (Rombach et al., 2021), and the idea extends to multimodal systems (see [TECH-MM-001]).
• Convolutional implementations. Image VAEs commonly use convolutional encoders and decoders (see [TECH-CNN-001]).

#Advantages

• Stable, likelihood-based training. The ELBO gives a single objective optimized by gradient descent, in contrast with the adversarial game of [TECH-GAN-001].
• Inference model. The encoder gives a fast mapping from data to latent variables (Kingma and Welling, 2013).
• Flexible latent structure. Continuous or discrete latents, and a learned prior, are possible (van den Oord et al., 2017).

#Limitations

• Approximation gap. The ELBO is only a lower bound; the tightness depends on how well $q_\phi$ matches the true posterior.
• Posterior collapse with strong decoders (Bowman et al., 2015; van den Oord et al., 2017).
• Sample quality in some settings. This article makes no comparison of sample quality against adversarial models.
• Gaussian assumptions. Simple factorized Gaussian encoders and fixed priors may limit expressiveness.

#Unresolved questions

1. Meaningful and disentangled latent representations. Whether latent coordinates correspond to human-interpretable factors is a research question; no disentanglement study is reviewed here.
2. Best role of autoencoders in large systems. They are widely used as compressors for diffusion models (Rombach et al., 2021), but what design of latent space gives the best downstream quality for different modalities is open.
3. Avoiding collapse while keeping informative latents in sequence models (Bowman et al., 2015).

As of 2026-10-15, the ELBO decomposition and the reparameterization trick are Established mathematical results, presented in the original paper and an introductory monograph (Kingma and Welling, 2013, 2019). Autoencoders remain widely used as learned compressors within larger generative pipelines (Rombach et al., 2021), while VAEs as stand-alone generators are less prominent for high-fidelity image synthesis than diffusion-based approaches. This is a statement about the sources reviewed here, not a measured survey of usage. The success of discrete-latent autoencoders is Well Supported by the VQ-VAE authors' reports on images, video and speech, though independent comparison across tasks was not examined for this draft.

This section describes plausible directions and does not register forecasts.

• Near term (0–3 years). Continued use of autoencoders as tokenizers and compressors for images, audio and video that feed larger generative or multimodal models.
• Medium term (3–10 years). Research on how latent representations should be structured for downstream generation and for reasoning in latent space is active, and no consensus has been confirmed.
• Long term (10+ years). Learned latent models as components of world models (see [AGI-WM-001]) is a hypothesis, not an established route.

• Foundational Concepts:
  • TECH-NN-001 Artificial Neural Networks: encoder and decoder networks.
  • TECH-BACKPROP-001 Backpropagation: gradients through stochastic nodes via reparameterization.
  • TECH-ML-001 Machine Learning: unsupervised and self-supervised learning context.
• Related Methods & Technologies:
  • TECH-GAN-001 Generative Adversarial Networks: the adversarial alternative.
  • TECH-DIFF-001 Diffusion Models: models that operate in an autoencoder's latent space.
  • TECH-CNN-001 Convolutional Neural Networks: common backbone of image encoders and decoders.
  • TECH-RNN-001 Recurrent Networks and LSTM: sequence VAEs.
  • TECH-TOK-001 Tokenization and Embeddings: discrete representations of inputs.
• Advanced Implications & Future Trajectories:
  • TECH-MM-001 Multimodal Foundation Models: latent codes across modalities.
  • AGI-WM-001 World Models: learned latent dynamics models.
  • CAP-GENAI-001 Generative Artificial Intelligence: the capability landscape of generative systems.