Tutorial on Variational Autoencoders

Carl Doersch

article2016arXiv2,043 citations

Derives the mathematical foundations and practical mechanics of Variational Autoencoders from first principles, bridging deep neural networks with probabilistic latent-variable modeling for readers without an advanced Bayesian background.

Listen

Generative modeling addresses the challenge of creating new, realistic data instances—such as images or 3D models—that reflect the complex statistical patterns of real-world datasets without copying them. Traditional methods for learning these distributions often suffered from severe drawbacks, including reliance on computationally prohibitive sampling procedures, restrictive structural assumptions, or crude approximations that impaired performance.

The article sets out to explain the theoretical foundations, optimization mechanics, and practical behavior of Variational Autoencoders (VAEs). It aims to demonstrate how pairing neural networks with variational inference provides an efficient, scalable solution for unsupervised generative modeling and structured prediction.

To evaluate this framework, the article outlines the mathematical formulation of latent variable models and demonstrates how the training objective can be efficiently optimized using standard backpropagation. The credibility of the framework is supported by empirical implementations on benchmark datasets like MNIST, testing both unconditional image synthesis and structured output generation through Conditional Variational Autoencoders (CVAEs).

The key findings show, first, that complex data distributions can be learned by mapping a simple, standard normal distribution through deep neural networks without manually hand-engineering the latent features. Second, optimization is made practical via the "reparameterization trick," which moves random sampling into an external input layer and allows standard gradient descent to update both the encoder and decoder simultaneously. Third, in practical experiments, CVAEs effectively handle ambiguous, one-to-many prediction tasks by producing sharp, plausible candidates, whereas traditional regression models fail by generating blurry averages of possible outcomes. Fourth, the model is resilient across a broad range of latent dimensions, performing well from 4 up to 1,000 latent variables, though severe performance degradation occurs if dimensions are set excessively low (under 4) or excessively high (around 10,000).

These findings imply that organizations can deploy generative models with weak structural assumptions and fast training pipelines, avoiding the high computational costs and slow inference speeds of legacy methods. Furthermore, for structured prediction problems where multiple valid outputs exist, adopting conditional latent variable models mitigates the risk of regression blur, leading to more realistic and usable system outputs.

For practical implementation, teams should select latent dimensions carefully within moderate bounds to prevent training degradation, and tuning the output variance hyperparameter is recommended when working with continuous data. Future theoretical and empirical efforts should focus on extending zero-approximation error proofs to multi-dimensional data and developing more robust perceptual metrics to evaluate sample quality.

The findings carry high confidence regarding the tractability and practical efficacy of the training framework. However, readers should remain aware of inherent limitations: current mathematical guarantees of zero approximation error are rigorously proven only for one-dimensional cases, and continuous latent spaces can occasionally generate unnatural "in-between" samples when transitioning across distinct data categories.

Carnegie Mellon University / UC Berkeley function approximators handwritten digit recognition stochastic variational autoencoders deep generative models variational Bayesian methods

arXiv: 1606.05908 cdoersch/vae_tutorial

Paper: Variational Inference: A Review for Statisticians, David M. Blei et al. (2016). This review establishes the fundamental mathematical framework of variational inference and the evidence lower bound (ELBO) upon which the VAE formulation is directly built.
Paper: Neural Variational Inference and Learning in Belief Networks, Andriy Mnih et al. (2014). It introduces the concept of amortized variational inference using neural networks to optimize a variational lower bound, providing crucial algorithmic context for VAEs.
Paper: Semi-supervised Learning with Deep Generative Models, Diederik P. Kingma et al. (2014). It demonstrates the foundational application of variational autoencoders to semi-supervised learning and deep latent-variable generative modeling.
Paper: Learning Structured Output Representation using Deep Conditional Generative Models, Kihyuk Sohn et al. (2015). It formulates the conditional variational autoencoder (CVAE) framework discussed in the tutorial for structured output prediction.
Paper: Variational Inference with Normalizing Flows, Danilo Jimenez Rezende et al. (2015). It presents normalizing flows to overcome the expressiveness limitations of standard Gaussian variational posteriors in latent-variable models.
Paper: Representation Learning: A Review and New Perspectives, Yoshua Bengio et al. (2012). It surveys the core principles of unsupervised representation learning and probabilistic autoencoding that motivate generative modeling with VAEs.

Paper: An Introduction to Variational Autoencoders, Diederik P. Kingma et al. (2019). This monograph extends the introductory tutorial into a comprehensive, advanced mathematical treatment of VAE theory, multi-layer generative models, and flow-based posteriors.
Paper: Improving Variational Inference with Inverse Autoregressive Flow, Diederik P. Kingma et al. (2016). It advances VAE posterior inference by introducing inverse autoregressive flows to capture rich latent dependencies in high-dimensional settings.
Paper: Neural Discrete Representation Learning, Aäron van den Oord et al. (2017). It extends the continuous VAE framework to learn discrete latent representations using vector quantization, addressing posterior collapse.
Paper: VAE with a VampPrior, Jakub M. Tomczak et al. (2018). It replaces the standard Gaussian prior of standard VAEs with a flexible, data-driven mixture-of-posteriors prior (VampPrior) to improve representation learning.
Paper: Wasserstein Auto-Encoders, Ilya Tolstikhin et al. (2018). It generalizes variational autoencoders to Wasserstein autoencoders, leveraging optimal transport theory to yield sharper generative samples.
Paper: Isolating Sources of Disentanglement in Variational Autoencoders, Ricky T. Q. Chen et al. (2018). It decomposes the ELBO objective to isolate total correlation, proposing an improved method for unsupervised disentangled representation learning in VAEs.
Paper: NVAE: A Deep Hierarchical Variational Autoencoder, Arash Vahdat et al. (2020). It designs deep hierarchical VAE architectures with specialized residual cells and regularizations to scale generative fidelity to high-resolution images.
Paper: Variational Graph Auto-Encoders, Thomas N. Kipf et al. (2016). It adapts the variational autoencoder framework to graph-structured data by combining graph convolutional networks with latent-variable inference.
Paper: Variational Autoencoders for Collaborative Filtering, Dawen Liang et al. (2018). It applies VAE principles to collaborative filtering with multinomial likelihoods for top-N recommendation systems.
Paper: Variational Diffusion Models, Diederik P. Kingma et al. (2021). It establishes a theoretical equivalence between diffusion models and continuous hierarchical VAEs to optimize exact data likelihoods.

Cover for Tutorial on Variational Autoencoders

Abstract

In just three years, Variational Autoencoders (VAEs) have emerged as one of the most popular approaches to unsupervised learning of complicated distributions. VAEs are appealing because they are built on top of standard function approximators (neural networks), and can be trained with stochastic gradient descent. VAEs have already shown promise in generating many kinds of complicated data, including handwritten digits, faces, house numbers, CIFAR images, physical models of scenes, segmentation, and predicting the future from static images. This tutorial introduces the intuitions behind VAEs, explains the mathematics behind them, and describes some empirical behavior. No prior knowledge of variational Bayesian methods is assumed.

Topics

function approximators handwritten digit recognition stochastic stochastic gradient descent supervised machine learning variational autoencoders variational bayesian variational auto encoders high resolution image

Keywords

variational autoencoders deep generative models variational Bayesian methods reparameterization trick Boltzmann Machines Helmholtz Machines Deep Belief Nets sparse autoencoders unsupervised learning stochastic gradient descent

1 Introduction
1.1 Preliminaries: Latent Variable Models
2 Variational Autoencoders
2.1 Setting up the objective
2.2 Optimizing the objective
2.3 Testing the learned model
2.4 Interpreting the objective
2.4.1 The error from 𝒟[Q(z|X)∥P(z|X)]\mathcal{D}[Q(z|X)\|P(z|X)]
2.4.2 The information-theoretic interpretation
2.4.3 VAEs and the regularization parameter
3 Conditional Variational Autoencoders
4 Examples
4.1 MNIST variational autoencoder
5 MNIST conditional variational autoencoder
A Proof in 1D that VAEs have zero approximation error given arbitrarily powerful learners.
References

Knowls

Knowl 1 — Variational Lower Bound (ELBO) for Latent Variable Models
equation
For an observed random variable $X \in \mathcal{X}$ , a continuous latent variable $z \in \mathcal{Z}$ , a prior distribution $P(z)$ , a generative decoding distribution $P(X|z)$ , and an approximate posterior variational distribution $Q(z|X)$ , the marginal log-likelihood $\log P(X)$ satisfies the identity:

$\log P(X) - \mathcal{D}_{\mathrm{KL}}[Q(z|X) \,\|\, P(z|X)] = \mathbb{E}_{z \sim Q(z|X)}\big[\log P(X|z)\big] - \mathcal{D}_{\mathrm{KL}}[Q(z|X) \,\|\, P(z)],$

where $\mathcal{D}_{\mathrm{KL}}[\cdot \,\|\, \cdot]$ denotes the Kullback-Leibler divergence.

Because $\mathcal{D}_{\mathrm{KL}}[Q(z|X) \,\|\, P(z|X)] \ge 0$ , the right-hand side provides a tractable lower bound on $\log P(X)$ , known as the Evidence Lower Bound (ELBO). Maximizing the ELBO with respect to the parameters of $Q$ and $P$ simultaneously pushes the approximate posterior $Q(z|X)$ towards the true posterior $P(z|X)$ while maximizing the marginal data log-likelihood $\log P(X)$ .
Knowl 2 — Reparameterization Trick for Continuous Latent Distributions
model/method
To train a variational autoencoder via backpropagation, gradients must flow through the sampling operation $z \sim Q(z|X) = \mathcal{N}(z \mid \mu(X), \Sigma(X))$ , where $\mu(X)$ and $\Sigma(X)$ are deterministic neural network outputs and $\Sigma(X)$ is diagonal. Direct sampling introduces a stochastic node with no well-defined gradient with respect to network parameters.

The reparameterization trick isolates the stochasticity by drawing an auxiliary noise vector $\epsilon \sim \mathcal{N}(0, I)$ independently of the network parameters and expressing $z$ as the deterministic transformation:

$z = \mu(X) + \Sigma^{1/2}(X) \odot \epsilon,$

where $\odot$ represents element-wise multiplication and $\Sigma^{1/2}(X)$ is the element-wise standard deviation. The stochastic gradient of the ELBO for a single training sample $X$ is computed from:

$\nabla \left( \log P\big(X \mid z = \mu(X) + \Sigma^{1/2}(X) \odot \epsilon\big) - \mathcal{D}_{\mathrm{KL}}[Q(z|X) \,\|\, P(z)] \right),$

which is continuous and differentiable with respect to all encoder and decoder parameters. This technique requires continuous latent spaces; discrete latent distributions induce step-function discontinuities that prevent backpropagation.
Knowl 3 — Closed-Form Gaussian-Gaussian KL Divergence
equation
When the variational posterior is parameterized as a multivariate Gaussian $Q(z|X) = \mathcal{N}(\mu(X), \Sigma(X))$ and the prior is a standard multivariate Gaussian $P(z) = \mathcal{N}(0, I)$ in a $k$ -dimensional latent space, the Kullback-Leibler divergence evaluates analytically as:

$\mathcal{D}_{\mathrm{KL}}[\mathcal{N}(\mu(X), \Sigma(X)) \,\|\, \mathcal{N}(0, I)] = \frac{1}{2} \left( \operatorname{tr}(\Sigma(X)) + \mu(X)^\top \mu(X) - k - \log \det(\Sigma(X)) \right).$

For a diagonal covariance matrix $\Sigma(X) = \operatorname{diag}(\sigma_1^2(X), \dots, \sigma_k^2(X))$ , this simplifies coordinate-wise to:

$\mathcal{D}_{\mathrm{KL}}[Q(z|X) \,\|\, P(z)] = \frac{1}{2} \sum_{j=1}^k \left( \sigma_j^2(X) + \mu_j^2(X) - 1 - \log(\sigma_j^2(X)) \right),$

where $\mu_j(X)$ and $\sigma_j^2(X)$ denote the mean and variance along the $j$ -th latent dimension.
Knowl 4 — Conditional Variational Autoencoder (CVAE) Framework
model/method
For conditional generative tasks and structured prediction where the mapping from an input condition $X$ to a target output $Y$ is one-to-many, a Conditional Variational Autoencoder (CVAE) conditions both the encoder $Q(z \mid Y, X)$ and the decoder $P(Y \mid z, X)$ on $X$ .

Assuming the prior over latent variables at inference time is independent of the condition, $P(z|X) = \mathcal{N}(0, I)$ , the variational lower bound on $\log P(Y|X)$ becomes:

$\log P(Y|X) - \mathcal{D}_{\mathrm{KL}}[Q(z|Y, X) \,\|\, P(z|Y, X)] = \mathbb{E}_{z \sim Q(z|Y, X)}\big[\log P(Y \mid z, X)\big] - \mathcal{D}_{\mathrm{KL}}[Q(z|Y, X) \,\|\, P(z|X)].$

During training, $Q(z \mid Y, X)$ generates latent codes from both the input condition $X$ and target $Y$ . At test time, the encoder is removed, latent codes are sampled from $z \sim \mathcal{N}(0, I)$ , and the conditional output is synthesized through the decoder $P(Y \mid z, X) = \mathcal{N}(f(z, X), \sigma^2 I)$ .
Knowl 5 — Zero Approximation Error of 1D VAEs for Smooth Distributions
theoretical result
Let $P_{\mathrm{gt}}(X)$ be a one-dimensional target probability density function that is strictly positive ( $P_{\mathrm{gt}}(X) > 0$ everywhere), infinitely differentiable, and has bounded derivatives. Consider a VAE with standard normal prior $P(z) = \mathcal{N}(0, 1)$ , Gaussian generative model $P_\sigma(X|z) = \mathcal{N}(X \mid f(z), \sigma^2)$ , and Gaussian variational posterior $Q_\sigma(z|X) = \mathcal{N}(z \mid \mu_\sigma(X), \Sigma_\sigma(X))$ .

As the decoder noise variance hyperparameter $\sigma \to 0$ , there exist deterministic functions $f$ , $\mu_\sigma$ , and $\Sigma_\sigma$ such that:
1. The induced marginal distribution $P_\sigma(X) = \int P_\sigma(X|z) P(z) \, dz$ converges to $P_{\mathrm{gt}}(X)$ .
2. The variational divergence vanishes everywhere:
$\lim_{\sigma \to 0} \mathcal{D}_{\mathrm{KL}}[Q_\sigma(z|X) \,\|\, P_\sigma(z|X)] = 0 \quad \text{for all } X.$

These functions are constructed via the cumulative distribution function (CDF) $F$ of $P_{\mathrm{gt}}$ and the CDF $G$ of $\mathcal{N}(0, 1)$ as $f(z) = F^{-1}(G(z))$ and $Q_\sigma(z|X) = \mathcal{N}(g(X), (g'(X)\sigma)^2)$ , where $g(X) = G^{-1}(F(X)) = f^{-1}(X)$ . Thus, with infinite learner capacity and vanishing decoder variance, 1D VAEs possess zero theoretical approximation error.
Knowl 6 — Decoder Variance as a Loss Weighting Parameter
theoretical result
In a VAE for continuous data with an isotropic Gaussian decoder $P(X|z) = \mathcal{N}(f(z), \sigma^2 I)$ , the log-likelihood term is:

$\log P(X|z) = -\frac{1}{2\sigma^2} \|X - f(z)\|^2 + C,$

where $C$ is a constant independent of $f$ . When substituting this into the ELBO, $\sigma^2$ acts as an explicit hyperparameter that controls the relative trade-off between the squared Euclidean reconstruction error and the latent regularization term $\mathcal{D}_{\mathrm{KL}}[Q(z|X) \,\|\, P(z)]$ .

In contrast, attempting to add a regularization parameter by scaling the prior variance $z \sim \mathcal{N}(0, \lambda I)$ has no effect on the model family or the objective: scaling parameters $\lambda$ are absorbed by setting $f'(z) = f(z/\lambda)$ , $\mu'(X) = \lambda \mu(X)$ , and $\Sigma'(X) = \lambda^2 \Sigma(X)$ . Furthermore, for discrete binary data with a Bernoulli output distribution, the reconstruction term is parameter-free cross-entropy, eliminating this regularization hyperparameter entirely.
Knowl 7 — Minimum Description Length Interpretation of the VAE Objective
theoretical result
The ELBO objective can be interpreted through information theory and the Minimum Description Length (MDL) principle as the total code length required to transmit a datapoint $X$ in a two-stage encoding scheme:
1. $-\log P(X)$ is the optimal theoretical description length of $X$ in nats.
2. $\mathcal{D}_{\mathrm{KL}}[Q(z|X) \,\|\, P(z)]$ represents the expected additional information (bits-back/information gain) required to communicate the latent code $z$ sampled from $Q(z|X)$ rather than the prior $P(z)$ .
3. $-\mathbb{E}_{z \sim Q}[\log P(X|z)]$ represents the expected description length required to reconstruct $X$ given the communicated latent variable $z$ .
The discrepancy between the true description length $-\log P(X)$ and the two-stage transmission cost equals $\mathcal{D}_{\mathrm{KL}}[Q(z|X) \,\|\, P(z|X)]$ , which corresponds to the coding inefficiency penalty paid for using an approximate posterior $Q(z|X)$ instead of the true posterior $P(z|X)$ .
Knowl 8 — Multimodal Modeling: CVAE versus Mean-Square Regression
empirical result
When conditioned on highly ambiguous or partial inputs (such as completing an entire handwritten digit given only a single column of noisy binary pixels), standard regression models trained with mean squared error minimize expected distance by generating a blurry average of all plausible target digits and styles.

In contrast, Conditional Variational Autoencoders (CVAEs) model the full conditional distribution $P(Y|X)$ . By sampling $z \sim \mathcal{N}(0, I)$ at test time, the CVAE samples distinct, sharp, and plausible individual digits from the multimodal distribution without blur.
Knowl 9 — Latent Dimensionality and Optimization Dynamics in VAEs
empirical result
The performance of a VAE depends on the dimensionality $k = \dim(z)$ of the latent space:
1. Too few latent dimensions (e.g., $k < 4$ on MNIST) restricts model capacity and prevents capturing sufficient factors of variation, degrading sample quality.
2. Moderate latent dimensions (e.g., $k = 1,000$ ) produce sharp, representative generative samples.
3. Excessively high latent dimensions (e.g., $k = 10,000$ ) degrade empirical performance. Although an optimal network could theoretically set unused latent dimensions to zero, stochastic gradient descent in practice struggles to keep $\mathcal{D}_{\mathrm{KL}}[Q(z|X) \,\|\, P(z)]$ small across excessively large latent vectors.

Coverage note — None was omitted; all substantive mathematical definitions, theoretical analyses, model formulations, and empirical insights presented in the tutorial are represented.

References

1.Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. ICLR, 2014.
2.Tim Salimans, Diederik Kingma, and Max Welling. Markov chain monte carlo and variational inference: Bridging the gap. ICML, 2015.
3.Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In ICML, 2014.
4.Tejas D Kulkarni, William F. Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. In NIPS, 2015.
5.Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In NIPS, 2014.
6.Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. In ICCV, 2015.
7.Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In NIPS, 2015.
8.Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert. An uncertain future: Forecasting from static images using variational autoencoders. In ECCV, 2016.
9.Alex Krizhevsky, Ilya Sutskever, and Geoff Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
10.Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607–609, 1996.
11.Honglak Lee, Alexis Battle, Rajat Raina, and Andrew Y Ng. Efficient sparse coding algorithms. In NIPS, 2006.
12.Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML, 2008.
13.Yoshua Bengio, Eric Thibodeau-Laufer, Guillaume Alain, and Jason Yosinski. Deep generative stochastic networks trainable by backprop. ICML, 2014.
14.Yoshua Bengio, Li Yao, Guillaume Alain, and Pascal Vincent. Generalized denoising auto-encoders as generative models. In NIPS, pages 899–907, 2013.
15.Luc Devroye. Sample-based non-uniform random variate generation. Springer-Verlag, New York, 1986.
16.Peter Dayan, Geoffrey E Hinton, Radford M Neal, and Richard S Zemel. The helmholtz machine. Neural computation, 7(5):889–904, 1995.
17.Geoffrey E Hinton, Peter Dayan, Brendan J Frey, and Radford M Neal. The" wake-sleep" algorithm for unsupervised neural networks. Science, 268(5214):1158–1161, 1995.
18.Geoffrey E Hinton, Simon Osindero, and Yee-Whye Teh. A fast learning algorithm for deep belief nets. Neural computation, 18(7):1527–1554, 2006.
19.Ruslan Salakhutdinov and Geoffrey E Hinton. Deep boltzmann machines. In International conference on artificial intelligence and statistics, pages 448–455, 2009.
20.Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, 1993.
21.Geoffrey E Hinton and Richard S Zemel. Autoencoders, minimum description length, and helmholtz free energy. In NIPS, 1994.
22.Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. In ACM-MM, 2014.
23.Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
24.Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models. ICLR, 2016.

Citation

MLA

Doersch, C. “Tutorial on Variational Autoencoders”. arXiv, 2016, http://arxiv.org/abs/1606.05908v3.

APA

Doersch, C. (2016). Tutorial on Variational Autoencoders. arXiv. http://arxiv.org/abs/1606.05908v3

Chicago

Doersch, C. 2016. “Tutorial on Variational Autoencoders”. arXiv. http://arxiv.org/abs/1606.05908v3.

Harvard

Doersch, C. (2016) “Tutorial on Variational Autoencoders”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1606.05908v3.

Vancouver

1. Doersch C (2016) Tutorial on Variational Autoencoders. arXiv

BibTeX

@article{doersch2016tutorial,
  title = {Tutorial on Variational Autoencoders},
  author = {Doersch, Carl},
  year = {2016},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1606.05908v3},
  eprint = {1606.05908}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

Tutorial on Variational Autoencoders

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Variational Lower Bound (ELBO) for Latent Variable Models

Knowl 2 — Reparameterization Trick for Continuous Latent Distributions

Knowl 3 — Closed-Form Gaussian-Gaussian KL Divergence

Knowl 4 — Conditional Variational Autoencoder (CVAE) Framework

Knowl 5 — Zero Approximation Error of 1D VAEs for Smooth Distributions

Knowl 6 — Decoder Variance as a Loss Weighting Parameter

Knowl 7 — Minimum Description Length Interpretation of the VAE Objective

Knowl 8 — Multimodal Modeling: CVAE versus Mean-Square Regression

Knowl 9 — Latent Dimensionality and Optimization Dynamics in VAEs

References

Citation

BibTeX

Source Code

Access the Paper