Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

variational lowerbound

The variational lower bound, also known as the evidence lower bound, is a mathematical objective function used in probabilistic machine learning and Bayesian inference to approximate the intractable marginal likelihood of observed data. Derived through variational principles and Jensen inequality, it establishes a computable lower threshold on the true log-likelihood of data under a latent variable model. The objective generally comprises two key components: an expected reconstruction term that evaluates how accurately the model reconstructs observed data from latent representations, and a regularization term that penalizes the statistical divergence between the approximate posterior distribution and a prior distribution. By maximizing this bound during optimization, probabilistic models can simultaneously learn generative parameters and bring the approximate posterior closer to the true posterior distribution.

1 item

Variational Lossy Autoencoder

Variational Lossy Autoencoder

Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, Pieter Abbeel

OrganizationsOpenAIUniversity of California Berkeley

Why you should read this

Reveals that the common failure of VAEs to use their latent codes when paired with powerful decoders isn't a bug but a controllable feature—by deliberately limiting what the decoder can model locally (like small texture patches), you can force the latent code to capture exactly the global structure you care about while achieving state-of-the-art density estimation.

Part of:Ilya Sutskever's reading list on deep learning

Representation learning seeks to expose certain aspects of observed data in a learned representation that's amenable to downstream tasks like classification. For instance, a good representation for 2D images might be one that describes only global structure and discards information about detailed texture. In this paper, we present a simple but principled method to learn such global representations by combining Variational Autoencoder (VAE) with neural autoregressive models such as RNN, MADE and PixelRNN/CNN. Our proposed VAE model allows us to have control over what the global latent code can learn and , by designing the architecture accordingly, we can force the global latent code to discard irrelevant information such as texture in 2D images, and hence the VAE only "autoencodes" data in a lossy fashion. In addition, by leveraging autoregressive models as both prior distribution p(z) and decoding distribution p(x|z), we can greatly improve generative modeling performance of VAEs, achieving new state-of-the-art results on MNIST, OMNIGLOT and Caltech-101 Silhouettes density estimation tasks.

Added

2026-02-21