A variational lossy autoencoder is a deep generative model that combines a variational autoencoder architecture with expressive autoregressive decoders to learn compact, high-level representations of data while deliberately discarding fine-grained, local details. By pairing latent variables with an autoregressive component capable of modeling local statistical dependencies such as textures on its own, the network prevents the latent code from wasting capacity on predictable patterns. Consequently, the latent space is constrained to retain only global structure and abstract features in a lossy manner, which enhances the quality of learned representations for downstream tasks as well as the model generative performance.
Representation learning seeks to expose certain aspects of observed data in a learned representation that's amenable to downstream tasks like classification. For instance, a good representation for 2D images might be one that describes only global structure and discards information about detailed texture. In this paper, we present a simple but principled method to learn such global representations by combining Variational Autoencoder (VAE) with neural autoregressive models such as RNN, MADE and PixelRNN/CNN. Our proposed VAE model allows us to have control over what the global latent code can learn and , by designing the architecture accordingly, we can force the global latent code to discard irrelevant information such as texture in 2D images, and hence the VAE only "autoencodes" data in a lossy fashion. In addition, by leveraging autoregressive models as both prior distribution p(z) and decoding distribution p(x|z), we can greatly improve generative modeling performance of VAEs, achieving new state-of-the-art results on MNIST, OMNIGLOT and Caltech-101 Silhouettes density estimation tasks.
