Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

neural autoregressive models

Neural autoregressive models are probabilistic generative neural networks that model the joint probability distribution of complex, high-dimensional data by decomposing it into a sequence of conditional probabilities. Using the chain rule of probability, these models predict each component of a data point, such as a pixel, word, or audio sample, conditioned on all previously observed or generated components. By employing neural architectures with recurrence or causal masking, such as recurrent neural networks, masked autoencoders, causal convolutional networks, and transformers, they capture complex sequential dependencies across variables. A defining advantage of neural autoregressive models is their ability to compute exact and tractable likelihoods, making them widely used for density estimation and step-by-step synthetic data generation across various modalities.

1 item

Variational Lossy Autoencoder

Variational Lossy Autoencoder

Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, Pieter Abbeel

OrganizationsOpenAIUniversity of California Berkeley

Why you should read this

Reveals that the common failure of VAEs to use their latent codes when paired with powerful decoders isn't a bug but a controllable feature—by deliberately limiting what the decoder can model locally (like small texture patches), you can force the latent code to capture exactly the global structure you care about while achieving state-of-the-art density estimation.

Part of:Ilya Sutskever's reading list on deep learning

Representation learning seeks to expose certain aspects of observed data in a learned representation that's amenable to downstream tasks like classification. For instance, a good representation for 2D images might be one that describes only global structure and discards information about detailed texture. In this paper, we present a simple but principled method to learn such global representations by combining Variational Autoencoder (VAE) with neural autoregressive models such as RNN, MADE and PixelRNN/CNN. Our proposed VAE model allows us to have control over what the global latent code can learn and , by designing the architecture accordingly, we can force the global latent code to discard irrelevant information such as texture in 2D images, and hence the VAE only "autoencodes" data in a lossy fashion. In addition, by leveraging autoregressive models as both prior distribution p(z) and decoding distribution p(x|z), we can greatly improve generative modeling performance of VAEs, achieving new state-of-the-art results on MNIST, OMNIGLOT and Caltech-101 Silhouettes density estimation tasks.

Added

2026-02-21