SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantization

Generative modeling commonly relies on vector-quantized variational autoencoders to represent complex data into compact, discrete codes. However, these systems frequently suffer from codebook collapse, a failure mode where the model utilizes only a tiny fraction of its available representational capacity. To bypass this issue, standard approaches rely on brittle training heuristics and complex workarounds, including manual codebook resets and extensive hyperparameter tuning. These challenges motivate the need for a principled, self-adjusting framework that reliably maximizes code capacity without manual interventions.

The article demonstrates and evaluates a framework called the stochastically quantized variational autoencoder (SQ-VAE). Its primary objective is to replace heuristic training with a mathematically grounded variational Bayesian formulation that incorporates stochastic quantization and self-annealing to ensure high codebook utilization.

To evaluate this framework, the authors conducted extensive empirical experiments spanning diverse vision and speech tasks. The datasets examined include standard image benchmarks (MNIST, Fashion-MNIST, CIFAR10, CelebA, and CelebAHQ) as well as speech corpora (VCTK and ZeroSpeech 2019). The approach introduces trainable probability distributions—specifically Gaussian distributions for continuous data and von Mises–Fisher distributions for categorical data—that enable the model to start with broad stochastic exploration and naturally transition into deterministic encoding as training converges.

The findings show that SQ-VAE consistently outperforms standard vector-quantized autoencoders across multiple domains. First, SQ-VAE achieves substantially higher codebook utilization (measured by perplexity) and lower reconstruction errors; for instance, on categorical face mask data, it cut pixel error roughly in half from 6.95% down to 3.51% while improving intersection-over-union scores from 59.7% to 74.6%. Second, the architecture scales predictably: increasing codebook size or dimensionality directly improves performance in SQ-VAE, whereas conventional models show little benefit and remain prone to collapse. Third, on continuous image benchmarks, SQ-VAE lowered reconstruction mean squared error from 1.33 to under 0.98 on CelebA and delivered superior image generation metrics. Finally, speech experiments confirmed clear gains in spectrogram reconstruction accuracy over baseline models.

These results demonstrate that generative pipelines can eliminate ad-hoc heuristics and fragile manual tuning without sacrificing fidelity. By stabilizing latent representations, SQ-VAE reduces engineering overhead, streamlines model deployment, and provides a dependable path toward data compression and multi-modal representation learning.

Organizations developing generative models or discrete compression systems should consider SQ-VAE as a drop-in replacement for standard vector-quantized architectures. Before production rollout for specialized downstream tasks, teams should conduct domain-specific validation; while spectrogram reconstruction improved in speech evaluations, discriminative phonetic unit discovery scores remained on par with baselines, indicating that representation quality depends on the end objective.

The article's conclusions are strongly supported across multiple random seeds, image scales, and modalities. However, minor limitations remain regarding specific hyperparameter variants in variable-length sequential data, as certain complex variance parameterizations showed training instability on speech and large-scale vision tasks.