Stochastic variational inference

Matt HoffmanDavid M. BleiChong WangJohn Paisley

article2012JMLR2,872 citations

Develops stochastic variational inference, a scalable algorithm that applies stochastic optimization to variational bounds, enabling complex Bayesian models to perform posterior inference on massive datasets containing millions of documents.

Listen

Modern data analysis increasingly involves massive collections of complex, high-dimensional observations such as millions of documents, user histories, or gene sequences. Probabilistic graphical models provide a principled way to encode assumptions about hidden structure, yet standard posterior inference methods, including traditional variational inference, require repeated passes over the entire data set and therefore cannot scale to these sizes.

The article develops stochastic variational inference, a general algorithm that casts mean-field variational inference as a stochastic optimization problem and solves it with noisy natural-gradient steps obtained by repeatedly subsampling the data. The method is derived for the broad class of models whose complete conditionals belong to the exponential family and is instantiated for latent Dirichlet allocation and its Bayesian nonparametric counterpart, the hierarchical Dirichlet process topic model.

On three large corpora—300,000 Nature articles, 1.8 million New York Times articles, and 3.8 million Wikipedia articles—stochastic variational inference converges faster and reaches better predictive likelihoods than batch variational inference run on subsets that fit in memory. The nonparametric model consistently outperforms its fixed-topic counterpart while automatically determining the number of topics needed.

These results show that complex Bayesian models can be applied to data sets orders of magnitude larger than previously feasible, without clusters or specialized hardware, thereby extending the practical reach of probabilistic modeling to web-scale collections and streaming sources. The approach also highlights the value of combining variational methods with stochastic optimization and natural gradients.

Further gains may come from extending the framework to nonconjugate models, incorporating structured variational families, and developing adaptive or variance-reduced gradient estimators. The main limitations are the conjugacy assumption required for closed-form updates, the need to tune the learning-rate schedule, and the truncation levels chosen for the nonparametric model; results are therefore most reliable when these modeling choices are validated on held-out data.

Adobe Carnegie Mellon University Princeton University University of California hierarchical dirichlet process inference posterior distributions stochastic variational inference mean-field variational inference stochastic optimization

arXiv: 1206.7051

Paper: Latent Dirichlet Allocation, David M. Blei et al. (2003). It introduces Latent Dirichlet Allocation and classical batch variational inference, providing the primary probabilistic model and baseline that stochastic variational inference aims to scale.
Paper: An Introduction to Variational Methods for Graphical Models, MICHAEL I. JORDAN et al. (1999). It provides foundational principles and derivations for mean-field variational approximations in graphical models, which underpin the optimization framework of stochastic variational inference.
Paper: A Natural Policy Gradient, Sham M. Kakade (2001). It develops the natural gradient formulation based on the Fisher information matrix, which is crucial for deriving the Riemannian natural-gradient updates in stochastic variational inference.
Paper: Bayesian Learning via Stochastic Gradient Langevin Dynamics, M. Welling et al. (2011). It establishes the paradigm of subsampling mini-batches to scale Bayesian posterior inference, motivating stochastic gradient techniques in large datasets.
Paper: Adaptive Subgradient Methods for Online Learning and Stochastic Optimization, John Duchi et al. (2011). It introduces adaptive stochastic subgradient methods that directly address the challenge of setting learning-rate schedules in stochastic optimization.
Paper: Unsupervised Learning by Probabilistic Latent Semantic Analysis, Thomas Hofmann (2001). It presents Probabilistic Latent Semantic Analysis, an essential conceptual predecessor to the topic modeling and latent variable paradigms optimized in stochastic variational inference.

Paper: Variational Inference: A Review for Statisticians, David M. Blei et al. (2016). It synthesizes modern variational inference for statisticians, positioning stochastic variational inference within the broader ecosystem of coordinate-ascent and scalable optimization techniques.
Paper: Stochastic Backpropagation and Approximate Inference in Deep Generative Models, Danilo Jimenez Rezende et al. (2014). It extends stochastic variational principles to deep generative models by introducing stochastic backpropagation through continuous latent variables.
Paper: Neural Variational Inference and Learning in Belief Networks, Andriy Mnih et al. (2014). It builds on stochastic variational inference by using neural networks to amortize inference and introducing variance-reduction techniques for nonconjugate belief networks.
Paper: Variational Inference with Normalizing Flows, Danilo Jimenez Rezende et al. (2015). It moves beyond the simple mean-field families used in stochastic variational inference by employing normalizing flows to construct flexible posterior approximations.
Paper: Improving Variational Inference with Inverse Autoregressive Flow, Diederik P. Kingma et al. (2016). It improves upon mean-field stochastic variational methods by introducing inverse autoregressive flows for richer, highly expressive latent variable posteriors.
Paper: Weight Uncertainty in Neural Network, C. Blundell et al. (2015). It adapts stochastic variational inference to quantify parameter uncertainty in deep neural networks via the Bayes by Backprop algorithm.
Paper: Variational Autoencoders for Collaborative Filtering, Dawen Liang et al. (2018). It applies scalable stochastic variational principles and amortized inference to large-scale recommender systems and implicit feedback datasets.
Paper: Accelerating Stochastic Gradient Descent using Predictive Variance Reduction, Rie Johnson et al. (2013). It addresses a key open direction of stochastic variational inference by introducing stochastic variance-reduced gradients to accelerate convergence.
Paper: An Introduction to Variational Autoencoders, Diederik P. Kingma et al. (2019). It comprehensively surveys variational autoencoders, illustrating the evolution of stochastic variational inference into deep amortized latent-variable modeling.

Cover for Stochastic variational inference

Abstract

We develop stochastic variational inference, a scalable algorithm for approximating posterior distributions. We develop this technique for a large class of probabilistic models and we demonstrate it with two probabilistic topic models, latent Dirichlet allocation and the hierarchical Dirichlet process topic model. Using stochastic variational inference, we analyze several large collections of documents: 300K articles from Nature, 1.8M articles from The New York Times, and 3.8M articles from Wikipedia. Stochastic inference can easily handle data sets of this size and outperforms traditional variational inference, which can only handle a smaller subset. (We also show that the Bayesian nonparametric topic model outperforms its parametric counterpart.) Stochastic variational inference lets us apply complex Bayesian models to massive data sets.

Topics

hierarchical dirichlet process inference posterior distributions probabilistic models variational approximation variational inference variational methods probabilistic inference deterministic

Keywords

stochastic variational inference mean-field variational inference stochastic optimization hierarchical Dirichlet processes approximate inference latent variable models stochastic gradient Langevin dynamic topic models Gibbs sampling

1 Introduction
2 Stochastic Variational Inference
2.1 Models with local and global hidden variables
2.2 Mean-field variational inference
2.3 The natural gradient of the ELBO
2.4 Stochastic variational inference
2.5 Extensions
3 Stochastic Variational Inference in Topic Models
3.1 Notation
3.2 Latent Dirichlet allocation
3.3 Bayesian nonparametric topic models with the HDP
4 Empirical Study
5 Discussion
References

Knowls

Knowl 1 — Conditionally Conjugate Exponential Family Model Class for Variational Inference
model/method
Consider a probabilistic model over $N$ observations $x = (x_1, \dots, x_N)$ , local latent variables $z = (z_1, \dots, z_N)$ where each local variable comprises $J$ components $z_n = (z_{n,1}, \dots, z_{n,J})$ , and global latent variables $\beta$ , governed by fixed hyperparameters $\alpha = (\alpha_1, \alpha_2)$ . The joint probability distribution factorizes into a global prior and a product of local likelihoods:
$p(x, z, \beta \mid \alpha) = p(\beta \mid \alpha) \prod_{n=1}^N p(x_n, z_n \mid \beta)$
The conditional distributions of each latent variable given the remaining hidden and observed variables (complete conditionals) belong to the exponential family:
$p(\beta \mid x, z, \alpha) = h(\beta) \exp\left\{ \eta_g(x, z, \alpha)^\top t(\beta) - a_g(\eta_g(x, z, \alpha)) \right\}$ $p(z_{nj} \mid x_n, z_{n,-j}, \beta) = h(z_{nj}) \exp\left\{ \eta_\ell(x_n, z_{n,-j}, \beta)^\top t(z_{nj}) - a_\ell(\eta_\ell(x_n, z_{n,-j}, \beta)) \right\}$
where $h(\cdot)$ is the base measure, $a(\cdot)$ is the log-normalizer, $t(\cdot)$ denotes the sufficient statistics, and $\eta(\cdot)$ is the natural parameter function.

Conjugacy between the global variable $\beta$ and the local contexts $(x_n, z_n)$ implies that the local likelihood is $p(x_n, z_n \mid \beta) = h(x_n, z_n) \exp\{ \beta^\top t(x_n, z_n) - a_\ell(\beta) \}$ and the prior is $p(\beta \mid \alpha) = h(\beta) \exp\{ \alpha^\top t(\beta) - a_g(\alpha) \}$ with $t(\beta) = (\beta, -a_\ell(\beta))$ . Consequently, the complete conditional natural parameter for $\beta$ decomposes into a prior term and a sum over all local sufficient statistics:
$\eta_g(x, z, \alpha) = \left( \alpha_1 + \sum_{n=1}^N t(x_n, z_n), \, \alpha_2 + N \right)$
Knowl 2 — Natural Gradient of the Evidence Lower Bound in Conjugate Exponential Families
theoretical result
In mean-field variational inference, the posterior distribution $p(\beta, z \mid x)$ is approximated by a factorized distribution:
$q(\beta, z) = q(\beta \mid \lambda) \prod_{n=1}^N \prod_{j=1}^J q(z_{nj} \mid \phi_{nj})$
where $q(\beta \mid \lambda) = h(\beta) \exp\{ \lambda^\top t(\beta) - a_g(\lambda) \}$ and $q(z_{nj} \mid \phi_{nj}) = h(z_{nj}) \exp\{ \phi_{nj}^\top t(z_{nj}) - a_\ell(\phi_{nj}) \}$ belong to the same exponential families as their complete conditionals.

The Evidence Lower Bound (ELBO) maximized by variational inference is:
$\mathcal{L}(\lambda, \phi) = \mathbb{E}_q\left[ \log p(x, z, \beta \mid \alpha) \right] - \mathbb{E}_q\left[ \log q(z, \beta) \right]$
The standard Euclidean gradient of the ELBO with respect to the global parameter $\lambda$ is:
$\nabla_\lambda \mathcal{L} = \nabla_\lambda^2 a_g(\lambda) \left( \mathbb{E}_q[\eta_g(x, z, \alpha)] - \lambda \right)$
where $G(\lambda) = \nabla_\lambda^2 a_g(\lambda)$ is the Fisher information matrix of $q(\beta \mid \lambda)$ .

The natural gradient $\hat{\nabla}_\lambda \mathcal{L}$ , defined with respect to the Riemannian metric induced by $G(\lambda)$ (measuring distance via symmetrized Kullback-Leibler divergence), premultiplies the Euclidean gradient by $G(\lambda)^{-1}$ :
$\hat{\nabla}_\lambda \mathcal{L} = G(\lambda)^{-1} \nabla_\lambda \mathcal{L} = \mathbb{E}_q[\eta_g(x, z, \alpha)] - \lambda$
Similarly, the natural gradient for the local parameter $\phi_{nj}$ is:
$\hat{\nabla}_{\phi_{nj}} \mathcal{L} = \mathbb{E}_q[\eta_\ell(x_n, z_{n,-j}, \beta)] - \phi_{nj}$
Setting the natural gradient to zero gives the closed-form coordinate ascent update, meaning a natural gradient ascent step with step size 1 is identical to a standard coordinate ascent update.
Knowl 3 — Stochastic Variational Inference Algorithm
algorithm
Stochastic Variational Inference (SVI) optimizes the Evidence Lower Bound over global variational parameters $\lambda$ using stochastic natural gradient ascent by subsampling data from the complete dataset of $N$ observations.
Input: Dataset of observations $x = (x_1, \dots, x_N)$ , prior hyperparameter $\alpha = (\alpha_1, \alpha_2)$ , batch size $S$ , forgetting rate $\kappa \in (0.5, 1]$ , delay $\tau \ge 0$ .
Output: Global variational parameter vector $\lambda$ .
Initialize global parameter vector $\lambda^{(0)}$ randomly.
Set step counter $t = 1$ .
repeat
    Sample a minibatch of $S$ data points $x_{t, 1}, \dots, x_{t, S}$ uniformly at random from the dataset.
    for each sampled data point $x_{t, s}$ in the minibatch do
        Compute local variational parameters $\phi_{t, s}$ by iteratively optimizing:
             $\phi_{t, s, j} = \mathbb{E}_{\lambda^{(t-1)}, \phi_{t, s, -j}}\left[ \eta_\ell(x_{t, s}, z_{t, s, -j}, \beta) \right]$ for all $j \in \{1, \dots, J\}$ .
        Compute the intermediate global parameter for observation $x_{t, s}$ :
             $\hat{\lambda}_{t, s} = \alpha + N \cdot \left( \mathbb{E}_{\phi_{t, s}}[t(x_{t, s}, z_{t, s})], \, 1 \right)$ .
    end for
    Compute average intermediate global parameter:
         $\hat{\lambda}_t = \frac{1}{S} \sum_{s=1}^S \hat{\lambda}_{t, s}$ .
    Compute step size:
         $\rho_t = (t + \tau)^{-\kappa}$ .
    Update global variational parameters:
         $\lambda^{(t)} = (1 - \rho_t) \lambda^{(t-1)} + \rho_t \hat{\lambda}_t$ .
    Increment $t \leftarrow t + 1$ .
until stopping criterion is reached.
return $\lambda^{(t)}$ .
The Robbins-Monro step-size sequence $\rho_t = (t + \tau)^{-\kappa}$ satisfies $\sum_{t=1}^\infty \rho_t = \infty$ and $\sum_{t=1}^\infty \rho_t^2 < \infty$ , ensuring asymptotic convergence to a stationary point of the ELBO.
Knowl 4 — Stochastic Variational Inference for Latent Dirichlet Allocation
algorithm
Latent Dirichlet Allocation (LDA) models a corpus of $D$ documents over vocabulary size $V$ using $K$ topics $\beta_k \sim \text{Dirichlet}(\eta)$ , document topic proportions $\theta_d \sim \text{Dirichlet}(\alpha)$ , word topic assignments $z_{dn} \sim \text{Multinomial}(\theta_d)$ , and observed words $w_{dn} \sim \text{Multinomial}(\beta_{z_{dn}})$ . SVI optimizes the global topic Dirichlet parameters $\lambda_{1:K}$ and local document parameters $\gamma_d$ and $\phi_{dn}$ .
Input: Corpus of $D$ documents, vocabulary size $V$ , number of topics $K$ , priors $\alpha$ and $\eta$ , step schedule parameters $\kappa \in (0.5, 1]$ and $\tau \ge 0$ .
Output: Variational topic Dirichlet parameters $\lambda_{1:K}$ .
Initialize $\lambda_{1:K}^{(0)}$ randomly such that $\lambda_{kv} - \eta \sim \text{Exponential}(100 D / (K V))$ .
Set step counter $t = 1$ .
repeat
    Sample a document $w_d = (w_{d,1}, \dots, w_{d,N})$ uniformly at random from the corpus.
    Initialize document Dirichlet parameter $\gamma_{dk} = 1$ for all $k \in \{1, \dots, K\}$ .
    repeat
        for each word index $n \in \{1, \dots, N\}$ do
            for each topic $k \in \{1, \dots, K\}$ do
                 $\phi_{dn}^k \propto \exp\left\{ \Psi(\gamma_{dk}) - \Psi\left(\sum_{j=1}^K \gamma_{dj}\right) + \Psi(\lambda_{k, w_{dn}}) - \Psi\left(\sum_{v=1}^V \lambda_{kv}\right) \right\}$
            end for
            Normalize $\phi_{dn}$ such that $\sum_{k=1}^K \phi_{dn}^k = 1$ .
        end for
        Update $\gamma_d = \alpha + \sum_{n=1}^N \phi_{dn}$ .
    until local parameters $\phi_d$ and $\gamma_d$ converge.
    for each topic $k \in \{1, \dots, K\}$ do
        Compute intermediate topic parameter:
             $\hat{\lambda}_k = \eta + D \sum_{n=1}^N \phi_{dn}^k w_{dn}$ .
        Compute step size $\rho_t = (t + \tau)^{-\kappa}$ .
        Update topic parameter:
             $\lambda_k^{(t)} = (1 - \rho_t) \lambda_k^{(t-1)} + \rho_t \hat{\lambda}_k$ .
    end for
    Increment $t \leftarrow t + 1$ .
until termination condition met.
return $\lambda_{1:K}$ .
Here $\Psi(\cdot)$ denotes the digamma function, and $w_{dn}$ is represented as an indicator vector over the vocabulary of length $V$ .
Knowl 5 — Truncated Stick-Breaking Formulation of the Hierarchical Dirichlet Process Topic Model
model/method
The Hierarchical Dirichlet Process (HDP) topic model provides a Bayesian nonparametric formulation where the number of topics is unbounded and determined by data. The model uses a two-level stick-breaking construction:
1. Topics $\beta_k \sim \text{Dirichlet}(\eta)$ for $k \in \{1, 2, \dots\}$ .
2. Corpus stick breaking proportions $v_k \sim \text{Beta}(1, \omega)$ , inducing global weights $\sigma_k(v) = v_k \prod_{j=1}^{k-1} (1 - v_j)$ .
3. For each document $d \in \{1, \dots, D\}$ : (a) Draw document-level topic pointers $c_{di} \sim \text{Multinomial}(\sigma(v))$ for $i \in \{1, 2, \dots\}$ . (b) Draw document breaking proportions $\pi_{di} \sim \text{Beta}(1, \alpha)$ , inducing document weights $\sigma_i(\pi_d) = \pi_{di} \prod_{j=1}^{i-1} (1 - \pi_{dj})$ . (c) For each word $n \in \{1, \dots, N\}$ : draw topic index assignment $z_{dn} \sim \text{Multinomial}(\sigma(\pi_d))$ and word $w_{dn} \sim \text{Multinomial}(\beta_{c_{d, z_{dn}}})$ .
To enable variational inference, the infinite representations are truncated to $K$ global topics/breaking proportions and $T$ document-level components ( $T \ll K$ ):
$q(\beta, v, z, c, \pi) = \left( \prod_{k=1}^K q(\beta_k \mid \lambda_k) q(v_k \mid a_k, b_k) \right) \prod_{d=1}^D \left( \prod_{i=1}^T q(c_{di} \mid \zeta_{di}) q(\pi_{di} \mid \gamma_{di}^{(1)}, \gamma_{di}^{(2)}) \prod_{n=1}^N q(z_{dn} \mid \phi_{dn}) \right)$
Under Beta and Dirichlet variational distributions, the expectations used during inference are:
$\mathbb{E}_q[\log v_k] = \Psi(a_k) - \Psi(a_k + b_k), \quad \mathbb{E}_q[\log(1 - v_k)] = \Psi(b_k) - \Psi(a_k + b_k)$ $\mathbb{E}_q[\log \sigma_k(v)] = \mathbb{E}_q[\log v_k] + \sum_{\ell=1}^{k-1} \mathbb{E}_q[\log(1 - v_\ell)]$
where $\Psi(\cdot)$ is the digamma function.
Knowl 6 — Stochastic Variational Inference for the Hierarchical Dirichlet Process Topic Model
algorithm
Stochastic variational inference fits the truncated two-level stick-breaking Hierarchical Dirichlet Process (HDP) topic model on a corpus of $D$ documents with vocabulary size $V$ , corpus truncation level $K$ , and document-level truncation level $T$ .
Input: Corpus of $D$ documents, vocabulary size $V$ , corpus truncation $K$ , document truncation $T$ , priors $\eta, \omega, \alpha$ , step schedule parameters $\kappa \in (0.5, 1]$ and $\tau \ge 0$ .
Output: Variational topic parameters $\lambda_{1:K}$ and corpus stick parameters $(a_{1:K}, b_{1:K})$ .
Initialize $\lambda_{1:K}^{(0)}$ randomly, and set $a_k^{(0)} = 1, b_k^{(0)} = \omega$ for all $k \in \{1, \dots, K\}$ .
Set step counter $t = 1$ .
repeat
    Sample a document $w_d = (w_{d,1}, \dots, w_{d,N})$ uniformly at random from the corpus.
    for $i \in \{1, \dots, T\}$ and $k \in \{1, \dots, K\}$ do
        Initialize $\zeta_{di}^k \propto \exp\left\{ \sum_{n=1}^N \mathbb{E}[\log \beta_{k, w_{dn}}] \right\}$ .
    end for
    for $n \in \{1, \dots, N\}$ and $i \in \{1, \dots, T\}$ do
        Initialize $\phi_{dn}^i \propto \exp\left\{ \sum_{k=1}^K \zeta_{di}^k \mathbb{E}[\log \beta_{k, w_{dn}}] \right\}$ .
    end for
    repeat
        for $i \in \{1, \dots, T\}$ do
             $\gamma_{di}^{(1)} = 1 + \sum_{n=1}^N \phi_{dn}^i$ .
             $\gamma_{di}^{(2)} = \alpha + \sum_{n=1}^N \sum_{j=i+1}^T \phi_{dn}^j$ .
            for $k \in \{1, \dots, K\}$ do
                 $\zeta_{di}^k \propto \exp\left\{ \mathbb{E}[\log \sigma_k(v)] + \sum_{n=1}^N \phi_{dn}^i \mathbb{E}[\log \beta_{k, w_{dn}}] \right\}$ .
            end for
            Normalize $\zeta_{di}$ such that $\sum_{k=1}^K \zeta_{di}^k = 1$ .
        end for
        for $n \in \{1, \dots, N\}$ do
            for $i \in \{1, \dots, T\}$ do
                 $\phi_{dn}^i \propto \exp\left\{ \mathbb{E}[\log \sigma_i(\pi_d)] + \sum_{k=1}^K \zeta_{di}^k \mathbb{E}[\log \beta_{k, w_{dn}}] \right\}$ .
            end for
            Normalize $\phi_{dn}$ such that $\sum_{i=1}^T \phi_{dn}^i = 1$ .
        end for
    until local parameters $\zeta_d, \gamma_d, \phi_d$ converge.
    for $k \in \{1, \dots, K\}$ do
         $\hat{\lambda}_{kv} = \eta + D \sum_{i=1}^T \zeta_{di}^k \sum_{n=1}^N \phi_{dn}^i w_{dn}^v$ for all $v \in \{1, \dots, V\}$ .
         $\hat{a}_k = 1 + D \sum_{i=1}^T \zeta_{di}^k$ .
         $\hat{b}_k = \omega + D \sum_{i=1}^T \sum_{\ell=k+1}^K \zeta_{di}^\ell$ .
    end for
    Compute learning rate $\rho_t = (t + \tau)^{-\kappa}$ .
    Update global parameters:
         $\lambda^{(t)} = (1 - \rho_t) \lambda^{(t-1)} + \rho_t \hat{\lambda}$ .
         $a^{(t)} = (1 - \rho_t) a^{(t-1)} + \rho_t \hat{a}$ .
         $b^{(t)} = (1 - \rho_t) b^{(t-1)} + \rho_t \hat{b}$ .
    Increment $t \leftarrow t + 1$ .
until termination condition met.
return $\lambda_{1:K}, a_{1:K}, b_{1:K}$ .
Knowl 7 — Per-Word Predictive Log-Likelihood Evaluation Metric for Topic Models
definition
Model fitness in topic modeling is measured by the predictive distribution over held-out words from unseen test documents rather than variational lower bounds on perplexity.

Each held-out test document's words $w$ are partitioned into an observed set $w_{\text{obs}}$ and a held-out evaluation set $w_{\text{ho}}$ , such that their unique vocabulary items are disjoint. Given global topics estimated from training data $\mathcal{D}$ , the document's topic proportions $q(\theta)$ are inferred using $w_{\text{obs}}$ .

The predictive distribution for a new word token $w_{\text{new}}$ is given by:
$p(w_{\text{new}} \mid \mathcal{D}, w_{\text{obs}}) \approx \sum_{k=1}^K \mathbb{E}_q[\theta_k] \, \mathbb{E}_q[\beta_{k, w_{\text{new}}}]$
where $\mathbb{E}_q[\beta_{k, v}] = \frac{\lambda_{kv}}{\sum_{v'=1}^V \lambda_{kv'}}$ and $\mathbb{E}_q[\theta_k] = \frac{\gamma_k}{\sum_{j=1}^K \gamma_j}$ (with analogous expectations derived from the two-level stick-breaking variational parameters for the HDP).

The evaluation metric evaluates the average log probability across all tokens in $w_{\text{ho}}$ under this predictive distribution. Higher values reflect superior predictive ability and better generalization.
Knowl 8 — Experimental Setup for Large-Scale Text Corpus Topic Modeling
experimental setup
Stochastic variational inference was evaluated against traditional batch coordinate ascent variational inference across three document collections:
- Nature: 350,000 articles (1869–2008), comprising 58M words with a vocabulary of 4,200 terms.
- The New York Times: 1.8M articles (1987–2007), comprising 461M words with a vocabulary of 8,000 terms.
- Wikipedia: 3.8M articles, comprising 482M words with a vocabulary of 7,700 terms.
Each corpus withheld a test set of 10,000 documents for computing per-word predictive log-likelihood.

Model parameter configurations:
- LDA: Topic counts $K \in \{25, 50, 100, 200, 300\}$ , Dirichlet prior $\alpha = 1/K$ , topic Dirichlet parameter $\eta = 0.01$ .
- HDP: Corpus truncation $K = 300$ , document truncation $T = 20$ , concentration hyperparameters $\gamma = 1$ and $\alpha = 1$ , topic Dirichlet parameter $\eta = 0.01$ .
- SVI parameters: Delay $\tau = 1$ , forgetting rates $\kappa \in \{0.5, 0.6, 0.7, 0.8, 0.9, 1.0\}$ , and minibatch sizes $S \in \{10, 50, 100, 500, 1000\}$ .
- Batch VI baseline: Trained on a 100,000-document subset of each corpus because full datasets exceeded batch capacity.

Knowl 9 — Predictive Log-Likelihood Comparison of SVI LDA and HDP Across Large Document Corpora

data/table

Comparison of per-word predictive log-likelihood on held-out test sets across three large document collections for Latent Dirichlet Allocation (LDA) at various topic counts $K$ and the Hierarchical Dirichlet Process (HDP) topic model ( $K=300, T=20$ ), trained using stochastic variational inference (SVI) with forgetting rate $\kappa = 0.9$ and minibatch size $S = 500$ .

Model	Nature	New York Times	Wikipedia
LDA 25	-7.24	-7.73	-7.44
LDA 50	-7.23	-7.68	-7.43
LDA 100	-7.26	-7.66	-7.41
LDA 200	-7.50	-7.78	-7.64
LDA 300	-7.86	-7.98	-7.74
HDP	-6.97	-7.38	-7.07

The HDP consistently outperforms LDA across all corpora and topic settings. For LDA, predictive likelihood degrades when $K \ge 200$ due to overfitting under symmetric exchangeable Dirichlet priors. In contrast, the HDP remains robust to overfitting despite a large truncation limit ( $K = 300$ ), facilitated by corpus-level stick-breaking proportions that assign non-uniform prior weights across topics.

Knowl 10 — Hyperparameter Sensitivity and Convergence Dynamics of SVI
empirical result
Analysis of SVI across forgetting rates $\kappa \in \{0.5, 0.6, 0.7, 0.8, 0.9, 1.0\}$ and minibatch sizes $S \in \{10, 50, 100, 500, 1000\}$ shows:
1. Forgetting rate $\kappa$ : Higher values of $\kappa$ (close to 0.9 or 1.0) consistently achieve superior predictive log-likelihood optima for both LDA and HDP. Lower values (e.g., $\kappa = 0.5$ ) decrease step sizes too slowly early on and converge to poorer local optima.
2. Minibatch size $S$ : Very small batches (e.g., $S = 10$ ) exhibit high gradient variance and hurt asymptotic model fit. Larger minibatch sizes ( $S = 500$ or $1000$ ) provide smoother, higher-quality convergence, with diminishing returns beyond $S = 500$ . Sensitivity to minibatch size was highest on The New York Times corpus and lowest on Wikipedia.
3. Computational speed: SVI on the complete corpus converges to high predictive log-likelihood significantly faster in wall-clock time than batch coordinate ascent variational inference operating on a 100,000-document subset.
Knowl 11 — Stochastic Variational Inference for Conditionally Conjugate Global Subvector Parameters
model/method
When the joint conditional distribution $p(\beta \mid x, z, \alpha)$ is not in a tractable exponential family, but the global parameter vector can be partitioned into $K$ subvectors $\beta = (\beta_1, \dots, \beta_K)$ such that each conditional distribution $p(\beta_k \mid x, z, \beta_{-k}, \alpha)$ is in a tractable exponential family:
$p(\beta_k \mid x, z, \beta_{-k}, \alpha) = h(\beta_k) \exp\left\{ \eta_g(x, z, \beta_{-k}, \alpha)^\top t(\beta_k) - a_g(\eta_g(x, z, \beta_{-k}, \alpha)) \right\}$
the conditional natural parameter decomposes into a global parameter term and a sum of local context terms:
$\eta_g(x, z, \beta_{-k}, \alpha) = \eta_g(\beta_{-k}, \alpha) + \sum_{n=1}^N \eta_g(x_n, z_n, \beta_{-k}, \alpha)$
Assigning each subvector an independent variational distribution $q(\beta_k \mid \lambda_k) = h(\beta_k) \exp\{ \lambda_k^\top t(\beta_k) - a_g(\lambda_k) \}$ , sampling a single observation $(x_i, z_i)$ produces an unbiased noisy natural gradient with respect to $\lambda_k$ :
$\hat{\nabla}_{\lambda_k} \mathcal{L}_i = -\lambda_k + \mathbb{E}_q[\eta_g(\beta_{-k}, \alpha)] + N \, \mathbb{E}_q[\eta_g(x_i, z_i, \beta_{-k}, \alpha)] \equiv -\lambda_k + \hat{\lambda}_k$
The global update with step size $\rho_t = (t + \tau)^{-\kappa}$ is:
$\lambda_k^{(t)} = (1 - \rho_t) \lambda_k^{(t-1)} + \rho_t \hat{\lambda}_k$
All subvectors $\lambda_1^{(t)}, \dots, \lambda_K^{(t)}$ are updated simultaneously using parameters from step $t-1$ , avoiding the sequential coordinate dependencies of batch variational inference.

Coverage note — All core theoretical formulations, generic and model-specific algorithms (LDA and HDP), evaluation metrics, and empirical findings have been captured; brief discussions of future research directions (such as nonconjugate extensions, collapsed variational inference, and adaptive step sizes) were omitted as they are not primary contributions of this paper.

References

1.A. Ahmed, M. Aly, J. Gonzalez, S. Narayanamurthy, and A. Smola. Scalable inference in latent variable models. In Web Search and Data Mining, New York, NY, USA, 2012.
2.S. Amari. Differential geometry of curved exponential families-curvatures and information loss. The Annals of Statistics, 10(2):357–385, 1982.
3.S. Amari. Natural gradient works efficiently in learning. Neural computation, 10(2):251–276, 1998.
4.C. Antoniak. Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. The Annals of Statistics, 2(6):1152–1174, 1974.
5.A. Asuncion, M. Welling, P. Smyth, and Y. Teh. On smoothing and inference for topic models. In Uncertainty in Artificial Intelligence, 2009.
6.H. Attias. Inferring parameters and structure of latent variable models by variational bayes. In Uncertainty in Artificial Intelligence, 1999.
7.H. Attias. A variational Bayesian framework for graphical models. In Neural Information Processing Systems, 2000.
8.J. Bernardo and A. Smith. Bayesian Theory. John Wiley & Sons Ltd., Chichester, 1994.
9.C. Bishop. Pattern Recognition and Machine Learning. Springer New York., 2006.
10.C. Bishop, D. Spiegelhalter, and J. Winn. VIBES: A variational inference engine for Bayesian networks. In Neural Information Processing Systems. Cambridge, MA, 2003.
11.D. Blackwell and J. MacQueen. Ferguson distributions via Pólya urn schemes. The Annals of Statistics, 1(2):353–355, 1973.
12.D. Blei. Probabilistic topic models. Communications of the ACM, 55(4):77–84, 2012.
13.D. Blei and M. Jordan. Variational inference for Dirichlet process mixtures. Journal of Bayesian Analysis, 1(1):121–144, 2006.
14.D. Blei and J. Lafferty. Dynamic topic models. In International Conference on Machine Learning, pages 113–120, 2006.
15.D. Blei and J. Lafferty. A correlated topic model of Science. Annals of Applied Statistics, 1(1): 17–35, 2007.
16.D. Blei, A. Ng, and M. Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993–1022, January 2003.
17.L. Bottou. On-line learning and stochastic approximations. In On-line Learning in Neural Networks, pages 9–42. Cambridge University Press, 1998.
18.L. Bottou. Stochastic learning. In Advanced Lectures on Machine Learning, pages 146–168. Springer, 2003.
19.L. Bottou and O. Bousquet. Learning using large datasets. In Mining Massive Datasets for Security. IOS Press, 2008.
20.Olivier Cappé and Eric Moulines. On-line expectation-maximization algorithm for latent data models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 71(3):593–613, 2009.
21.M. Collins, S. Dasgupta, and R. Schapire. A generalization of principal component analysis to the exponential family. In Neural Information Processing Systems, 2002.
22.A. Dempster, N. Laird, and D. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B, 39:1–38, 1977.
23.M. Do Carmo. Riemannian Geometry. Birkhäuser, 1992.
24.A. Doucet, N. De Freitas, and N. Gordon. An introduction to sequential Monte Carlo methods. Springer, 2001.
25.E. Erosheva. Bayesian estimation of the grade of membership model. Bayesian Statistics, 7:501–510, 2003.
26.M. Escobar and M. West. Bayesian density estimation and inference using mixtures. Journal of the American Statistical Association, 90:577–588, 1995.
27.T. Ferguson. A Bayesian analysis of some nonparametric problems. The Annals of Statistics, 1: 209–230, 1973.
28.S. Fine, Y. Singer, and N. Tishby. The hierarchical hidden Markov model: Analysis and applications. Machine Learning, 32:41–62, 1998.
29.E. Fox, E. Sudderth, M. Jordan, and A. Willsky. An HDP-HMM for systems with state persistence. In International Conference on Machine Learning, 2008.
30.E. Fox, E. Sudderth, M. Jordan, and A. Willsky. Bayesian Nonparametric Inference of Switching Dynamic Linear Models. IEEE Transactions on Signal Processing, 59(4):1569–1585, 2011a.
31.E. Fox, E. Sudderth, M. Jordan, and A. Willsky. A Sticky HDP-HMM with Application to Speaker Diarization. Annals of Applied Statistics, 5(2A):1020–1056, 2011b.
32.S. Geisser. The predictive sample reuse method with applications. Journal of the American Statistical Association, 70:320–328, 1975.
33.A. Gelfand and A. Smith. Sampling based approaches to calculating marginal densities. Journal of the American Statistical Association, 85:398–409, 1990.
34.A. Gelman and J. Hill. Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge University Press, 2007.
35.S. Geman and D. Geman. Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6:721–741, 1984.
36.S. Gershman and D. Blei. A tutorial on Bayesian nonparametric models. Journal of Mathematical Psychology, 56:1–12, 2012.
37.S. Gershman, M. Hoffman, and D. Blei. Nonparametric variational inference. In International Conference on Machine Learning, 2012.
38.Z. Ghahramani and M. Beal. Variational inference for Bayesian mixtures of factor analysers. In Neural Information Processing Systems, 2000.
39.Z. Ghahramani and M. Beal. Propagation algorithms for variational Bayesian learning. In Neural Information Processing Systems, pages 507–513, 2001.
40.Z. Ghahramani and M. Jordan. Factorial hidden Markov models. Machine Learning, 31(1), 1997.
41.M. Girolami and S. Rogers. Variational Bayesian multinomial probit regression with Gaussian process priors. Neural Computation, 18(8), 2006.
42.P. Gopalan, D. Mimno, S. Gerrish, M. Freedman, and D. Blei. Scalable inference of overlapping communities. In Neural Information Processing Systems, 2012.
43.T. Griffiths and M. Steyvers. Finding scientific topics. Proceedings of the National Academy of Science, 101:5228–5235, 2004.
44.W. Hastings. Monte Carlo sampling methods using Markov chains and their applications. Biometrika, 57:97–109, 1970.
45.G. Hinton and D. Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Computational Learning Theory, pages 5–13. ACM, 1993.
46.N. Hjort, C. Holmes, P. Muller, and S. Walker, editors. Bayesian Nonparametrics. Cambridge University Press, 2010.
47.M. Hoffman, D. Blei, and F. Bach. On-line learning for latent Dirichlet allocation. In Neural Information Processing Systems, 2010a.
48.M. Hoffman, D. Blei, and P. Cook. Bayesian nonparametric matrix factorization for recorded music. In International Conference on Machine Learning, 2010b.
49.A. Honkela, M. Tornio, T. Raiko, and J. Karhunen. Natural conjugate gradient in variational inference. In Neural Information Processing Systems, 2008.
50.T. Jaakkola. Variational Methods for Inference and Estimation in Graphical Models. PhD thesis, Massachusetts Institute of Technology, 1997.
51.M. Jordan, editor. Learning in Graphical Models. MIT Press, Cambridge, MA, 1999.
52.M. Jordan, Z. Ghahramani, T. Jaakkola, and L. Saul. Introduction to variational methods for graphical models. Machine Learning, 37:183–233, 1999.
53.R. Kalman. A new approach to linear filtering and prediction problems a new approach to linear filtering and prediction problems,". Transaction of the AMSE: Journal of Basic Engineering, 82: 35–45, 1960.
54.D. Knowles and T. Minka. Non-conjugate variational message passing for multinomial and binary regression. In Neural Information Processing Systems, 2011.
55.D. Koller and N. Friedman. Probabilistic Graphical Models: Principles and Techniques. The MIT Press, 2009.
56.S. Kullback and R.A. Leibler. On information and sufficiency. The Annals of Mathematical Statistics, 22(1):79–86, 1951.
57.P. Liang, M. Jordan, and D. Klein. Learning semantic correspondences with less supervision. In Association of Computational Linguisitics, 2009.
58.J. Mairal, J. Bach, J. Ponce, and G. Sapiro. Online learning for matrix factorization and sparse coding. Journal of Machine Learning Research, 11:19–60, 2010.
59.J. Maritz and T. Lwin. Empirical Bayes methods. Monographs on Statistics and Applied Probability. Chapman & Hall, London, 1989.
60.P. McCullagh and J. A. Nelder. Generalized Linear Models. London: Chapman and Hall, 1989.
61.N. Metropolis, A. Rosenbluth, M. Rosenbluth, M. Teller, and E. Teller. Equations of state calculations by fast computing machines. Journal of Chemical Physics, 21:1087–1092, 1953.
62.D. Mimno, M. Hoffman, and D. Blei. Sparse stochastic inference for latent Dirichlet allocation. In International Conference on Machine Learning, 2012.
63.T. Minka and J. Lafferty. Expectation-propagation for the generative aspect model. In Uncertainty in Artificial Intelligence (UAI), 2002.
64.K. Murphy. Machine Learning: A Probabilistic Approach. MIT Press, 2012.
65.R. Neal. Markov chain sampling methods for Dirichlet process mixture models. Journal of Computational and Graphical Statistics, 9(2):249–265, 2000.
66.R. Neal and G. Hinton. A view of the EM algorithm that justifies incremental, sparse, and other variants. In Learning in graphical models, pages 355–368. MIT Press, 1999.
67.D. Newman, A. Asuncion, P. Smyth, and M. Welling. Distributed algorithms for topic models. Journal of Machine Learning Research, 10:1801–1828, 2009.
68.J. Paisley and L. Carin. Nonparametric factor analysis with beta process priors. In International Conference on Machine Learning, 2009.
69.J. Paisley, D. Blei, and M. Jordan. Variational Bayesian inference with stochastic search. In International Conference on Machine Learning, 2012a.
70.J. Paisley, C. Wang, and D. Blei. The discrete infinite logistic normal distribution. Bayesian Analysis, 7(2):235–272, 2012b.
71.J. Paisley, C. Wang, D. Blei, and M. Jordan. Nested hierarchical Dirichlet processes. arXiv preprint arXiv:1210.6738, 2012c.
72.G. Parisi. Statistical Field Theory. Perseus Books, 1988.
73.J. Pearl. Probabilistic reasoning in intelligent systems: Networks of plausible inference. Morgan Kaufmann, 1988. ISBN 1558604790.
74.C. Peterson and J. Anderson. A mean field theory learning algorithm for neural networks. Complex Systems, 1(5):995–1019, 1987.
75.J. Pitman. Combinatorial Stochastic Processes. Lecture Notes for St. Flour Summer School. Springer-Verlag, New York, NY, 2002.
76.J. Platt, E. Kıcıman, and D. Maltz. Fast variational inference for large-scale internet diagnosis. Neural Information Processing Systems, 2008.
77.L. Rabiner. A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77:257–286, 1989.
78.R. Ranganath, C. Wang, D. Blei, and E. Xing. An adaptive learning rate for stochastic variational inference. In International Conference on Machine Learning, 2013.
79.H. Robbins and S. Monro. A stochastic approximation method. The Annals of Mathematical Statistics, 22(3):400–407, 1951.
80.C. Robert and G. Casella. Monte Carlo Statistical Methods. Springer Texts in Statistics. Springer-Verlag, New York, NY, 2004.
81.R. Salakhutdinov and A. Mnih. Bayesian probabilistic matrix factorization using Markov chain Monte Carlo. In International Conference on Machine learning, pages 880–887, 2008.
82.M. Sato. Online model selection based on the variational Bayes. Neural Computation, 13(7):1649–1681, 2001.
83.L. Saul and M. Jordan. Exploiting tractable substructures in intractable networks. Neural Information Processing Systems, 1996.
84.L. Saul, T. Jaakkola, and M. Jordan. Mean field theory for sigmoid belief networks. Journal of Artificial Intelligence Research, 4:61–76, 1996.
85.J. Sethuraman. A constructive definition of Dirichlet priors. Statistica Sinica, 4:639–650, 1994.
86.A. Smola and S. Narayanamurthy. An architecture for parallel topic models. In Very Large Databases, 2010.
87.J. Spall. Introduction to stochastic search and optimization: Estimation, simulation, and control. John Wiley and Sons, 2003.
88.C. Spearman. "General intelligence," objectively determined and measured. The American Journal of Psychology, pages 201–292, 1904.
89.Y. Teh, M. Jordan, M. Beal, and D. Blei. Hierarchical Dirichlet processes. Journal of the American Statistical Association, 101(476):1566–1581, 2006a.
90.Y. Teh, D. Newman, and M. Welling. A collapsed variational Bayesian inference algorithm for latent Dirichlet allocation. In Neural Information Processing Systems, 2006b.
91.Y. Teh, K. Kurihara, and M. Welling. Collapsed variational inference for HDP. In Neural Information Processing Systems, 2007.
92.M. Tipping and C. Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 61(3):611–622, 1999.
93.M. Wahabzada and K. Kersting. Larger residuals, less work: Active document scheduling for latent dirichlet allocation. In European Conference on Machine Learning, 2011.
94.M. Wainwright and M. Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1(1–2):1–305, 2008.
95.H. Wallach, D. Mimno, and A. McCallum. Rethinking LDA: Why priors matter. In Neural Information Processing Systems. 2009.
96.C. Wang. Variational Bayesian approach to canonical correlation analysis. IEEE Transactions on Neural Networks, 2006.
97.C. Wang and D. Blei. Variational inference in nonconjugate models. Journal of Machine Learning Research, 2013.
98.C. Wang, D. Blei, and D. Heckerman. Continuous time dynamic topic models. In Uncertainty in Artificial Intelligence, 2008.
99.C. Wang, J. Paisley, and D. Blei. Online variational inference for the hierarchical Dirichlet process. In Artificial Intelligence and Statistics, 2011.
100.S. Waterhouse, D. MacKay, and T. Robinson. Bayesian methods for mixtures of experts. Neural Information Processing Systems, pages 351–357, 1996.
101.M. Welling and Y. Teh. Bayesian learning via stochastic gradient Langevin dynamics. In International Conference on Machine Learning, 2011.
102.W. Wiegerinck. Variational approximations between mean field theory and the junction tree algorithm. In Uncertainty in Artificial Intelligence, 2000.
103.E. Xing, M. Jordan, and S. Russell. A generalized mean field algorithm for variational inference in exponential families. In Uncertainty in Artificial Intelligence, 2003.

Citation

MLA

Hoffman, M., et al. “Stochastic Variational Inference”. arXiv, 2012, https://doi.org/10.48550/arxiv.1206.7051.

APA

Hoffman, M., Blei, D. M., Wang, C., & Paisley, J. (2012). Stochastic Variational Inference. arXiv. https://doi.org/10.48550/arxiv.1206.7051

Chicago

Hoffman, M., D. M. Blei, C. Wang, and J. Paisley. 2012. “Stochastic Variational Inference”. Preprint, ArXiv. https://doi.org/10.48550/arxiv.1206.7051.

Harvard

Hoffman, M. et al. (2012) “Stochastic Variational Inference”. arXiv. Available at: https://doi.org/10.48550/arxiv.1206.7051.

Vancouver

1. Hoffman M, Blei DM, Wang C, Paisley J (2012) Stochastic Variational Inference. https://doi.org/10.48550/arxiv.1206.7051

BibTeX

@misc{https://doi.org/10.48550/arxiv.1206.7051,
  doi = {10.48550/ARXIV.1206.7051},
  url = {https://arxiv.org/abs/1206.7051},
  author = {Hoffman, Matt and Blei, David M. and Wang, Chong and Paisley, John},
  keywords = {Machine Learning (stat.ML), Artificial Intelligence (cs.AI), Computation (stat.CO), Methodology (stat.ME), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {Stochastic Variational Inference},
  publisher = {arXiv},
  year = {2012},
  copyright = {arXiv.org perpetual, non-exclusive license}
}

Metadata:DOI registry

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Stochastic variational inference

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Conditionally Conjugate Exponential Family Model Class for Variational Inference

Knowl 2 — Natural Gradient of the Evidence Lower Bound in Conjugate Exponential Families

Knowl 3 — Stochastic Variational Inference Algorithm

Knowl 4 — Stochastic Variational Inference for Latent Dirichlet Allocation

Knowl 5 — Truncated Stick-Breaking Formulation of the Hierarchical Dirichlet Process Topic Model

Knowl 6 — Stochastic Variational Inference for the Hierarchical Dirichlet Process Topic Model

Knowl 7 — Per-Word Predictive Log-Likelihood Evaluation Metric for Topic Models

Knowl 8 — Experimental Setup for Large-Scale Text Corpus Topic Modeling

Knowl 9 — Predictive Log-Likelihood Comparison of SVI LDA and HDP Across Large Document Corpora

Knowl 10 — Hyperparameter Sensitivity and Convergence Dynamics of SVI

Knowl 11 — Stochastic Variational Inference for Conditionally Conjugate Global Subvector Parameters

References

Citation

BibTeX

Access the Paper