Supervised Topic Models

article2007NeurIPS1,828 citations

Proposes supervised latent Dirichlet allocation (sLDA) to jointly model text content and document labels, enabling the discovery of topic representations directly optimized for predictive accuracy over traditional two-stage regression approaches.

Listen

Organizations increasingly need to extract actionable intelligence and predictions from large collections of electronic text, such as public policy records, consumer reviews, and internal communications. Traditional topic modeling techniques uncover the broad themes in a corpus without supervision, but these unsupervised themes often capture irrelevant general patterns rather than the specific features needed to forecast an external outcome, such as a sentiment rating, financial metric, or legislative vote.

The article introduces and evaluates supervised latent Dirichlet allocation, a statistical modeling framework that jointly discovers underlying text themes while simultaneously predicting an external response variable for each document.

To evaluate this framework, the authors tested the model across two distinct real-world datasets: a collection of 5,006 movie reviews used to predict numeric ratings, and legislative records from the 109th and 110th U.S. Senates (288 and 213 amendments, respectively) used to predict ideological voting tone based on amendment text. Using five-fold cross-validation, the model was benchmarked against two standard industry baselines: standard regularized linear regression (the lasso) and a two-stage pipeline where unsupervised topics are extracted first and then fed into a separate regression model.

The empirical findings show that jointly modeling text and outcomes delivers superior predictive performance. First, the supervised topic approach consistently outperformed two-stage unsupervised topic regression across both datasets and across varying numbers of topics. Second, on the legislative amendment data, the supervised topic model substantially outperformed regularized regression, achieving an 80% relative improvement in predictive explanatory power for the 109th Senate (a predictive R-squared of 0.27 versus 0.15) and a 43% relative improvement for the 110th Senate (0.23 versus 0.16). Third, on the movie review data, the model slightly outperformed the regularized baseline with an explanatory power of 0.432 compared to 0.426. Finally, unlike standard regression methods that function as black-box predictors, the proposed model successfully identified intuitive, interpretable thematic word clusters directly linked to positive or negative outcome values.

These results demonstrate that when the primary objective is outcome prediction, statistical dimensionality reduction must be guided directly by the target metric. Relying on disconnected two-stage text analysis pipelines degrades predictive accuracy and increases analytical risk. Furthermore, because this framework accommodates various outcome data types—such as continuous values, counts, and binary classifications—it provides decision-makers with a flexible tool that combines quantitative forecasting with qualitative, interpretable insights.

For organizations managing complex text repositories, the article supports adopting supervised topic models over disjointed unsupervised workflows when labeled outcome data exists. Practitioners should explore extensions of the framework, including semi-supervised implementations where only a subset of documents have associated outcome labels, or integrating additional metadata covariates alongside document text. Decision-makers should note that model performance is constrained by text quality and vocabulary preprocessing choices, and legislative outcomes remain inherently difficult to predict solely from text because external political factors also drive voting behavior. Overall, confidence in the model's predictive advantage over conventional two-stage topic regression remains high.

Princeton University University of California Berkeley maximum likelihood text document topic model supervised topic models latent Dirichlet allocation mixed-membership models

arXiv: 1003.0783 blei-lab/class-slda

Paper: Latent Dirichlet Allocation, David M. Blei et al. (2003). It introduces the fundamental unsupervised Latent Dirichlet Allocation model and variational inference framework upon which supervised LDA is directly built.
Paper: Probabilistic Latent Semantic Analysis, Thomas Hofmann (1999). It develops probabilistic latent semantic analysis, providing the foundational generative aspect model that prefigured Dirichlet-based topic modeling.
Paper: Indexing By Latent Semantic Analysis, Scott Deerwester et al. (1990). It provides the foundational conceptual basis for representing text corpora in low-dimensional latent semantic spaces.
Paper: Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales, Bo Pang et al. (2005). It introduces the problem of predicting continuous or multi-point ratings from review text, which serves as a core benchmark task in supervised LDA.
Paper: Thumbs up? Sentiment Classification using Machine Learning Techniques, Bo Pang et al. (2002). It establishes standard machine learning benchmarks for document-level sentiment and review classification that motivate supervised topic modeling applications.

Paper: Stochastic variational inference, Matt Hoffman et al. (2012). It scales variational inference for latent Dirichlet allocation and related topic models to massive datasets via stochastic optimization.
Paper: Learning Word Vectors for Sentiment Analysis, Andrew L. Maas et al. (2011). It extends joint probabilistic topic and rating modeling to continuous word vectors for sentiment and polarity prediction.
Paper: Semi-supervised Learning with Deep Generative Models, Diederik P. Kingma et al. (2014). It advances beyond linear supervised topic models by unifying generative latent variable modeling and classification via deep neural variational inference.

Abstract

We introduce supervised latent Dirichlet allocation (sLDA), a statistical model of labelled documents. The model accommodates a variety of response types. We derive an approximate maximum-likelihood procedure for parameter estimation, which relies on variational methods to handle intractable posterior expectations. Prediction problems motivate this research: we use the fitted model to predict response values for new documents. We test sLDA on two real-world problems: movie ratings predicted from reviews, and the political tone of amendments in the U.S. Senate based on the amendment text. We illustrate the benefits of sLDA versus modern regularized regression, as well as versus an unsupervised LDA analysis followed by a separate regression.

Topics

maximum likelihood text document topic model variational methods latent factor latent variable

Keywords

supervised topic models latent Dirichlet allocation mixed-membership models approximate posterior inference variational inference algorithm topic proportions Dirichlet distribution generative process

1 Introduction
2 Supervised latent Dirichlet allocation
3 Computation with supervised LDA
3.1 Posterior inference
3.2 Parameter estimation
3.3 Prediction
3.4 Examples
4 Empirical study
5 Discussion
References

Knowls

Knowl 1 — Supervised Latent Dirichlet Allocation Generative Model
model/method
Supervised latent Dirichlet allocation (sLDA) is a generative model that jointly models a document consisting of $N$ words $w_{1:N} = (w_1, \dots, w_N)$ from a vocabulary of size $V$ and an associated continuous or discrete response variable $y \in \mathbb{R}$ .

The model parameters comprise:
- $K$ topic multinomial distributions $\beta_{1:K}$ , where each $\beta_k = (\beta_{k,1}, \dots, \beta_{k,V}) \in \Delta^{V-1}$ represents term probabilities for topic $k$ .
- A Dirichlet prior parameter vector $\alpha \in \mathbb{R}_+^K$ .
- Generalized linear model (GLM) regression coefficients $\eta \in \mathbb{R}^K$ .
- A GLM dispersion parameter $\delta \in \mathbb{R}_+$ .
The generative process for each document-response pair is as follows:
1. Draw topic proportions $\theta \sim \text{Dirichlet}(\alpha)$ .
2. For each word index $n \in \{1, \dots, N\}$ : a. Draw topic assignment indicator vector $z_n \sim \text{Multinomial}(\theta, 1)$ , where $z_n \in \{0, 1\}^K$ and $\sum_{k=1}^K z_{n,k} = 1$ . b. Draw word $w_n \sim \text{Multinomial}(\beta_{z_n}, 1)$ .
3. Define the document's empirical topic frequency vector as: $\bar{z} = \frac{1}{N} \sum_{n=1}^N z_n$
4. Draw the response variable $y$ from an exponential dispersion family conditioned on $\bar{z}$ : $p(y \mid z_{1:N}, \eta, \delta) = h(y, \delta) \exp\left( \frac{(\eta^\top \bar{z})y - A(\eta^\top \bar{z})}{\delta} \right)$ where $h(y, \delta)$ is the base measure, $\eta^\top \bar{z}$ is the natural parameter acting as the linear predictor, and $A(\eta^\top \bar{z})$ is the log-normalizer (cumulant generating function).
Knowl 2 — Variational Inference for Supervised Latent Dirichlet Allocation
model/method
The document-level posterior distribution of the latent topic proportions $\theta$ and topic assignment vectors $z_{1:N} = (z_1, \dots, z_N)$ given words $w_{1:N}$ and response $y$ is approximated by a fully factorized mean-field variational family: $q(\theta, z_{1:N} \mid \gamma, \phi_{1:N}) = q(\theta \mid \gamma) \prod_{n=1}^N q(z_n \mid \phi_n)$ where $\gamma \in \mathbb{R}_+^K$ is a variational Dirichlet parameter vector, and each $\phi_n \in \Delta^{K-1}$ parameterizes a categorical distribution such that $\mathbb{E}_q[z_n] = \phi_n$ .

The variational distribution is optimized to maximize the Evidence Lower Bound (ELBO): $\mathcal{L}(w_{1:N}, y \mid \alpha, \beta_{1:K}, \eta, \delta) = \mathbb{E}_q[\log p(\theta \mid \alpha)] + \sum_{n=1}^N \mathbb{E}_q[\log p(z_n \mid \theta)] + \sum_{n=1}^N \mathbb{E}_q[\log p(w_n \mid z_n, \beta_{1:K})] + \mathbb{E}_q[\log p(y \mid z_{1:N}, \eta, \delta)] + H(q)$ where $H(q) = -\mathbb{E}_q[\log q(\theta, z_{1:N})]$ is the variational entropy.

Block coordinate-ascent updates optimize this bound:
- The Dirichlet update is identical to unsupervised LDA: $\gamma^{\text{new}} \leftarrow \alpha + \sum_{n=1}^N \phi_n$
- The gradient of the ELBO with respect to the variational multinomial $\phi_n$ accounts for both word likelihood and response likelihood: $\frac{\partial \mathcal{L}}{\partial \phi_n} = \mathbb{E}_q[\log \theta] + \log \beta_{\cdot, w_n} - \log \phi_n + \mathbf{1} + \left( \frac{y}{N\delta} \right) \eta - \frac{1}{\delta} \frac{\partial}{\partial \phi_n} \mathbb{E}_q[A(\eta^\top \bar{Z})]$ where $\bar{Z} = \frac{1}{N} \sum_{n=1}^N z_n$ , $\beta_{\cdot, w_n} = (\beta_{1, w_n}, \dots, \beta_{K, w_n})^\top$ , $\mathbf{1}$ is the vector of ones, and $\mathbb{E}_q[\log \theta_i] = \Psi(\gamma_i) - \Psi\left( \sum_{j=1}^K \gamma_j \right)$ with $\Psi(\cdot)$ denoting the digamma function.
Knowl 3 — Variational Expectation-Maximization for sLDA Parameter Estimation
algorithm
Variational Expectation-Maximization (variational EM) estimates the corpus-wide parameters $\alpha$ , $\beta_{1:K}$ , $\eta$ , and $\delta$ across a training corpus $\mathcal{D} = \{(w_{d,1:N_d}, y_d)\}_{d=1}^D$ by optimizing the corpus-level lower bound $\mathcal{L} = \sum_{d=1}^D \mathcal{L}_d$ .
Input: Corpus of document-response pairs $(w_{d,1:N_d}, y_d)_{d=1}^D$ , number of topics $K$ , convergence threshold $\epsilon$
Output: Fitted parameters $\beta_{1:K}$ , $\eta$ , $\delta$ , $\alpha$
Initialize $\beta_{1:K}$ randomly, $\eta$ uniformly on $[-1, 1]$ , $\delta$ to empirical variance of $y$ , and $\alpha = 1/K$
repeat
    // E-step: Document-level variational inference
    for $d = 1$ to $D$ do
        repeat
            Update $\gamma_d \leftarrow \alpha + \sum_{n=1}^{N_d} \phi_{d,n}$
            for $n = 1$ to $N_d$ do
                Update $\phi_{d,n}$ by coordinate ascent or gradient step on $\frac{\partial \mathcal{L}_d}{\partial \phi_{d,n}}$
                Normalize $\phi_{d,n}$ to sum to 1
            end for
        until per-document ELBO $\mathcal{L}_d$ converges to within $\epsilon$
    end for
    // M-step: Corpus-level parameter updates
    for $k = 1$ to $K$ and word $w = 1$ to $V$ do
         $\hat{\beta}_{k,w} \propto \sum_{d=1}^D \sum_{n=1}^{N_d} 1(w_{d,n} = w) \phi_{d,n,k}$
    end for
    Update $\eta$ and $\delta$ using the corpus ELBO gradients:
     $\frac{\partial \mathcal{L}}{\partial \eta} = \frac{1}{\delta}\left( \sum_{d=1}^D \bar{\phi}_d y_d - \sum_{d=1}^D \mathbb{E}_{q_d}[\mu(\eta^\top \bar{Z}_d)\bar{Z}_d] \right)$ where $\bar{\phi}_d = \frac{1}{N_d}\sum_{n=1}^{N_d} \phi_{d,n}$
    Update $\alpha$ using Dirichlet maximum-likelihood on expected topic proportions
until corpus-level ELBO converges to within $\epsilon$
return $\beta_{1:K}, \eta, \delta, \alpha$
Knowl 4 — Response Prediction for Unlabeled Documents in sLDA
algorithm
Given a fitted sLDA model with parameters $\{\alpha, \beta_{1:K}, \eta, \delta\}$ and a newly observed unlabeled document $w_{1:N} = (w_1, \dots, w_N)$ , predicting the response $y$ corresponds to approximating the posterior expectation $\mathbb{E}[Y \mid w_{1:N}, \alpha, \beta_{1:K}, \eta, \delta] = \mathbb{E}[\mu(\eta^\top \bar{Z}) \mid w_{1:N}, \alpha, \beta_{1:K}]$ , where $\mu(\cdot) = \mathbb{E}_{\text{GLM}}[Y \mid \cdot]$ .
Input: New document words $w_{1:N}$ , fitted model $\{\alpha, \beta_{1:K}, \eta, \delta\}$ , convergence threshold $\epsilon$
Output: Predicted scalar response $\hat{y}$
Initialize $\gamma \leftarrow \alpha + N/K$ and $\phi_n \leftarrow 1/K$ for all $n = 1, \dots, N$
repeat
     $\gamma \leftarrow \alpha + \sum_{n=1}^N \phi_n$
    for $n = 1$ to $N$ do
        for $k = 1$ to $K$ do
             $\phi_{n,k} \propto \beta_{k, w_n} \exp\left( \Psi(\gamma_k) - \Psi\left(\sum_{j=1}^K \gamma_j\right) \right)$
        end for
        Normalize $\phi_n$ such that $\sum_{k=1}^K \phi_{n,k} = 1$
    end for
until variational parameters $\gamma, \phi_{1:N}$ converge to within $\epsilon$
Compute $\bar{\phi} = \frac{1}{N} \sum_{n=1}^N \phi_n$
Compute prediction $\hat{y} \approx \mathbb{E}_q[\mu(\eta^\top \bar{Z})]$
// For Gaussian response: $\hat{y} = \eta^\top \bar{\phi}$
// For Poisson response: $\hat{y} = \prod_{n=1}^N \left( K - 1 + \sum_{i=1}^K \phi_{n,i} \exp(\eta_i / N) \right)$
return $\hat{y}$
Knowl 5 — Exact Variational Updates and Estimation for Gaussian Response sLDA
model/method
When the response distribution is Gaussian with mean $\mu = \eta^\top \bar{z}$ and variance $\sigma^2 = \delta$ , the base measure is $h(y, \delta) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp(-y^2 / (2\sigma^2))$ and the log-normalizer is $A(\eta^\top \bar{z}) = \frac{1}{2}(\eta^\top \bar{z})^2$ .
1. Second Moment of Topic Frequencies: $\mathbb{E}_q[\bar{Z}\bar{Z}^\top] = \frac{1}{N^2}\left( \sum_{n=1}^N \sum_{m \neq n} \phi_n \phi_m^\top + \sum_{n=1}^N \text{diag}(\phi_n) \right)$
2. Coordinate Ascent Update for $\phi_j$ : $\phi_j^{\text{new}} \propto \exp\left( \mathbb{E}_q[\log \theta] + \log \beta_{\cdot, w_j} + \left(\frac{y}{N\sigma^2}\right)\eta - \frac{1}{2N^2\sigma^2}\left[ 2(\eta^\top \phi_{-j})\eta + (\eta \circ \eta) \right] \right)$ where $\phi_{-j} = \sum_{n \neq j} \phi_n$ , $\eta \circ \eta$ is the elementwise square, and $\beta_{\cdot, w_j} = (\beta_{1,w_j}, \dots, \beta_{K,w_j})^\top$ .
3. Closed-Form M-Step for GLM Parameters: Let $y = (y_1, \dots, y_D)^\top \in \mathbb{R}^D$ and $X \in \mathbb{R}^{D \times K}$ be the design matrix whose $d$ -th row is $\bar{Z}_d$ . The updates are: $\hat{\eta}^{\text{new}} = \left( \mathbb{E}_q[X^\top X] \right)^{-1} \mathbb{E}_q[X]^\top y = \left( \sum_{d=1}^D \mathbb{E}_{q_d}[\bar{Z}_d \bar{Z}_d^\top] \right)^{-1} \left( \sum_{d=1}^D \bar{\phi}_d y_d \right)$ $\hat{\sigma}^{2,\text{new}} = \frac{1}{D} \left( y^\top y - y^\top \mathbb{E}_q[X] \left( \mathbb{E}_q[X^\top X] \right)^{-1} \mathbb{E}_q[X]^\top y \right)$ This update differs from ordinary least squares of $y$ on $\mathbb{E}_q[X]$ because $\mathbb{E}_q[X^\top X] \neq \mathbb{E}_q[X]^\top \mathbb{E}_q[X]$ .
Knowl 6 — Variational Inference and Estimation for Overdispersed Poisson Response sLDA
model/method
When the response variable $y \in \mathbb{N}_0$ is count data modeled by an overdispersed Poisson distribution with mean parameter $\lambda = \exp(\eta^\top \bar{z})$ and dispersion $\delta$ , the density is: $p(y \mid \lambda, \delta) = \frac{1}{y!} \exp\left( \frac{y\log\lambda - \lambda}{\delta} \right)$ Here, $A(\eta^\top \bar{z}) = \mu(\eta^\top \bar{z}) = \exp(\eta^\top \bar{z})$ .
1. Expected Log-Normalizer and Derivative: $\mathbb{E}_q[A(\eta^\top \bar{Z})] = \prod_{n=1}^N \mathbb{E}_q\left[ \exp\left( \frac{1}{N}\eta^\top Z_n \right) \right] = \prod_{n=1}^N \left( K - 1 + \sum_{i=1}^K \phi_{n,i} \exp(\eta_i / N) \right) =: C$ $\frac{\partial}{\partial \phi_n} \mathbb{E}_q[A(\eta^\top \bar{Z})] = C_{-n} \exp(\eta / N)$ where $C_{-n} = \prod_{m \neq n} \left( K - 1 + \sum_{i=1}^K \phi_{m,i} \exp(\eta_i / N) \right)$ .
2. Expected Mean-Feature Product: $\mathbb{E}_q[\mu(\eta^\top \bar{Z})\bar{Z}] = \frac{\exp(\eta/N)}{N} \sum_{n=1}^N C_{-n}\phi_n$
3. M-Step Parameter Optimization:
- The gradient of the corpus ELBO w.r.t. $\eta$ is optimized numerically via convex optimization: $\frac{\partial \mathcal{L}}{\partial \eta} = \frac{1}{\delta}\left( \sum_{d=1}^D \bar{\phi}_d y_d - \sum_{d=1}^D \frac{\exp(\eta/N_d)}{N_d} \sum_{n=1}^{N_d} C_{d,-n}\phi_{d,n} \right)$
- The dispersion parameter update has an exact closed form: $\hat{\delta}^{\text{new}} = \frac{\sum_{d=1}^D \hat{\eta}_{\text{new}}^\top \bar{\phi}_d y_d}{\sum_{d=1}^D C_d(\hat{\eta}_{\text{new}})}$
Knowl 7 — Empirical Predictive Performance of sLDA on Movie Ratings and Senate Ideology
empirical result
Supervised LDA was evaluated on two real-world continuous-response prediction tasks using 5-fold cross-validation. Performance was measured by Pearson correlation and predictive $R^2$ ( $pR^2 = 1 - \frac{\sum (y - \hat{y})^2}{\sum (y - \bar{y})^2}$ ):
1. Movie Review Rating Prediction:
- Corpus: 5,006 movie reviews paired with star ratings (log-transformed), with a 2,180-word vocabulary.
- Results: sLDA achieved a peak $pR^2$ of $0.432$ , improving upon regularized linear regression via lasso ( $pR^2 = 0.426$ ) and substantially outperforming unsupervised LDA followed by linear regression (which plateaued at $pR^2 \approx 0.24$ ).
1. U.S. Senate Amendment Political Tone Prediction:
- Task: Predict amendment ideal-point discrimination parameters $\beta_i$ (derived from roll call voting records) based on amendment text.
- 109th Senate (288 amendments, 2,084 vocabulary words): sLDA achieved a best $pR^2$ of $0.27$ , an 80% relative improvement over lasso ( $pR^2 = 0.15$ ) and outperforming unsupervised LDA + regression ( $pR^2 \approx 0.22$ ).
- 110th Senate (213 amendments, 1,653 vocabulary words): sLDA achieved a best $pR^2$ of $0.23$ , a 43% relative improvement over lasso ( $pR^2 = 0.16$ ) and outperforming unsupervised LDA + regression ( $pR^2 \approx 0.18$ ).
Across all settings, jointly estimating topics conditioned on the response variable produced topic representations with higher predictive accuracy than two-stage unsupervised LDA feature extraction.

Coverage note — None was omitted; the extracted knowls fully cover the sLDA model formulation, variational inference, variational EM parameter estimation, out-of-fold prediction, exact Gaussian and Poisson solutions, and empirical cross-validation results.

References

1.Bickel, P. and Doksum, K. (2007). Mathematical Statistics: Basic Ideas and Selected Topics, volume 1. Pearson Prentice Hall, Upper Saddle River, NJ, 2nd edition.
2.Bishop, C. (2006). Pattern Recognition and Machine Learning. Springer New York.
3.Blei, D. and Jordan, M. (2003). Modeling annotated data. In Proceedings of the 26th annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 127–134. ACM Press.
4.Blei, D. and Lafferty, J. (2009). Topic models. In Srivastava, A. and Sahami, M., editors, Text Mining: Theory and Applications. Taylor and Francis.
5.Blei, D., Ng, A., and Jordan, M. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993–1022.
6.Braun, M. and McAuliffe, J. (2010). Variational inference for large-scale models of discrete choice. Journal of the American Statistical Association.
7.Brown, L. (1986). Fundamentals of Statistical Exponential Families. Institute of Mathematical Statistics, Hayward, CA.
8.Chang, J. and Blei, D. (2010). Hierarchical relational models for document networks. Annals of Applied Statistics.
9.Chang, J., Boyd-Graber, J., Wang, C., Gerrish, S., and Blei, D. (2009). Reading tea leaves: How humans interpret topic models. In Neural Information Processing Systems.
10.Clinton, J., Jackman, S., and Rivers, D. (2004). The statistical analysis of roll call data. American Political Science Review, 98(2):355–370.
11.Dempster, A., Laird, N., and Rubin, D. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B, 39:1–38.
12.Erosheva, E. (2002). Grade of membership and latent structure models with application to disability survey data. PhD thesis, Carnegie Mellon University, Department of Statistics.
13.Erosheva, E., Fienberg, S., and Lafferty, J. (2004). Mixed-membership models of scientific publications. Proceedings of the National Academy of Science, 97(22):11885–11892.
14.Fei-Fei, L. and Perona, P. (2005). A Bayesian hierarchical model for learning natural scene categories. IEEE Computer Vision and Pattern Recognition, pages 524–531.
15.Flaherty, P., Giaever, G., Kumm, J., Jordan, M., and Arkin, A. (2005). A latent variable model for chemogenomic profiling. Bioinformatics, 21(15):3286–3293.
16.Fukumizu, K., Bach, F., and Jordan, M. (2004). Dimensionality reduction for supervised learning with reproducing kernel hilbert spaces. Journal of Machine Learning Research, 5:73–99.
17.Griffiths, T., Steyvers, M., Blei, D., and Tenenbaum, J. (2005). Integrating topics and syntax. In Saul, L. K., Weiss, Y., and Bottou, L., editors, Advances in Neural Information Processing Systems 17, pages 537–544, Cambridge, MA. MIT Press.
18.Hastie, T., Tibshirani, R., and Friedman, J. (2001). The Elements of Statistical Learning. Springer.
19.Jordan, M., Ghahramani, Z., Jaakkola, T., and Saul, L. (1999). Introduction to variational methods for graphical models. Machine Learning, 37:183–233.
20.Marlin, B. (2003). Modeling user rating profiles for collaborative filtering. In Neural Information Processing Systems.
21.McCallum, A., Pal, C., Druck, G., and Wang, X. (2006). Multi-conditional learning: Generative/discriminative training for clustering and classification. In AAAI.
22.McCullagh, P. and Nelder, J. A. (1989). Generalized Linear Models. London: Chapman and Hall.
23.Mimno, D. and McCallum, A. (2007). Organizing the OCA: Learning faceted subjects from a library of digital books. In Joint Conference on Digital Libraries.
24.Pang, B. and Lee, L. (2005). Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the Association of Computational Linguistics.
25.Pritchard, J., Stephens, M., and Donnelly, P. (2000). Inference of population structure using multilocus genotype data. Genetics, 155:945–959.
26.Quelhas, P., Monay, F., Odobez, J., Gatica-Perez, D., Tuyelaars, T., and Van Gool, L. (2005). Modeling scenes with local descriptors and latent aspects. In ICCV.
27.Ramage, D., Hall, D., Nallapati, R., and Manning, C. (2009). Labeled LDA: A supervised topic model for credit attribution in muli-labeled corpora. In Empirical Methods in Natural Language Processing.
28.Ronning, G. (1989). Maximum likelihood estimation of Dirichlet distributions. Journal of Statistcal Computation and Simulation, 34(4):215–221.
29.Rosen-Zvi, M., Griffiths, T., Steyvers, M., and Smith, P. (2004). The author-topic model for authors and documents. In Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence, pages 487–494. AUAI Press.
30.Steyvers, M. and Griffiths, T. (2006). Probabilistic topic models. In Landauer, T., McNamara, D., Dennis, S., and Kintsch, W., editors, Latent Semantic Analysis: A Road to Meaning. Laurence Erlbaum.
31.Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B (Methodological), 58(1):267–288.
32.Wainwright, M. and Jordan, M. (2008). Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1(1–2):1–305.
33.Wallach, H., Mimno, D., and McCallum, A. (2009). Rethinking lda: Why priors matter. In Bengio, Y., Schuurmans, D., Lafferty, J., Williams, C. K. I., and Culotta, A., editors, Advances in Neural Information Processing Systems 22, pages 1973–1981.
34.Wang, C., Blei, D., and Li, F. (2009). Simultaneous image classification and annotation. In Computer Vision and Pattern Recognition.
35.Wei, X. and Croft, B. (2006). LDA-based document models for ad-hoc retrieval. In SIGIR.

Citation

MLA

Blei, D. M., and J. D. McAuliffe. “Supervised Topic Models”. arXiv, 2010, http://arxiv.org/abs/1003.0783v1.

APA

Blei, D. M., & McAuliffe, J. D. (2010). Supervised Topic Models. arXiv. http://arxiv.org/abs/1003.0783v1

Chicago

Blei, D. M., and J. D. McAuliffe. 2010. “Supervised Topic Models”. arXiv. http://arxiv.org/abs/1003.0783v1.

Harvard

Blei, D.M. and McAuliffe, J.D. (2010) “Supervised Topic Models”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1003.0783v1.

Vancouver

1. Blei DM, McAuliffe JD (2010) Supervised Topic Models. arXiv

BibTeX

@article{blei2010supervised,
  title = {Supervised Topic Models},
  author = {Blei, David M. and McAuliffe, Jon D.},
  year = {2010},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1003.0783v1},
  eprint = {1003.0783}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

Supervised Topic Models

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Supervised Latent Dirichlet Allocation Generative Model

Knowl 2 — Variational Inference for Supervised Latent Dirichlet Allocation

Knowl 3 — Variational Expectation-Maximization for sLDA Parameter Estimation

Knowl 4 — Response Prediction for Unlabeled Documents in sLDA

Knowl 5 — Exact Variational Updates and Estimation for Gaussian Response sLDA

Knowl 6 — Variational Inference and Estimation for Overdispersed Poisson Response sLDA

Knowl 7 — Empirical Predictive Performance of sLDA on Movie Ratings and Senate Ideology

References

Citation

BibTeX

Source Code

Access the Paper