Text Classification from Labeled and Unlabeled Documents using EM

K. NigamA. McCallumS. ThrunTom Michael Mitchell

article2000Machine Learning3,259 citations

Demonstrates how combining Expectation-Maximization with naive Bayes leverages abundant unlabeled text to significantly reduce classification error and labeled data requirements, while introducing practical extensions to address violated generative model assumptions.

Listen

This paper addresses the high cost of obtaining labeled training examples for statistical text classifiers, which must be hand-labeled by people, while unlabeled documents are often easy to collect in large quantities from sources such as the web, newsgroups, and news feeds. The work evaluates whether a large pool of unlabeled documents can be combined with a small number of labeled ones to produce more accurate classifiers.

The authors combine the Expectation-Maximization algorithm with a naive Bayes generative model to iteratively assign probabilistic labels to the unlabeled documents and re-estimate classifier parameters until convergence. They test the approach on three real-world collections—20 Newsgroups articles, university web pages, and Reuters newswire stories—using training sets that range from a few dozen to several thousand labeled documents and up to 10,000 unlabeled ones. They also examine two practical extensions: a tunable weight on the unlabeled data and the use of multiple mixture components per class.

The experiments show that unlabeled data can reduce classification error by up to 30 percent. With only 300 labeled documents on 20 Newsgroups, accuracy rises from 52 percent to 66 percent; with just 40 labeled documents, accuracy improves from 27 percent to 43 percent. In one case the method achieves the same 70 percent accuracy target with roughly one-third as many labeled examples. When the data fit the model assumptions well, basic EM works reliably; when the assumptions are violated, performance can drop slightly, but the two extensions largely eliminate the degradation while preserving most of the gains.

These results matter because they demonstrate a practical way to lower the labeling burden that currently limits the deployment of text classification systems in many domains. The approach is especially valuable in settings where only a few dozen or a few hundred labeled examples can be obtained.

The authors recommend applying the basic EM procedure when labeled data are scarce, selecting the unlabeled-data weight and the number of mixture components per class by cross-validation on the labeled set, and falling back to ordinary naive Bayes when abundant labeled data are already available. They note that the method’s success depends on how closely the data match the assumed mixture model and that cross-validation can under-select components when the labeled set is extremely small; additional labeled data or more robust model-selection techniques would increase reliability in those edge cases.

Carnegie Mellon University Just Research bayes classifier naive bayes naive bayes classifiers Expectation‑Maximization Naive Bayes Unlabeled Data

Nigam et al (2000).pdf

Paper: A comparison of event models for naive bayes text classification, Andrew McCallum et al. (1998). Reading this comparison of multinomial and binary event models for naive Bayes text classification provides essential baseline context for the generative text model used in the source.

Paper: Latent Dirichlet Allocation, David M. Blei et al. (2003). This work generalizes document modeling beyond class-conditional mixture components to Latent Dirichlet Allocation, representing a major evolution in generative text corpora analysis.

Cover for Text Classification from Labeled and Unlabeled Documents using EM

Abstract

This paper shows that the accuracy of learned text classifiers can be improved by augmenting a small number of labeled training documents with a large pool of unlabeled documents. This is important because in many text classification problems obtaining training labels is expensive, while large quantities of unlabeled documents are readily available.

We introduce an algorithm for learning from labeled and unlabeled documents based on the combination of Expectation-Maximization (EM) and a naive Bayes classifier. The algorithm first trains a classifier using the available labeled documents, and probabilistically labels the unlabeled documents. It then trains a new classifier using the labels for all the documents, and iterates to convergence. This basic EM procedure works well when the data conform to the generative assumptions of the model. However these assumptions are often violated in practice, and poor performance can result. We present two extensions to the algorithm that improve classification accuracy under these conditions: (1) a weighting factor to modulate the contribution of the unlabeled data, and (2) the use of multiple mixture components per class. Experimental results, obtained using text from three different real-world tasks, show that the use of unlabeled data reduces classification error by up to 30%.

1. Introduction
2. Argument for the value of unlabeled data
3. The probabilistic framework
4. Text classification with naive Bayes
4.1. The generative model
4.2. Training a classifier
4.3. Using a classifier
4.4. Discussion
5. Incorporating unlabeled data with EM
5.1. Basic EM
5.2. Discussion
5.3. Augmented EM
6. Experimental results
6.1. Datasets and protocol
6.2. EM with unlabeled data increases accuracy
6.3. *Varying the weight of the unlabeled data*
6.4. Multiple mixture components per class
7. Related work
8. Summary and conclusions
Acknowledgments
Notes

Knowls

Knowl 1 — Expectation-Maximization Algorithm for Multinomial Naive Bayes Text Classification
algorithm
The basic semi-supervised Expectation-Maximization (EM) algorithm combines a small labeled document set $\mathcal{D}^l$ and a large unlabeled document set $\mathcal{D}^u$ ( $\{d_1, \dots, d_{|\mathcal{D}|}\} = \mathcal{D} = \mathcal{D}^l \cup \mathcal{D}^u$ ) to estimate the parameters $\theta = \{\theta_{w_t \mid c_j}, \theta_{c_j}\}$ of a generative multinomial mixture model over vocabulary $V = \{w_1, \dots, w_{|V|}\}$ and classes $C = \{c_1, \dots, c_{|C|}\}$ .
Input: Labeled documents $\mathcal{D}^l = \{(d_i, y_i)\}$ , unlabeled documents $\mathcal{D}^u = \{d_i\}$ , vocabulary $V$ , class set $C$ , convergence threshold $\epsilon = 0.05$
Output: Estimated parameters $\hat{\theta} = \{\hat{\theta}_{w_t \mid c_j}, \hat{\theta}_{c_j}\}$ for document classification
Initialize parameters $\hat{\theta}$ using only labeled documents $\mathcal{D}^l$ via Laplace-smoothed MAP estimates:
for each class $c_j \in C$ do
     $\hat{\theta}_{c_j} \leftarrow \frac{1 + \sum_{d_i \in \mathcal{D}^l} P(y_i = c_j \mid d_i)}{|C| + |\mathcal{D}^l|}$
    for each word $w_t \in V$ do
         $\hat{\theta}_{w_t \mid c_j} \leftarrow \frac{1 + \sum_{d_i \in \mathcal{D}^l} N(w_t, d_i) P(y_i = c_j \mid d_i)}{|V| + \sum_{s=1}^{|V|} \sum_{d_i \in \mathcal{D}^l} N(w_s, d_i) P(y_i = c_j \mid d_i)}$
repeat
    // E-step: Compute expected class memberships for all unlabeled documents
    for each document $d_i \in \mathcal{D}^u$ and class $c_j \in C$ do
         $P(c_j \mid d_i; \hat{\theta}) \leftarrow \frac{\hat{\theta}_{c_j} \prod_{k=1}^{|d_i|} P(w_{d_i, k} \mid c_j; \hat{\theta})}{\sum_{r=1}^{|C|} \hat{\theta}_{c_r} \prod_{k=1}^{|d_i|} P(w_{d_i, k} \mid c_r; \hat{\theta})}$
    // M-step: Re-estimate parameters over all documents (labeled and probabilistically labeled)
    for each class $c_j \in C$ do
         $\hat{\theta}_{c_j} \leftarrow \frac{1 + \sum_{d_i \in \mathcal{D}} P(y_i = c_j \mid d_i)}{|C| + |\mathcal{D}|}$
        for each word $w_t \in V$ do
             $\hat{\theta}_{w_t \mid c_j} \leftarrow \frac{1 + \sum_{d_i \in \mathcal{D}} N(w_t, d_i) P(y_i = c_j \mid d_i)}{|V| + \sum_{s=1}^{|V|} \sum_{d_i \in \mathcal{D}} N(w_s, d_i) P(y_i = c_j \mid d_i)}$
    Compute change $\Delta l_c$ in complete data log-likelihood
until $\Delta l_c < \epsilon$
return $\hat{\theta}$
Here $N(w_t, d_i)$ denotes the count of word $w_t$ in document $d_i$ , and $w_{d_i, k}$ is the word token at position $k$ in document $d_i$ . For labeled documents $d_i \in \mathcal{D}^l$ , $P(y_i = c_j \mid d_i)$ is deterministic ( $1$ if $y_i = c_j$ and $0$ otherwise). Laplace smoothing corresponds to a Dirichlet prior over parameters with hyperparameter $\alpha = 2$ .
Knowl 2 — EM-λ: Modulating the Influence of Unlabeled Data in Semi-Supervised Parameter Estimation
model/method
When generative model assumptions fail or natural clusters in the data do not align with target classes, large volumes of unlabeled data can dominate parameter estimation in standard Expectation-Maximization and degrade classification accuracy. The EM- $\lambda$ method modulates the relative contribution of unlabeled documents using a discount parameter $\lambda \in [0, 1]$ .

The objective is the weighted complete data log-likelihood:

$l_c(\theta \mid \mathcal{D}; z) = \log P(\theta) + \sum_{d_i \in \mathcal{D}^l} \sum_{j=1}^{|C|} z_{ij} \log(P(c_j \mid \theta) P(d_i \mid c_j; \theta)) + \lambda \left( \sum_{d_i \in \mathcal{D}^u} \sum_{j=1}^{|C|} z_{ij} \log(P(c_j \mid \theta) P(d_i \mid c_j; \theta)) \right)$

where $z_{ij}$ is the binary indicator for document $d_i$ belonging to class $c_j$ , $\mathcal{D}^l$ is the labeled set, and $\mathcal{D}^u$ is the unlabeled set. Defining document weight $\Lambda(i)$ as:

$\Lambda(i) = \begin{cases} \lambda & \text{if } d_i \in \mathcal{D}^u \\ 1 & \text{if } d_i \in \mathcal{D}^l \end{cases}$

the M-step word probability and class prior estimators with Laplace smoothing are modified to:

$\hat{\theta}_{w_t \mid c_j} = \frac{1 + \sum_{i=1}^{|\mathcal{D}|} \Lambda(i) N(w_t, d_i) P(y_i = c_j \mid d_i)}{|V| + \sum_{s=1}^{|V|} \sum_{i=1}^{|\mathcal{D}|} \Lambda(i) N(w_s, d_i) P(y_i = c_j \mid d_i)}$

$\hat{\theta}_{c_j} = \frac{1 + \sum_{i=1}^{|\mathcal{D}|} \Lambda(i) P(y_i = c_j \mid d_i)}{|C| + |\mathcal{D}^l| + \lambda |\mathcal{D}^u|}$

When $\lambda = 1$ , the model reverts to standard EM; when $\lambda = 0$ , the algorithm reduces to supervised naive Bayes on labeled data alone. Intermediate values of $\lambda$ allow leveraging unlabeled data while preventing it from overpowering labeled supervision.
Knowl 3 — Multiple Mixture Components per Class for Semi-Supervised Text Classification
model/method
To handle classes that contain multiple distinct sub-topics or complex word co-occurrence dependencies (such as a broad negative class in one-vs-all classification), the one-to-one correspondence between mixture components and classes is relaxed to a deterministic many-to-one mapping.

Let $t_a \in T$ denote the target classes (topics) and $c_j \in C$ denote the underlying mixture components, where $P(t_a \mid c_j) \in \{0, 1\}$ indicates whether component $c_j$ belongs to class $t_a$ .

In this formulation:
1. Labeled Document Membership: For a labeled document $(d_i, y_i)$ , component membership probabilities $P(c_j \mid d_i; \hat{\theta})$ are clamped to zero for all components where $P(y_i \mid c_j) = 0$ . For components associated with class $y_i$ , $P(c_j \mid d_i; \hat{\theta})$ is allowed to vary between $0$ and $1$ according to the model posterior and is normalized to sum to $1$ .
2. Initialization: Components belonging to the same class are initialized by performing a randomized E-step that samples $P(c_j \mid d_i; \hat{\theta})$ uniformly across the components assigned to class $y_i$ .
3. Classification Inference: An unclassified test document $d_i$ is classified by summing posterior probabilities over all mixture components assigned to each target class $t_a$ :
$P(t_a \mid d_i; \hat{\theta}) = \sum_{c_j} P(t_a \mid c_j) \frac{P(c_j \mid \hat{\theta}) \prod_{k=1}^{|d_i|} P(w_{d_i, k} \mid c_j; \hat{\theta})}{\sum_{r=1}^{|C|} P(c_r \mid \hat{\theta}) \prod_{k=1}^{|d_i|} P(w_{d_i, k} \mid c_r; \hat{\theta})}$
Knowl 4 — Unified Augmented EM Algorithm for Text Classification
algorithm
The unified augmented EM algorithm incorporates both unlabeled data weighting ( $\lambda$ ) and multiple mixture components per class.
Input: Labeled set $\mathcal{D}^l$ , unlabeled set $\mathcal{D}^u$ , vocabulary $V$ , class set $T$ , component-to-class mapping $P(t_a \mid c_j) \in \{0, 1\}$ , weighting factor $\lambda \in [0, 1]$ , threshold $\epsilon = 0.05$
Output: Trained classifier $\hat{\theta} = \{\hat{\theta}_{w_t \mid c_j}, \hat{\theta}_{c_j}\}$
// Initialize component memberships for labeled documents
for each document $d_i \in \mathcal{D}^l$ do
    for each mixture component $c_j$ do
        if $P(y_i \mid c_j) = 1$ then
            Assign $P(c_j \mid d_i; \hat{\theta})$ randomly from uniform distribution
        else
             $P(c_j \mid d_i; \hat{\theta}) \leftarrow 0$
    Normalize $P(c_j \mid d_i; \hat{\theta})$ over all $c_j$
// Initial M-step using labeled data only
Estimate initial $\hat{\theta}$ using labeled documents and their initial component assignments via Laplace-smoothed MAP
repeat
    // E-step
    for each document $d_i \in \mathcal{D}^u$ and component $c_j$ do
         $P(c_j \mid d_i; \hat{\theta}) \leftarrow \frac{\hat{\theta}_{c_j} \prod_{k=1}^{|d_i|} P(w_{d_i, k} \mid c_j; \hat{\theta})}{\sum_{r=1}^{|C|} \hat{\theta}_{c_r} \prod_{k=1}^{|d_i|} P(w_{d_i, k} \mid c_r; \hat{\theta})}$

    for each document $d_i \in \mathcal{D}^l$ and component $c_j$ do
        if $P(y_i \mid c_j) = 1$ then
             $P(c_j \mid d_i; \hat{\theta}) \leftarrow \frac{\hat{\theta}_{c_j} \prod_{k=1}^{|d_i|} P(w_{d_i, k} \mid c_j; \hat{\theta})}{\sum_{r: P(y_i \mid c_r)=1} \hat{\theta}_{c_r} \prod_{k=1}^{|d_i|} P(w_{d_i, k} \mid c_r; \hat{\theta})}$
        else
             $P(c_j \mid d_i; \hat{\theta}) \leftarrow 0$
    // M-step with discount weight lambda
    for each component $c_j$ do
         $\hat{\theta}_{c_j} \leftarrow \frac{1 + \sum_{d_i \in \mathcal{D}} \Lambda(i) P(c_j \mid d_i; \hat{\theta})}{|C| + |\mathcal{D}^l| + \lambda |\mathcal{D}^u|}$
        for each word $w_t \in V$ do
             $\hat{\theta}_{w_t \mid c_j} \leftarrow \frac{1 + \sum_{d_i \in \mathcal{D}} \Lambda(i) N(w_t, d_i) P(c_j \mid d_i; \hat{\theta})}{|V| + \sum_{s=1}^{|V|} \sum_{d_i \in \mathcal{D}} \Lambda(i) N(w_s, d_i) P(c_j \mid d_i; \hat{\theta})}$
    Compute change $\Delta l_c$ in complete log-likelihood
until $\Delta l_c < \epsilon$
return $\hat{\theta}$
Knowl 5 — Generative Assumptions of Semi-Supervised Multinomial Mixture Models
assumption
The basic probabilistic framework for semi-supervised naive Bayes text classification relies on four core assumptions about document generation:
1. Mixture Model Generation: Every document $d_i$ is generated by a finite mixture model parameterized by $\theta$ , where a component $c_j$ is chosen with probability $P(c_j \mid \theta)$ , and the document is subsequently generated according to component distribution $P(d_i \mid c_j; \theta)$ .
2. Component-to-Class Correspondence: In the basic model, there is an exact one-to-one mapping between mixture components $c_j$ and target class labels $y_i$ .
3. Conditional Word Independence (Unigram Assumption): Given mixture component $c_j$ , the probability of each word token $w_{d_i, k}$ in document $d_i$ is statistically independent of its position $k$ and of all preceding or surrounding words in the document: $P(w_{d_i, k} \mid c_j; \theta; w_{d_i, q}, q < k) = P(w_{d_i, k} \mid c_j; \theta)$ .
4. Class-Independent Document Length: Document length $|d_i|$ is generated independently of the mixture component $c_j$ and class label: $P(|d_i| \mid c_j) = P(|d_i|)$ .
When real-world text violates these assumptions (e.g., multi-topic documents, semantic word correlations, multi-modal class distributions), unsupervised clustering via basic EM can find mixture modes that do not correspond to the true classification boundary, potentially degrading classifier accuracy compared to supervised learning.
Knowl 6 — Semi-Supervised Classification Gains on 20 Newsgroups
empirical result
On the 20 Newsgroups dataset (20,017 articles across 20 classes, tested on 4,000 held-out documents using 10,000 unlabeled training documents):
- Small Labeled Datasets: With 300 labeled documents (15 per class), supervised naive Bayes achieves 52% classification accuracy, whereas basic EM with 10,000 unlabeled documents achieves 66% accuracy, representing a 30% reduction in classification error ( $p < 0.05$ ). With only 20 labeled documents (1 per class), naive Bayes achieves 20% accuracy, while EM reaches 35%.
- Labeled Sample Efficiency: To reach 70% classification accuracy, supervised naive Bayes requires 2,000 labeled documents, whereas EM achieves 70% accuracy with only 600 labeled documents combined with 10,000 unlabeled documents (a reduction in labeled data requirements by more than a factor of three).
- Large Labeled Datasets: With 5,500 labeled documents (275 per class), the learning curve nears its plateau, and EM improves accuracy from 76% to 78%.
- Small Unlabeled Data Anomaly: Adding a very small number of unlabeled documents (e.g., $<1,000$ ) to a small labeled set can temporarily hurt accuracy because extreme, over-confident naive Bayes posterior probabilities in the E-step distribute small unlabeled sample mass too sharply.
Knowl 7 — Mitigation of Classifier Degradation Using EM-λ on WebKB
empirical result
On the WebKB dataset (4,199 web pages across 4 classes: student, faculty, course, project, tested via leave-one-university-out using 2,500 unlabeled documents and a 300-word vocabulary chosen by mutual information):
- Low Supervision Regime: With 4 labeled documents (1 per class), basic EM ( $\lambda = 1$ ) improves test accuracy from 40% (supervised naive Bayes) to 55%.
- High Supervision Degradation: With 240 labeled documents, basic EM degrades performance to 79%, compared to 81% for supervised naive Bayes ( $p < 0.05$ ), caused by a misfit between the single-component generative model assumptions and the true document distribution.
- EM- $\lambda$ Modulation: Across labeled training set sizes, optimal classification accuracy is achieved at intermediate values of $\lambda$ . Smaller labeled sets favor larger $\lambda$ (heavier reliance on unlabeled clustering), whereas larger labeled sets favor smaller $\lambda$ . Selecting $\lambda$ via leave-one-out cross-validation on the labeled training set automatically assigns a weight of only $\approx 5\%$ to unlabeled documents at 240 labeled examples, achieving 82% accuracy and preventing the accuracy degradation of basic EM.

Knowl 8 — Performance of Multi-Component EM on Binary Reuters Categorization

data/table

On the Reuters-21578 dataset evaluated across its ten most populous categories in a binary classification setting (10 positive and 40 negative labeled training documents, 7,000 unlabeled documents, and 3,299 ModApte test documents), the negative class covers 89 topics and strongly violates the single-component assumption.

The table below reports the precision-recall breakeven points for:

NB1: Standard naive Bayes (1 component per class, labeled data only)
NB*: Naive Bayes with multiple mixture components for the negative class (labeled data only, optimal number of components)
EM1: Basic EM with 1 component per class (labeled and unlabeled data)
EM*: Augmented EM with multiple mixture components for the negative class (labeled and unlabeled data, optimal number of components; median optimal component count in parentheses)

Category	NB1	NB*	EM1	EM*	EM* vs. NB1	EM* vs. NB*
acq	69.4	74.3 (4)	70.7	83.9 (10)	+14.5	+9.6
corn	44.3	47.8 (3)	44.6	52.8 (5)	+8.5	+5.0
crude	65.2	68.3 (2)	68.2	75.4 (8)	+10.2	+7.1
earn	91.1	91.6 (1)	89.2	89.2 (1)	-1.9	-2.4
grain	65.7	66.6 (2)	67.0	72.3 (8)	+6.3	+5.7
interest	44.4	54.9 (5)	36.8	52.3 (5)	+7.9	-2.6
money-fx	49.4	55.3 (15)	40.3	56.9 (10)	+7.5	+1.6
ship	44.3	51.2 (4)	34.1	52.5 (7)	+8.2	+1.3
trade	57.7	61.3 (3)	56.1	61.8 (3)	+4.1	+0.5
wheat	56.0	67.4 (10)	52.9	67.8 (10)	+11.8	+0.4

Basic EM with a single negative component (EM1) performs worse than supervised naive Bayes (NB1) on 6 out of 10 categories. Combining multiple mixture components with unlabeled data (EM*) yields substantial performance gains over both single-component EM and supervised naive Bayes ( $p < 0.05$ across all trials), while supporting a more expressive model with higher optimal component counts (median 5 to 10 components) than NB*.

Knowl 9 — Suboptimality of Short-Cut Cross-Validation for EM Model Selection
limitation
Selecting hyperparameters—such as the number of mixture components or the unlabeled data weight $\lambda$ —using leave-one-out cross-validation requires evaluating multiple parameter configurations. To make this computationally feasible, a computational short-cut runs EM once to convergence on all data, and then subtracts the word counts of each labeled document in turn during test evaluation.

This short-cut biases the model towards the held-out document because the EM cluster parameters were estimated with that document included. As a consequence, cross-validation systematically underestimates the optimal number of mixture components for multi-component EM (e.g., selecting 1 component instead of the test-optimal 10 components for categories like acq and wheat), leading to cross-validated performance (EM*CV) that falls short of optimal semi-supervised potential (EM*).

Coverage note — None was omitted. All primary models, algorithms, generative assumptions, empirical findings across the three benchmark datasets, and model selection limitations have been captured.

References

1.Blum, A. & Mitchell, T. (1998). Combining labeled and unlabeled data with co-training. Proceedings of the 11th Annual Conference on Computational Learning Theory (COLT '98) (pp. 92–100).
2.Castelli, V. & Cover, T.M. (1995). On the exponential value of labeled samples. Pattern Recognition Letters, 16(1), 105–111.
3.Cheeseman, P. & Stutz, J. (1996). Bayesian classification (AutoClass): Theory and results. In U. Fayyad, G. Piatetsky-Shapiro, P. Smyth, & R. Uthurusamy (Eds.), Advances in knowledge discovery and data mining. MIT Press.
4.Cohen, W.W. & Singer, Y. (1996). Context-sensitive learning methods for text categorization. SIGIR '96: Proceedings of the Nineteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 307–315).
5.Cover, T.M. & Thomas, J.A. (1991). Elements of information theory. New York: John Wiley and Sons.
6.Craven, M., DiPasquo, D., Freitag, D., McCallum, A., Mitchell, T., Nigam, K., & Slattery, S. (1998). Learning to extract symbolic knowledge from the World Wide Web. Proceedings of the Fifteenth National Conference on Artificial Intellligence (AAAI-98) (pp. 509–516).
7.Dagan, I. & Engelson, S.P. (1995). Committee-based sampling for training probabilistic classifiers. Machine Learning: Proceedings of the Twelfth International Conference (ICML '95) (pp. 150–157).
8.Dempster, A.P., Laird, N.M., & Rubin, D.B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B, 39(1), 1–38.
9.Dietterich, T.G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7), 1895–1923.
10.Domingos, P. & Pazzani, M. (1997). On the optimality of the simple Bayesian classifier under zero-one loss. Machine Learning, 29, 103–130.
11.Friedman, J.H. (1997). On bias, variance, 0/1-loss, and the curse-of-dimensionality. Data Mining and Knowledge Discovery, 1(1), 55–77.
12.Ghahramani, Z. & Jordan, M.I. (1994). Supervised learning from incomplete data via an EM approach. In Advances in neural information processing systems 6 (pp. 120–127). Morgan Kaufmann.
13.Jaakkola, T.S. & Jordan, M.I. (1998). Improving the mean field approximation via the use of mixture distributions. In M. I. Jordan (Ed.), Learning in graphical models. Kluwer Academic Publishers.
14.Joachims, T. (1997). A probabilistic analysis of the Rocchio algorithm with TFIDF for text categorization. Machine Learning: Proceedings of the Fourteenth International Conference (ICML '97) (pp. 143–151).
15.Joachims, T. (1998). Text categorization with Support Vector Machines: Learning with many relevant features. Machine Learning: ECML-98, Tenth European Conference on Machine Learning (pp. 137–142).
16.Koller, D. & Sahami, M. (1997). Hierarchically classifying documents using very few words. Machine Learning: Proceedings of the Fourteenth International Conference (ICML '97) (pp. 170–178).
17.Lang, K. (1995). Newsweeder: Learning to filter netnews. Machine Learning: Proceedings of the Twelfth International Conference (ICML '95) (pp. 331–339).
18.Larkey, L.S. & Croft, W.B. (1996). Combining classifiers in text categorization. SIGIR '96: Proceedings of the Nineteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 289–297).
19.Lewis, D.D. (1992). An evaluation of phrasal and clustered representations on a text categorization task. SIGIR '92: Proceedings of the Fifteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 37–50).
20.Lewis, D.D. (1995). A sequential algorithm for training text classifiers: Corrigendum and additional data. SIGIR Forum, 29(2), 13–19.
21.Lewis, D.D. (1998). Naive (Bayes) at forty: The independence assumption in information retrieval. Machine Learning: ECML-98, Tenth European Conference on Machine Learning (pp. 4–15).
22.Lewis, D.D. & Gale, W.A. (1994). A sequential algorithm for training text classifiers. SIGIR '94: Proceedings of the Seventeenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 3–12).
23.Lewis, D.D. & Knowles, K.A. (1997). Threading electronic mail: A preliminary study. Information Processing and Management, 33(2), 209–217.
24.Lewis, D.D. & Ringuette, M. (1994). A comparison of two learning algorithms for text categorization. Third Annual Symposium on Document Analysis and Information Retrieval (pp. 81–93).
25.Li, H. & Yamanishi, K. (1997). Document classification using a finite mixture model. Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics (pp. 39–47).
26.Liere, R. & Tadepalli, P. (1997). Active learning with committees for text categorization. Proceedings of the Fourteenth National Conference on Artificial Intelligence (AAAI-97) (pp. 591–596).
27.McCallum, A. & Nigam, K. (1998). A comparison of event models for naive Bayes text classification. AAAI-98 Workshop on Learning for Text Categorization. Tech. Rep. WS-98-05, AAAI Press. http://www.cs.cmu.edu/∼mccallum.
28.McCallum, A.K. & Nigam, K. (1998). Employing EM in pool-based active learning for text classification. Machine Learning: Proceedings of the Fifteenth International Conference (ICML '98) (pp. 350–358).
29.McCallum, A., Rosenfeld, R., Mitchell, T., & Ng, A. (1998). Improving text clasification by shrinkage in a hierarchy of classes. Machine Learning: Proceedings of the Fifteenth International Conference (ICML '98) (pp. 359–367).
30.McLachlan, G. & Basford, K. (1988). Mixture models. New York: Marcel Dekker.
31.McLachlan, G.J. & Krishnan, T. (1997). The EM algorithm and extensions. New York: John Wiley and Sons.
32.Miller, D.J. & Uyar, H.S. (1997). A mixture of experts classifier with learning based on both labelled and unlabelled data. In Advances in Neural Information Processing Systems 9 (pp. 571–577). The MIT Press.
33.Mitchell, T.M. (1997). Machine learning. New York: McGraw-Hill.
34.Ng, A.Y. (1997). Preventing “overfitting” of cross-validation data. Machine Learning: Proceedings of the Fourteenth International Conference (ICML '97) (pp. 245–253).
35.Pazzani, M.J., Muramatsu, J., & Billsus, D. (1996). Syskill & Webert: Identifying interesting Web sites. Proceedings of the Thirteenth National Conference on Artificial Intelligence (AAAI-96) (pp. 54–59).
36.Rissanen, J. (1983). A universal prior for integers and estimation by minimum description length. Annals of Statistics, 11(2), 416–431.
37.Robertson, S.E. & Sparck-Jones, K. (1976). Relevance weighting of search terms. Journal of the American Society for Information Science, 27(3), 129–146.
38.Rocchio, J. (1971). Relevance feedback in information retrieval. In G. Salton (Ed.), The SMART retrieval system: experiments in automatic document processing. Englewood Cliffs, NJ: Prentice Hall.
39.Sahami, M., Dumais, S., Heckerman, D., & Horvitz, E. (1998). A Baysian approach to filtering junk e-mail. AAAI-98 Workshop on Learning for Text Categorization. Tech. Rep. WS-98-05, AAAI Press. http://robotics.stanford.edu/users/sahami/papers.html.
40.Salton, G. (1991). Developments in automatic text retrieval. Science, 253(5023), 974–980.
41.Schuurmans, D. (1997). A new metric-based approach to model selection. Proceedings of the Fourteenth National Conference on Artificial Intelligence (AAAI-97) (pp. 552–558).
42.Shahshahani, B. & Landgrebe, D. (1994). The effect of unlabeled samples in reducing the small sample size problem and mitigating the Hughes phenomenon. IEEE Transactions on Geoscience and Remote Sensing, 32(5), 1087–1095.
43.Shavlik, J. & Eliassi-Rad, T. (1998). Intelligent agents for web-based tasks: An advice-taking approach. AAAI-98 Workshop on Learning for Text Categorization. Tech. Rep. WS-98-05, AAAI Press. http://www.cs.wisc.edu/∼shavlik/mlrg/publications.html.
44.Stolcke, A. & Omohundro, S.M. (1994). Best-first model merging for hidden Markov model induction. Tech. Rep. TR-94-003, ICSI, University of California, Berkeley. http:// www.icsi.berkeley.edu/techreports/1994.html.
45.Yang, Y. (1994). Expert network: Effective and efficient learning from human decisions in text categorization and retrieval. SIGIR '94: Proceedings of the Seventeenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 13–22).
46.Yang, Y. (1999). An evaluation of statistical approaches to text categorization. Journal of Information Retrieval, 1, 69–90.
47.Yang, Y. & Pederson, J.O. (1997). Feature selection in statistical learning of text categorization. Machine Learning: Proceedings of the Fourteenth International Conference (ICML '97) (pp. 412–420).

Citation

MLA

Nigam, K., et al. “Text Classification from Labeled and Unlabeled Documents Using EM”. Machine Learning, vol. 39, nos. 2-3, 2000, pp. 103–34, https://doi.org/10.1023/A:1007692713085.

APA

Nigam, K., Mccallum, A. K., Thrun, S., & Mitchell, T. (2000). Text Classification from Labeled and Unlabeled Documents using EM. Machine Learning, 39(2-3), 103–134. https://doi.org/10.1023/A:1007692713085

Chicago

Nigam, K., A. K. Mccallum, S. Thrun, and T. Mitchell. 2000. “Text Classification from Labeled and Unlabeled Documents Using EM”. Machine Learning 39 (2-3): 103–34. https://doi.org/10.1023/A:1007692713085.

Harvard

Nigam, K. et al. (2000) “Text Classification from Labeled and Unlabeled Documents using EM”, Machine Learning, 39(2-3), pp. 103–134. Available at: https://doi.org/10.1023/A:1007692713085.

Vancouver

1. Nigam K, Mccallum AK, Thrun S, Mitchell T (2000) Text Classification from Labeled and Unlabeled Documents using EM. Machine Learning 39:103–134

BibTeX

@article{Nigam_2000, title={Text Classification from Labeled and Unlabeled Documents using EM}, volume={39}, ISSN={1573-0565}, url={http://dx.doi.org/10.1023/A:1007692713085}, DOI={10.1023/a:1007692713085}, number={2-3}, journal={Machine Learning}, publisher={Springer Science and Business Media LLC}, author={Nigam, Kamal and Mccallum, Andrew Kachites and Thrun, Sebastian and Mitchell, Tom}, year={2000}, month=May, pages={103–134} }

Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Text Classification from Labeled and Unlabeled Documents using EM

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Expectation-Maximization Algorithm for Multinomial Naive Bayes Text Classification

Knowl 2 — EM-λ: Modulating the Influence of Unlabeled Data in Semi-Supervised Parameter Estimation

Knowl 3 — Multiple Mixture Components per Class for Semi-Supervised Text Classification

Knowl 4 — Unified Augmented EM Algorithm for Text Classification

Knowl 5 — Generative Assumptions of Semi-Supervised Multinomial Mixture Models

Knowl 6 — Semi-Supervised Classification Gains on 20 Newsgroups

Knowl 7 — Mitigation of Classifier Degradation Using EM-λ on WebKB

Knowl 8 — Performance of Multi-Component EM on Binary Reuters Categorization

Knowl 9 — Suboptimality of Short-Cut Cross-Validation for EM Model Selection

References

Citation

BibTeX

Access the Paper