Topics over time: a non-Markov continuous-time model of topical trends

Xuerui WangAndrew McCallum

article2006KDD1,465 citations

Proposes a continuous-time topic model that associates each topic with a continuous distribution over document timestamps, capturing temporal topical trends and improving timestamp prediction without relying on time discretization or Markov assumptions.

Listen

Standard text analysis and topic modeling techniques often process collections of documents as static snapshots, failing to account for how vocabulary and discussion subjects evolve over time. When time is ignored, models struggle to separate historical events or transient tasks that share similar vocabularies, leading to mixed and inaccurate topic summaries. Existing temporal methods often address this by slicing data into discrete time windows or relying on step-by-step transition assumptions, which introduces arbitrary boundaries and struggles with gaps in activity.

The article evaluates Topics over Time (TOT), a statistical model designed to jointly capture word co-occurrence patterns and their continuous occurrence across time. The objective is to demonstrate that treating time as a continuous variable directly tied to topics produces sharper, more event-specific themes and enables accurate temporal predictions.

The approach models documents by treating topic meanings as fixed while allowing the prominence and co-occurrence of topics to change over a continuous timeline. Each topic is parameterized with a continuous Beta distribution over normalized timestamps, avoiding the need to slice time into artificial bins. The authors tested this method across three diverse, real-world data sources: 208 U.S. Presidential State-of-the-Union addresses spanning over two centuries, 2,326 machine learning research papers published over 17 years, and nine months of personal email archives consisting of 13,300 messages.

The analysis produced several key findings. First, TOT effectively eliminated historical and contextual blending, successfully isolating distinct historical eras and short-lived events that conventional models merged. Second, across all three data sets, the model produced statistically more distinct topics, as measured by higher average divergence scores between topic word distributions. Third, when tasked with predicting the publication decade of presidential addresses from text alone, TOT achieved nearly double the exact-match accuracy of standard models (19% versus 10%) while reducing average prediction error by approximately 20%. Finally, the model mapped clear thematic shifts over time, such as tracking how classification research transitioned from neural network implementations to support vector machines and boosting methods.

These findings indicate that integrating continuous temporal data significantly improves automated document categorization, discovery, and archiving. For decision-makers managing large organizational knowledge bases, email repositories, or historical records, this framework provides clearer trend identification and reduces the risk of confounding unrelated initiatives. The model provides an effective, computationally straightforward mechanism to track shifts in strategic priorities without the complexity or artificial boundaries of discrete time-slice modeling.

Organizations handling evolving textual data should consider adopting continuous-time topic frameworks for document search, categorization, and trend analysis. Because the framework is modular and computationally efficient, technical teams can integrate it directly into existing organizational text-mining pipelines. Future development should explore applying this continuous-time formulation to richer network models, such as analyzing shifts in group dynamics and social roles over time.

A primary limitation of this evaluation is the assumption that individual topic word definitions remain constant rather than shifting dynamically over time. Additionally, the standard formulation uses a single-peaked Beta distribution per topic, which may require alternative distributions to capture topics that recur across multiple distinct eras. While confidence in the model's comparative performance on the tested corpora is high, practitioners applying it to operational environments should tune the weighting parameter between the text and temporal data sources to ensure optimal topic balance.

University of Massachusetts Amherst continuous time email markov chains dynamic topic models non-Markov continuous-time model topical trends

Wang et al (2006).pdf

Paper: Latent Dirichlet Allocation, David M. Blei et al. (2003). Introduces Latent Dirichlet Allocation (LDA), the foundational generative probabilistic model that Topics over Time directly modifies by incorporating continuous timestamps.
Paper: The Author-Topic Model for Authors and Documents, Michal Rosen-Zvi et al. (2004). Introduces the Author-Topic model, demonstrating how to condition LDA topic mixtures on document-level non-textual metadata.
Paper: Probabilistic Latent Semantic Analysis, Thomas Hofmann (1999). Establishes probabilistic latent semantic analysis and the aspect model framework that underlies hierarchical Bayesian topic modeling.
Paper: Relevance-Based Language Models, Victor Lavrenko et al. (2001). Pioneers the formal integration of continuous time distributions into probabilistic language models for document retrieval.

Paper: Latent Dirichlet allocation (LDA) and topic modeling: models, applications, a survey, Hamed Jelodar et al. (2017). Provides a comprehensive survey charting the major extensions, inference strategies, and downstream applications that evolved from LDA and its temporal adaptations.
Paper: Supervised Topic Models, David M. Blei et al. (2007). Extends the generative framework of LDA to jointly model latent themes and external response variables under supervision.
Paper: Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora, Daniel Ramage et al. (2009). Applies supervised multi-label conditioning to LDA latent topic structures for explicit credit attribution.
Paper: Reading Tea Leaves: How Humans Interpret Topic Models, Jonathan D. Chang et al. (2009). Develops formal human-in-the-loop evaluation frameworks to validate whether topic distributions capture meaningful semantic themes.
Paper: Optimizing Semantic Coherence in Topic Models, David Mimno et al. (2011). Introduces automated semantic coherence metrics to evaluate and optimize the quality of latent topic mixtures.
Paper: Collaborative topic modeling for recommending scientific articles, Chong Wang et al. (2011). Combines probabilistic topic modeling with collaborative filtering to drive document recommendations.
Paper: BERTopic: Neural topic modeling with a class-based TF-IDF procedure, Maarten Grootendorst (2022). Modernizes dynamic and continuous topic modeling by coupling transformer embeddings with clustering and class-based TF-IDF.

Cover for Topics over time: a non-Markov continuous-time model of topical trends

Abstract

This paper presents an LDA-style topic model that captures not only the low-dimensional structure of data, but also how the structure changes over time. Unlike other recent work that relies on Markov assumptions or discretization of time, here each topic is associated with a continuous distribution over timestamps, and for each generated document, the mixture distribution over topics is influenced by both word co-occurrences and the document’s timestamp. Thus, the meaning of a particular topic can be relied upon as constant, but the topics’ occurrence and correlations change significantly over time. We present results on nine months of personal email, 17 years of NIPS research papers and over 200 years of presidential state-of-the-union addresses, showing improved topics, better timestamp prediction, and interpretable trends.

Topics

continuous time email markov chains topic model continuous time models timestamping email messages

Keywords

dynamic topic models non-Markov continuous-time model topical trends Latent Dirichlet Allocation topic discovery Gibbs sampling graphical models

Categories and Subject Descriptors
General Terms
Keywords
1. INTRODUCTION
2. TOPICS OVER TIME
3. RELATED WORK
4. DATA SETS
4.1 State-of-the-Union Addresses
4.2 A Researcher's Email
4.3 NIPS Papers
5. EXPERIMENTAL RESULTS
5.1 Topics Discovered for Addresses
5.2 Topics Discovered for Email
5.3 Topics Discovered for NIPS
5.4 Time Prediction
5.5 Topic Distribution Profile over Time
5.6 Topic Co-occurrences over Time
6. CONCLUSIONS
7. ACKNOWLEDGMENTS
8. REFERENCES
APPENDIX
A. GIBBS SAMPLING DERIVATION FOR TOT

Knowls

Knowl 1 — Topics over Time Generative Model
model/method
Topics over Time (TOT) is a continuous-time generative topic model that jointly generates text tokens and document timestamps without discretizing time or applying Markov transition assumptions across time slices. In TOT, the semantic meaning of each topic (its distribution over vocabulary words) is assumed constant over time, while the prominence, occurrence, and co-occurrence of topics vary continuously.

Let $D$ be the number of documents, $T$ the number of topics, $V$ the vocabulary size, and $N_d$ the number of word tokens in document $d$ . Timestamps $t_d$ are observed continuous values normalized to the interval $[0, 1]$ across the entire dataset. The generative process is:
1. For each topic $z \in \{1, \dots, T\}$ , draw a word distribution $\phi_z \sim \text{Dirichlet}(\beta)$ .
2. For each document $d \in \{1, \dots, D\}$ , draw a topic mixture $\theta_d \sim \text{Dirichlet}(\alpha)$ .
3. For each word token index $i \in \{1, \dots, N_d\}$ in document $d$ :
  
  (a) Draw a topic assignment $z_{di} \sim \text{Multinomial}(\theta_d)$ ;
  
  (b) Draw a word $w_{di} \sim \text{Multinomial}(\phi_{z_{di}})$ ;
  
  (c) Draw a timestamp $t_{di} \sim \text{Beta}(\psi_{z_{di}})$ , where $\psi_z = (\psi_{z1}, \psi_{z2})$ are topic-specific Beta distribution shape parameters.
During training on typical datasets where a single timestamp $t_d$ is associated with document $d$ , every token's timestamp observation $t_{di}$ is set to $t_d$ .
Knowl 2 — Collapsed Gibbs Sampling Inference for Topics over Time
algorithm
Exact inference in Topics over Time (TOT) is intractable, so collapsed Gibbs sampling is used by integrating out the per-document topic distributions $\theta_d$ and per-topic word distributions $\phi_z$ .

Let $w_{di}$ be the $i$ -th token in document $d$ , $t_{di} \in [0, 1]$ its normalized timestamp, and $z_{di}$ its latent topic assignment. Let $\mathbf{z}_{-di}$ denote the topic assignments of all tokens excluding $w_{di}$ . Let $m_{dz}^{-di}$ denote the count of tokens in document $d$ assigned to topic $z$ excluding token $w_{di}$ , and let $n_{zv}^{-di}$ denote the count of times vocabulary word $v \in \{1, \dots, V\}$ is assigned to topic $z$ excluding token $w_{di}$ . The full conditional distribution for sampling $z_{di}$ is:

$P(z_{di} = z \mid \mathbf{w}, \mathbf{t}, \mathbf{z}_{-di}, \alpha, \beta, \Psi) \propto (m_{dz}^{-di} + \alpha_z) \frac{n_{zw_{di}}^{-di} + \beta_{w_{di}}}{\sum_{v=1}^V (n_{zv}^{-di} + \beta_v)} \frac{(1 - t_{di})^{\psi_{z1}-1} t_{di}^{\psi_{z2}-1}}{B(\psi_{z1}, \psi_{z2})}$

where $\alpha$ and $\beta$ are symmetric Dirichlet hyperparameter vectors (fixed in practice to $\alpha_z = 50/T$ and $\beta_v = 0.1$ ), and $B(\psi_{z1}, \psi_{z2}) = \frac{\Gamma(\psi_{z1})\Gamma(\psi_{z2})}{\Gamma(\psi_{z1} + \psi_{z2})}$ is the Beta function.
Input: Word tokens $\mathbf{w}$ , timestamps $\mathbf{t} \in [0, 1]$ , topic count $T$ , hyperparameters $\alpha, \beta$ , total iterations $N_{\text{iter}}$
Output: Topic assignments $\mathbf{z}$ , Beta parameters $\Psi = \{\psi_z\}_{z=1}^T$ , posterior estimates for $\theta$ and $\phi$
Initialize topic assignments $z_{di} \in \{1, \dots, T\}$ uniformly at random for all tokens
for iter = 1 to $N_{\text{iter}}$ do
    for $d = 1$ to $D$ do
        for $i = 1$ to $N_d$ do
            Decrement counts $m_{d, z_{di}}$ and $n_{z_{di}, w_{di}}$
            Sample new $z_{di} = z$ from $P(z_{di} = z \mid \mathbf{w}, \mathbf{t}, \mathbf{z}_{-di}, \alpha, \beta, \Psi)$
            Increment counts $m_{d, z}$ and $n_{z, w_{di}}$
        end for
    end for
    for $z = 1$ to $T$ do
        Calculate sample mean $\bar{t}_z$ and sample variance $s_z^2$ of timestamps assigned to topic $z$
         $\psi_{z1} \leftarrow \bar{t}_z (\frac{\bar{t}_z(1 - \bar{t}_z)}{s_z^2} - 1)$
         $\psi_{z2} \leftarrow (1 - \bar{t}_z) (\frac{\bar{t}_z(1 - \bar{t}_z)}{s_z^2} - 1)$
    end for
end for
Compute posterior estimates $\theta_{dz} = \frac{m_{dz} + \alpha_z}{\sum_{z'} (m_{dz'} + \alpha_{z'})}$ and $\phi_{zv} = \frac{n_{zv} + \beta_v}{\sum_{v'} (n_{zv'} + \beta_{v'})}$
Knowl 3 — Method-of-Moments Estimation of Topic Beta Parameters
equation
In the Topics over Time (TOT) model, the shape parameters $\psi_z = (\psi_{z1}, \psi_{z2})$ of the continuous Beta distribution over normalized timestamps $t \in [0, 1]$ for each topic $z \in \{1, \dots, T\}$ are updated at each iteration of collapsed Gibbs sampling via the method of moments.

Let $N_z = \sum_{d=1}^D m_{dz}$ be the total number of tokens assigned to topic $z$ . The sample mean $\bar{t}_z$ and biased sample variance $s_z^2$ of timestamps assigned to topic $z$ are computed as:

$\bar{t}_z = \frac{1}{N_z} \sum_{d=1}^D \sum_{i: z_{di} = z} t_{di}$

$s_z^2 = \frac{1}{N_z} \sum_{d=1}^D \sum_{i: z_{di} = z} (t_{di} - \bar{t}_z)^2$

The parameter updates are given by:

$\hat{\psi}_{z1} = \bar{t}_z \left( \frac{\bar{t}_z(1 - \bar{t}_z)}{s_z^2} - 1 \right)$

$\hat{\psi}_{z2} = (1 - \bar{t}_z) \left( \frac{\bar{t}_z(1 - \bar{t}_z)}{s_z^2} - 1 \right)$
Knowl 4 — Document Timestamp Prediction and Topic Mixture Inversion in TOT
model/method
The Topics over Time (TOT) model supports two operations involving continuous time:
1. Timestamp Prediction given Words: Given an unstamped document $d$ with $N_d$ word tokens and inferred topic assignments $\mathbf{z}_d = (z_{d1}, \dots, z_{dN_d})$ , its continuous timestamp $t \in [0, 1]$ is predicted by maximizing the likelihood under the per-topic Beta distributions:
$\hat{t} = \arg\max_{t} \prod_{i=1}^{N_d} p(t \mid \psi_{z_{di}}) = \arg\max_{t} \prod_{i=1}^{N_d} \frac{t^{\psi_{z_{di}1}-1} (1 - t)^{\psi_{z_{di}2}-1}}{B(\psi_{z_{di}1}, \psi_{z_{di}2})}$

In practice, this optimization can be evaluated over candidate discretized timestamps to identify the most probable time slice.
1. Topic Mixture Conditioned on Time: The expected mixture weight of topic $z$ at a specific continuous timestamp $t$ is calculated by inverting the generative model via Bayes' rule:
$E[\theta_z \mid t] = P(z \mid t) = \frac{p(t \mid z) P(z)}{\sum_{z'=1}^T p(t \mid z') P(z')} = \frac{\text{Beta}(t; \psi_{z1}, \psi_{z2}) P(z)}{\sum_{z'=1}^T \text{Beta}(t; \psi_{z'1}, \psi_{z'2}) P(z')}$

where $P(z)$ is the prior probability of topic $z$ across the corpus (often assumed uniform or estimated empirically from token counts).
Knowl 5 — Modality Balancing between Continuous Time and Discrete Words
model/method
In generative models combining continuous document-level features (such as timestamps) and discrete bag-of-words text, the likelihood contribution from a single continuous observation can be overpowered by the product of $N_d$ discrete word likelihoods.

To balance the influence of discrete text tokens and continuous time during inference, the temporal likelihood is scaled relative to the text likelihood. In Topics over Time (TOT), associating the document timestamp $t_d$ with every individual word token $w_{di}$ within document $d$ acts as generating $N_d$ identically distributed samples from the document's mixture of topic-specific Beta distributions. This is equivalent to applying an inverse document length scaling weight $1/N_d$ to the combined text likelihood relative to the time likelihood, ensuring that timestamp evidence effectively guides topic formation.

Knowl 6 — Topic Distinctness across Temporal Datasets

data/table

Topics over Time (TOT) produces topic word distributions that are more distinct from one another than standard Latent Dirichlet Allocation (LDA), as measured by the average pairwise symmetric Kullback-Leibler (KL) divergence between topic word distributions $\phi_z$ . Across three corpora with $T = 50$ topics:

Because Beta temporal distributions penalize multi-modal time spans, TOT separates distinct historical events that share vocabulary into separate, tighter topics, preventing the conflation of temporally distant phenomena.

Knowl 7 — Decade Timestamp Prediction Performance on State-of-the-Union Addresses

data/table

Timestamp prediction accuracy was evaluated on the US Presidential State-of-the-Union Address corpus (spanning 1790 to 2002, partitioned into 6,427 three-paragraph documents). Given only the text of an address segment, the model predicts the decade in which it was delivered. Error was measured by decade classification accuracy, average L1 distance to the correct decade (number of decades off), and expected L1 error $E(L1)$ :

TOT nearly doubles the exact decade prediction accuracy of LDA (0.19 vs. 0.10) and achieves an approximate 20% relative reduction in L1 error.

Knowl 8 — Temporal Event Disambiguation in TOT versus LDA
empirical result
When applied to corpora spanning long time horizons, Latent Dirichlet Allocation (LDA) confounds temporally separated events that share common vocabulary, whereas Topics over Time (TOT) isolates events into narrow, temporally localized topics:
1. US Presidential Addresses (1790–2002): LDA blends the Mexican-American War (1846–1848) with World War I (1914–1918) due to shared military terms. TOT correctly isolates the Mexican-American War to 1846–1850 and creates distinct topics for World War I, the Cold War, and the Panama Canal construction (1904–1914).
2. NIPS Proceedings (1987–2003): LDA mixes Recurrent Neural Networks with Markov models into a single broad dynamical systems topic. TOT cleanly separates the waning of Recurrent Neural Networks in the 1990s from the subsequent rise of Markov models.
3. Email Archive (9 months): LDA merges seasonal tasks (such as spring faculty recruiting) into general faculty communication. TOT identifies localized seasonal topics and distinguishes toolkit development (MALLET) from general repository usage (CVS).
Knowl 9 — Continuous-Time Dynamic Topic Co-occurrence Tracking
model/method
TOT enables tracking how topic co-occurrence patterns evolve across time while maintaining static topic word vocabularies. Two topics $z_1$ and $z_2$ are defined to strongly co-occur in document $d$ if both of their inferred mixture weights exceed a specified threshold $h$ :

$\theta_{d, z_1} > h \quad \text{and} \quad \theta_{d, z_2} > h$

with $h$ typically set to $2/T$ where $T$ is the total topic count. Counting the frequency of documents meeting this criterion across continuous time reveals dynamic structural shifts in a field. For instance, in NIPS conference papers, the topic "classification" strongly co-occurred with "neural network learning" and "digit recognition" in the late 1980s and early 1990s, but shifted to co-occur with "support vector machines (SVMs)", "boosting", and "probabilistic mixture models" in later years.

Coverage note — None was omitted; all primary models, mathematical formulations, inference procedures, prediction mechanics, and experimental evaluations were converted into standalone knowls.

References

1.C. Andrieu, N. de Freitas, A. Doucet, and M. Jordan. An introduction to MCMC for machine learning. Machine Learning, 50:5–43, 2003.
2.D. Blei, A. Ng, and M. Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993–1022, 2003.
3.D. M. Blei and J. D. Lafferty. Dynamic topic models. In Proceedings of the 23rd International Conference on Machine Learning, 2006.
4.E. Erosheva, S. Fienberg, and J. Lafferty. Mixed membership models of scientific publications. Proceedings of the National Academy of Sciences, 101(Suppl. 1), 2004.
5.T. Griffiths and M. Steyvers. Finding scientific topics. Proceedings of the National Academy of Sciences, 101(suppl. 1):5228–5235, 2004.
6.T. Griffiths, M. Steyvers, D. Blei, and J. Tenenbaum. Integrating topics and syntax. In Advances in Neural Information Processing Systems (NIPS) 17, 2004.
7.J. Kleinberg. Bursty and hierarchical structure in streams. In Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2002.
8.P. Kumaraswamy. A generalized probability density function for double-bounded random processes. Journal of Hydrology, 46:79–88, 1980.
9.R. E. Madsen, D. Kauchak, and C. Elkan. Modeling word burstiness using the Dirichlet distribution. In Proceedings of the 22nd International Conference on Machine Learning, 2005.
10.A. McCallum, A. Corrada-Emanuel, and X. Wang. Topic and role discovery in social networks. In Proceedings of 19th International Joint Conference on Artificial Intelligence, 2005.
11.U. Nodelman, C. Shelton, and D. Koller. Continuous time Bayesian networks. In Proceedings of the 18th Conference on Uncertainty in Artificial Intelligence, pages 378–387, 2002.
12.M. Rosen-Zvi, T. Griffiths, M. Steyvers, and P. Smyth. The author-topic model for authors and documents. In Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence, 2004.
13.P. Sarkar and A. Moore. Dynamic social network analysis using latent space models. In The 19th Annual Conference on Neural Information Processing Systems, 2005.
14.X. Song, C.-Y. Lin, B. L. Tseng, and M.-T. Sun. Modeling and predicting personal information dissemination behavior. In Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2005.
15.R. Swan and D. Jensen. Timemines: Constructing timelines with statistical models of word usage. In The 6th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining Workshop on Text Mining, pages 73–80, 2000.
16.Y. W. Teh, M. I. Jordan, M. J. Beal, and D. M. Blei. Hierarchical Dirichlet processes. Technical report, UC Berkeley Statistics TR-653, 2004.
17.X. Wang and A. McCallum. A note on topical n-grams. Technical report, UMass UM-CS-2005-071, 2005.
18.X. Wang, N. Mohanty, and A. McCallum. Group and topic discovery from relations and text. In The 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining Workshop on Link Discovery: Issues, Approaches and Applications, pages 28–35, 2005.

Citation

MLA

Wang, X., and A. McCallum. “Topics over Time”. Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2006, pp. 424–33, https://doi.org/10.1145/1150402.1150450.

APA

Wang, X., & McCallum, A. (2006). Topics over time. Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 424–433. https://doi.org/10.1145/1150402.1150450

Chicago

Wang, X., and A. McCallum. 2006. “Topics over Time”. Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 424–33. https://doi.org/10.1145/1150402.1150450.

Harvard

Wang, X. and McCallum, A. (2006) “Topics over time”, Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, pp. 424–433. Available at: https://doi.org/10.1145/1150402.1150450.

Vancouver

1. Wang X, McCallum A (2006) Topics over time. In: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, pp 424–433

BibTeX

@inproceedings{Wang_2006, series={KDD06}, title={Topics over time: a non-Markov continuous-time model of topical trends}, url={http://dx.doi.org/10.1145/1150402.1150450}, DOI={10.1145/1150402.1150450}, booktitle={Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining}, publisher={ACM}, author={Wang, Xuerui and McCallum, Andrew}, year={2006}, month=Aug, pages={424–433}, collection={KDD06} }

Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Model	State-of-the-Union Addresses	Email	NIPS Papers
TOT	0.6266	0.6416	0.5728
LDA	0.5965	0.5943	0.5421

Model	L1 Error (decades)	E(L1)	Accuracy
TOT	1.98	2.02	0.19
LDA	2.51	2.58	0.10

Topics over time: a non-Markov continuous-time model of topical trends

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Topics over Time Generative Model

Knowl 2 — Collapsed Gibbs Sampling Inference for Topics over Time

Knowl 3 — Method-of-Moments Estimation of Topic Beta Parameters

Knowl 4 — Document Timestamp Prediction and Topic Mixture Inversion in TOT

Knowl 5 — Modality Balancing between Continuous Time and Discrete Words