Elamur Research
☰
home / papers / 2006 / Dynamic Topic Models

Dynamic Topic Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "author copy (cs.columbia.edu) — read only" does not allow redistributing the text itself. Original — cs.columbia.edu~blei/papers/BleiLafferty2006a.pdf

Summary

Classic topic models such as LDA treat a collection as an unordered bag of documents. That ignores the fact that archives like journals, news and email change their content over time. Blei and Lafferty propose dynamic topic models (DTM), which split a corpus into time slices (e.g., years) and let each of K topics drift gradually from one slice to the next. They do not use Dirichlet priors, because these do not chain well over time. Instead they place Gaussian state space dynamics on the natural (log-ratio) parameters of the topic multinomials, and also on the mean of a logistic normal prior over per-document topic proportions. Exact posterior inference is intractable because the model is nonconjugate. The authors therefore derive variational approximations in which the variational parameters act as pseudo-observations, smoothed either by a Kalman filter or by nonparametric wavelet regression. They apply the model to OCR'ed articles of the journal Science from 1880 to 2000. DTM predicts next-year documents better than static LDA baselines, and it also gives an interpretable view of how scientific themes and vocabulary shifted.

Key points

  • Time dynamics: Topic parameters follow a random walk, β_t,k ~ N(β_t−1,k, σ²I), and the proportion mean evolves the same way, α_t ~ N(α_t−1, δ²I). Topic correlation dynamics are left out.
  • Inference: Two variational schemes are proposed, variational Kalman filtering and variational wavelet regression. The wavelet method is illustrated with the Haar basis, while the experiments use Daubechies wavelets. Document-level updates follow LDA-style coordinate ascent, and topic-level parameters are fit with conjugate gradient.
  • Data and qualitative results: The corpus is the Science archive (1880–2000). A 20-topic model shows each topic's top words at ten-year intervals and traces how individual word probabilities change over time.
  • Prediction experiment: For every 5th year from 1900 to 2000, the models were trained on all earlier articles and evaluated on the following year's documents using a variational upper bound on their negative log likelihood. DTM was compared against LDA-prev (previous year only) and LDA-all (all prior articles) and showed better predictive accuracy.
  • Future work: The authors propose autoregressive drift terms, heteroscedastic noise, and birth–death or Galton-Watson processes so that topics can appear and disappear over time.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

119 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 117 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1145/1143844.1143859

@inproceedings{blei2006dynamic,
  title = {{Dynamic Topic Models}},
  author = {Blei, David M. and Lafferty, John D.},
  booktitle = {ICML 2006},
  year = {2006},
  doi = {10.1145/1143844.1143859},
  url = {https://doi.org/10.1145/1143844.1143859}
}

References

15 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aitchison, J. (1982). The statistical analysis of compositional data. Journal of the Royal Statistical Society, Series B, 44(2):139–177. find
  2. Blei, D., Ng, A., and Jordan, M. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993–1022. find our card
  3. Blei, D. M. and Lafferty, J. D. (2006). Correlated topic models. In Weiss, Y., Schölkopf, B., and Platt, J., editors, Advances in Neural Information Processing Systems 18. MIT Press, Cambridge, MA. find
  4. Buntine, W. and Jakulin, A. (2004). Applying discrete PCA in data analysis. In Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence, pages 59–66. AUAI Press. find
  5. Erosheva, E. (2002). Grade of membership and latent structure models with application to disability survey data. PhD thesis, Carnegie Mellon University, Department of Statistics. find
  6. Fei-Fei, L. and Perona, P. (2005). A Bayesian hierarchical model for learning natural scene categories. IEEE Computer Vision and Pattern Recognition. find
  7. Griffiths, T. and Steyvers, M. (2004). Finding scientific topics. Proceedings of the National Academy of Science, 101:5228–5235. find
  8. Kalman, R. (1960). A new approach to linear filtering and prediction problems. Transaction of the AMSE: Journal of Basic Engineering, 82:35–45. find
  9. McCallum, A., Corrada-Emmanuel, A., and Wang, X. (2004). The author-recipient-topic model for topic and role discovery in social networks: Experiments with Enron and academic email. Technical report, University of Massachusetts, Amherst. find
  10. Pritchard, J., Stephens, M., and Donnelly, P. (2000). Inference of population structure using multilocus genotype data. Genetics, 155:945–959. find
  11. Rosen-Zvi, M., Griffiths, T., Steyvers, M., and Smith, P. (2004). The author-topic model for authors and documents. In Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence, pages 487–494. AUAI Press. find
  12. Sivic, J., Rusell, B., Efros, A., Zisserman, A., and Freeman, W. (2005). Discovering objects and their location in images. In International Conference on Computer Vision (ICCV 2005). find
  13. Snelson, E. and Ghahramani, Z. (2006). Sparse Gaussian processes using pseudo-inputs. In Weiss, Y., Schölkopf, B., and Platt, J., editors, Advances in Neural Information Processing Systems 18, Cambridge, MA. MIT Press. find
  14. Wasserman, L. (2006). All of Nonparametric Statistics. Springer. find
  15. West, M. and Harrison, J. (1997). Bayesian Forecasting and Dynamic Models. Springer. find