Elamur Research
☰
home / papers / 2009 / Reading Tea Leaves: How Humans Interpret Topic Models

Reading Tea Leaves: How Humans Interpret Topic Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "NeurIPS proceedings — open access, read only" does not allow redistributing the text itself. Original — proceedings.neurips.cc/paper files/paper/2009/file/f92586a25bb3145facd64ab20fd554ff-Paper.pdf

Summary

Topic models are often valued because their latent topics look meaningful to people, yet evaluations usually rely on perplexity or held-out likelihood, which ignore whether humans can interpret the topics. Chang et al. address this gap with two human-judgment tasks. In word intrusion, subjects see the five most probable words of a topic plus one "intruder" and must find the odd word. The intruder is unlikely in that topic but likely in another. In topic intrusion, subjects read a document's title and opening sentences and pick which of four topics does not belong: three are the model's top topics for the document and one is a random low-probability topic. Agreement is scored as model precision and topic log odds. The authors ran large-scale user studies on pLSI, LDA and CTM, trained on New York Times and Wikipedia articles with several topic counts. They found that better predictive likelihood often goes with less interpretable topics, and CTM fared worst for humans. This challenges likelihood-driven model selection and supports evaluating topic models on human-centered criteria.

Key points

  • Models and data: pLSI, LDA and CTM were compared at 50, 100 and 150 topics, on two corpora (New York Times and Wikipedia).
  • Two metrics:
    • Model precision is the share of subjects who spot the true intruding word.
    • Topic log odds (TLO) compares the model's probability for the true intruder topic with its probability for the subject's choice. Its maximum value is 0.
  • Main result: CTM usually had the best predictive likelihood but scored lowest on both human tasks, while LDA and pLSI generally did better. Regression lines of the human scores against likelihood slope downward.
  • Lexical correlates: model precision correlated negatively with the WordNet sense count of the displayed words (Spearman about −0.235) and slightly positively with their Jiang-Conrath similarity (about 0.109).
  • Per-document TLO examples: documents about narrow topics such as "Lindy Hop" score high, while broad, multi-theme articles such as "Book" score low.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

215 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 208 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{chang2009reading,
  title = {{Reading Tea Leaves: How Humans Interpret Topic Models}},
  author = {Chang, Jonathan and Boyd-Graber, Jordan and Gerrish, Sean and Wang, Chong and Blei, David M.},
  booktitle = {NeurIPS 2009 (NIPS 22)},
  year = {2009}
}

References

25 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Blei, D., J. Lafferty. Text Mining: Theory and Applications, chap. Topic Models. Taylor and Francis, 2009. find
  2. Mimno, D., A. Mccallum. Organizing the OCA: learning faceted subjects from a library of digital books. In JCDL. 2007. find
  3. Hofmann, T. Probabilistic latent semantic analysis. In UAI. 1999. find
  4. Blei, D., A. Ng, M. Jordan. Latent Dirichlet allocation. JMLR, 3:993–1022, 2003. find our card
  5. Blei, D. M., J. D. Lafferty. Correlated topic models. In NIPS. 2005. find
  6. Landauer, T., S. Dumais. Solutions to Plato’s problem: The latent semantic analsyis theory of acquisition, induction, and representation of knowledge. Psychological Review, 2(104):211–240, 1997. find
  7. Landauer, T. K. On the computational basis of learning and cognition: Arguments from LSA. The Psychology of Learning and Motivation, 41:43–84, 2002. find
  8. Deerwester, S., S. Dumais, T. Landauer, et al. Indexing by latent semantic analysis. Journal of the American Society of Information Science, 41(6):391–407, 1990. find
  9. Berry, M. W., S. T. Dumais, T. A. Letsche. Computational methods for intelligent information access. In Supercomputing. 1995. find
  10. Titov, I., R. McDonald. A joint model of text and aspect ratings for sentiment summarization. In HLT. 2008. find
  11. Wei, X., B. Croft. LDA-based document models for ad-hoc retrieval. In SIGIR. 2006. find
  12. Boyd-Graber, J. L., D. M. Blei, X. Zhu. Probabalistic walks in semantic hierarchies as a topic model for WSD. In HLT. 2007. find
  13. Wallach, H. M., I. Murray, R. Salakhutdinov, et al. Evaluation methods for topic models. In ICML. 2009. find
  14. Griffiths, T., M. Steyvers. Probabilistic topic models. In T. Landauer, D. McNamara, S. Dennis, W. Kintsch, eds., Latent Semantic Analysis: A Road to Meaning. Laurence Erlbaum, 2006. find
  15. Hall, D., D. Jurafsky, C. D. Manning. Studying the history of ideas using topic models. In EMNLP. 2008. find
  16. Mei, Q., X. Shen, C. Zhai. Automatic labeling of multinomial topic models. In KDD. 2007. find
  17. Erosheva, E., S. Fienberg, J. Lafferty. Mixed-membership models of scientific publications. PNAS, 101(Suppl 1):5220 — 5227, 2004. find
  18. Dempster, A., N. Laird, D. Rubin, et al. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B, 39(1):1–38, 1977. find
  19. Schmid, H. Probabilistic part-of-speech tagging using decision trees. In Proceedings of International Conference on New Methods in Language Processing. 1994. find
  20. Loper, E., S. Bird. NLTK: the natural language toolkit. In Proceedings of the ACL-02 Workshop on Effective tools and methodologies for teaching natural language processing and computational linguistics. 2002. find
  21. Teh, Y. W., K. Kurihara, M. Welling. Collapsed variational inference for HDP. In NIPS. 2008. find
  22. Snow, R., B. O’Connor, D. Jurafsky, et al. Cheap and fast—but is it good? evaluating non-expert annotations for natural language tasks. In EMNLP. 2008. find
  23. Deng, J., W. Dong, R. Socher, et al. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR. 2009. find
  24. Miller, G. A. Nouns in WordNet: A lexical inheritance system. International Journal of Lexicography, 3(4):245–264, 1990. find
  25. Jiang, J. J., D. W. Conrath. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings on International Conference on Research in Computational Linguistics. 1997. find