Elamur Research
☰
home / papers / 2014 / Machine Reading Tea Leaves: Automatically Evaluating Topic Coherence and Topic Model Quality

Machine Reading Tea Leaves: Automatically Evaluating Topic Coherence and Topic Model Quality

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/E14-1056.pdf

Summary

Topic models such as LDA are often shown directly to people, for example in the digital humanities. However, there is no agreed way to judge whether their topics make sense to humans. This paper compares two families of interpretability measures. The first is word intrusion, an indirect test from Chang et al. (2009) in which annotators try to spot an inserted foreign word. The second is observed coherence, a direct human rating of how well the top words belong together.

The authors automate word intrusion for the first time. They do this by ranking candidate words with SVM rank, using word-association features computed over large reference corpora. They also propose replacing PMI with normalised PMI (NPMI) in the coherence measure of Newman et al. (2010).

At the level of whole models, both approaches track human judgements closely. At the level of single topics, the best automatic coherence measures match human agreement, while automated word intrusion falls slightly short. The two human task formulations also diverge systematically. The authors release an open-source toolkit implementing all the measures, which gives practitioners a practical, validated way to evaluate topic quality.

Key points

  • Data: The study reuses the Chang et al. (2009) annotations, which cover two collections:
    • WIKI: 10,000 English Wikipedia articles.
    • NEWS: 8,447 New York Times articles from 1987–2007.
  • Models and topics: Topics come from LDA, PLSI and CTM, each with T = 50, 100 and 150. This gives 900 topics per collection, reduced to 876 for WIKI after removing corrupted topics.
  • Reference corpora: Lexical probabilities are estimated from:
    • NEWS-FULL: 1.2M NYT articles from 1994–2004, taken from the English Gigaword.
    • WIKI-FULL: 3.3M Wikipedia articles.
  • Automated word intrusion:
    • The features are PMI, two conditional-probability measures (CP1, CP2) and NPMI.
    • At the model level, correlation with human scores is highest when the reference corpus matches the topics' domain. The best value reported in Table 1 is a Pearson r of 0.947.
  • NPMI versus PMI: NPMI removes PMI's tendency to favour rare words.
  • Comparing the human tasks: Human word-intrusion and coherence scores agree only moderately at the topic level (r ≈ 0.6).
  • Toolkit: The code is available at github.com/jhlau/topic_interpretability.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

128 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 126 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/e14-1056

@inproceedings{lau2014machine,
  title = {{Machine Reading Tea Leaves: Automatically Evaluating Topic Coherence and Topic Model Quality}},
  author = {Lau, Jey Han and Newman, David and Baldwin, Timothy},
  booktitle = {Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {2014},
  doi = {10.3115/v1/e14-1056},
  url = {https://doi.org/10.3115/v1/e14-1056}
}

References

26 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. N. Aletras and M. Stevenson. 2013a. Evaluating topic coherence using distributional semantics. In Proceedings of the Tenth International Workshop on Computational Semantics (IWCS-10), pages 13–22, Potsdam, Germany. find
  2. N. Aletras and M. Stevenson. 2013b. Representing topics using images. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT 2013), pages 158–167, Atlanta, USA. find
  3. D. Blei and J. Lafferty. 2005. Correlated topic models. In Advances in Neural Information Processing Systems 17 (NIPS-05), pages 147–154, Vancouver, Canada. find
  4. D.M. Blei, A.Y. Ng, and M.I. Jordan. 2003. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993–1022. find our card
  5. L. Bolelli, Ş. Ertekin, and C.L. Giles. 2009. Topic and trend detection in text collections using Latent Dirichlet Allocation. In Proceedings of ECIR 2009, pages 776–780, Toulouse, France. find
  6. G. Bouma. 2009. Normalized (pointwise) mutual information in collocation extraction. In Proceedings of the Biennial GSCL Conference, pages 31–40, Potsdam, Germany. find
  7. J. Chang, J. Boyd-Graber, S. Gerrish, C. Wang, and D. Blei. 2009. Reading tea leaves: How humans interpret topic models. In Advances in Neural Information Processing Systems 21 (NIPS-09), pages 288–296, Vancouver, Canada. find our card
  8. A. Haghighi and L. Vanderwende. 2009. Exploring content models for multi-document summarization. In Proceedings of the North American Chapter of the Association for Computational Linguistics – Human Language Technologies 2009 (NAACL HLT 2009), pages 362–370. find
  9. D. Hall, D. Jurafsky, and C.D. Manning. 2008. Studying the history of ideas using topic models. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (EMNLP 2008), pages 363–371, Honolulu, USA. find
  10. M. Hall, P. Clough, and M. Stevenson. 2012. Evaluating the use of clustering for automatically organising digital library collections. In Proceedings of the Second International Conference on Theory and Practice of Digital Libraries, pages 323–334, Paphos, Cyprus. find
  11. T. Hofmann. 1999. Probabilistic latent semantic indexing. In Proceedings of 22nd International ACM-SIGIR Conference on Research and Development in Information Retrieval (SIGIR’99), pages 50–57, Berkeley, USA. find
  12. T. Joachims. 2006. Training linear SVMs in linear time. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2006), Philadelphia, USA. find
  13. J.H. Lau, D. Newman, S. Karimi, and T. Baldwin. 2010. Best topic word selection for topic labelling. In Proceedings of the 23rd International Conference on Computational Linguistics (COLING 2010), Posters Volume, pages 605–613, Beijing, China. find
  14. J.H. Lau, K. Grieser, D. Newman, and T. Baldwin. 2011. Automatic labelling of topic models. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL HLT 2011), pages 1536– 1545, Portland, USA. find
  15. J.H. Lau, N. Collier, and T. Baldwin. 2012a. Online trend analysis with topic models: \#twitter trends detection topic model online. In Proceedings of the 24th International Conference on Computational Linguistics (COLING 2012), pages 1519– 1534, Mumbai, India. find
  16. J.H. Lau, P. Cook, D. McCarthy, D. Newman, and T. Baldwin. 2012b. Word sense induction for novel sense detection. In Proceedings of the 13th Conference of the EACL (EACL 2012), pages 591–601, Avignon, France. find
  17. J.H. Lau, T. Baldwin, and D. Newman. 2013. On collocations and topic models. ACM Transactions on Speech and Language Processing, 10(3):10:1– 10:14. find
  18. A McCallum, G.S. Mann, and D Mimno. 2006. Bibliometric impact measures leveraging topic analysis. In Proceedings of the 6th ACM/IEEE-CS Joint Conference on Digital Libraries 2006 (JCDL’06), pages 65–74, Chapel Hill, USA. find
  19. D. Mimno, H. Wallach, E. Talley, M. Leenders, and A. McCallum. 2011. Optimizing semantic coherence in topic models. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP 2011), pages 262–272, Edinburgh, UK. find our card
  20. G. Minnen, J. Carroll, and D. Pearce. 2001. Applied morphological processing of English. Natural Language Engineering, 7(3):207–223. find
  21. C. Musat, J. Velcin, S. Trausan-Matu, and M.A. Rizoiu. 2011. Improving topic evaluation using conceptual knowledge. In Proceedings of the 22nd International Joint Conference on Artificial Intelligence (IJCAI-2011), pages 1866–1871, Barcelona, Spain. find
  22. D. Newman, J.H. Lau, K. Grieser, and T. Baldwin. 2010. Automatic evaluation of topic coherence. In Proceedings of Human Language Technologies: The 11th Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL HLT 2010), pages 100–108, Los Angeles, USA. find our card
  23. M. Paul and R. Girju. 2010. A two-dimensional topic-aspect model for discovering multi-faceted topics. In Proceedings of the 24th Annual Conference on Artificial Intelligence (AAAI-10), Atlanta, USA. find
  24. J. Reisinger, A. Waters, B. Silverthorn, and R.J. Mooney. 2010. Spherical topic models. In Proceedings of the 27th International Conference on Machine Learning (ICML 2010), pages 903–910, Haifa, Israel. find
  25. X. Wang, A. McCallum, and X. Wei. 2007. Topical n-grams: Phrase and topic discovery, with an application to information retrieval. In Proceedings of the Seventh IEEE International Conference on Data Mining (ICDM 2007), pages 697–702, Omaha, USA. find
  26. B. Zhao and E.P. Xing. 2007. HM-BiTAM: Bilingual topic exploration, word alignment, and translation. In Advances in Neural Information Processing Systems (NIPS 2007), pages 1689–1696, Vancouver, Canada. find