Elamur Research
☰
home / papers / 2015 / Exploring the Space of Topic Coherence Measures

Exploring the Space of Topic Coherence Measures

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "author copy (svn.aksw.org) — read only" does not allow redistributing the text itself. Original — svn.aksw.org/papers/2015/WSDM Topic Evaluation/public.pdf

Summary

Topic models generate word lists for topics, but nothing guarantees that people can interpret them. Automatic coherence scores are therefore needed to judge topic quality. Röder, Both and Hinneburg show that earlier proposals can all be broken into a few interchangeable components. These proposals include UCI, UMass, the context-vector method of Aletras and Stevenson, and set-based coherences from scientific philosophy (Shogenji, Olsson, Fitelson). The components are:

  • how the top-word set is divided into subset pairs (segmentation),
  • how word probabilities are estimated from a reference corpus,
  • which confirmation measure relates the subsets,
  • how the resulting values are aggregated.

Recombining these parts produces a large configuration space. The authors search it exhaustively, scoring each candidate by its correlation with human topic ratings from all publicly available benchmarks. Several combinations never tried before outperform every published measure, and the strongest is named C_V. The study offers a shared, reproducible basis for comparing coherence measures. It also suggests uses beyond topic modelling, such as filtering search hits, matching ads to pages and picking among ambiguous dictionary translations.

Key points

  • In total, 237,912 coherence measures were tested on six benchmarks. The authors believe this is the largest benchmark set used for topic coherence so far. Human ratings come from three sources: Chang et al. (prepared by Lau et al.), Aletras and Stevenson, and Rosner et al.
  • The framework has four parts: segmentation, probability estimation, confirmation measure and aggregation. Segmentation determines how many confirmation values must be computed and therefore drives runtime. Schemes that compare a word against all subsets of the other words grow exponentially with the number of top words N.
  • Fitelson's coherence, borrowed from scientific philosophy and new to topic evaluation, performed well.
  • Data and tools for reproducing the calculations are released as Palmetto (github.com/AKSW/Palmetto).
  • In a demonstration, 200 LDA topics learned from English Wikipedia were ranked with C_V. Scores ran from 0.94 for clearly interpretable topics down to 0.27 for incoherent ones. The authors note that systems like TopicExplorer can use such scores to hide topics users would find confusing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

141 papers of our library cite it (of the 140673 whose bibliographies we read).

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1145/2684822.2685324

@inproceedings{roder2015exploring,
  title = {{Exploring the Space of Topic Coherence Measures}},
  author = {Röder, Michael and Both, Andreas and Hinneburg, Alexander},
  booktitle = {WSDM 2015},
  year = {2015},
  doi = {10.1145/2684822.2685324},
  url = {https://doi.org/10.1145/2684822.2685324}
}

References

18 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. N. Aletras and M. Stevenson. Evaluating topic coherence using distributional semantics. In Proc. of the 10th Int. Conf. on Computational Semantics (IWCS’13), pages 13–22, 2013. find
  2. L. Alsumait, D. Barbará, J. Gentle, and C. Domeniconi. Topic significance ranking of lda generative models. In Proc. of the Europ. Conf. on Machine Learning and Knowledge Disc. in Databases: Part I, ECML PKDD ’09, pages 67–82, 2009. find
  3. D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet allocation. Journal of Machine Learning Research, 3:993–1022, 2003. find our card
  4. G. Bouma. Normalized (pointwise) mutual information in collocation extraction. In From Form to Meaning: Processing Texts Automatically, Proc. of the Biennial GSCL Conference 2009, pages 31–40, Tübingen, 2009. find
  5. L. Bovens and S. Hartmann. Bayesian Epistemology. Oxford University Press, 2003. find
  6. J. Chang, S. Gerrish, C. Wang, J. L. Boyd-graber, and D. M. Blei. Reading tea leaves: How humans interpret topic models. In Advances in Neural Information Processing Systems 22, pages 288–296. 2009. find our card
  7. I. Douven and W. Meijs. Measuring coherence. Synthese, 156(3):405–425, 2007. find
  8. B. Fitelson. A probabilistic theory of coherence. Analysis, 63(279):194–199, 2003. find
  9. A. Hinneburg, R. Preiss, and R. Schröder. Topicexplorer: Exploring document collections with topic models. In ECML/PKDD (2), pages 838–841, 2012. find
  10. J. H. Lau, D. Newman, and T. Baldwin. Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In Proc. of the Europ. Chap. of the Assoc. for Comp. Ling., 2014. find our card
  11. D. Mimno and D. Blei. Bayesian checking for topic models. In Proc. of the Conf. on Empirical Methods in Natural Language Processing, pages 227–237. 2011. find
  12. D. Mimno, H. M. Wallach, E. Talley, M. Leenders, and A. McCallum. Optimizing semantic coherence in topic models. In Proc. of the Conf. on Empirical Methods in Natural Language Processing, pages 262–272. 2011. find our card
  13. D. Newman, J. H. Lau, K. Grieser, and T. Baldwin. Automatic evaluation of topic coherence. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 100–108. 2010. find our card
  14. E. Olsson. What is the problem of coherence and truth? The Jour. of Philosophy, 99(5):246–272, 2002. find
  15. M. Röder, M. Speicher, and R. Usbeck. Investigating quality raters’ performance using interface evaluation methods. In Informatik 2013, pages 137–139. GI, 2013. find
  16. F. Rosner, A. Hinneburg, M. Röder, M. Nettling, and A. Both. Evaluating topic coherence measures. CoRR, abs/1403.6397, 2014. — [arXiv:1403.6397](https://arxiv.org/abs/1403.6397) find
  17. T. Shogenji. Is coherence truth conducive? Analysis, 59(264):338–345, 1999. find
  18. K. Stevens, P. Kegelmeyer, D. Andrzejewski, and D. Buttler. Exploring topic coherence over many models and many topics. In Proc. of the 2012 Joint Conf. on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL ’12, pages 952–961, 2012. find