Elamur Research⌕
☰
home / papers / 2013 / Evaluating Topic Coherence Using Distributional Semantics

Evaluating Topic Coherence Using Distributional Semantics

Summary

The paper addresses how to automatically judge whether the top words of a topic produced by a topic model such as LDA form an interpretable, coherent set. Earlier work by Newman et al. (2010b) averaged pairwise PMI between topic words. The authors instead represent each topic word as a context vector built from co-occurrence statistics in Wikipedia, using a ±5-word window. Vectors are weighted with PMI or NPMI, raised to a power γ that boosts strongly associated features. Coherence is then the average pairwise similarity between vectors (cosine, Dice or Jaccard) or the mean cosine to the topic centroid. They test three vector spaces: a full space, a Reduced Semantic Space that keeps the most related features per word, and a Topic Word Space that uses only the topic's own words as features. To evaluate, they built a new public data set of 300 LDA topics with crowdsourced human ratings. The Topic Word Space correlated best with humans and consistently beat earlier methods. This offers a more reliable automatic check of topic quality.

Key points

  • The data set has 300 topics, each shown as its 10 most probable words, from three collections (100 per collection):
    • NYT: 47,229 GigaWord articles.
    • 20NG: 20 Newsgroups.
    • Genomics: 30,000 MEDLINE articles from TREC-Genomics.
  • Topics were generated with gensim LDA, and coherence ratings were collected via CrowdFlower.
  • Under Spearman correlation with human ratings, average pairwise PMI (Newman et al.) scored about 0.71 on NYT, 0.73 on 20NG and 0.75 on Genomics. Averaging NPMI instead added roughly 0.03 on every set.
  • Cosine similarity in the Topic Word Space with PMI weighting reached 0.76 (NYT), 0.79 (20NG) and 0.80 (Genomics), which is 0.05–0.06 above average PMI.
  • The co-document frequency measure of Mimno et al. (2011) did worst: −0.39 on NYT, 0.34 on 20NG and −0.4 on Genomics.
  • Cosine generally outperformed Dice and Jaccard, and centroid similarity was about as good as cosine. Polysemous context features hurt the Reduced Semantic Space on NYT and Genomics.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

63 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 61 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{aletras2013evaluating,
  title = {{Evaluating Topic Coherence Using Distributional Semantics}},
  author = {Aletras, Nikolaos and Stevenson, Mark},
  booktitle = {Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers},
  year = {2013},
  url = {https://aclanthology.org/W13-0102/}
}

References

25 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Loulwah AlSumait, Daniel Barbará, James Gentle, and Carlotta Domeniconi. Topic Significance Ranking of LDA Generative Models. Machine Learning and Knowledge Discovery in Databases, pages 67–82, 2009. find
  2. David Andrzejewski, Xiaojin Zhu, and Mark Craven. Incorporating Domain Knowledge into Topic Modeling via Dirichlet Forest Priors. In Proceedings of the International Conference on Machine Learning, pages 25–32, 2009. find
  3. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latent Dirichlet Allocation. Journal of Machine Learning Research, 3:993–1022, 2003. find our card
  4. G. Bouma. Normalized (pointwise) mutual information in collocation extraction. In Proceedings of the International Conference of the German Society for Computational Linguistics and Language Technology (GSCL ’09), Potsdam, Germany, 2009. find
  5. Jordan Boyd-Graber, David Blei, and Xiaojin Zhu. A topic model for word sense disambiguation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL ’07), pages 1024–1033, 2007. find
  6. Allison June-Barlow Chaney and David M. Blei. Visualizing topic models. In Proceedings of the 6th International AAAI Conference on Weblogs and Social Media (IWSCM), 2012. find
  7. Jonathan Chang, Jordan Boyd-Graber, and Sean Gerrish. Reading Tea Leaves: How Humans Interpret Topic Models. Neural Information, pages 1–9, 2009. find our card
  8. James R. Curran. From distributional to semantic similarity. Ph.D. Thesis, University of Edinburgh, 2003. find
  9. Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6): 391–407, 1990. find
  10. Gregory Grefenstette. Explorations in Automatic Thesaurus Discovery. Springer, 1994. find
  11. Aria Haghighi and Lucy Vanderwende. Exploring content models for multi-document summarization. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 362–370, Boulder, Colorado, 2009. find
  12. Zellig Sabbettai Harris. Distributional structure. Word, 10:146–162, 1954. find
  13. Thomas Hofmann. Probabilistic latent semantic indexing. In Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’99), pages 50–57, Berkeley, California, United States, 1999. find
  14. Aminul Md. Islam and Diana Inkpen. Second Order Co-occurrence PMI for Determining the Semantic Similarity of Words. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC ’06), pages 1033–1038, 2006. find
  15. Jey Han Lau, Karl Grieser, David Newman, and Timothy Baldwin. Automatic labelling of topic models. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1536–1545, Portland, Oregon, USA, June 2011. find
  16. David Mimno, Hanna Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. Optimizing semantic coherence in topic models. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 262–272, Edinburgh, Scotland, UK, 2011. find our card
  17. Claudiu C. Musat, Julien Velcin, Stefan Trausan-Matu, and Marian A. Rizoiu. Improving topic evaluation using conceptual knowledge. In Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence (IJCAI ’11), pages 1866–1871, Barcelona, Spain, 2011. find
  18. David Newman, Timothy Baldwin, Lawrence Cavedon, Eric Huang, Sarvnaz Karimi, David Martinez, Falk Scholer, and Justin Zobel. Visualizing search results and document collections using topic maps. Web Semantics: Science, Services and Agents on the World Wide Web, 8(23):169–175, 2010a. find
  19. David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. Automatic evaluation of topic coherence. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT ’10), pages 100–108, Los Angeles, California, 20… find our card
  20. Sebastian Padó and Mirella Lapata. Constructing semantic space models from parsed corpora. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 128–135, Sapporo, Japan, 2003. find
  21. Judita Preiss. Identifying comparable corpora using LDA. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 558–562, Montréal, Canada, 2012. find
  22. Eduardo H. Ramirez, Ramon Brena, Davide Magatti, and Fabio Stella. Topic Model Validation. Neurocomputing, 76(1):125–133, 2012. find
  23. Peter D. Turney and Patrick Pantel. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37:141–188, 2010. find
  24. Hanna M. Wallach, Iain Murray, Ruslan Salakhutdinov, and David Mimno. Evaluation Methods for Topic Models. In Proceedings of the 26th Annual International Conference on Machine Learning (ICML ’09), pages 1105–1112, Montreal, Quebec, Canada, 2009. find
  25. Xing Wei and W. Bruce Croft. LDA-based Document Models for Ad-hoc Retrieval. In Proceedings of the 29th annual international ACM SIGIR conference on Research and Development in Information Retrieval (SIGIR ’06), pages 178–185, 2006. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aletras-2013-evaluating-topic-coherence-distributional-semantics/