Elamur Research
☰
home / papers / 2010 / Automatic Evaluation of Topic Coherence

Automatic Evaluation of Topic Coherence

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/N10-1012.pdf

Summary

The paper proposes topic coherence evaluation as a new task: judging whether the top words of a learned topic form an interpretable whole that relates to one concept. Until then, topic models were mostly assessed by perplexity or by downstream performance, neither of which shows whether people can make sense of the topics. The authors trained LDA (collapsed Gibbs sampling) on a news collection (NEWS) and a book collection (BOOKS) and represented each topic by its ten most probable words. Human annotators then rated these topics for coherence. The authors compared those ratings with many automatic scorers built on WordNet, Wikipedia and Google. Most scorers compare all word pairs in a topic and combine the pair scores with a mean or median. A simple pointwise mutual information (PMI) measure, using word co-occurrence counts from Wikipedia, matched human judgments about as well as the annotators matched each other. This makes cheap, automatic, human-aligned evaluation of topic quality feasible.

Key points

  • Setup: Each topic is represented by its top-10 words, which gives 45 word pairs per topic. Pair scores are aggregated by mean or median, and neither aggregation was clearly better.
  • Evaluation: Scorers were compared with gold-standard scores from nine annotators using Spearman ρ. The upper bound was inter-annotator agreement, computed by comparing each annotator with the mean or median of the remaining annotators.
  • Wikipedia (most robust): Wikipedia-based PMI was the best or near-best method on both datasets, reaching or exceeding inter-annotator agreement. It was followed by MIW, RACO and DocSim.
  • Google: The Google TITLES method slightly exceeded inter-annotator agreement on NEWS but dropped sharply on BOOKS. The authors attribute this to the BOOKS topics containing fewer proper names, more varied word classes, and a genre that is less represented on the web.
  • WordNet (weak and uneven): The WordNet::Similarity measures (PATH, LCH, WUP, HSO, JCN, LIN, RES, LESK, VECTOR) performed poorly overall and worked better on BOOKS. JCN and LCH, which are strong on standard lexical similarity tasks, did very badly here. LESK was the most consistent WordNet method.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

115 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 112 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{newman2010automatic,
  title = {{Automatic Evaluation of Topic Coherence}},
  author = {Newman, David and Lau, Jey Han and Grieser, Karl and Baldwin, Timothy},
  booktitle = {NAACL-HLT 2010},
  year = {2010},
  url = {https://aclanthology.org/N10-1012/}
}

References

35 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. E Agirre, E Alfonseca, K Hall, J Kravalova, M Paşca, and A Soroa. 2009. A study on similarity and relatedness using distributional and WordNet-based approaches. In Proc. of HLT: NAACL 2009, pages 19– 27, Boulder, Colorado. find
  2. S Banerjee and T Pedersen. 2002. An adapted Lesk algorithm for word sense disambiguation using WordNet. Proc. of CICLing’02, pages 136–145. find
  3. DM Blei, AY Ng, and MI Jordan. 2003. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993–1022. find our card
  4. S Brody and M Lapata. 2009. Bayesian word sense induction. In Proc. of EACL 2009, pages 103–111, Athens, Greece. find
  5. A Budanitsky and G Hirst. 2005. Evaluating WordNet-based Measures of Lexical Sematic Relatedness. Computational Linguistics, 32(1):13–47. find
  6. WL Buntine and A Jakulin. 2004. Applying discrete PCA in data analysis. In Proc. of UAI 2004, pages 59–66. find
  7. J Chang, J Boyd-Graber, S Gerris, C Wang, and D Blei. 2009. Reading tea leaves: How humans interpret topic models. In Proc. of NIPS 2009. find our card
  8. H Daume III. 2009. Non-parametric bayesian areal linguistics. In Proc. of HLT: NAACL 2009, pages 593– 601, Boulder, USA. find
  9. Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society of Information Science, 41(6). find
  10. C Fellbaum, editor. 1998. WordNet: An Electronic Lexical Database. MIT Press, Cambridge, USA. find
  11. E Gabrilovich and S Markovitch. 2007. Computing semantic relatedness using Wikipedia-based explicit semantic analysis. In Proc. of IJCAI’07, pages 1606– 1611, Hyderabad, India. find
  12. T Griffiths and M Steyvers. 2004. Finding scientific topics. In Proc. of the National Academy of Sciences, volume 101, pages 5228–5235. find
  13. T Griffiths and M Steyvers. 2006. Probabilistic topic models. In Latent Semantic Analysis: A Road to Meaning. find
  14. A Haghighi and L Vanderwende. 2009. Exploring content models for multi-document summarization. In Proc. of HLT: NAACL 2009, pages 362–370, Boulder, USA. find
  15. G Hirst and D St-Onge. 1998. Lexical chains as representations of context for the detection and correction of malapropism. In Fellbaum (Fellbaum, 1998), pages 305–332. find
  16. T Hofmann. 2001. Unsupervised learning by probabilistic latent semantic analysis. Machine Learning, 42(1):177–196. find
  17. JJ Jiang and DW Conrath. 1997. Semantic similarity based on corpus statistics and lexical taxonomy. In Proc. of COLING’97, pages 19–33, Taipei, Taiwan. find
  18. C Leacock, G A Miller, and M Chodorow. 1998. Using corpus statistics and WordNet relations for sense identification. Computational Linguistics, 24(1):147–65. find
  19. M Lesk. 1986. Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from an ice cream cone. In Proc. of SIGDOC’86, pages 24–26, Toronto, Canada. find
  20. D Lin. 1998. Automatic retrieval and clustering of similar words. In Proc. of COLING/ACL’98, pages 768– 774, Montreal, Canada. find
  21. C-Y Lin. 2004. ROUGE: a package for automatic evaluation of summaries. In Proc. of the ACL 2004 Workshop on Text Summarization Branches Out (WAS 2004), pages 74–81, Barcelona, Spain. find
  22. Q Mei, X Shen, and CX Zhai. 2007. Automatic labeling of multinomial topic models. In Proc. of KDD 2007, pages 490–499. find
  23. D Milne and IH Witten. 2008. An effective, low-cost measure of semantic relatedness obtained from Wikipedia links. In Proc. of AAAI Workshop on Wikipedia and Artificial Intelligence, pages 25–30, Chicago, USA. find
  24. H Misra, O Cappe, and F Yvon. 2008. Using LDA to detect semantically incoherent documents. In Proc. of CoNLL 2008, pages 41–48, Manchester, England. find
  25. D Newman, S Karimi, and L Cavedon. 2009. External evaluation of topic models. In Proc. of ADCS 2009, pages 11–18, Sydney, Australia. find
  26. D Newman, T Baldwin, L Cavedon, S Karimi, D Martinez, and J Zobel. to appeara. Visualizing document collections and search results using topic mapping. Journal of Web Semantics. D Newman, Y Noh, E Talley, S Karimi, and T Baldwin. to appearb. Evaluating topic models for digital libraries. In Proc. of… find
  27. K Papineni, S Roukos, T Ward, and W-J Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proc. of ACL 2002, pages 311–318, Philadelphia, USA. find our card
  28. P Pecina. 2008. Lexical Association Measures: Collocation Extraction. Ph.D. thesis, Charles University. find
  29. P Resnik. 1995. Using information content to evaluate semantic similarity in a taxonomy. In Proc. of IJ-CAI’95, pages 448–453, Montreal, Canada. find our card
  30. H Schütze. 1998. Automatic word sense discrimination. Computational Linguistics, 24(1):97–123. find
  31. M Strübe and SP Ponzetto. 2006. WikiRelate! computing semantic relateness using Wikipedia. In Proc. of AAAI’06, pages 1419–1424, Boston, USA. find
  32. Q Sun, R Li, D Luo, and X Wu. 2008. Text segmentation with LDA-based Fisher kernel. In Proc. of ACL-08: HLT, pages 269–272. find
  33. HM Wallach, I Murray, R Salakhutdinov, and DM Mimno. 2009. Evaluation methods for topic models. In Proc. of ICML 2009, page 139. find
  34. D Widdows and K Ferraro. 2008. Semantic Vectors: A scalable open source package and online technology management application. In Proc. of LREC 2008, Marrakech, Morocco. find
  35. Z Wu and M Palmer. 1994. Verb selection and lexical selection. In Proc. of ACL’94, pages 133–138, Las Cruces, USA. find