Elamur Research
☰
home / papers / 2011 / Optimizing Semantic Coherence in Topic Models

Optimizing Semantic Coherence in Topic Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/D11-1024.pdf

Summary

Topic models such as LDA often produce topics that domain experts immediately see as flawed, and this undermines trust. The problem is worst for the smallest topics in models with many topics. The authors worked with two NINDS experts on about 300,000 NIH grant and journal abstracts. The experts rated 148 of 500 LDA topics and labeled their failure types (chained, intruded, random, unbalanced). The authors then propose an intrinsic coherence score based on how often a topic's top words co-occur in documents of the training corpus itself. It needs neither human judges nor external resources such as Wikipedia, and it matches expert ratings well. Building on this score, they design a topic model that replaces LDA's standard Pólya (DCM) word distribution with a generalized Pólya urn, so that drawing one word can raise the probability of related words. The new model increases average coherence, improves the weakest topics and converges in far fewer Gibbs iterations. However, its final held-out likelihood falls below LDA's, and its reduction in bad topics is not statistically significant. The work offers a cheap, self-contained way to detect and reduce junk topics.

Key points

  • Expert annotation of 148 NIH topics: 90 good, 21 intermediate, 37 bad. Among flawed topics, 23 were chained, 21 intruded, 3 random and 15 unbalanced (multiple labels were allowed). The two experts never split good versus bad.
  • The coherence metric uses only document frequency and co-document frequency of top words from the modeled corpus. It flags problems, such as chained topics, that word-intrusion tests miss.
  • The generalized Pólya urn model reaches its final coherence level within about 50 Gibbs iterations. Each iteration costs roughly 2–3× LDA's time. Results held for T ∈ {100, 200, 300}, using 5-fold cross-validation with 3 initializations each.
  • In a blind expert re-evaluation at T = 200, coherence predicted bad topics with AUC 0.83 for the new model and 0.80 for LDA.
  • Bad topics made up 13.5% of the new model's topics versus 16.5% for LDA. This difference is not significant at p = 0.05.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

145 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 143 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mimno2011optimizing,
  title = {{Optimizing Semantic Coherence in Topic Models}},
  author = {Mimno, David and Wallach, Hanna and Talley, Edmund and Leenders, Miriam and McCallum, Andrew},
  booktitle = {EMNLP 2011},
  year = {2011},
  url = {https://aclanthology.org/D11-1024/}
}

References

16 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Loulwah AlSumait, Daniel Barbara, James Gentle, and Carlotta Domeniconi. 2009. Topic significance ranking of LDA generative models. In ECML. find
  2. David Andrzejewski, Xiaojin Zhu, and Mark Craven. 2009. Incorporating domain knowledge into topic modeling via Dirichlet forest priors. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 25–32. find
  3. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993–1022, January. find our card
  4. K.R. Canini, L. Shi, and T.L. Griffiths. 2009. Online inference of topics with latent Dirichlet allocation. In Proceedings of the 12th International Conference on Artificial Intelligence and Statistics. find
  5. Jonathan Chang, Jordan Boyd-Graber, Chong Wang, Sean Gerrish, and David M. Blei. 2009. Reading tea leaves: How humans interpret topic models. In Advances in Neural Information Processing Systems 22, pages 288–296. find our card
  6. Kenneth Church and Patrick Hanks. 1990. Word association norms, mutual information, and lexicography. Computational Linguistics, 6(1):22–29. find our card
  7. Gabriel Doyle and Charles Elkan. 2009. Accounting for burstiness in topic models. In ICML. find
  8. S. Geman and D. Geman. 1984. Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transaction on Pattern Analysis and Machine Intelligence 6, pages 721–741. find
  9. Thomas L. Griffiths and Mark Steyvers. 2004. Finding scientific topics. Proceedings of the National Academy of Sciences, 101(suppl. 1):5228–5235. find
  10. Matthew Hoffman, David Blei, and Francis Bach. 2010. Online learning for latent dirichlet allocation. In NIPS. find
  11. Hosan Mahmoud. 2008. Pólya Urn Models. Chapman & Hall/CRC Texts in Statistical Science. find
  12. Qiaozhu Mei, Xuehua Shen, and ChengXiang Zhai. 2007. Automatic labeling of multinomial topic models. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 490–499. find
  13. David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. 2010. Automatic evaluation of topic coherence. In Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics. find our card
  14. Yee Whye Teh, Dave Newman, and Max Welling. 2006. A collapsed variational Bayesian inference algorithm for lat ent Dirichlet allocation. In Advances in Neural Information Processing Systems 18. find
  15. Hanna Wallach, Iain Murray, Ruslan Salakhutdinov, and David Mimno. 2009. Evaluation methods for topic models. In Proceedings of the 26th Interational Conference on Machine Learning. find
  16. Xing Wei and Bruce Croft. 2006. LDA-based document models for ad-hoc retrival. In Proceedings of the 29th Annual International SIGIR Conference. find