Elamur Research⌕
☰
home / papers / 2015 / Improving Topic Models with Latent Feature Word Representations

Improving Topic Models with Latent Feature Word Representations

Paper · 2015 · Dat Quoc Nguyen, Richard Billingsley, Lan Du et al.

Summary

Standard topic models such as LDA learn topics only from word co-occurrence within the training collection. When that collection is small or consists of short texts like tweets or news titles, the topics are estimated from very little evidence. Nguyen et al. address this by bringing in knowledge from huge external corpora through pre-trained word embeddings. They build two models, LF-LDA and LF-DMM, on top of LDA and the one-topic-per-document DMM. In both, each word comes from a mixture. With probability λ it is drawn from a latent feature component, which scores vocabulary items by the dot product of a learned topic vector and a fixed pre-trained word vector. Otherwise it is drawn from the usual Dirichlet multinomial. Inference combines Gibbs sampling with MAP estimation of the topic vectors. The authors test both Google word2vec and Stanford GloVe embeddings. Across six datasets, the new models beat their baselines on topic coherence and generally improve document clustering and classification. The gains are largest for small or short-text corpora, which shows that embeddings can offset data sparsity in topic modeling.

Key points

  • Models: LF-LDA and LF-DMM keep the structure of LDA and DMM. They replace only the topic-to-word step with a two-part mixture whose weight is set by the hyper-parameter λ.
  • Best mixture weights:
    • λ = 1 (pure embedding-based topic-word distributions) gave the best topic coherence. Coherence was measured with the Lau et al. (2014) method, using Wikipedia co-occurrence statistics.
    • λ = 0.6 worked best for clustering and classification, which depend more on corpus-specific properties.
  • Classification results:
    • On N20small and Twitter, with the number of topics set to the number of gold labels (20 and 4), w2v-LDA scored over 5% higher F1 than LDA.
    • With T = 80, w2v-DMM beat DMM by 5.4% F1 on TMN and by 2.9% on TMNtitle.
  • Robustness:
    • The embeddings helped even when the domain differed sharply from the corpus they were trained on, as with Twitter.
    • Google word2vec and Stanford GloVe vectors performed almost identically.
  • Limitations:
    • The approach runs slowly on very large corpora.
    • The LF-DMM sampler depends on an approximation.
    • Words missing from either embedding set were excluded.
    • Embeddings trained only on the topic-modeling corpus performed worse than the pre-trained ones.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Macquarie UniversitySydney · Australia44 works · 5,778 citations🔬Santa Fe InstituteSanta Fe · United States7 works · 646 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksGeneral Social Sciences · 254 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Topic modeling · 47 works

Cite

Cite as Dat Quoc Nguyen, Richard Billingsley, Lan Du and Mark Johnson. 2015. Improving Topic Models with Latent Feature Word Representations. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00140

@article{nguyen2015improving,
  title = {{Improving Topic Models with Latent Feature Word Representations}},
  author = {Dat Quoc Nguyen, Richard Billingsley, Lan Du and Mark Johnson},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.1162/tacl\_a\_00140},
  url = {https://doi.org/10.1162/tacl_a_00140}
}

References

53 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning (Information Science and Statistics). find
  2. Springer-Verlag New York, Inc. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet Allocation. Journal of Machine Learning Research, 3:993–1022.
  3. David M. Blei. 2012. Probabilistic Topic Models. Communications of the ACM, 55(4):77–84. find
  4. John A. Bullinaria and Joseph P. Levy. 2007. Extracting semantic representations from word co-occurrence statistics: A computational study. Behavior Research Methods, 39(3):510–526. find
  5. Deng Cai, Qiaozhu Mei, Jiawei Han, and Chengxiang Zhai. 2008. Modeling Hidden Topics on Document Manifold. In Proceedings of the 17th ACM Conference on Information and Knowledge Management, pages 911–920. find
  6. Ziqiang Cao, Sujian Li, Yang Liu, Wenjie Li, and Heng Ji. 2015. A Novel Neural Topic Model and Its Supervised Extension. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pages 2210–2216. find
  7. Ana Cardoso-Cachopo. 2007. Improving Methods for Single-label Text Categorization. find
  8. PhD Thesis, Instituto Superior Tecnico, Universidade Tecnica de Lisboa. Jonathan Chang, Sean Gerrish, Chong Wang, Jordan L. Boyd-graber, and David M. Blei. 2009. Reading Tea Leaves: How Humans Interpret Topic Models. In Advances in Neural Information Processing Systems 22, pages 288–296.
  9. Ronan Collobert and Jason Weston. 2008. A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. In Proceedings of the 25th International Conference on Machine Learning, pages 160–167. find
  10. Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. 1990. Indexing by Latent Semantic Analysis. Journal of the American Society for Information Science, 41(6):391– 407. find
  11. Jacob Eisenstein, Amr Ahmed, and Eric Xing. 2011. Sparse Additive Generative Models of Text. In Proceedings of the 28th International Conference on Machine Learning, pages 1041–1048. find
  12. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach. In Proceedings of the 28th International Conference on Machine Learning, pages 513–520. find
  13. Thomas L. Griffiths and Mark Steyvers. 2004. Finding scientific topics. Proceedings of the National Academy of Sciences of the United States of America, 101(Suppl 1):5228–5235. find
  14. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA Data Mining Software: An Update. ACM SIGKDD Explorations Newsletter, 11(1):10–18. find
  15. Bo Han, Paul Cook, and Timothy Baldwin. 2012. Automatically Constructing a Normalisation Dictionary for Microblogs. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 421–432. find
  16. Swapnil Hingmire, Sandeep Chougule, Girish K. Palshikar, and Sutanu Chakraborti. 2013. Document Classification by Topic Labeling. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval, pages 877–880. find
  17. Liangjie Hong and Brian D. Davison. 2010. Empirical Study of Topic Modeling in Twitter. In Proceedings of the First Workshop on Social Media Analytics, pages 80–88. find
  18. Seungil Huh and Stephen E. Fienberg. 2012. Discriminative Topic Modeling Based on Manifold Learning. ACM Transactions on Knowledge Discovery from Data, 5(4):20:1–20:25. find
  19. Mark Johnson. 2010. PCFGs, Topic Models, Adaptor Grammars and Learning Topical Collocations and the Structure of Proper Names. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1148–1157. find
  20. Michael I. Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. 1999. An Introduction to Variational Methods for Graphical Models. Machine Learning, 37(2):183–233. find
  21. Simon Lacoste-Julien, Fei Sha, and Michael I. Jordan. 2009. DiscLDA: Discriminative Learning for Dimensionality Reduction and Classification. In Advances in Neural Information Processing Systems 21, pages 897–904. find
  22. Han Jey Lau, David Newman, and Timothy Baldwin. 2014. Machine Reading Tea Leaves: Automatically Evaluating Topic Coherence and Topic Model Quality. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 530–539.
  23. D. C. Liu and J. Nocedal. 1989. On the Limited Memory BFGS Method for Large Scale Optimization. Mathematical Programming, 45(3):503–528. find
  24. Yang Liu, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. 2015. Topical Word Embeddings. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pages 2418–2424. find
  25. Yue Lu, Qiaozhu Mei, and ChengXiang Zhai. 2011. Investigating task performance of probabilistic topic models: an empirical study of PLSA and LDA. Information Retrieval, 14:178–203. find
  26. Kevin Lund and Curt Burgess. 1996. Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instruments, & Computers, 28(2):203–208. Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. find
  27. Cambridge University Press. Andrew McCallum. 2002. MALLET: A Machine Learning for Language Toolkit. find
  28. Rishabh Mehrotra, Scott Sanner, Wray Buntine, and Lexing Xie. 2013. Improving LDA Topic Models for Microblogs via Tweet Pooling and Automatic Labeling. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 889–892. find
  29. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. In Advances in Neural Information Processing Systems 26, pages 3111–3119.
  30. David Mimno, Hanna M. Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. 2011. Optimizing Semantic Coherence in Topic Models. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 262– 272.
  31. David Newman, Chaitanya Chemudugunta, and Padhraic Smyth. 2006. Statistical Entity-Topic Models. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 680–686. find
  32. David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. 2010. Automatic Evaluation of Topic Coherence. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 100–108.
  33. Kamal Nigam, AK McCallum, S Thrun, and T Mitchell. 2000. Text Classification from Labeled and Unlabeled Documents Using EM. Machine learning, 39:103– 134.
  34. Michael Paul and Mark Dredze. 2015. SPRITE: Generalizing Topic Models with Structured Priors. Transactions of the Association for Computational Linguistics, 3:43–57. find
  35. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532–1543. James Petterson, Wray Buntine, Shravan M. Narayana-murthy, Tibério S. Caetano…
  36. Xuan-Hieu Phan, Cam-Tu Nguyen, Dieu-Thu Le, Le-Minh Nguyen, Susumu Horiguchi, and Quang-Thuy Ha. 2011. A Hidden Topic-Based Framework Toward Building Applications with Short Web Documents. IEEE Transactions on Knowledge and Data Engineering, 23(7):961–976. find
  37. John C. Platt. 1999. Fast Training of Support Vector Machines using Sequential Minimal Optimization. In Bernhard Schölkopf, Christopher J. C. Burges, and Alexander J. Smola, editors, Advances in kernel methods, pages 185–208. find
  38. Ian Porteous, David Newman, Alexander Ihler, Arthur Asuncion, Padhraic Smyth, and Max Welling. 2008. Fast Collapsed Gibbs Sampling for Latent Dirichlet Allocation. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 569–577. find
  39. Christian P. Robert and George Casella. 2004. Monte Carlo Statistical Methods (Springer Texts in Statistics). find
  40. Springer-Verlag New York, Inc. Mehran Sahami and Timothy D. Heilman. 2006. A Web-based Kernel Function for Measuring the Similarity of Short Text Snippets. In Proceedings of the 15th International Conference on World Wide Web, pages 377– 386. find
  41. Ruslan Salakhutdinov and Geoffrey Hinton. 2009. Replicated Softmax: an Undirected Topic Model. In Advances in Neural Information Processing Systems 22, pages 1607–1614. find
  42. Richard Socher, John Bauer, Christopher D. Manning, and Ng Andrew Y. 2013. Parsing with Compositional Vector Grammars. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 455–465. find
  43. Nitish Srivastava, Ruslan Salakhutdinov, and Geoffrey Hinton. 2013. Modeling Documents with a Deep Boltzmann Machine. In Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence, pages 616–624. find
  44. Keith Stevens, Philip Kegelmeyer, David Andrzejewski, and David Buttler. 2012. Exploring Topic Coherence over Many Models and Many Topics. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 952–961. find
  45. Yee W Teh, David Newman, and Max Welling. 2006. A Collapsed Variational Bayesian Inference Algorithm for Latent Dirichlet Allocation. In Advances in Neural Information Processing Systems 19, pages 1353–1360. find
  46. Kristina Toutanova and Mark Johnson. 2008. A Bayesian LDA-based Model for Semi-Supervised Part-of-speech Tagging. In Advances in Neural Information Processing Systems 20, pages 1521–1528. find
  47. Daniele Vitale, Paolo Ferragina, and Ugo Scaiella. 2012. Classification of Short Texts by Deploying Topical Annotations. In Proceedings of the 34th European Conference on Advances in Information Retrieval, pages 376–387. find
  48. Jianshu Weng, Ee-Peng Lim, Jing Jiang, and Qi He. 2010. TwitterRank: Finding Topic-sensitive Influential Twitterers. In Proceedings of the Third ACM International Conference on Web Search and Data Mining, pages 261–270. find
  49. Pengtao Xie and Eric P. Xing. 2013. Integrating Document Clustering and Topic Modeling. In Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence, pages 694–703. find
  50. Xiaohui Yan, Jiafeng Guo, Yanyan Lan, and Xueqi Cheng. 2013. A Biterm Topic Model for Short Texts. In Proceedings of the 22Nd International Conference on World Wide Web, pages 1445–1456. find
  51. Jianhua Yin and Jianyong Wang. 2014. A Dirichlet Multinomial Mixture Model-based Approach for Short Text Clustering. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 233–242. find
  52. Ke Zhai and Jordan L. Boyd-graber. 2013. Online Latent Dirichlet Allocation with Infinite Vocabulary. In Proceedings of the 30th International Conference on Machine Learning, pages 561–569. find
  53. Wayne Xin Zhao, Jing Jiang, Jianshu Weng, Jing He, Ee-Peng Lim, Hongfei Yan, and Xiaoming Li. 2011. Comparing Twitter and Traditional Media Using Topic Models. In Proceedings of the 33rd European Conference on Advances in Information Retrieval, pages 338–349. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nguyen-2015-improving-topic-models-latent-feature/