Elamur Research⌕
☰
home / papers / 2024 / TopicGPT: A Prompt-based Topic Modeling Framework

TopicGPT: A Prompt-based Topic Modeling Framework

Paper · 2024 · Chau Minh Pham, Alexander Hoyle, Simeng Sun et al.

Summary

Classic topic models such as LDA describe each theme as a list of probable words. Users must interpret these lists themselves and have little say over how specific or how formatted the topics are. TopicGPT addresses this by having large language models do the topic modeling through prompting, in two stages. In the first stage, GPT-4 reads a sample of documents, starting from just two human-written seed topics, and proposes labeled topics with one-sentence descriptions. It then merges near-duplicates and drops rare topics. In the second stage, GPT-3.5-turbo assigns topics to each document and cites a supporting passage from it, with a self-correction step for invalid outputs. On Wikipedia articles and Congressional bills, TopicGPT's topics match human annotations more closely than LDA, SeededLDA and BERTopic do. Users can edit or constrain topics without retraining anything. This makes the approach a verifiable, human-centered tool for content analysis. Open-source models remain weak at the generation stage.

Key points

  • On Wikipedia, TopicGPT reaches a harmonic mean purity of 0.74, compared with 0.64 for the best baseline. It also improves on normalized mutual information and adjusted Rand index. Only 30.3% of its topics are misaligned with human labels, versus 62.4% for LDA.
  • During refinement, Sentence-Transformer embeddings flag topic pairs with cosine similarity ≥ 0.5. The LLM then merges those that are near-duplicates, and topics that occur rarely are removed using a frequency threshold.
  • Using GPT-4 for generation and GPT-3.5-turbo for assignment costs roughly $100 per dataset.
  • Mistral-7B-Instruct assigns topics competently but fails to follow the generation instructions. As generators, Mistral produced 1,418 topics and GPT-3.5-turbo produced 151, mostly overly narrow ones.
  • Code is available at github.com/chtmp223/topicGPT. Known limitations include truncating documents to fit the context window, testing on English only, and reliance on closed-source models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Maryland, College ParkCollege Park · United States132 works · 25,380 citations🎓University of Massachusetts AmherstAmherst Center · United States106 works · 66,692 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Topic modeling · 47 works

Cite

Cite as Chau Minh Pham, Alexander Hoyle, Simeng Sun, Philip Resnik and Mohit Iyyer. 2024. TopicGPT: A Prompt-based Topic Modeling Framework. NAACL 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{pham2024topicgpt,
  title = {{TopicGPT: A Prompt-based Topic Modeling Framework}},
  author = {Chau Minh Pham, Alexander Hoyle, Simeng Sun, Philip Resnik and Mohit Iyyer},
  booktitle = {NAACL 2024},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.164},
  url = {https://doi.org/10.18653/v1/2024.naacl-long.164}
}

References

66 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. E Scott Adler and John Wilkerson. 2018. Congressional bills project: 1995-2018. find
  2. Enrique Amigó, Julio Gonzalo, Javier Artiles, and Felisa Verdejo. 2009. A comparison of extrinsic clustering evaluation metrics based on formal constraints. Information retrieval, 12:461–486. find
  3. David Andrzejewski and Xiaojin Zhu. 2009. Latent Dirichlet Allocation with Topic-in-Set Knowledge. In Proceedings of the NAACL HLT 2009 Workshop on Semi-supervised Learning for Natural Language Processing, pages 43–48, Boulder, Colorado. Association for Computational Linguistics. find
  4. Christian Baden, Christian Pipal, Martijn Schoonvelde, and Mariken A.C.G. van der Velden. 2021. Three gaps in computational text analysis methods for social sciences: A research agenda. Communication Methods and Measures, 16:1 – 18. find
  5. Federico Bianchi, Silvia Terragni, and Dirk Hovy. 2021. Pre-training is a hot topic: Contextualized document embeddings improve topic coherence. 2964 9
  6. David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993–1022.
  7. Hsiu-Fang Hsieh and Sarah E Shannon. 2005. Three approaches to qualitative content analysis. Qualitative health research, 15(9):1277–1288. find
  8. Jonathan Chang, Sean Gerrish, Chong Wang, Jordan Boyd-Graber, and David Blei. 2009. Reading tea leaves: How humans interpret topic models. Advances in neural information processing systems, 22.
  9. Yuening Hu, Jordan Boyd-Graber, Brianna Satinoff, and Alison Smith. 2014. Interactive topic modeling. Machine learning, 95:423–469. find
  10. Robert Chew, John Bollenbacher, Michael Wenger, Jessica Speer, and Annice Kim. 2023. Llm-assisted content analysis: Using large language models to support deductive coding. find
  11. Jason Chuang, Sonal Gupta, Christopher Manning, and Jeffrey Heer. 2013. Topic model diagnostics: Assessing domain relevance via topical alignment. In Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 612–620, Atlanta, G… find
  12. Caitlin Doogan and Wray Buntine. 2021. Topic model or topic twaddle? re-evaluating semantic interpretability measures. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3824–3848, Online. Associati… find
  13. Ryan J. Gallagher, Kyle Reing, David Kale, and Greg Ver Steeg. 2017. Anchored Correlation Explanation: Topic Modeling with Minimal Domain Knowledge. Transactions of the Association for Computational Linguistics, 5:529–542. find
  14. Thomas Griffiths, Michael Jordan, Joshua Tenenbaum, and David Blei. 2003. Hierarchical topic models and the nested chinese restaurant process. Advances in neural information processing systems, 16. find
  15. Tom Griffiths. 2002. Gibbs sampling in the generative model of latent dirichlet allocation. Standford University, 518(11):1–3. find
  16. Maarten Grootendorst. 2022. Bertopic: Neural topic modeling with a class-based tf-idf procedure.
  17. Alexander Hoyle, Pranav Goel, Andrew Hian-Cheong, Denis Peskov, Jordan Boyd-Graber, and Philip Resnik. 2021. Is Automated Topic Model Evaluation Broken? The Incoherence of Coherence. In Advances in Neural Information Processing Systems, volume 34, pages 2018–2033. Curran Associates, Inc. find
  18. Alexander Hoyle, Rupak Sarkar, Pranav Goel, and Philip Resnik. 2023. Natural language decompositions of implicit content enable better text representations. find
  19. Alexander Miserlis Hoyle, Pranav Goel, Rupak Sarkar, and Philip Resnik. 2022. Are neural topic models broken? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5321–5344, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  20. Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting. In Findings of Empirical Methods in Natural Language Processing.
  21. Lawrence Hubert and Phipps Arabie. 1985. Comparing partitions. Journal of classification, 2:193–218. find
  22. Jagadeesh Jagarlamudi, Hal Daumé III, and Raghavendra Udupa. 2012. Incorporating Lexical Priors into Topic Models. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 204–213, Avignon, France. Association for Computational Linguistics… find
  23. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  24. Damir Korenčić, Strahil Ristov, Jelena Repar, and Jan Šnajder. 2021. A Topic Coverage Approach to Evaluation of Topic Models. IEEE Access, 9:123280– 123312. ArXiv:2012.06274 [cs]. — [arXiv:2012.06274](https://arxiv.org/abs/2012.06274) find
  25. Helvi Kyngäs. 2020. Inductive content analysis. The application of content analysis in nursing science research, pages 13–21. find
  26. Jey Han Lau, Karl Grieser, David Newman, and Timothy Baldwin. 2011. Automatic labelling of topic models. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 1536–1545. find
  27. Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023. Bactrian-x : A multilingual replicable instruction-following model with low-rank adaptation. find
  28. Marsha M Linehan, Judith L Goodstein, Stevan L Nielsen, and John A Chiles. 1983. Reasons for staying alive when you are thinking of killing yourself: the reasons for living inventory. Journal of consulting and clinical psychology, 51(2):276. find
  29. Sengjie Liu and Christopher G Healey. 2023. Abstractive summarization of large document collections using gpt. arXiv preprint arXiv:2310.05690. — [arXiv:2310.05690](https://arxiv.org/abs/2310.05690) find
  30. Andrew Kachites McCallum. 2002. Mallet: A machine learning for language toolkit. Http://www.cs.umass.edu/ mccallum/mallet. 2965 10 find
  31. Qiaozhu Mei, Xuehua Shen, and ChengXiang Zhai. 2007. Automatic labeling of multinomial topic models. In Knowledge Discovery and Data Mining. find
  32. Marina Meilă. 2007. Comparing clusterings—an information based distance. Journal of multivariate analysis, 98(5):873–895. find
  33. Yu Meng, Jiaxin Huang, Guangyuan Wang, Zihan Wang, Chao Zhang, Yu Zhang, and Jiawei Han. 2020. Discriminative Topic Mining via Category-Name Guided Text Embedding. In Proceedings of The Web Conference 2020, pages 2121–2132. ArXiv:1908.07162 [cs]. — [arXiv:1908.07162](https://arxiv.org/abs/1908.07162) find
  34. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. Regularizing and optimizing LSTM language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
  35. David Mimno, Wei Li, and Andrew McCallum. 2007. Mixtures of hierarchical topics with pachinko allocation. In Proceedings of the 24th international conference on Machine learning, pages 633–640. find
  36. David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. 2010. Automatic evaluation of topic coherence. In Human language technologies: The 2010 annual conference of the North American chapter of the association for computational linguistics, pages 100–108.
  37. Sergey I Nikolenko, Sergei Koltcov, and Olessia Koltsova. 2017. Topic modelling for qualitative studies. Journal of Information Science, 43(1):88–102. find
  38. OpenAI. 2023. Gpt-4 technical report.
  39. John Paisley, Chong Wang, David M Blei, and Michael I Jordan. 2014. Nested hierarchical dirichlet processes. IEEE transactions on pattern analysis and machine intelligence, 37(2):256–270. find
  40. Forough Poursabzi-Sangdeh, Jordan Boyd-Graber, Leah Findlater, and Kevin Seppi. 2016. Alto: Active learning with topic overviews for speeding label induction and document labeling. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pag… find
  41. Daniel Ramage, Christopher D. Manning, and Susan Dumais. 2011. Partially labeled topic models for interpretable text mining. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 457–465, San Diego California USA. ACM. find
  42. William M Rand. 1971. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association, 66(336):846–850. find
  43. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
  44. Philip Resnik, Pranav Goel, Alexander Hoyle, Rupak Sarkar, Josh Hagedorn, Maeve Gearing, and Carol Bruce. 2022. A step-by-step protocol for curation of topic models by subject matter experts. In New Directions in Analyzing Text as Data. find
  45. Philip Resnik, Bolei Ma, Alexander Hoyle, Pranav Goel, Rupak Sarkar, Maeve Gearing, Anna-Carolina Haensch, and Frauke Kreuter. 2024a. A topic-oriented protocol for content analysis of text. Manuscript in preparation. find
  46. Philip Resnik, Katherine Musacchio Schafer, Rebecca Resnik, Josh Hagedorn, and Jonathan Singer. 2024b. Reasons to live instead of dying by suicide: New insights from a computer assisted content analysis. Manuscript in preparation. find
  47. Emil Rijcken, Floortje Scheepers, Kalliopi Zervanou, Marco Spruit, Pablo Mosteiro, and Uzay Kaymak. 2023. Towards Interpreting Topic Models with Chat-GPT: The 20th World Congress of the International Fuzzy Systems Association. find
  48. Claude Elwood Shannon. 1948. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423. find
  49. Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning.
  50. Suzanna Sia, Ayush Dalmia, and Sabrina J. Mielke. 2020. Tired of topic models? clusters of pretrained word embeddings make for fast and good topics too! In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1728–1736, Online. Association for Computa… find
  51. Akash Srivastava and Charles Sutton. 2017. Autoencoding variational inference for topic models. arXiv preprint arXiv:1703.01488. — [arXiv:1703.01488](https://arxiv.org/abs/1703.01488) find
  52. Dominik Stammbach, Vilém Zouhar, Alexander Hoyle, Mrinmaya Sachan, and Elliott Ash. 2023. Revisiting automated topic model evaluation with large language models. find
  53. Alexander Strehl and Joydeep Ghosh. 2002. Cluster ensembles—a knowledge reuse framework for combining multiple partitions. Journal of machine learning research, 3(Dec):583–617. 2966 11 find
  54. Simeng Sun, Yang Liu, Shuohang Wang, Chenguang Zhu, and Mohit Iyyer. 2023. Pearl: Prompting large language models to plan and execute actions over long documents. find
  55. Robert H Tai, Lillian R Bentley, Xin Xia, Jason M Sitt, Sarah C Fankhauser, Ana M Chicas-Mosier, and Barnas G Monteith. 2023. Use of large language models to aid analysis of textual data. bioRxiv, pages 2023–07. find
  56. Yee Whye Teh, Michael I Jordan, Matthew J Beal, and David M Blei. 2006. Hierarchical Dirichlet Processes. Journal of the American Statistical Association, 101(476):1566–1581. find
  57. Laure Thompson and David Mimno. 2020. Topic Modeling with Contextualized Word Representation Clusters. ArXiv:2010.12626 [cs]. — [arXiv:2010.12626](https://arxiv.org/abs/2010.12626) find
  58. Danya F Vears and Lynn Gillam. 2022. Inductive content analysis: A guide for beginning qualitative researchers. Focus on Health Professional Education: A Multi-disciplinary Journal, 23(1):111–127. find
  59. Nguyen Xuan Vinh, Julien Epps, and James Bailey. 2009. Information theoretic measures for clusterings comparison: is a correction for chance necessary? In Proceedings of the 26th annual international conference on machine learning, pages 1073–1080. find
  60. Vijay Viswanathan, Kiril Gashteovski, Carolin Lawrence, Tongshuang Wu, and Graham Neubig. 2023. Large language models enable few-shot clustering.
  61. Hanna Wallach, David Mimno, and Andrew McCallum. 2009. Rethinking lda: Why priors matter. Advances in neural information processing systems, 22. find
  62. Xiaojun Wan and Tianming Wang. 2016. Automatic labeling of topic models using text summaries. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2297–2305, Berlin, Germany. Association for Computational Linguistics. find
  63. Zihan Wang, Jingbo Shang, and Ruiqi Zhong. 2023. Goal-driven explainable clustering via language descriptions. find
  64. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359.
  65. Yuwei Zhang, Zihan Wang, and Jingbo Shang. 2023. ClusterLLM: Large Language Models as a Guide for Text Clustering. ArXiv:2305.14871 [cs]. find
  66. Ying Zhao. 2005. Criterion Functions for Document Clustering. Ph.D. thesis, University of Minnesota, USA. AAI3180039. 2967 12 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pham-2024-topicgpt-prompt-based-topic-modeling/