Elamur Research⌕
☰
home / papers / 2026 / When Numbers Tell Half the Story: Human-Metric Alignment in Topic Model Evaluation

When Numbers Tell Half the Story: Human-Metric Alignment in Topic Model Evaluation

Summary

Judging topic models is difficult, particularly on specialized scholarly text. Automated scores such as coherence and diversity may not reflect human perception. The standard human test, word intrusion, only checks whether a single topic is internally consistent. The authors propose Topic Word Mixing (TWM), a new annotation task in which judges decide whether a word set comes from one topic or blends two. This gives a human-grounded measure of how well topics are separated. They build six models (LDA, NMF, Top2Vec, BERTopic, CFMF, CFMF-emb) on a large philosophy of science corpus. They then compare coherence and diversity scores with almost 4,000 human judgments from word intrusion and TWM. Intrusion outcomes and coherence scores sometimes disagree, which questions co-occurrence-based coherence in niche domains and points toward embedding-based alternatives. The authors also find that building intrusion items requires attention to how semantically close intruders are to topic words. TWM seems to track diversity metrics, at least on this corpus. The annotated data and the task-generation code are publicly released.

Key points

  • Corpus: full texts of eight philosophy of science journals (e.g., Synthese, Erkenntnis, Philosophy of Science) from 1931 to 2017. It contains 16,917 articles, about 65M tokens and 23.7k types. The roughly 6% of non-English items (mainly German, Dutch, French) were machine-translated into English.
  • Models were trained with K = 10, 25 and 50. For Top2Vec and BERTopic, K was steered through the HDBSCAN minimum cluster size, giving 48 and 54 topics at the largest setting. Document embeddings came from stella_en_1.5B_v5, except for Top2Vec, which uses Doc2Vec.
  • Automated evaluation used Coherence C_V and topic diversity. Human evaluation used word intrusion (TWI) and the new TWM task, with 10 annotators producing nearly 4,000 annotations.
  • Coherence metrics do not consistently match word intrusion results in this specialized domain. TWM appears to align with diversity metrics.
  • The released resources are the annotated TWI/TWM dataset and the code for generating such tasks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/387fiwstpipw

@inproceedings{prouteau2026when,
  title = {{When Numbers Tell Half the Story: Human-Metric Alignment in Topic Model Evaluation}},
  author = {Prouteau, Thibault and Lareau, Francis and Dugue, Nicolas and Lamirel, Jean-Charles and Malaterre, Christophe},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/387fiwstpipw},
  url = {https://doi.org/10.63317/387fiwstpipw}
}

References

34 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nikolaos Aletras and Mark Stevenson. 2013. Evaluating topic coherence using distributional semantics. In Proceedings of the 10th international conference on computational semantics (IWCS 2013)–Long Papers, pages 13–22. find our card
  2. Dimo Angelov. 2020. Top2vec: Distributed representations of topics. arXiv preprint arXiv:2008.09470. find
  3. David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993–1022. find our card
  4. Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander. 2013. Density-based clustering based on hierarchical density estimates. In Pacific-Asia conference on knowledge discovery and data mining, pages 160–172. Springer. find
  5. Jonathan Chang, Sean Gerrish, Chong Wang, Jordan Boyd-graber, and David Blei. 2009. Reading Tea Leaves: How Humans Interpret Topic Models. In Advances in Neural Information Processing Systems, volume 22. Curran Associates, Inc. find our card
  6. Adji B Dieng, Francisco JR Ruiz, and David M Blei. 2020. Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8:439–453. find our card
  7. Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah A. Smith. 2015. Sparse Overcomplete Word Vector Representations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (V… find
  8. Joseph L Fleiss and Jacob Cohen. 1973. The equivalence of weighted kappa and the intra-class correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3):613–619. find
  9. Bernd Fritzke. 1994. A growing neural gas network learns topologies. Advances in neural information processing systems, 7. find
  10. Maarten Grootendorst. 2022. Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint arXiv:2203.05794. find our card
  11. Alexander Hoyle, Pranav Goel, Andrew Hian-Cheong, Denis Peskov, Jordan Boyd-Graber, and Philip Resnik. 2021. Is automated topic model evaluation broken? the incoherence of coherence. Advances in neural information processing systems, 34:2018–2033. find
  12. Jean-Charles Lamirel, Nicolas Dugué, and Pascal Cuxac. 2016. New efficient clustering quality indexes. In 2016 International joint conference on neural networks (IJCNN), pages 3649–3657. IEEE. find
  13. Jean-Charles Lamirel, Francis Lareau, and Christophe Malaterre. 2025. Old but not obsolete: Bag-of-words vs. embeddings in topic modeling. In 20th International Conference on Scientometrics & Informetrics, pages 2275–2282. find
  14. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174. find
  15. Jey Han Lau, David Newman, and Timothy Baldwin. 2014. Machine Reading Tea Leaves: Automatically Evaluating Topic Coherence and Topic Model Quality. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 530– 539, Gothenburg, Sweden. Asso… find our card
  16. Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In International conference on machine learning, pages 1188–1196. PMLR. find our card
  17. Daniel D Lee and H Sebastian Seung. 1999. Learning the parts of objects by non-negative matrix factorization. nature, 401(6755):788– 791. find
  18. Christophe Malaterre and Francis Lareau. 2022. The early days of contemporary philosophy of science: Novel insights from machine translation and topic-modeling of non-parallel multilingual corpora. Synthese, 200(3):242. find
  19. Christophe Malaterre and Francis Lareau. 2023. The emergence of astrobiology: a topic-modeling perspective. Astrobiology, 23(5):496– 512. find
  20. George A Miller. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review, 63(2):81. find
  21. David Mimno, Hanna Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. 2011. Optimizing semantic coherence in topic models. In Proceedings of the 2011 conference on empirical methods in natural language processing, pages 262–272. find our card
  22. Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistic… find our card
  23. Brian Murphy, Partha Talukdar, and Tom Mitchell. 2012. Learning Effective and Interpretable Semantic Models using Non-Negative Sparse Embedding. In Proceedings of COLING 2012, pages 1933–1950, Mumbai, India. find
  24. The COLING 2012 Organizing Committee. find
  25. David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. 2010. Automatic evaluation of topic coherence. In Human language technologies: The 2010 annual conference of the North American chapter of the association for computational linguistics, pages 100–108. find our card
  26. Maximilian Noichl. 2021. Modeling the structure of recent philosophy. Synthese, 198(6):5089– 5100. find
  27. Thibault Prouteau, Nicolas Dugué, Nathalie Camelin, and Sylvain Meignier. 2022. Are embedding spaces interpretable? results of an intrusion detection evaluation on a large french corpus. In LREC 2022. find
  28. Michael Röder, Andreas Both, and Alexander Hinneburg. 2015. Exploring the space of topic coherence measures. In Proceedings of the eighth ACM international conference on Web search and data mining, pages 399–408. find our card
  29. Anant Subramanian, Danish Pruthi, Harsh Jhamtani, Taylor Berg-Kirkpatrick, and Eduard Hovy. 2018. SPINE: SParse Interpretable Neural Embeddings. In Thirty-Second AAAI Conference on Artificial Intelligence. find
  30. Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. 2020-2025. Label Studio: Data labeling software. Open source software available from https://github.com/HumanSignal/label-studio. ↗
  31. Nam Khanh Tran, Sergej Zerr, Kerstin Bischoff, Claudia Niederée, and Ralf Krestel. 2013. Topic cropping: Leveraging latent topics for the analysis of small corpora. In International Conference on Theory and Practice of Digital Libraries, pages 297–308. Springer. find
  32. Hanna M Wallach, Iain Murray, Ruslan Salakhutdinov, and David Mimno. 2009. Evaluation methods for topic models. In Proceedings of the 26th annual international conference on machine learning, pages 1105–1112. find
  33. Xiaohao Yang, He Zhao, Dinh Phung, Wray Buntine, and Lan Du. 2025. LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models. find
  34. Dun Zhang, Jiacheng Li, Ziyang Zeng, and Fulong Wang. 2025. Jasper and stella: distillation of sota embedding models. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/prouteau-2026-when-numbers-tell-half-story/