Elamur Research⌕
☰
home / papers / 2026 / Can a Large Language Model Replace Humans at Rating Lexical Semantic Relations Strength?

Can a Large Language Model Replace Humans at Rating Lexical Semantic Relations Strength?

Paper · 2026 · André Fernandes dos Santos and José Paulo Leal

Summary

Benchmarks of lexical semantic relations (SR), meaning how similar or related two words are, rely on human annotators. These annotators are costly and hard to recruit, so the datasets stay small. The authors test whether LLMs could stand in for both the annotators and classic semantic measure (SM) algorithms. They queried 15 commercial models from OpenAI, Google, Mistral and Anthropic, asking each to rate word pairs from a broad collection of English and Portuguese out-of-context SR datasets. The model ratings were compared with averaged human scores using correlation, and set against reported inter-annotator agreement (IAA), a word-embedding baseline and earlier SM results. To check whether models had simply memorised benchmarks seen in training, the study included PAP900, a new Portuguese dataset that was unpublished when the experiments ran. The study also looked at whether dataset traits such as language, relation type, part of speech and annotator expertise affect performance. Overall, LLM ratings correlated strongly with human judgements. This suggests LLMs can complement or partly replace crowd-sourced annotation and traditional SMs when building and using SR resources.

Key points

  • The study addresses four research questions:
    • how LLMs compare with SM algorithms,
    • how LLMs compare with human IAA,
    • whether training-data exposure inflates results,
    • which dataset characteristics matter.
  • Models tested include:
    • gpt-3.5-turbo, gpt-4-turbo, gpt-4o and gpt-4o-mini;
    • gemini-1.5-pro/flash/flash-8b;
    • claude-3-haiku, claude-3-sonnet and claude-3-5-sonnet;
    • mistral-large/small, open-mistral-nemo and ministral-3b/8b.
  • A fastText-style cosine baseline (ftwe-cosine) was also included.
  • Around 45 dataset partitions were evaluated with Pearson's r. Examples include MC30, RG65, SimLex-999, WS353, MEN3000, SimVerb3500, MayoSRS/UMNSRS and the Portuguese LX-SimLex-999/LX-WS353.
  • PAP900, with separate similarity and relatedness partitions, served as a contamination-free test set for the exposure question.
  • This work extends the authors' 2024 study, which focused on MC30 and prompt verbosity. Related prior work reported that ChatGPT reached r = 0.604 on SimLex-999 pairs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universidade do PortoPorto · Portugal14 works · 1,992 citations🔬INESC TECPorto · Portugal11 works · 1,180 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksLexical semantics & word embeddings · 63 works

Cite

Cite as André Fernandes dos Santos and José Paulo Leal. 2026. Can a Large Language Model Replace Humans at Rating Lexical Semantic Relations Strength?. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli.a.587

@article{santos2026can,
  title = {{Can a Large Language Model Replace Humans at Rating Lexical Semantic Relations Strength?}},
  author = {André Fernandes {dos Santos} and José Paulo Leal},
  journal = {Computational Linguistics},
  year = {2026},
  doi = {10.1162/coli.a.587},
  url = {https://doi.org/10.1162/coli.a.587}
}

References

45 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Achiam, Josh, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023, GPT-4 technical report. Technical report, OpenAI.
  2. Agirre, Eneko, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pasca, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and WordNet-based approaches. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the A…
  3. Dataset partition reword26\#g26 rg65\#table1 scws2003\#main semeval17\#main simlex999\#main simverb3500\#dev sl7576\#main srw2034\#rw tr9856\#main umnsrs\#rel umnsrs\#sim umnsrsMod\#rel umnsrsMod\#sim word19k\#test word19k\#train wp300\#wp ws353\#combined ws353split\#rel ws353split\#sim yp130\#verbpairs zie55\#B0 zie55\#… find
  4. AI, Mistral. 2025a. Mistral AI API. https://docs.mistral.ai/api/ ↗
  5. AI, Mistral. 2025b. Terms of use. https://mistral.ai/terms\#terms-of-service Aiyappa, Rachith, Jisun An, Haewoon Kwak, and Yong-yeol Ahn. 2023. Can we trust the evaluation on ChatGPT? In Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), pages 47–54. https://… ↗
  6. AlMousa, Mohannad, Rachid Benlamri, and Richard Khoury. 2022. A novel word sense disambiguation approach using WordNet knowledge graph. Computer Speech & Language, 74:101337. https://doi.org/10.1016/j.csl.2021.101337 Anthropic. 2024a. Anthropic models overview. https://docs.anthropic.com /claude/do… — [doi:10.1016/j.csl.2021.101337](https://doi.org/10.1016/j.csl.2021.101337) ↗
  7. Anthropic. 2024b. The Claude 3 model family: Opus, Sonnet, Haiku. Technical report, Anthropic. find
  8. Anthropic. 2024c. Tool use (function calling). https://docs.anthropic.com/claude/docs/tool-use. Accessed: May 9, 2024. ↗
  9. Anthropic. 2025a. Is my data used for model training? \| Anthropic Privacy Center. https://privacy.anthropic.com/en/articles/7996868-is-my-data-used -for-model-training Anthropic. 2025b. Messages. https://docs.anthropic.com/en/api/messages API, Gemini. 2025. Generating content. https://ai.google.de… ↗
  10. Aston, Guy and Lou Burnard. 2020. The BNC Handbook: Exploring the British National Corpus with SARA. Edinburgh University Press. https://doi.org/10.1515/9780748628889 Bai, Yuntao, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini,… ↗
  11. Anthropic. Baker, Simon, Roi Reichart, and Anna Korhonen. 2014. An unsupervised model for instance level subcategorization acquisition. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 278–289. https://doi.org/10.3115/v1/D14-1034 Ballatore, An… ↗
  12. Boslaugh, Sarah. 2012. Statistics in a Nutshell. find
  13. O’Reilly Media, Inc. Boyd-Graber, Jordan, Christiane Fellbaum, Daniel Osherson, and Robert Schapire. 2006. Adding dense, weighted connections to WordNet. In Proceedings of the Third International WordNet Conference, pages 29–36. find
  14. Bruni, Elia, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. Journal of Artificial Intelligence Research, 49:1–47. https://doi.org/10.1613/jair.4135 Camacho-Collados, Jose, Mohammad Taher Pilehvar, Nigel Collier, and Roberto Navigli. 2017. SemEval-2017 Task 2: Multiling… ↗
  15. Carlini, Nicholas, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations. https://doi.org/10.52202/075280-1708 Carlini, Nicholas, Chang… ↗
  16. Cer, Daniel, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity – Multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055. https://doi.org/10.18653/v1/S17-2001 ↗
  17. Chandrasekaran, Dhivya and Vijay Mago. 2022. Evolution of semantic similarity – A survey. ACM Computing Surveys, 54(2):1–37. https://doi.org/10.1145/3440755 Chen, Zugang, Jia Song, and Yaping Yang. 2018. An approach to measuring semantic relatedness of geographic terminologies using a thesaurus and… ↗
  18. Christiano, Paul F., Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30. find
  19. Cohere. 2024. Models. https://docs.cohere.com/docs/models. Accessed: May 8, 2024. Costa, Teresa and José Paulo Leal. 2016. Semantic measures: How similar? How related? In International Conference on Web Engineering, pages 431–438. https://doi.org/10.1007/978-3-319 -38791-8 29 De Deyne, Simon, Chun… ↗
  20. Journal of Quantitative Linguistics, 27(3):244–271. https://doi.org/10.1080/09296174.2019.1570897 Dolan, William B., Chris Quirk, and Chris Brockett. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In Proceedings of the 20th International Con… ↗
  21. Grabb, Declan. 2023. The impact of prompt engineering in large language model performance: A psychiatric example. find
  22. Journal of Medical Artificial Intelligence, 6. https://doi.org/10.21037/jmai-23-71 Gracia, Jorge and Eduardo Mena. 2008. Web-based measure of semantic relatedness. In Web Information Systems Engineering - WISE 2008: 9th International Conference, Auckland, New Zealand, September 1–3, 2008. Proceeding… ↗
  23. Huang, Guangjian, Xingtu Zhu, Shahbaz Hassan Wasti, and Yuncheng Jiang. 2023. Multi-knowledge resources-based semantic similarity models with application for movie recommender system. Artificial Intelligence Review, 56(2):2151–2182. https://doi.org/10.1007/s10462-023 -10573-6 Hussain, Muhammad Jawa… ↗
  24. Liu, Ying, Genevieve B. Melton, and Rui Zhang. 2024. Exploring large language models for acronym, symbol sense disambiguation, and semantic similarity and relatedness assessment. AMIA Summits on Translational Science Proceedings, 2024:324. find
  25. Luong, Minh Thang, Richard Socher, and Christopher D. Manning. 2013. Better word representations with recursive neural networks for morphology. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 104–113. Martinez-Gil, Jorge and José F. Aldana-Montes. 2013…
  26. Mikolov, Tomas, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018).
  27. Miller, George A. and Walter G. Charles. 1991. Contextual correlates of semantic similarity. Language and Cognitive Processes, 6(1):1–28. https://doi.org/10.1080/01690969108406936 Mistral AI. 2024a. Large enough. https://mistral.ai/news/mistral-large-2407. Accessed: March 10, 2025. ↗
  28. Mistral AI. 2024b. Mistral NeMo. https://mistral.ai/news/mistral-nemo. Accessed: May 8, 2024. ↗
  29. Mistral AI. 2024c. Un Ministral, des Ministraux. https://mistral.ai/news/ministraux. Accessed: May 8, 2024. ↗
  30. Mistral AI. 2025. Mistral Small 3. https://mistral.ai/news/mistral-small-3. Accessed: May 8, 2024. Mistral AI large language models. 2024. Function calling. https://docs.mistral .ai/capabilities/function calling/. Accessed: May 9, 2024. Musker, Sam, Alex Duchnowski, Raphaël Millière, and Ellie Pa… ↗
  31. OpenAI. 2025b. Enterprise privacy at OpenAI. https://openai.com/enterprise-privacy/ OpenAI API. 2024. Function calling. https://platform.openai.com/docs/guides/function-calling. Accessed: May 9, 2024. ↗
  32. Pakhomov, Serguei, Bridget McInnes, Terrence Adam, Ying Liu, Ted Pedersen, and Genevieve B. Melton. 2010. Semantic similarity and relatedness between clinical terms: An experimental study. In AMIA Annual Symposium Proceedings, volume 2010, page 572, American Medical Informatics Association. find
  33. Pakhomov, Serguei V. S., Greg Finley, Reed McEwan, Yan Wang, and Genevieve B. Melton. 2016. Corpus domain effects on distributional semantic modeling of medical terms. Bioinformatics (Oxford, England), 32(23):3635–3644. https://doi.org/10.1093/bioinformatics/btw529, PubMed: 27531100 Pedersen, Ted,… ↗
  34. Pilehvar, Mohammad Taher and Roberto Navigli. 2015. From senses to texts: An all-in-one graph-based approach for measuring semantic similarity. Artificial Intelligence, 228:95–128. https://doi.org/10.1016/j.artint.2015 .07.005 Pirró, Giuseppe. 2012. Reword: Semantic relatedness in the Web of data.… — [doi:10.1016/j.artint.2015](https://doi.org/10.1016/j.artint.2015) ↗
  35. Richie, Russell, Sachin Grover, and Fuchiang Rich Tsui. 2022. Inter-annotator agreement is not the ceiling of machine learning performance: Evidence from a comprehensive set of simulations. In Proceedings of the 21st Workshop on Biomedical Language Processing, pages 275–284. https://doi.org /10.1865… ↗
  36. Sahoo, Pranab, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2024. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927. https://doi.org/10.48550/arXiv.2402.07927 Sainz, Oscar, Jon Campos, I… ↗
  37. Salle, Alexandre, Marco Idiart, and Aline Villavicencio. 2016. Matrix factorization using window sampling and negative sampling for improved word representations. In The 54th Annual Meeting of the Association for Computational Linguistics, page 419. https://doi.org/10.18653/v1/P16-2068 Sanderson, M… ↗
  38. Shah, Chirag. 2024. From prompt engineering to prompt science with human in the loop. arXiv preprint arXiv:2401.04122. https://doi.org/10.1145/3709599 — [doi:10.1145/3709599](https://doi.org/10.1145/3709599) ↗
  39. Shapiro, Samuel Sanford and Martin B. Wilk. 1965. An analysis of variance test for normality (complete samples). Biometrika, 52(3–4):591–611. https://doi.org/10.1093/biomet/52.3-4.591 Shaqiri, Mirlinda, Teuta Iljazi, Lazim Kamberi, and Rushadije Ramani-Halili. 2023. Differences between the correlat… — [doi:10.1093/biomet/52.3-4.591](https://doi.org/10.1093/biomet/52.3-4.591) ↗
  40. Silberer, Carina and Mirella Lapata. 2014. Learning grounded meaning representations with autoencoders. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 721–732. https://doi.org/10.3115/v1/P14-1068 ↗
  41. Slimani, Thabet. 2013. Description and evaluation of semantic similarity measures approaches. International Journal of Computer Applications, 80(10):25–33. https://doi.org/10.5120/13897-1851 ↗
  42. Speer, Robyn, Joshua Chin, and Catherine Havasi. 2017. ConceptNet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31. https://doi.org/10.1609/aaai.v31i1.11164 Szumlanski, Sean, Fernando Gomez, and Valerie K. Sims. 2013.… ↗
  43. Trott, Sean. 2024. Can large language models help augment English psycholinguistic datasets? Behavior Research Methods, 56(6):6082–6100. https://doi.org/10.3758/s13428-024-02337-z, PubMed: 38261264 Vulić, Ivan, Simon Baker, Edoardo Maria Ponti, Ulla Petti, Ira Leviant, Kelly Wing, Olga Majewska, E… — [doi:10.3758/s13428-024-02337-z](https://doi.org/10.3758/s13428-024-02337-z) ↗
  44. Yang, Dongqiang and David Powers. 2006. Verb similarity on the taxonomy of WordNet. In The Third International WordNet Conference: GWC 2006. find
  45. Zhang, Ziqi, Anna Lisa Gentile, and Fabio Ciravegna. 2013. Recent advances in methods of lexical semantic relatedness—A survey. Natural Language Engineering, 19(4):411–479. https://doi.org/10.1017/S1351324912000125 Ziegler, C., K. Simon, and G. Lausen. 2006. Automatic computation of semantic proxim… — [doi:10.1145/1183614.1183682](https://doi.org/10.1145/1183614.1183682) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/santos-2026-can-large-language-model-replace/