Elamur Research⌕
☰
home / papers / 2026 / Prompting Instruction-tuned LLMs for Semantic Similarity Values

Prompting Instruction-tuned LLMs for Semantic Similarity Values

Summary

Gathering human ratings of lexical-semantic properties is expensive, so researchers increasingly ask LLMs to generate such ratings. This paper tests how closely prompted decoder models match human judgements of word-pair similarity and which prompting set-ups work best. The authors use the SimLex-999 benchmark in English, Dutch and Mandarin Chinese. They compare nine prompt designs: zero-, one- and few-shot variants, different numeric ranges, a categorical scale, an adaptation of the original annotator guidelines, a chat-style format and a one-pair-per-prompt version. They also compare several models: GPT-3.5, GPT-4o, Mistral-7b-v0.3 in base and Instruct versions, gpt-oss-20b, Doubao-pro and Qwen2.5-7b-Instruct. As an embedding baseline, they use cosine scores from OpenAI's text-embedding-3-large. Agreement with human scores is strong across languages and models. Querying a single pair at a time beats batching many pairs, and gpt-oss-20b aligns best with human ratings. Error analysis shows that the generative models still conflate true similarity with mere relatedness, as older static and contextual embeddings did. The findings matter for anyone using LLM scores to build or extend psycholinguistic norms, especially for languages other than English.

Key points

  • The benchmark is SimLex-999: 999 English pairs, each scored 0–10 by roughly 50 US native speakers. The study also uses its Dutch and Mandarin Chinese counterparts. For Chinese, it compares GPT-4o, Doubao-pro and Qwen2.5-7b-Instruct.
  • The study defines nine English prompt templates (F-1 to F-9). F-7 reuses the human questionnaire, with synonym examples (cup/mug) contrasted against related-but-dissimilar pairs (car/tyre). F-9 asks about one word pair per prompt.
  • The comparison includes instruction-tuned versus non-instruction-tuned Mistral-7b-v0.3, as well as prompting versus text-embedding-3-large embeddings.
  • Earlier work reported a correlation of 0.86 for GPT-4 on SimLex-999 (Trott, 2024), against 0.4–0.6 for embedding models. Embeddings extracted from decoders correlate weakly with human ratings: ρ = 0.179 (Arnold et al.), and ρ < 0.1 for the Dutch models Falcon-7B, BLOOM-560m and Schaapje-2B.
  • Code and data are available at github.com/XanderSnelder/semantic-similarity-prompting.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

2 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 1 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3kbjxx6989dg

@inproceedings{snelder2026prompting,
  title = {{Prompting Instruction-tuned LLMs for Semantic Similarity Values}},
  author = {Snelder, Xander Akiko and Huang, Yunchong and Bloem, Jelke},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3kbjxx6989dg},
  url = {https://doi.org/10.63317/3kbjxx6989dg}
}

References

64 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Katherine Abramski, Clara Lavorati, Giulio Rossetti, and Massimo Stella. 2024. LLM-generated word association norms. In HHAI 2024: Hybrid Human AI Systems for the Social Good, pages 3–12. IOS Press. find
  2. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. 2025. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925. find our card
  3. Alaa Alzahrani, Wafa Aljuaythin, Hassan Alshumrani, Alaa Mamoun Saleh, and Mohamed M Mostafa. 2025. Kalimah norms: Ratings for 2,467 Modern Standard Arabic words on two scales. Behavior Research Methods, 57(7):1–20. find
  4. Stefan Arnold, Marian Fietta, and Dilara Yesilbas. 2024. Routing in sparsely-gated language models responds to context. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 15–22. find
  5. Mohammad Atari, Mona J Xue, Peter S Park, Damián E Blasi, and Joseph Henrich. 2023. Which humans? find
  6. Siamak Barzegar, Brian Davis, Manel Zarrouk, Siegfried Handschuh, and Andre Freitas. 2018. find
  7. SemR-11: A multi-lingual gold-standard for semantic similarity and relatedness for eleven languages. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
  8. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. Advances in Neural Information Processing Systems, 3:1137–1155. find
  9. Lizzy Brans and Jelke Bloem. 2024. SimLex-999 for Dutch. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 14832–14845. find
  10. Lizzy Brans and Jelke Bloem. in press. Multi-SimLex for Dutch: Benchmarking embedding-and prompt-based model performance on semantic similarity. In Proceedings of the Fifteenth International Conference on Language Resources and Evaluation (LREC 2026). find
  11. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  12. Marc Brysbaert, Gonzalo Martínez, and Pedro Reviriego. 2025. Moving beyond word frequency based on tally counting: AI-generated familiarity estimates of words and phrases are an interesting additional index of language knowledge. Behavior Research Methods, 57(1):1–15. find
  13. Jose Camacho-Collados, Mohammad Taher Pilehvar, Nigel Collier, and Roberto Navigli. 2017. SemEval-2017 task 2: Multilingual and cross-lingual semantic word similarity. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), page 15–26. find
  14. Billy Chiu, Sampo Pyysalo, Ivan Vulić, and Anna Korhonen. 2018. Bio-SimVerb and Bio-SimLex: wide-coverage evaluation sets of word similarity in biomedicine. BMC Bioinformatics, 19(1). find
  15. Gabriella Chronis and Katrin Erk. 2020. When is a bishop not like a rook? when it’s like a rabbi! multi-prototype BERT embeddings for estimating semantic relationships. Proceedings of the 24th Conference on Computational Natural Language Learning, page 227–244. find
  16. Simon De Deyne. 2024. Evaluating human-like similarity biases at every scale in large language models: Evidence from remote and basic-level triads. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 46. find
  17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1:… find
  18. Doubao Team. 2024. 8 key moments of Doubao Big Model in 2024. Accessed: 2025-01-15. find
  19. Anton Ehrmanntraut, Thora Hagen, Leonard Konle, and Fotis Jannidis. 2021. Type- and token-based word embeddings in the digital humanities. Workshop on Computational Humanities Research. find
  20. Lev Finkelstein, Evgeniy Gabrilovich1, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2002. Placing search in context: the concept revisited. ACM Transactions on Information Systems, 20:116–131. find
  21. Daniela Gerz, Ivan Vulić, Felix Hill, Roi Reichart, and Anna Korhonen. 2016. SimVerb-3500: A large-scale evaluation set of verb similarity. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. find
  22. Anna Gladkova and Aleksandr Drozd. 2016. Intrinsic evaluations of word embeddings: What can we do better? In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pages 36–42. find
  23. Hila Gonen, Srini Iyer, Terra Blevins, Noah Smith, and Luke Zettlemoyer. 2023. Demystifying prompts in language models via perplexity estimation. Findings of the Association for Computational Linguistics: EMNLP 2023, page 10136–10148. find
  24. Jack Grieve, Sara Bartl, Matteo Fuoli, Jason Grafmiller, Weihang Huang, Alejandro Jawerbaum, Akira Murakami, Marcus Perlman, Dana Roemling, and Bodo Winter. 2025. The sociolinguistic foundations of language modeling. Frontiers in Artificial Intelligence, 7:1472411. find
  25. Zellig S Harris. 1954. Distributional structure. Word, 10(2-3):146–162. find
  26. Yoshihiko Hayashi. 2025. Evaluating LLMs’ capability to identify lexical semantic equivalence: Probing with the word-in-context task. In Proceedings of the 31st International Conference on Computational Linguistics, pages 6985–6998, Abu Dhabi, UAE. Association for Computational Linguistics. find
  27. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. SimLex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695. find our card
  28. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  29. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438. find our card
  30. Jaap Jumelet, Willem Zuidema, and Arabella Sinclair. 2024. Do language models exhibit human-like structural priming effects? In Findings of the Association for Computational Linguistics: ACL 2024, pages 14727–14742, Bangkok, Thailand. Association for Computational Linguistics. find
  31. Jon A Krosnick and Leandre R Fabrigar. 1997. Designing rating scales for effective measurement in surveys. Survey measurement and process quality, pages 141–164. find
  32. Jonghyun Lee, Dojun Park, Jiwoo Lee, Hoekeon Choi, and Sung-Eun Lee. 2025. Exploring multimodal perception in large language models through perceptual strength ratings. IEEE Access, 13:176751–176769. find
  33. Alina Leidinger, Robert van Rooij, and Ekaterina Shutova. 2023. The language of prompting: What linguistic properties make a prompt successful? Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9210– 9232. find
  34. Alessandro Lenci. 2018. Distributional models of word meaning. Annual review of Linguistics, 4(1):151–171. find
  35. Ira Leviant and Roi Reichart. 2015. Separated by an un-common language: Towards judgment language informed vector space modeling. find
  36. Jeannette Littlemore, Paula Pérez Sobrino, David Houghton, Jinfang Shi, and Bodo Winter. 2018. What makes a good metaphor? A cross-cultural study of computer-generated metaphor appreciation. Metaphor and Symbol, 33(2):101–122. find
  37. Chaoqun Liu, Wenxuan Zhang, Yiran Zhao, Anh Tuan Luu, and Lidong Bing. 2025. Is translation all you need? A study on solving multilingual tasks with large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: H… find
  38. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). find our card
  39. Bolei Ma. 2024. Evaluating lexical aspect with large language models. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 123–131. find
  40. Gonzalo Martínez, Javier Conde, Pedro Reviriego, and Marc Brysbaert. 2024. AI-generated estimates of familiarity, concreteness, valence, and arousal for over 100,000 Spanish words. Quarterly Journal of Experimental Psychology, page 17470218241306694. find
  41. Gonzalo Martínez, Juan Diego Molero, Sandra González, Javier Conde, Marc Brysbaert, and Pedro Reviriego. 2025. Using large language models to estimate features of multi-word expressions: Concreteness, valence, arousal. Behavior Research Methods, 57(1):1–11. find
  42. Natalja Menold. 2020. Rating-scale labeling in online surveys: An experimental comparison of verbal and numeric rating scales with respect to measurement quality and respondents’ cognitive processes. Sociological Methods & Research, 49(1):79–107. find
  43. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of ICLR (Workshop Track). find
  44. Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madelei… find our card
  45. OpenAI. 2022. Introducing ChatGPT. Retrieved April 8, 2024 from https://openai.com/blog/chatgpt\#OpenAI. ↗
  46. OpenAI. 2023. GPT-4 Technical Report. Retrieved April 14, 2024 from https://doi.org/10.48550/arXiv.2303.08774. ↗ our card
  47. OpenAI. 2024a. Hello GPT-4o — OpenAI. Retrieved June 20, 2024 from https://openai.com/index/hello-gpt-4o/. ↗
  48. OpenAI. 2024b. Models — OpenAI API. Retrieved April 8, 2024 from https://platform.openai.com/docs/models. ↗
  49. OpenAI. 2024c. New embedding mod-Sean Trott. 2024. Can large language models help els and API updates. Retrieved April 8, augment English psycholinguistic datasets? Be-2024 from https://openai.com/blog/new-embedding-models-and-api-updates. havior Research Methods, pages 1–19. ↗
  50. OpenAI. 2024d. Pricing. Retrieved April 11, 2024 from https://openai.com/pricing. ↗
  51. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2024. Train… find
  52. Mohammad Taher Pilehvar, Dimitri Kartsaklis, Victor Prokhorov, and Nigel Collier. 2018. Card-660: Cambridge Rare Word Dataset - a reliable benchmark for infrequent word representation models. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, page 1391–1401. find
  53. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng… find
  54. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  55. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842– 866. find our card
  56. Timo Schick and Hinrich Schütze. 2022. True few-shot learning with Prompts—A real-world perspective. Transactions of the Association for Computational Linguistics, 10:716–731. find
  57. Tobias Schnabel, Igor Labutov, David Mimno, and Thorsten Joachims. 2015. Evaluation methods for unsupervised word embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 298– 307, Lisbon, Portugal. Association for Computational Linguistics. find
  58. Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7222–7240, Online. Association for Computational… find our card
  59. Ivan Vulić, Simon Baker, Edoardo Maria Ponti, Ulla Petti, Ira Leviant, Kelly Wing, Olga Majewska, Eden Bar, Matt Malone, Thierry Poibeau, Roi Reichart, and Anna Korhonen. 2020. Multi-SimLex: A large-scale evaluation of multilingual and crosslingual lexical semantic similarity. Computational Linguist… find
  60. Albert Webson and Ellie Pavlick. 2022. Do prompt-based models really understand the meaning of their prompts? Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2300–2344. find our card
  61. Qinyuan Ye, Maxamed Axmed, Reid Pryzant, and Fereshte Khani. 2023. Prompt engineering a prompt engineer. find
  62. Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. find our card
  63. Yanlu Zhong, Simon Todd, Nicole Xu, and Laurel Brehm. 2025. Evaluating LLMs as proxies for humans in psycholinguistic ratings: A comparison of statistical knowledge. Research Methods in Applied Linguistics, 4(3):100274. find
  64. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/snelder-2026-prompting-instruction-tuned-llms-semantic/