Elamur Research⌕
☰
home / papers / 2024 / Can LLMs assist with Ambiguity? A Quantitative Evaluation of various Large Language Models on Word Sense Disambiguation

Can LLMs assist with Ambiguity? A Quantitative Evaluation of various Large Language Models on Word Sense Disambiguation

Paper · 2024 · T.G.D.K. Sumanathilaka, Nicholas Micallef and Julian Hough

Summary

Large language models are tested here on Word Sense Disambiguation (WSD), especially for highly polysemous English words with many noun and verb senses, where classic supervised systems struggle because labelled data is scarce. The authors build a knowledge base of sense entries (gloss, synonyms, tags) from FEWS. Commercial and open-source LLMs then choose the right sense, guided by prompts developed with a human in the loop. Prompts were refined in stages: plain prompts, sentence augmentation, and a RAG-like setup that pairs the model with the KB. The final prompt adds POS tags, synonyms, aspect-based narrowing of candidate senses, and few-shot chain-of-thought reasoning. Models were scored on whether the gold sense was their single best answer (prediction level) or among their top two (suggestion level). Large frontier models reach high accuracy, while small open models fall well behind. Targeted prompt tricks recover part of the cases GPT models originally got wrong. This matters for translation, retrieval and misinformation-sensitive applications such as Cyber Threat Intelligence.

Key points

  • Evaluation data: 1050 FEWS test instances (400 nouns, 300 verbs, 300 adjectives, 50 adverbs), scored with FEWS sense tags.
  • Models tested: GPT-3.5 Turbo, GPT-4 Turbo, Gemini, Llama-2-70B, Llama-3-70B, Yi 34B, Gemma 7B, Mixtral and others. The extracted table also lists DeepSeek R1, o4-mini, Virtuoso-Large, GPT-4.1 mini and Gemini 2.0 Flash.
  • Best performers: Virtuoso-Large leads at suggestion level, while DeepSeek R1 and GPT-4 Turbo lead at prediction level, with top all-word scores of roughly 0.88–0.89. Llama-3-70B is the strongest open-source model but stays below GPT-4 Turbo. Exact per-model figures are hard to read reliably from the extracted table.
  • Part of speech: Nouns and adjectives are easier to disambiguate, and verbs remain the hardest category.
  • Error recovery: Prompt and parameter tuning was applied to earlier mistakes (234 instances for GPT-3.5, 191 for GPT-4) and corrected roughly 20–30% of them. The best result (0.30, GPT-4) came from prompt chaining combined with aspect-based sense filtering and synonyms. Self-consistency with majority voting also helped GPT-4.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Swansea UniversitySwansea · United Kingdom3 works · 14 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 worksWord sense disambiguation · 83 works

Cite

Cite as T.G.D.K. Sumanathilaka, Nicholas Micallef and Julian Hough. 2024. Can LLMs assist with Ambiguity? A Quantitative Evaluation of various Large Language Models on Word Sense Disambiguation. arXiv:2411.18337.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2411.18337 · arXiv 2411.18337

@misc{sumanathilaka2024can,
  title = {{Can LLMs assist with Ambiguity? A Quantitative Evaluation of various Large Language Models on Word Sense Disambiguation}},
  author = {T.G.D.K. Sumanathilaka, Nicholas Micallef and Julian Hough},
  year = {2024},
  doi = {10.48550/arxiv.2411.18337},
  eprint = {2411.18337},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2411.18337}
}

References

34 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abeysiriwardana, M., & Sumanathilaka, D. (2024). A Survey on Lexical Ambiguity Detection and Word Sense Disambiguation. https://doi.org/10.48550/ARXIV.2403.161 29 AI, 01, Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., Yu, K., Liu, P., Liu, Q., Yue… ↗
  2. Arazzi, M., Arikkat, D. R., Nicolazzo, S., Nocera, A., A., R. R. K., P., V., & Conti, M. (2023). NLP-Based Techniques for Cyber Threat Intelligence (arXiv:2311.08807). arXiv. http://arxiv.org/abs/2311.08807 — [arXiv:2311.08807](https://arxiv.org/abs/2311.08807) ↗
  3. Barba, E., Procopio, L., & Navigli, R. (2021). ConSeC: Word Sense Disambiguation as Continuous Sense Comprehension. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 1492–1503. https://doi.org/10.18653/v1/2021.emnlp-main.112 ↗
  4. Barba, E., Procopio, L., & Navigli, R. (2022). ExtEnD: Extractive Entity Disambiguation. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2478–2488. https://doi.org/10.18653/v1/2022.acl-long.177 ↗
  5. Bevilacqua, M., & Navigli, R. (2020). Breaking Through the 80% Glass Ceiling: Raising the State of the Art in Word Sense Disambiguation by Incorporating Knowledge Graph Information. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2854–2864. https://doi.org/10…
  6. Blevins, T., Joshi, M., & Zettlemoyer, L. (2021). FEWS: Large-Scale, Low-Shot Word Sense Disambiguation with the Dictionary. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, 455–465. https://doi.org/10.18653/v1/2021.eacl-main.3… ↗
  7. Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs (arXiv:2305.14314). arXiv. http://arxiv.org/abs/2305.14314 Gemma Team, Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., Ta… ↗
  8. Guo, Z., Jin, R., Liu, C., Huang, Y., Shi, D., Supryadi, Yu, L., Liu, Y., Li, J., Xiong, B., & Xiong, D. (2023a). Evaluating Large Language Models: A Comprehensive Survey (arXiv:2310.19736). arXiv. http://arxiv.org/abs/2310.19736 ↗
  9. Guo, Z., Jin, R., Liu, C., Huang, Y., Shi, D., Supryadi, Yu, L., Liu, Y., Li, J., Xiong, B., & Xiong, D. (2023b). Evaluating Large Language Models: A Comprehensive Survey (arXiv:2310.19736). arXiv. http://arxiv.org/abs/2310.19736 ↗
  10. Huang, L., Sun, C., Qiu, X., & Huang, X. (2020). GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge (arXiv:1908.07245). arXiv. http://arxiv.org/abs/1908.07245 Ji, D., & Xiao, G. (Eds.). (2013). Chinese Lexical Semantics: 13th Workshop, CLSW 2012, Wuhan, China, July 6-8, 2012, Revised… ↗
  11. Jia, L., Tang, J., Li, M., You, J., Ding, J., & Chen, Y. (2021). TWE‐WSD: An effective topical word embedding based word sense disambiguation. CAAI Transactions on Intelligence Technology, 6(1), 72–79. https://doi.org/10.1049/cit2.12006 Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B… ↗
  12. Kang, H., Blevins, T., & Zettlemoyer, L. (2023). Translate to Disambiguate: Zero-shot Multilingual Word Sense Disambiguation with Pretrained Language Models (arXiv:2304.13803). arXiv. http://arxiv.org/abs/2304.13803 ↗
  13. Koppula, N., Rao, K. S., & VeeraSekharReddy, B. (2021). find
  14. Word Sense Disambiguation Using Context Dependent Methods. 2021 5th International Conference on Trends in Electronics and Informatics (ICOEI), 1582–1590. https://doi.org/10.1109/ICOEI51242.2021. 9452823 Laatar, R., Aloulou, C., & Belguith, L. H. (2023). Evaluation of Stacked Embeddings for Arabic Wo… — [doi:10.1109/icoei51242.2021](https://doi.org/10.1109/icoei51242.2021) ↗
  15. Le, M., Postma, M., Urbani, J., & Vossen, P. (2018). A Deep Dive into Word Sense Disambiguation with LSTM. find
  16. Li, Y., Chen, J., Li, Y., Yu, T., Chen, X., & Zheng, H.-T. (2023). Embracing ambiguity: Improving similarity-oriented tasks with contextual synonym knowledge. Neurocomputing, 555, 126583. https://doi.org/10.1016/j.neucom.2023.126 583 Loureiro, D., Rezaee, K., Pilehvar, M. T., & Camacho-Collados, J.… ↗
  17. Luo, F., Liu, T., Xia, Q., Chang, B., & Sui, Z. (2018). Incorporating Glosses into Neural Word Sense Disambiguation (arXiv:1805.08028). arXiv. http://arxiv.org/abs/1805.08028 ↗
  18. Mente, R., Aland, S., & Chendage, B. (2022). Review of Word Sense Disambiguation and It’S Approaches. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4097221 — [doi:10.2139/ssrn.4097221](https://doi.org/10.2139/ssrn.4097221) ↗
  19. Nguyen, Q.-P., Vo, A.-D., Shin, J.-C., & Ock, C.-Y. (2018). Effect of Word Sense Disambiguation on Neural Machine Translation: A Case Study in Korean. IEEE Access, 6, 38512–38523. https://doi.org/10.1109/ACCESS.2018.28 51281 Orlando, R., Conia, S., Brignone, F., Cecconi, F., & Navigli, R. (2021). AM… ↗
  20. Ortega-Martín, M., García-Sierra, Ó., Ardoiz, A., Álvarez, J., Armenteros, J. C., & Alonso, A. (2023). Linguistic ambiguity analysis in ChatGPT (arXiv:2302.06426). arXiv. http://arxiv.org/abs/2302.06426 ↗
  21. Pan, X., Chen, Z., & Komachi, M. (2023). Query Generation Using GPT-3 for CLIP-Based Word Sense Disambiguation for Image Retrieval. Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), 417–422. https://doi.org/10.18653/v1/2023.starsem-1.36 ↗
  22. Pasini, T., Raganato, A., & Navigli, R. (2021). XL-WSD: An Extra-Large and Cross-Lingual Evaluation Framework for Word Sense Disambiguation. Proceedings of the AAAI Conference on Artificial Intelligence, 35(15), 13648–13656. https://doi.org/10.1609/aaai.v35i15.17609 — [doi:10.1609/aaai.v35i15.17609](https://doi.org/10.1609/aaai.v35i15.17609) ↗
  23. Raganato, A., Camacho-Collados, J., & Navigli, R. (2017). Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, 99– 110. https://doi.org/1…
  24. Rudnick, A. (2011). Towards Cross-Language Word Sense Disambiguation for Quechua. find
  25. Saidi, R., Jarray, F., Kang, J., & Schwab, D. (2023). GPT-2 Contextual Data Augmentation for Word Sense Disambiguation. find
  26. Sainz, O., de Lacalle, O. L., Agirre, E., & Rigau, G. (2023). What do Language Models know about word senses? Zero-Shot WSD with Language Models and Domain Inventories. find
  27. Scarlini, B., Pasini, T., & Navigli, R. (2020). Sense-Annotated Corpora for Word Sense Disambiguation in Multiple Languages and Domains. find
  28. Song, Y., Ong, X. C., Ng, H. T., & Lin, Q. (2021). Improved Word Sense Disambiguation with Enhanced Sense Representations. Findings of the Association for Computational Linguistics: EMNLP 2021, 4311–4320. https://doi.org/10.18653/v1/2021.findings-emnlp.365 ↗
  29. Subasinghe, T. (2020). SINSENSE - WORD SENSE DISAMBIGUATION TOOL FOR SINHALA LANGUAGE. find
  30. Sumanathilaka, D., Micallef, N., & Hough, J. (2024). Assessing GPT’s Potential for Word Sense Disambiguation: A Quantitative Evaluation on Prompt Engineering Techniques. IEEE 15th Control and System Graduate Research Colloquium (ICSGRC), 17 Aug 2024, Shah Alam, Selangor, Malaysia. https://doi.org/97…
  31. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019). SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems.
  32. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 353–355. https://doi.org/10.18653/…
  33. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models (arXiv:2203.11171). arXiv. http://arxiv.org/abs/2203.11171 ↗
  34. Yoon, H. S., Yoon, E., Harvill, J., Yoon, S., Hasegawa-Johnson, M., & Yoo, C. D. (2022). SMSMix: Sense-Maintained Sentence Mixup for Word Sense Disambiguation (arXiv:2212.07072). arXiv. http://arxiv.org/abs/2212.07072 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sumanathilaka-2024-llms-assist-ambiguity-quantitative-evaluation/