Elamur Research⌕
☰
home / papers / 2026 / CUCLASIC at SemEval-2026 Task 5: LLM Prompting Strategies for Rating Ambiguous Word Senses

CUCLASIC at SemEval-2026 Task 5: LLM Prompting Strategies for Rating Ambiguous Word Senses

Paper · 2026 · Federico Ortega Riba, Jasper Wilkerson and Kelsey Lafreniere Adams

Summary

The paper describes a system for SemEval-2026 Task 5. The task treats word sense disambiguation as graded plausibility rating rather than a single correct label. Systems receive short English narratives from the AmbiStory dataset, each built around a homonym with two plausible readings. They must predict how well a given sense fits, matching the averaged 1–5 Likert judgments of five human annotators. The authors tested six large language models, drawn from both the Gemini and Llama lines, using zero-shot, 3-shot and 5-shot prompting. The prompts used an expert-linguist persona and asked for strict JSON output. They compared two kinds of demonstrations: contentious items with high annotator disagreement, and items with near-unanimous ratings. The near-unanimous demonstrations worked better. Gemini 3-Flash performed best, reaching 0.75 on the official metric. Across models, the authors found a strong tendency to avoid middle ratings, especially 3, even though 3 is the most common human score. This points to overconfidence as the main obstacle when LLMs must model graded, subjective ambiguity.

Key points

  • Data and metrics: AmbiStory supplies 2,280 training items, each with a precontext, an ambiguous sentence and an optional ending. Systems are scored on Spearman correlation and on accuracy within one standard deviation of the human mean, and the two are averaged.
  • Models: gemini-2.5-pro, gemini-3-pro-preview, gemini-3-flash-preview, Llama4 Maverick-17B-128E, Llama4 Scout-17B-16E and Llama 3.3-70B-Instruct. All were run via API at temperature 0.0.
  • Best result: Gemini 3-Flash scored 0.75 overall, with Spearman 0.73 and accuracy 0.77. Its per-score accuracy was lowest for predictions of 3 (0.69).
  • Model behavior: Llama models predicted 5 almost every time. Most Gemini models gave mostly 1s and 5s. The authors attribute the weaker result of the larger Pro model to inverse scaling and RLHF-induced overconfidence.
  • Open questions: The authors did not run ablations of individual prompt fields. They also did not test in a controlled way why low-standard-deviation examples help. Code and prompts are available on GitHub (fede-ortega/SemEval-2026-SharedTask-5).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Colorado BoulderBoulder · United States69 works · 51,069 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksWord sense disambiguation · 83 works

Cite

Cite as Federico Ortega Riba, Jasper Wilkerson and Kelsey Lafreniere Adams. 2026. CUCLASIC at SemEval-2026 Task 5: LLM Prompting Strategies for Rating Ambiguous Word Senses. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.121

@inproceedings{riba2026cuclasic,
  title = {{CUCLASIC at SemEval-2026 Task 5: LLM Prompting Strategies for Rating Ambiguous Word Senses}},
  author = {Federico Ortega Riba, Jasper Wilkerson and Kelsey Lafreniere Adams},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.121},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.121}
}

References

14 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pierpaolo Basile, Lucia Siciliani, Elio Musacchio, and Giovanni Semeraro. 2025. Exploring the word sense disambiguation capabilities of large language models. arXiv preprint arXiv:2503.08662. — [arXiv:2503.08662](https://arxiv.org/abs/2503.08662) find
  2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  3. Philip Edmonds and Scott Cotton. 2001. SENSEVAL-2: Overview. In Proceedings of SENSEVAL-2 Second International Workshop on Evaluating Word Sense Disambiguation Systems, pages 1–5, Toulouse, France. Association for Computational Linguistics.
  4. Janosch Gehring, Selina Meyer, and Michael Roth. 2026. SemEval-2026 task 5: Rating plausibility of word senses in ambiguous stories through narrative understanding. In Proceedings of the 20th International Workshop on Semantic Evaluation, San Diego, California. Association for Computational Linguist… find
  5. Raihan Kibria, Sheikh Intiser Uddin Dipta, and Muhammad Abdullah Adnan. 2024. On functional competence of LLMs for linguistic disambiguation. In Proceedings of the 28th Conference on Computational Natural Language Learning, pages 143–160, Miami, FL, USA. Association for Computational Linguistics. find
  6. Robert Krovetz and W Bruce Croft. 1989. Word sense disambiguation using machine-readable dictionaries. In Proceedings of the 12th annual international ACM SIGIR conference on Research and development in information retrieval, pages 127–136. find
  7. Federico Martelli, Najla Kalach, Gabriele Tola, and Roberto Navigli. 2021. SemEval-2021 task 2: Multilingual and cross-lingual word-in-context disambiguation (MCL-WiC). In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 24–36, Online. Association for Compu… find
  8. Rada Mihalcea, Timothy Chklovski, and Adam Kilgarriff. 2004. The senseval-3 English lexical sample task. In Proceedings of SENSEVAL-3, the Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text, pages 25–28, Barcelona, Spain. Association for Computational Linguis…
  9. Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2025. Large language models: A survey. Preprint, arXiv:2402.06196.
  10. Fengfei Sun, Ningke Li, Kailong Wang, and Lorenz Goette. 2025. Large language models are overconfident and amplify human bias. arXiv preprint arXiv:2505.02151. — [arXiv:2505.02151](https://arxiv.org/abs/2505.02151) find
  11. Liqiang Ming, Sheng-hua Zhong, and Luncong Li. 2025. Towards general-domain word sense disambiguation: Distilling large language model into compact disambiguator. In Conference on Empirical Methods in Natural Language Processing, pages 884–897. Association for Computational Linguistics. find
  12. Jean Veronis and Nancy Ide. 1990. Word sense disambiguation with very large neural networks extracted from machine readable dictionaries. In COLING 1990 Volume 2: Papers presented to the 13th International Conference on Computational Linguistics. find
  13. Andrea Moro and Roberto Navigli. 2015. SemEval-2015 task 13: Multilingual all-words sense disambiguation and entity linking. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 288–297, Denver, Colorado. Association for Computational Linguistics. find
  14. Roberto Navigli, David Jurgens, and Daniele Vannella. 2013. SemEval-2013 task 12: Multilingual word sense disambiguation. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/riba-2026-cuclasic-semeval-2026-task-5/