Elamur Research⌕
☰
home / papers / 2026 / SwanNLP at SemEval-2026 Task 5: An LLM-based Framework for Plausibility Scoring in Narrative Word Sense Disambiguation

SwanNLP at SemEval-2026 Task 5: An LLM-based Framework for Plausibility Scoring in Narrative Word Sense Disambiguation

Paper · 2026 · Deshan Sumanathilaka, Nicholas Micallef, Julian Hough et al.

Summary

This paper describes the SwanNLP system for SemEval-2026 Task 5. The task asks systems to predict how plausible humans find a given sense of a homonym inside a short story, on a 1–5 scale. The data come from the AmbiStory dataset. Each instance has a multi-sentence precontext, an ambiguous sentence and sometimes an ending. Five Prolific annotators rated every instance.

The authors compared three strategies:

  • LoRA fine-tuning of small LLMs (Gemma 4B, Qwen 4B). They tried four reasoning designs: single-annotator scoring, simulating five annotators, plausibility reasoning guided by GLOSSGPT sense predictions, and difficulty-aware reasoning.
  • Dynamic few-shot prompting of commercial models. Examples were retrieved per difficulty category, RAG-style, and combined with CoT reasoning.
  • Ensembling. Earlier predictions were combined to imitate agreement among several annotators.

Large commercial models with retrieved in-context examples came closest to human judgments, and GPT-4o was the strongest. Ensembling added a small further gain. The team finished 10th overall. The work matters because it shows that narrative-level sense plausibility remains hard, especially when humans themselves disagree.

Key points

  • AmbiStory size: 2322 training stories, 600 dev and 942 test, covering 188, 42 and 76 unique senses respectively.
  • Evaluation metrics: Spearman correlation with the mean human rating, and accuracy within one standard deviation of that mean.
  • Small models: Difficulty-aware fine-tuning gave the best results among small models. Qwen 4B reached a test Spearman of 0.491 and accuracy of 0.631. Simulating five annotators tended to underperform single-annotator setups.
  • Commercial models: GPT-4o reached a test Spearman of 0.741 even zero-shot, and dynamic few-shot prompting added modest gains. One example per category worked best on test, while three per category worked best on dev.
  • Ensembles and ranking: On test, the Linear Regression Ensemble gave the best accuracy (0.797) and XGBOOST the best Spearman (0.724). The final leaderboard result was 10th place with an average score of 0.760.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Swansea UniversitySwansea · United Kingdom3 works · 14 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Word sense disambiguation · 83 works

Cite

Cite as Deshan Sumanathilaka, Nicholas Micallef, Julian Hough and Saman Jayasinghe. 2026. SwanNLP at SemEval-2026 Task 5: An LLM-based Framework for Plausibility Scoring in Narrative Word Sense Disambiguation. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.109

@inproceedings{sumanathilaka2026swannlp,
  title = {{SwanNLP at SemEval-2026 Task 5: An LLM-based Framework for Plausibility Scoring in Narrative Word Sense Disambiguation}},
  author = {Deshan Sumanathilaka, Nicholas Micallef, Julian Hough and Saman Jayasinghe},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.109},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.109}
}

References

18 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohamad Ballout, Anne Dedert, Nohayr Muhammad Abdelmoneim, Ulf Krumnack, Gunther Heidemann, and Kai-Uwe Kühnberger. 2024. Fool me if you can! an adversarial dataset to investigate the robustness of lms in word sense disambiguation. In Proceedings of the 2024 Conference on Empirical Methods in Natura… find
  2. Pierpaolo Basile, Lucia Siciliani, Elio Musacchio, and Giovanni Semeraro. 2025. Exploring the word sense disambiguation capabilities of large language models. arXiv preprint arXiv:2503.08662. — [arXiv:2503.08662](https://arxiv.org/abs/2503.08662) find
  3. Michele Bevilacqua, Tommaso Pasini, Alessandro Raganato, and Roberto Navigli. 2021. Recent trends in word sense disambiguation: A survey. In International joint conference on artificial intelligence, pages 4330–4338. International Joint Conference on Artificial Intelligence, Inc. find
  4. Terra Blevins, Mandar Joshi, and Luke Zettlemoyer. 2021. FEWS: Large-scale, low-shot word sense disambiguation with the dictionary. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 455–465, Online. Association for Comp… find
  5. Samuel Cahyawijaya, Ruochen Zhang, Holy Lovenia, Jan Christian Blaise Cruz, Hiroki Nomoto, and Alham Fikri Aji. 2024. Thank you, stingray: Multilingual large language models can not (yet) disambiguate cross-lingual word sense. arXiv preprint arXiv:2410.21573. — [arXiv:2410.21573](https://arxiv.org/abs/2410.21573) find
  6. Janosch Gehring, Selina Meyer, and Michael Roth. 2026. SemEval-2026 task 5: Rating plausibility of word senses in ambiguous stories through narrative understanding. In Proceedings of the 20th International Workshop on Semantic Evaluation, San Diego, California. Association for Computational Linguist… find
  7. Janosch Gehring and Michael Roth. 2025. AmbiStory: A challenging dataset of lexically ambiguous short stories. In Proceedings of the 14th Joint Conference on Lexical and Computational Semantics (*SEM 2025), pages 152–171, Suzhou, China. Association for Computational Linguistics. find
  8. Debanjan Ghosh, Kai North, Ekaterina Kochmar, Mamoru Komachi, and Marcos Zampieri, editors. 2026. Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). Association for Computational Linguistics, San Diego, United States. find
  9. Alessandro Raganato, Jose Camacho-Collados, and Roberto Navigli. 2017. Word sense disambiguation: A unified evaluation framework and empirical comparison. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 99–1…
  10. Oscar Sainz, Oier Lopez de Lacalle, Eneko Agirre, and German Rigau. 2023. What do Language Models know about word senses? Zero-Shot WSD with Language Models and Domain Inventories. find
  11. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Ng. 2008. Cheap and fast – but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 254–263, Honolulu, Hawaii. Association fo…
  12. Deshan Sumanathilaka, Nicholas Micallef, and Julian Hough. 2024a. Assessing gpt’s potential for word sense disambiguation: A quantitative evaluation on prompt engineering techniques. In 2024 IEEE 15th Control and System Graduate Research Colloquium (ICSGRC), pages 204–209. IEEE. find
  13. Deshan Sumanathilaka, Nicholas Micallef, and Julian Hough. 2025a. Exploring the impact of temperature on large language models: A case study for classification task based on word sense disambiguation. In 2025 7th International Conference on Natural Language Processing (ICNLP), pages 178–182. IEEE. find
  14. Deshan Sumanathilaka, Nicholas Micallef, and Julian Hough. 2025b. GlossGPT: GPT for Word Sense Disambiguation using Few-shot Chain-of-Thought Prompting. Procedia Computer Science. find
  15. Deshan Sumanathilaka, Nicholas Micallef, and Julian Hough. 2026. An exploration-analysis-disambiguation reasoning framework for word sense disambiguation with low-parameter llms. arXiv preprint arXiv:2603.05400. find
  16. Deshan Koshala Sumanathilaka, Nicholas Micallef, and Julian Hough. 2024b. Can llms assist with ambiguity? a quantitative evaluation of various large language models on word sense disambiguation. In Proceedings of the First International Conference on Natural Language Processing and Artificial Intell…
  17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  18. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sumanathilaka-2026-swannlp-semeval-2026-task-5/