Elamur Research⌕
☰
home / papers / 2026 / Ambirig at SemEval-2026 Task 5: Distributional Ordinal Modelling for Ambiguous Word Senses in Narrative Contexts

Ambirig at SemEval-2026 Task 5: Distributional Ordinal Modelling for Ambiguous Word Senses in Narrative Contexts

Paper · 2026 · Soumyajit Roy

Summary

SemEval-2026 Task 5 asks systems to score how plausible a given sense of a homonym is within a short English story, on a 1–5 scale. Human raters often disagree on these scores. The author frames the task as predicting an ordinal distribution over ratings rather than a single number. The system is a GlossBERT-style cross-encoder built on DeBERTa-v3-base. Its input pairs the sense definition and an example usage with the story, and the target word is wrapped in quotes. A softmax head outputs probabilities for the five ratings. It is trained against the empirical annotator vote distribution using a squared Earth Mover's Distance (EMD) loss over CDFs. With only 2,280 training items, the model reached Spearman 0.593 and accuracy 0.730 on the development set, and 0.49 and 66.6% on the blind test set. It outperformed MSE regression, contrastive, CORAL and entropy-based variants. TAPT, NLI initialisation and complementarity constraints all lowered results. The study suggests that, for truly ambiguous data, a loss that preserves disagreement matters more than model scale or auxiliary training.

Key points

  • Data: 2,280 training and 588 development instances. Each instance has a three-sentence precontext, an ambiguous sentence, an optional ending and a candidate sense.
  • Setup: microsoft/deberta-v3-base, AdamW with lr 1.5e-5, batch 8, 5 epochs and 300-token inputs. DeBERTa-Large (435M) overfit.
  • Development-set results:
    • EMD: ρ = 0.593, Acc = 0.730.
    • Entropy loss: 0.586.
    • CORAL: 0.518.
    • MSE regression: ≈0.00, because it collapsed to predicting around 3.4–3.5.
  • Failed additions:
    • Complementarity constraint: ρ = 0.135.
    • TAPT: ρ = 0.501.
    • NLI initialisation (cross-encoder/nli-deberta-v3-base): ρ = 0.447.
    • Adding a pairwise ranking loss brought no gain.
  • On high-variance items, EMD kept a lower MAE than a Cross-Entropy baseline. Code is announced at github.com/roysammy/ambirig.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Word sense disambiguation · 87 works

Cite

Cite as Soumyajit Roy. 2026. Ambirig at SemEval-2026 Task 5: Distributional Ordinal Modelling for Ambiguous Word Senses in Narrative Contexts. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.84

@inproceedings{roy2026ambirig,
  title = {{Ambirig at SemEval-2026 Task 5: Distributional Ordinal Modelling for Ambiguous Word Senses in Narrative Contexts}},
  author = {Soumyajit Roy},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.84},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.84}
}

References

10 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2009. Evaluation measures for ordinal regression. In 2009 Ninth International Conference on Intelligent Systems Design and Applications, pages 283–287. find
  2. Michele Bevilacqua, Tommaso Pasini, Alessandro Raganato, and Roberto Navigli. 2021. Recent trends in word sense disambiguation: A survey. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 4330– 4338. International Joint Conferences on Artifici… find
  3. Janosch Gehring, Selina Meyer, and Michael Roth. 2026. SemEval-2026 task 5: Rating plausibility of word senses in ambiguous stories through narrative understanding. In Proceedings of the 20th International Workshop on Semantic Evaluation, San Diego, California. Association for Computational Linguist… find
  4. Xin Geng and Quan Zhao. 2014. Label distribution learning. CoRR, abs/1408.6027. — [arXiv:1408.6027](https://arxiv.org/abs/1408.6027) find
  5. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. A… find
  6. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. DeBERTav3: Improving deBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. In The Eleventh International Conference on Learning Representations.
  7. Le Hou, Chen-Ping Yu, and Dimitris Samaras. 2017. Squared earth mover’s distance-based loss for training deep neural networks. Preprint, arXiv:1611.05916. — [arXiv:1611.05916](https://arxiv.org/abs/1611.05916) find
  8. Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing Huang. 2019. GlossBERT: BERT for word sense disambiguation with gloss knowledge. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-I…
  9. Ling Li and Hsuan-tien Lin. 2006. Ordinal regression by extended binary classification. In Advances in Neural Information Processing Systems, volume 19. MIT Press. find
  10. Roberto Navigli. 2009. Word sense disambiguation: A survey. ACM Comput. Surv., 41(2). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/roy-2026-ambirig-semeval-2026-task-5/