Elamur Research⌕
☰
home / papers / 2026 / blue at SemEval-2026 Task 5: NarrBERT : Narrative-Aware BERT for Word Sense Disambiguation

blue at SemEval-2026 Task 5: NarrBERT : Narrative-Aware BERT for Word Sense Disambiguation

Paper · 2026 · Rhea Singhal, Krish Sharma, Lakksh Sharma et al.

Summary

Team blue's entry to SemEval-2026 Task 5 (AmbiStory) tackles graded word sense disambiguation. Instead of selecting one correct sense, a system must give each candidate sense of an ambiguous word a 1–5 plausibility rating that tracks averaged human judgments. The input is a five-sentence story whose ending may resolve the ambiguity. The authors extend the Gloss-BERT idea from single sentences to whole narratives. A bert-base-uncased cross-encoder reads the joined pre-context, ambiguous sentence and ending as one segment, and the sense gloss with its example usage as the other. A linear head, trained with MSE, regresses a score. Raw outputs tend to bunch in the middle of the scale, so a least-squares linear calibration fitted on the development set stretches them across the full range. On dev, the tuned model reaches Spearman 0.66, compared with 0.52 for a sentence-transformer bi-encoder. On the hidden test set it scores 0.4866 Spearman and 0.6613 Accuracy-within-SD. The authors attribute the drop to overfitting and dev/test distribution shift. The results suggest that joint token-level attention between story and definition outperforms embedding similarity for this task.

Key points

  • Input format: [CLS] narrative [SEP] definition + example [SEP], with max length 512 tokens and WordPiece tokenization. Truncation was rare and mostly cut the example field.
  • Training setup: AdamW with lr 1e-5 and linear decay, effective batch size 16 (8 × 2 gradient accumulation), 10 epochs, FP16, and checkpoint selection by dev Spearman rather than loss.
  • Dev ablation (Spearman):
    • Sentence-Transformer bi-encoder: 0.52
    • Initial BERT-Base cross-encoder: 0.58
    • Tuned BERT-Base cross-encoder: 0.66
    • BERT-Large cross-encoder: 0.64; BERT-Base was kept for efficiency.
  • Official test results: Spearman 0.4866 and Accuracy-within-SD 0.6613 (615 of 930 instances). Adding dropout 0.2 and weight decay 0.01 did not close the dev–test gap.
  • Code is available at github.com/RheaSinghal/SemEval-Task5. The system is built with PyTorch and Hugging Face transformers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Thapar Institute of Engineering & TechnologyPatiāla · India6 works · 11 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Word sense disambiguation · 83 works

Cite

Cite as Rhea Singhal, Krish Sharma, Lakksh Sharma and Jatin Bedi. 2026. blue at SemEval-2026 Task 5: NarrBERT : Narrative-Aware BERT for Word Sense Disambiguation. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.306

@inproceedings{singhal2026blue,
  title = {{blue at SemEval-2026 Task 5: NarrBERT : Narrative-Aware BERT for Word Sense Disambiguation}},
  author = {Rhea Singhal, Krish Sharma, Lakksh Sharma and Jatin Bedi},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.306},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.306}
}

References

11 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  2. Katrin Erk and Diana McCarthy. 2009. Graded word sense assignment. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 440–449, Singapore. Association for Computational Linguistics. find
  3. Janosch Gehring, Selina Meyer, and Michael Roth. 2026. SemEval-2026 task 5: Rating plausibility of word senses in ambiguous stories through narrative understanding. In Proceedings of the 20th International Workshop on Semantic Evaluation, San Diego, California. Association for Computational Linguist… find
  4. Christian Hadiwinoto, Hwee Tou Ng, and Wee Chung Gan. 2019. Improved word sense disambiguation using pre-trained contextualized word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 5300–5309, Hong Kong, China. Association for Computat… find
  5. Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing Huang. 2019. GlossBERT: BERT for word sense disambiguation with gloss knowledge. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 3509–3514, Hong Kong, China. Association for Computational Linguistics.
  6. Roberto Navigli. 2009. Word sense disambiguation: A survey. ACM Computing Surveys (CSUR), 41(2):1– 69. find
  7. Alessandro Raganato, Jose Camacho-Collados, and Roberto Navigli. 2017. Word sense disambiguation: A unified evaluation framework and empirical comparison. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 99–110, Valencia, Spain. As…
  8. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  9. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  10. Dayu Yuan, Julian Richardson, Ryan Doherty, Colin Evans, and Eric Altendorf. 2016. Semi-supervised word sense disambiguation with neural models. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1374–1385, Osaka, Japan. find
  11. The COL-ING 2016 Organizing Committee. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/singhal-2026-blue-semeval-2026-task-5/