Elamur Research⌕
☰
home / papers / 2026 / Team p1 at SemEval-2026 Task 5: Semantic Bridge - Augmented Encoding for Word Sense Plausibility

Team p1 at SemEval-2026 Task 5: Semantic Bridge - Augmented Encoding for Word Sense Plausibility

Paper · 2026 · Pawan Rajpoot

Summary

This paper presents a submission to SemEval 2026 Task 5. The task asks systems to score, on a 1–5 scale, how well a given sense of an ambiguous English homonym fits a short story. A prediction counts as correct if it lies within max(SD, 1.0) of the mean of five annotators' ratings. Homonyms do not overlap between train, dev and test.

Rather than asking an LLM for the final number, the author uses LLMs as feature producers. DeepSeek-Chat-V3 generates chain-of-thought fit labels (POOR/PARTIAL/STRONG) in English, Hindi and Korean, gives 4-shot ratings, and translates stories into Hindi and Korean. The idea is that translation forces a commitment to one sense. These reasoning outputs are distilled into Qwen-2.5-7B with LoRA.

System A feeds eight numeric features into XGBoost with Huber loss. System B fine-tunes the bge-reranker-v2-m3 cross-encoder on Korean stories plus English and Korean reasoning text.

An equal-weight average of the two reaches 92.37% test accuracy, above the estimated 89.2% human ceiling. This suggests that representing the same LLM signals in different ways yields complementary errors.

Key points

  • Dataset: 3,798 story–sense pairs (2,280 train / 588 dev / 930 test) covering 361 unique homonyms, disjoint across splits. Mean annotator SD is 0.95.
  • Test accuracy: System A scores 89.78%, System B (retrained on train+dev) 89.35%, and the 50/50 ensemble 859/930 = 92.37%. The ensemble reaches Spearman ρ = 0.8384 and MAE = 0.53.
  • DeepSeek-V3 baselines: 4-shot prompting reaches 81.6%. The author's zero-shot run reached 71.29%, versus the 79.0% reported for the benchmark.
  • Ablations:
    • System B on dev scores 71.43% with no enrichment, 86.90% with English only, and 88.27% with English + Korean (best). Adding Hindi lowers it to 85.88%.
    • In System A, the 4-shot score alone as a regressor input gives only 76.88%.
  • Error analysis: The ensemble makes 71 errors. 82% are shrinkage toward the scale midpoint, and 54% miss the tolerance by under 0.3. "disappear" is the worst homonym.
  • Released resources: Code is at github.com/pawan2411/semeval26-task5, and the fine-tuned model is pawan2411/semstage7b on Hugging Face.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Word sense disambiguation · 83 works

Cite

Cite as Pawan Rajpoot. 2026. Team p1 at SemEval-2026 Task 5: Semantic Bridge - Augmented Encoding for Word Sense Plausibility. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.388

@inproceedings{rajpoot2026team,
  title = {{Team p1 at SemEval-2026 Task 5: Semantic Bridge - Augmented Encoding for Word Sense Plausibility}},
  author = {Pawan Rajpoot},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.388},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.388}
}

References

21 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Carmen Banea, Rada Mihalcea, and Janyce Wiebe. 2011. Word sense disambiguation with multilingual features. In Proceedings of the Ninth International Conference on Computational Semantics (IWCS 2011). find
  2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642. Association for Computational Linguistics.
  3. Katrin Erk and Diana McCarthy. 2009. Graded word sense assignment. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 440–449. Association for Computational Linguistics. find
  4. Janosch Gehring, Selina Meyer, and Michael Roth. 2026. SemEval-2026 task 5: Rating plausibility of word senses in ambiguous stories through narrative understanding. In Proceedings of the 20th International Workshop on Semantic Evaluation, San Diego, California. Association for Computational Linguist… find
  5. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, pages 1321–1330. find
  6. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
  7. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shanen Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  8. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. BGE M3-Embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216.
  9. Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing Huang. 2019. GlossBERT: BERT for word sense disambiguation with gloss knowledge. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-I…
  10. Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794. ACM. find
  11. Els Lefever and Véronique Hoste. 2010. SemEval-2010 task 3: Cross-lingual word sense disambiguation. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 15–20. Association for Computational Linguistics. find
  12. Alexis Conneau, Karttikeya Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association… find
  13. DeepSeek-AI. 2024. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437. find
  14. Mona Diab and Philip Resnik. 2002. An unsupervised method for word sense tagging using parallel corpora. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 255–262. Association for Computational Linguistics. find
  15. Thomas G. Dietterich. 2000. Ensemble methods in machine learning. In International Workshop on Multiple Classifier Systems, pages 1–15. Springer. find
  16. Daniel Loureiro, Jose Camacho-Collados, and Luis Espinosa-Anke. 2021. Analysis and evaluation of language models for word sense disambiguation. Computational Linguistics, 47(2):387–443. find
  17. George A. Miller. 1995. WordNet: A lexical database for English. Communications of the ACM, 38(11):39– 41.
  18. Roberto Navigli. 2009. Word sense disambiguation: A survey. ACM Computing Surveys, 41(2):1–69. find
  19. John C. Platt. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers, pages 61–74. MIT Press. find
  20. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35.
  21. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Wang, Bowen Zheng, Chengyuan Yu, Dayiheng Li, and 1 others. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rajpoot-2026-team-p1-semeval-2026-task/