Elamur Research⌕
☰
home / papers / 2026 / PEU Lab at SemEval-2026 Task 4: Pairwise Text Comparison using RoBERTa and Ranking Loss

PEU Lab at SemEval-2026 Task 4: Pairwise Text Comparison using RoBERTa and Ranking Loss

Paper · 2026 · Hangchao Ma, Jiaxu Dao, Jinli Tong et al.

Summary

This system paper addresses Track A of SemEval-2026 Task 4 (Narrative Story Similarity). Given an anchor story and two candidates, a system must decide which candidate is narratively closer to the anchor. The PEU Lab team wanted a cheaper alternative to LLM-based pipelines.

Their approach scores each anchor–candidate pair with a shared-weight (Siamese) RoBERTa-Large encoder. A linear head on the <s> token turns each pair into a scalar similarity score. Instead of cross-entropy, training uses a margin ranking loss, which requires the preferred pair to outscore the other by a fixed margin. The training data combines the small official dev set with the organizers' synthetic triplets. Five models are trained via 5-fold cross-validation, and their scores are averaged at inference.

On the official test set, the ensemble reaches 64.50% accuracy, 2.5 points below the GPT-4o-mini baseline. Ablations indicate that the ranking objective helps more than switching encoders. The work suggests that a fine-tuned ~355M-parameter model is a practical, locally deployable option for comparative narrative reasoning.

Key points

  • Training corpus: 200 official dev triplets plus 2,097 synthetic triplets (2,297 total, 50.8% positive). The test set has 400 triplets, and the code is released on GitHub (mhchhh/SemEval2026-Task-4).
  • Setup: roberta-large, AdamW, learning rate 1e-5, batch size 32, max length 512, 10 epochs per fold, margin γ = 0.5, and FP16 AMP on a single A100.
  • Test accuracy:
    • RoBERTa-Base (ranking): 59.80%
    • DeBERTa-Large (CE): 61.35%
    • RoBERTa-Large (CE): 62.15%
    • Single-fold RoBERTa-Large (ranking): 63.20%
    • 5-fold ensemble: 64.50%
    • GPT-4o-mini baseline: 67.00%
  • The 5-fold mean ensemble adds +1.30 points over a single model.
  • Error analysis: accuracy falls to 56.4% on triplets longer than 650 words, which the authors attribute to truncation at the 512-token limit. They propose long-context models (e.g., Longformer) and event-graph structure as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Puer University.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Hangchao Ma, Jiaxu Dao, Jinli Tong, Zhuoying Li, Qingsong Zhou and Xiuzhong Tang. 2026. PEU Lab at SemEval-2026 Task 4: Pairwise Text Comparison using RoBERTa and Ranking Loss. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.136

@inproceedings{ma2026peu,
  title = {{PEU Lab at SemEval-2026 Task 4: Pairwise Text Comparison using RoBERTa and Ranking Loss}},
  author = {Hangchao Ma, Jiaxu Dao, Jinli Tong, Zhuoying Li, Qingsong Zhou and Xiuzhong Tang},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.136},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.136}
}

References

3 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sumit Chopra, Raia Hadsell, and Yann LeCun. 2005. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 1, pages 539–546. IEEE. find
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  3. Hans Ole Hatzel, Ekaterina Artemova, Haimo Stiemer, Evelyn Gius, and Chris Biemann. 2026. SemEval-2026 Task 4: Narrative similarity and narrative representation learning. In Proceedings of the 20th International Workshop on Semantic Evaluation find

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ma-2026-peu-lab-semeval-2026-task/