Elamur Research⌕
☰
home / papers / 2026 / MarSan at SemEval-2026 Task 4: Narrative Similarity via Sentence-BERT Metric Learning with Triple-Derived Losses

MarSan at SemEval-2026 Task 4: Narrative Similarity via Sentence-BERT Metric Learning with Triple-Derived Losses

Paper · 2026 · Maryam Najafi, Ehsan Tavan and Simon Colreavy

Summary

SemEval-2026 Task 4 (NSNRL) asks systems to judge narrative similarity. Here similarity means shared plot logic, causal development and outcomes rather than shared wording. Each test item is a triple of an anchor story and two candidates. Track A asks which candidate is closer to the anchor. Track B asks for one embedding per story, and these must reproduce hidden triple orderings through cosine distance. The MarSan team uses a single model for both tracks: a Sentence-BERT bi-encoder built on gte-large-en-v1.5, with mean pooling and L2-normalised vectors. No sizeable official training set exists, so they fine-tune mainly on LLM-generated synthetic triples whose negatives vary in difficulty. They keep the development split only for checkpoint selection. Training mixes an in-batch contrastive loss with an explicit triplet margin constraint. The submitted system achieves 65.50% for Track B, with a Track A result of 65.00%. It beats the organisers' story-embedding baseline on Track B but falls well short of the leading entries. The work shows that one inexpensive embedding model can handle both tracks, and it also shows where single-vector story encodings fall short.

Key points

  • Data: Stories come from the English part of TELL-ME-AGAIN and are limited to summaries of 4–8 sentences.
    • Track A has 39 sample, 200 dev and 400 test triples.
    • Track B covers 1,436 unique stories.
    • The organisers released 1,900 synthetic triples; the local training pool used 1,700 synthetic instances.
  • Objective: The loss combines MultipleNegativesRankingLoss (L_MN), which shapes the global structure of the embedding space, with TripletLoss (L_tri) on cosine distance, which enforces the local ordering. An equivalent margin-ranking form exists, but only one of the two is optimised. Fine-tuning ran for 4 epochs with a maximum sequence length of 256.
  • Ablation: Track A accuracy rises step by step from 56.25% to 65.00%, and Track B from 63.25% to 65.50%. The starting point is an all-MiniLM-L6-v2 baseline.
    • Switching to gte-large-en-v1.5 gives the largest single improvement.
    • Text normalisation and optional proper-noun masking add a small final gain.
    • Used alone, L_MN helps more than L_tri, but combining the two works best.
  • Reference points: The GPT-4o-mini prompting baseline reaches 67.00% on Track A. The top leaderboard system, COGNAC, scores 78.00% on Track A and 72.00% on Track B.
  • Error analysis: Errors arise when candidates match the anchor on different aspects, such as event structure versus outcome, and when summaries leave out plot steps. Future work targets better hard negatives, multi-facet embeddings and distillation from cross-encoders. Code: github.com/MaryNJ1995/narrative_similarity2026.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: University of Limerick.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Maryam Najafi, Ehsan Tavan and Simon Colreavy. 2026. MarSan at SemEval-2026 Task 4: Narrative Similarity via Sentence-BERT Metric Learning with Triple-Derived Losses. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.33

@inproceedings{najafi2026marsan,
  title = {{MarSan at SemEval-2026 Task 4: Narrative Similarity via Sentence-BERT Metric Learning with Triple-Derived Losses}},
  author = {Maryam Najafi, Ehsan Tavan and Simon Colreavy},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.33},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.33}
}

References

23 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Prithviraj Ammanabrolu, Ethan Tien, Wesley Cheung, Zhaochen Luo, William Ma, Lara J. Martin, and Mark O. Riedl. 2020. Story realization: Expanding plot events into sentences. In Proceedings of AAAI.
  2. Ralph Allan Bradley and Milton E. Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3–4):324– 345. find
  3. Nathanael Chambers and Daniel Jurafsky. 2008. Unsupervised learning of narrative event chains. In Proceedings of ACL-08: HLT. find
  4. Snigdha Chaturvedi, Shashank Srivastava, and Dan Roth. 2018. Where have I heard this story before? identifying narrative similarity in movie remakes. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol… find
  5. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of ICML.
  6. Hans Ole Hatzel, Ekaterina Artemova, Haimo Stiemer, Evelyn Gius, and Chris Biemann. 2026. SemEval-2026 Task 4: Narrative similarity and narrative representation learning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, CA, USA. Association for Comp… find
  7. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of NAACL-HLT. find
  8. Dong Nguyen, Dolf Trieschnigg, and Mariët Theune. 2014. Using crowdsourcing to investigate perception of narrative similarity. In Proceedings of the 23rd ACM International Conference on Information and Knowledge Management (CIKM ’14), pages 321–330. ACM. find
  9. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of EMNLP-IJCNLP.
  10. Nils Reiter. 2014. Discovering Structural Similarities in Narrative Texts Using Event Alignment Algorithms. Phd thesis, Heidelberg University. find
  11. Nils Reiter, Anette Frank, and Oliver Hellwig. 2014. An nlp-based cross-document approach to narrative structure discovery. Literary and Linguistic Computing, 29(4):583–605. find
  12. Belen Saldias and Deb Roy. 2020. Exploring aspects of similarity between spoken personal narratives by disentangling them into narrative clause types. In Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events, pages 78–86, Online. Association for Computational Lin… find
  13. Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of CVPR.
  14. Hans Ole Hatzel and Chris Biemann. 2024. Tell me again! a large-scale dataset of multiple summaries for the same story. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 15732–15741, Torino, Italia. ELR… find
  15. Jocelyn Shen, Maarten Sap, Pedro Colon-Hernandez, Hae Park, and Cynthia Breazeal. 2023. Modeling empathic similarity in personal narratives. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6237–6252, Singapore. Association for Computational Linguistic… find
  16. Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In Proceedings of KDD. find
  17. Louis L. Thurstone. 1927. A law of comparative judgment. Psychological Review, 34(4):273–286. find
  18. Elektra Kypridemou and Loizos Michael. 2013. Narrative similarity as common summary. In 2013 Workshop on Computational Models of Narrative, volume 32 of OpenAccess Series in Informatics (OA-SIcs), pages 129–146. Schloss Dagstuhl–Leibniz-Zentrum für Informatik. find
  19. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. In arXiv preprint arXiv:1807.03748.
  20. Elektra Kypridemou and Loizos Michael. 2014. Narrative similarity as common summary: Evaluation of behavioral and computational aspects. Literary and Linguistic Computing, 29(4):532–560. find
  21. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281.
  22. Kilian Q. Weinberger and Lawrence K. Saul. 2009. Distance metric learning for large margin nearest neighbor classification. In Proceedings of NeurIPS. find
  23. Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, and 1 others. 2024. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval. arXiv preprint arXiv:2407.19669.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/najafi-2026-marsan-semeval-2026-task-4/