Elamur Research⌕
☰
home / papers / 2026 / Cryptix at SemEval-2026 Task 4: Zero-Shot Bi-Encoder Modeling for Narrative Story Similarity - A Sentence Transformer Approach

Cryptix at SemEval-2026 Task 4: Zero-Shot Bi-Encoder Modeling for Narrative Story Similarity - A Sentence Transformer Approach

Paper · 2026 · Sushmitha M, Sarath Kumar P, Thanalaxmi S et al.

Summary

The paper presents Team Cryptix's entry to SemEval-2026 Task 4, which asks systems to judge narrative similarity between story summaries. The judgment rests on theme, course of action and outcome, not word overlap. Data come as triples: an anchor story and two candidates, one of which is narratively closer. The team entered Track A, which makes a direct choice between candidates, and Track B, which outputs story embeddings scored by cosine distance. They did no training at all. The pretrained sentence-transformers/all-mpnet-base-v2 bi-encoder encodes each summary separately, truncated to 2000 characters, using mean pooling and L2 normalization. Cosine similarity then picks the closer candidate. Track A accuracy was 59.50%, ranking 37th, while Track B accuracy was 57.50%, ranking 25th. This beats chance but falls well short of the GPT-4o-mini baseline on Track A. The authors back this up with ablations and margin and error analyses. The work matters as a reproducible zero-shot reference point. It shows that general sentence encoders mainly capture topical overlap and miss causal progression and outcome alignment.

Key points

  • Official baselines were GPT-4o-mini at 73.00% (Track A) and 58.00% (Track B), and MiniLM-L6-v2 at 62.00% and 56.00%. Cryptix slightly beats MiniLM on Track B only.
  • Development-set ablations (MPNet baseline 61.5%):
    • Removing normalization had no effect, at 61.5%.
    • Sentence-level averaging dropped accuracy to 56.5%.
    • Truncating at 1000 characters gave 62.0%.
    • Swapping in RoBERTa-Large gave 57.5%.
  • The authors caution that 1–2 point differences may not be statistically significant, citing McNemar's test as one way to check.
  • Errors cluster near the decision boundary. The average cosine margin is 0.0606 for wrong predictions versus 0.1101 for correct ones.
  • The main failure mode is shared themes despite different event structure. Other failure types are:
    • missing structural parallels when wording differs;
    • diverging outcomes;
    • surface lexical bias.
  • Proposed next steps are:
    • fine-tuning on the training triples with triplet or MNR loss;
    • cross-encoders;
    • event-graph representations;
    • ensembles with generative LLM judgments.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Sushmitha M, Sarath Kumar P, Thanalaxmi S and Beulah A. 2026. Cryptix at SemEval-2026 Task 4: Zero-Shot Bi-Encoder Modeling for Narrative Story Similarity - A Sentence Transformer Approach. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.168

@inproceedings{m2026cryptix,
  title = {{Cryptix at SemEval-2026 Task 4: Zero-Shot Bi-Encoder Modeling for Narrative Story Similarity - A Sentence Transformer Approach}},
  author = {Sushmitha M, Sarath Kumar P, Thanalaxmi S and Beulah A},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.168},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.168}
}

References

6 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. In Advances in Neural Information Processing Systems, volume 33, pages 16156–16170.
  2. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada.…
  3. Nathanael Chambers and Dan Jurafsky. 2008. Unsupervised learning of narrative event chains. In Proceedings of ACL-08: HLT, pages 789–797, Columbus, Ohio. Association for Computational Linguistics. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  5. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  6. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT- find

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/m-2026-cryptix-semeval-2026-task-4/