Elamur Research⌕
☰
home / papers / 2026 / CuriosAI at SemEval-2026 Task 4: A Comprehensive Study of Zero-Shot versus Fine-Tuned Approaches for Narrative Similarity

CuriosAI at SemEval-2026 Task 4: A Comprehensive Study of Zero-Shot versus Fine-Tuned Approaches for Narrative Similarity

Paper · 2026 · Yuki Shibata, Hiroki Takushima, Fumika Beppu et al.

Summary

The paper describes a SoftBank team's entry to SemEval-2026 Task 4, which deals with judging how similar English short stories are at the narrative level. Track A asks which of two candidate stories is closer to an anchor. Track B asks for a fixed-size embedding per story. The authors compared three families of approaches:

  • off-the-shelf sentence encoders used without training, scoring candidates by cosine similarity;
  • multi-step prompting of GPT-4o and GPT-5;
  • fine-tuning bi-encoders and a cross-encoder on roughly 3,800 synthetic, LLM-generated training samples.

The untrained Sentence-T5-XL turned out to be the strongest system, reaching 67.5% on the 200-item development set. Every fine-tuned variant performed worse, losing between 2 and 18 points, and longer training made results steadily worse even as training loss fell. The authors attribute this to noisy synthetic labels, a stylistic gap between synthetic and real stories, and a different notion of "similarity" in the generated data. The study is a useful negative result: zero-shot performance should be checked before investing in fine-tuning on synthetic data.

Key points

  • Data: 200 development instances, 400 Track A test triplets and 849 Track B test narratives. Accuracy was the Track A metric.
  • Zero-shot development accuracy:
    • Sentence-T5-XL: 67.5% (95% CI 61.0–74.0)
    • mxbai-embed-large-v1: 66.0%
    • all-mpnet-base-v2: 63.5%
    • all-MiniLM-L6-v2: 59.5%
    • paraphrase-MiniLM-L3-v2: 57.0%
  • Fine-tuning results:
    • Sentence-T5-XL, fine-tuned with a triplet margin loss: fell from 67.5% to 61.5% after 1 epoch and to 49.5% after 10 epochs.
    • ms-marco-MiniLM-L-6-v2 cross-encoder: never beat its 52.0% zero-shot baseline.
  • Official test results (zero-shot Sentence-T5-XL): Track A reached 73.50% (rank 8) and Track B reached 63.00% (rank 15).
  • LLM prompting and errors: The GPT-4o/GPT-5 prompting could not be fully evaluated because of API rate limits and content filtering. Manual error analysis showed the model favours surface lexical overlap over deeper plot and theme alignment.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢SoftBank Corp.Tokyo · Japan2 works · 1 citation

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Yuki Shibata, Hiroki Takushima, Fumika Beppu, Aiswariya Manoj Kumar, Daichi Yamaga and Takayuki Hori. 2026. CuriosAI at SemEval-2026 Task 4: A Comprehensive Study of Zero-Shot versus Fine-Tuned Approaches for Narrative Similarity. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.62

@inproceedings{shibata2026curiosai,
  title = {{CuriosAI at SemEval-2026 Task 4: A Comprehensive Study of Zero-Shot versus Fine-Tuned Approaches for Narrative Similarity}},
  author = {Yuki Shibata, Hiroki Takushima, Fumika Beppu, Aiswariya Manoj Kumar, Daichi Yamaga and Takayuki Hori},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.62},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.62}
}

References

10 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. 1. Negative results: W…
  2. Hans Ole Hatzel, Ekaterina Artemova, Haimo Stiemer, Evelyn Gius, and Chris Biemann. 2026. SemEval-2026 Task 4: Narrative similarity and narrative representation learning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, CA, USA. Association for Comp… find
  3. Mixedbread.ai. 2024. mxbai-embed-large-v1: Mixedbread.ai embedding model. Model card. find
  4. Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. 2021. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. arXiv preprint arXiv:2108.08877.
  5. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, and 1 others. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems.
  6. Andrew J Reagan, Lewis Mitchell, Dilan Kiley, Christopher M Danforth, and Peter Sheridan Dodds. 2016. Computational approaches to narrative similarity. PLOS ONE, 11(1):e0146279. find
  7. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982– 3992,…
  8. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: Masked and permuted pre-training for language understanding. In Advances in Neural Information Processing Systems.
  9. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems. find
  10. Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel E Ho. 2021. When does pretraining help? assessing self-supervised learning for law and the casehold dataset. arXiv preprint arXiv:2104.08671. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shibata-2026-curiosai-semeval-2026-task-4/