Elamur Research⌕
☰
home / papers / 2026 / Team CV at SemEval-2026 Task 4: Prompting LLMs and Benchmarking Embedding Models for Narrative Story Similarity

Team CV at SemEval-2026 Task 4: Prompting LLMs and Benchmarking Embedding Models for Narrative Story Similarity

Paper · 2026 · Chandan Kumar R S and Vinay Ulli

Summary

Team CV took part in both tracks of SemEval-2026 Task 4. The task asks systems to judge how similar short Wikipedia plot summaries from the Tell-Me-Again Corpus are as narratives, based on theme, event course and outcomes rather than names or settings. In Track A, a system sees an anchor and two candidate stories and picks the closer one. In Track B, each story is embedded on its own, and cosine similarity decides which candidate is closer.

For Track A, the authors tested five prompting setups on Qwen-2.5 7B and compared them with QLoRA fine-tuning of small Gemma-3 and Qwen models on about 1,900 synthetic triples. They then ran few-shot prompting across ten LLMs. For Track B, they benchmarked twelve embedding models ranging from 384 to 4096 dimensions and also fine-tuned Qwen3 Embedding 4B.

Simple few-shot prompting outperformed both elaborate reasoning and fine-tuning. Fine-tuning suffered from a mismatch between synthetic and human-annotated data. Both final submissions beat the official baselines, which suggests that general-purpose models can rival a dedicated narrative encoder. All systems still fall well short of the 89% human ceiling.

Key points

  • Track A prompting (dev, Qwen-2.5 7B):
    • 3-shot prompting reached 64.00%, while zero-shot stayed near chance (51.00%).
    • Chain-of-thought scored 61.50% and pairwise scoring 62.00%.
    • The best QLoRA model reached only 57.50%.
    • With 3-shot prompting, LLaMA-3.3-70B scored 75.00% on dev.
  • Track B embeddings (dev):
    • text-embedding-3-large (3072-d) led with 67.00%.
    • It was followed by E5-Large-v2 (1024-d, 66.50%) and Gemini Embedding 001 (66.00%).
    • Qwen3 Embedding 8B (4096-d) reached only 60.00%, so dimensionality alone does not predict performance.
    • all-MiniLM-L6-v2 scored 43.50%.
  • Fine-tuning: fine-tuning Qwen3 Embedding 4B on synthetic triples slightly lowered accuracy, from 64.50% to 63.50%.
  • Official test results:
    • Track A reached 70.75% with LLaMA-3.3-70B, against 67.00% for the GPT-4o-mini baseline.
    • Track B reached 64.50% with text-embedding-3-large, against 63.25% for STORY-EMB.
  • Error analysis: the authors examined 50 dev errors from LLaMA-3.3-70B. They found the model tends to follow topical keywords or shared genre instead of abstract plot structure. The fine-tuned models and dev embeddings are available in a HuggingFace collection.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

Cite

Cite as Chandan Kumar R S and Vinay Ulli. 2026. Team CV at SemEval-2026 Task 4: Prompting LLMs and Benchmarking Embedding Models for Narrative Story Similarity. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.374

@inproceedings{s2026team,
  title = {{Team CV at SemEval-2026 Task 4: Prompting LLMs and Benchmarking Embedding Models for Narrative Story Similarity}},
  author = {Chandan Kumar R S and Vinay Ulli},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.374},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.374}
}

References

14 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. BAAI Research Team. 2024. Bge-large-en-v1.5: BAAI general embedding model. https://huggingface.co/BAAI/bge-large-en-v1.5 . Accessed 2026-03-03. ↗
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), pages 1877– 19…
  3. Snigdha Chaturvedi, Shashank Srivastava, and Dan Roth. 2018. Where have i heard this story before? identifying narrative similarity in movie remakes. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol… find
  4. Xi Chen, Ali Zeynali, Chico Camargo, Fabian Flöck, Devin Gaffney, Przemyslaw Grabowicz, Scott A. Hale, David Jurgens, and Mattia Samory. 2022. Semeval-2022 task 8: Multilingual news article similarity. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Associat… find
  5. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023). ArXiv:2305.14314.
  6. Bernhard Fisseni and Benedikt Löwe. 2012. Which dimensions of narratives are relevant for human judgments of story equivalence? In Proceedings of the Workshop on Computational Models of Narrative (CMN 2012). find
  7. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  8. Hans Ole Hatzel, Ekaterina Artemova, Haimo Stiemer, Natalia Fedorova, Evelyn Gius, and Chris Biemann. 2026. Semeval-2026 task 4: Narrative story similarity and narrative representation learning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). Association for… find
  9. Hans Ole Hatzel and Chris Biemann. 2024a. Story embeddings – narrative-focused representations of fictional stories. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5931–5943, Miami, Florida, USA. Association for Computational Linguistics. find
  10. Hans Ole Hatzel and Chris Biemann. 2024b. Tell me again! a large-scale dataset of multiple summaries for the same story. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Torino, Italia. ELRA and ICCL. find
  11. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. Preprint, arXiv:2308.03281.
  12. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
  13. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533.
  14. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Fei Xia, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022).

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/s-2026-team-cv-semeval-2026-task/