Elamur Research⌕
☰
home / papers / 2026 / schmerle at SemEval-2026 Task 4: Exploring Large Language Model Prompting Strategies for Low-Resource Narrative Similarity Detection

schmerle at SemEval-2026 Task 4: Exploring Large Language Model Prompting Strategies for Low-Resource Narrative Similarity Detection

Paper · 2026 · Maximilian Schmerle and Nils Constantin Hellwig

Summary

The paper addresses SemEval-2026 Task 4, Track A. Each item is a triple: an anchor story and two candidates. The system must decide which candidate is narratively closer to the anchor, judged by abstract theme, course of action and outcome. The authors use no fine-tuning and no extra annotation. They only prompt Gemma 3 (27B), run locally through Ollama, and force answers into a fixed JSON schema.

They cross three prompt templates with five structural strategies:

  • Templates: a near-copy of the official task description, a structured template with labelled sections, and a stripped-down "cognitive load reduction" version.
  • Strategies: zero-/few-shot prompting, story summarization, keyword extraction, per-aspect splitting with majority voting, and pairwise scoring of each candidate against the anchor.

The structured template did best overall. Its strongest pairing was with pairwise comparison, which reached 72.50% accuracy on the test set. The officially submitted zero-shot configuration scored 67.75%. This shows that careful prompt design alone gives a competitive, cheap and reproducible baseline for narrative similarity.

Key points

  • Setup: Gemma 3 (27B) ran on a single NVIDIA RTX PRO 6000 (96 GB) via Ollama v0.17.0. The authors stress that the model also fits 24 GB consumer GPUs.
  • Data:
    • Few-shot examples came from the sample split (39 items), using 0, 6, 12 or 18 class-balanced shots.
    • The dev split (200 items) was used to pick the submission, and results are reported on the test split (400 items).
    • The synthetic data (1,897 items) was not used.
    • Every configuration was run with three seeds.
  • Best result: structured template with pairwise scoring (1–10 scale), 72.50% on test. Roughly 30% of pairwise comparisons ended in a tie, and ties defaulted to Text B.
  • Leaderboard: the submission scored 67.75% and placed 23rd of 44 teams, about 11 points behind the leader, COGNAC (78.00%). The introduction instead says 20th of 42.
  • Few-shot effects:
    • Standard prompting peaked at about 6 shots.
    • Summarization peaked at about 12 shots.
    • Keyword extraction got steadily worse as more shots were added.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of RegensburgRegensburg · Germany5 works · 14 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Low-resource languages · 287 works

Cite

Cite as Maximilian Schmerle and Nils Constantin Hellwig. 2026. schmerle at SemEval-2026 Task 4: Exploring Large Language Model Prompting Strategies for Low-Resource Narrative Similarity Detection. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.257

@inproceedings{schmerle2026schmerle,
  title = {{schmerle at SemEval-2026 Task 4: Exploring Large Language Model Prompting Strategies for Low-Resource Narrative Similarity Detection}},
  author = {Maximilian Schmerle and Nils Constantin Hellwig},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.257},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.257}
}

References

31 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Roland Barthes and Lionel Duisit. 1975. An introduction to the structural analysis of narrative. New Literary History, 6(2):237–272. find
  2. Regina Barzilay and Mirella Lapata. 2005. Modeling local coherence: An entity-based approach. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 141–148, Ann Arbor, Michigan. Association for Computational Linguistics. find
  3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  4. Jia He, Mukund Rungta, David Koleczek, Arshdeep Sekhon, Franklin X Wang, and Sadid Hasan. 2024. Does prompt formatting have any impact on llm performance? Preprint, arXiv:2411.10541. — [arXiv:2411.10541](https://arxiv.org/abs/2411.10541) find
  5. Nikitas . Karanikolas, Eirini Manga, Nikoletta Samaridi, Vaios Stergiopoulos, Eleni Tousidou, and Michael Vassilakopoulos. 2025. Strengths and weaknesses of llm-based and rule-based nlp technologies and their potential synergies. Electronics, 14(15). find
  6. Wendy G. Lehnert. 1981. Plot units and narrative summarization. Cognitive Science, 5(4):293–331. find
  7. Claude Lévi-Strauss. 1955. The structural study of myth. The Journal of American Folklore, 68(270):428–444. find
  8. Yuetian Mao, Junjie He, and Chunyang Chen. 2025. From prompts to templates: A systematic prompt template analysis for real-world llmapps. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, FSE Companion ’25, page 75–86, New York, NY, USA. Association… find
  9. Sania Nayab, Giulio Rossolini, Giorgio Buttazzo, Nicolamaria Manes, and Fabrizio Giacomelli. 2024. Concise thoughts: Impact of output length on llm reasoning and cost. ArXiv, abs/2407.19825. — [arXiv:2407.19825](https://arxiv.org/abs/2407.19825) find
  10. Nathanael Chambers and Dan Jurafsky. 2008. Unsupervised learning of narrative event chains. In Proceedings of ACL-08: HLT, pages 789–797, Columbus, Ohio. Association for Computational Linguistics. find
  11. Zabir Al Nazi, Md. Rajib Hossain, and Faisal Al Mamun. 2025. Evaluation of open and closed-source llms for low-resource language with zero-shot, few-shot, and chain-of-thought prompting. Natural Language Processing Journal, 10:100124. find
  12. Jon Chun. 2024. AIStorySimilarity: Quantifying story similarity using narrative for search, IP infringement, and guided creativity. In Proceedings of the 28th Conference on Computational Natural Language Learning, pages 161–177, Miami, FL, USA. Association for Computational Linguistics. find
  13. Ahmed Hamza Osman, Naomie Salim, Mohammed Salem Binwahlan, Rihab Alteeb, and Albaraa Abuobieda. 2012. An improved plagiarism detection scheme based on semantic role labeling. Applied Soft Computing, 12(5):1493–1502. find
  14. Washington Cunha, Leonardo Rocha, and Marcos André Gonçalves. 2025. A thorough benchmark of automatic text classification: From traditional approaches to large language models. Preprint, arXiv:2504.01930. — [arXiv:2504.01930](https://arxiv.org/abs/2504.01930) find
  15. David Elson and Kathleen McKeown. 2010. Automatic attribution of quoted speech in literary narrative. Proceedings of the AAAI Conference on Artificial Intelligence, 24(1):1013–1019. find
  16. Tanzir Pial and Steven Skiena. 2023. GNAT: A general narrative alignment tool. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14636–14652, Singapore. Association for Computational Linguistics. find
  17. Karl Pichotta and Raymond Mooney. 2016. Learning statistical scripts with lstm recurrent neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1). find
  18. Gérard Genette. 1980. Narrative Discourse: An Essay in Method. Cornell University Press, Ithaca, NY. Foreword by Jonathan Culler. find
  19. Vladimir Propp. 1968. Morphology of the Folk Tale, 2nd edition, volume 10 of Bibliographical and Special Series. University of Texas Press, Austin. Translated by Laurence Scott. find
  20. Hans Ole Hatzel, Ekaterina Artemova, Haimo Stiemer, Evelyn Gius, and Chris Biemann. 2026. SemEval-2025 task 4: Narrative similarity and narrative representation learning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, CA, USA. find
  21. Seyed Mahmoud Sajjadi Mohammadabadi, Burak Cem Kara, Can Eyupoglu, Can Uzay, Mehmet Serkan Tosun, and Oktay Karakuş. 2025. A survey of large language models: Evolution, architectures, adaptation, benchmarking, applications, challenges, and societal implications. Electronics, 14(18). find
  22. Roger C. Schank and Robert P. Abelson. 1977. Scripts, Plans, Goals, and Understanding: An Inquiry into Human Knowledge Structures. The Artificial Intelligence Series. Lawrence Erlbaum Associates, Hillsdale, NJ. find
  23. Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Tao Yu. Selective annotation makes language models better few-shot learners. find
  24. Madhumita Sushil, Travis Zack, Divneet Mandair, Zhiwei Zheng, Ahmed Wali, Yan-Ning Yu, Yuwei Quan, Dmytro Lituiev, and Atul J Butte. 2024. A comparative study of large language model-based zero-shot inference and task-specific supervised classification of breast cancer pathology reports. Journal of… find
  25. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  26. Tzvetan Todorov. 1981. Introduction to Poetics, volume 1 of Theory and History of Literature. University of Minnesota Press, Minneapolis. find
  27. Bibek Upadhayay, Vahid Behzadan, and Amin Karbasi. 2024. Cognitive overload attack:prompt injection for long context. Preprint, arXiv:2410.11272. — [arXiv:2410.11272](https://arxiv.org/abs/2410.11272) find
  28. Ming Wang, Yuanzhong Liu, Xiaoyu Liang, Songlian Li, Yijie Huang, Xiaoming Zhang, Sijia Shen, Chaofeng Guan, Daling Wang, Shi Feng, Huaiwen Zhang, Yifei Zhang, Minghui Zheng, and Chi Zhang. 2024. Langgpt: Rethinking structured reusable prompt design framework for llms from the programming language.… — [arXiv:2402.16929](https://arxiv.org/abs/2402.16929) find
  29. Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatgpt. In Proceedings of the 30th Conference on Pattern Languages of Programs, PLoP ’23, U…
  30. Zheng Zhang, Jinyi Li, Yihuai Lan, Xiang Wang, and Hao Wang. 2025. An empirical study on prompt compression for large language models. Preprint, arXiv:2505.00019. — [arXiv:2505.00019](https://arxiv.org/abs/2505.00019) find
  31. Lixing Zhu, Runcong Zhao, Lin Gui, and Yulan He. 2023. Are NLP models good at tracing thoughts: An overview of narrative understanding. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10098–10121, Singapore. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schmerle-2026-schmerle-semeval-2026-task-4/