Elamur Research⌕
☰
home / papers / 2026 / hermeneutichools at SemEval-2026 Task 4: Multiperspectivity as a Resource for Narrative Similarity Prediction

hermeneutichools at SemEval-2026 Task 4: Multiperspectivity as a Resource for Narrative Similarity Prediction

Paper · 2026 · Max Upravitelev, Veronika Solopova, Jing Yang et al.

Summary

The paper addresses SemEval-2026 Task 4 (Track A). Each item is an anchor story and two candidate stories, and a system must pick the candidate closer to the anchor. The authors treat the existence of several valid readings of a narrative as something a model can use, not a defect of the benchmark.

They build 31 LLM personas, each defined by a system prompt. These fall into two groups:

  • Practitioners [P]: framework-driven, such as literary, feminist and postcolonial critics.
  • Lay People [L]: intuitive characters, such as a student or a football player.

Each persona runs on three open-weight models, and the predictions are combined by majority vote. The motivation is the Condorcet Jury Theorem, even though LLM voters are not truly independent.

Simple cross-model voting reaches 0.7025 on the test set, and an optimized subset reaches 0.705 (rank 12 of 46). Accuracy grows with ensemble size. Practitioner personas are individually weaker but make less correlated errors.

Outputs using gender-focused vocabulary tend to be wrong more often. The authors read this as evidence that single-ground-truth benchmarks may penalize legitimate interpretations.

Key points

  • Setup: 31 personas × 3 models (Gemma 3 27B-it, Qwen3-14B, gpt-oss-20b) give 93 votes per item. The dev split has 200 triplets and the test split has 400.
  • Majority voting: On the dev data, cross-model voting reached 75.8%, against 67.7% for the best individual configuration. Gains level off at about 30 personas. gpt-oss-20b personas average only about 57% individually.
  • Optimized subset vs. simple voting: An exhaustively searched k=8 subset scored 82.0% on dev but only 0.705 on test, versus 0.7025 for plain voting. The authors suggest the subset overfit to dev.
  • Error correlation: Pairwise error correlation was r=.388 for [P] and r=.461 for [L], about 19% lower for Practitioners. Lay personas dominate the top individual results.
  • Gender vocabulary and cost: Gender-related vocabulary shows a weak but consistent negative link with accuracy across all persona types and models. The full ensemble takes about 56 s per sample on one H100.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬German Research Centre for Artificial IntelligenceKaiserslautern · Germany77 works · 29,479 citations🎓Technische Universität BerlinBerlin · Germany22 works · 957 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

Cite

Cite as Max Upravitelev, Veronika Solopova, Jing Yang, Charlott Jakob, Premtim Sahitaj, Ariana Sahitaj and Vera Schmitt. 2026. hermeneutichools at SemEval-2026 Task 4: Multiperspectivity as a Resource for Narrative Similarity Prediction. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.230

@inproceedings{upravitelev2026hermeneutichools,
  title = {{hermeneutichools at SemEval-2026 Task 4: Multiperspectivity as a Resource for Narrative Similarity Prediction}},
  author = {Max Upravitelev, Veronika Solopova, Jing Yang, Charlott Jakob, Premtim Sahitaj, Ariana Sahitaj and Vera Schmitt},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.230},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.230}
}

References

30 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mousumi Akter and Shubhra Kanti Karmaker Santu. 2024. Fans: a facet-based narrative similarity metric. Preprint, arXiv:2309.04823. — [arXiv:2309.04823](https://arxiv.org/abs/2309.04823) find
  2. Gerardo Bárcena-Ruiz and Richard de Jesús Gil-Herrera. 2024. Application of condorcet’s jury theorem for enhancing sentiment analysis performance using bert transformers: A case study for spanish. Advances in Artificial Intelligence, page 5. find
  3. Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1):289–300. find
  4. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. O’Reilly Media, Inc. find
  5. Hyeong Kyu Choi, Xiaojin Zhu, and Yixuan Li. 2025. Debate or vote: Which yields better decisions in multi-agent large language models? In Advances in Neural Information Processing Systems. Spotlight. find
  6. Jon Chun. 2024. AIStorySimilarity: Quantifying story similarity using narrative for search, IP infringement, and guided creativity. In Proceedings of the 28th Conference on Computational Natural Language Learning, pages 161–177, Miami, FL, USA. Association for Computational Linguistics. find
  7. Jennifer Cooke, editor. 2020. The New Feminist Literary Studies. Twenty-First-Century Critical Revisions. Cambridge University Press, Cambridge. find
  8. Chen Han, Wenzhen Zheng, and Xijin Tang. 2025. Debate-to-detect: Reformulating misinformation detection as a real-world debate with large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 15114–15129, Suzhou, China. Association for Comp… find
  9. Hans Ole Hatzel, Ekaterina Artemova, Haimo Stiemer, Evelyn Gius, and Chris Biemann. 2026. SemEval-2026 Task 4: Narrative similarity and narrative representation learning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, CA, USA. Association for Comp… find
  10. Hans Ole Hatzel and Chris Biemann. 2024. Story embeddings — narrative-focused representations of fictional stories. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5931–5943, Miami, Florida, USA. Association for Computational Linguistics. find
  11. Lars Benedikt Kaesberg, Jonas Becker, Jan Philip Wahle, Terry Ruas, and Bela Gipp. 2025. Voting or consensus? decision-making in multi-agent debate. In Findings of the Association for Computational Linguistics: ACL 2025, pages 11640–11671, Vienna, Austria. Association for Computational Linguistics. find
  12. Junseok Kim, Nakyeong Yang, and Kyomin Jung. 2025. Persona is a double-edged sword: Rethinking the impact of role-play prompts in zero-shot reasoning tasks. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of… find
  13. Cody Kommers, Ruth Ahnert, Maria Antoniak, Emmanouil Benetos, Steve Benford, Mercedes Bunz, Baptiste Caramiaux, Shauna Concannon, Martin Dis-ley, James Dobson, Yali Du, Edgar Duéñez-Guzmán, Kerry Francksen, Evelyn Gius, Jonathan Gray, Ryan Heuser, Sarah Immel, Richard So, Sang Leigh, and 19 others.… find
  14. Ludmila I. Kuncheva and Christopher J. Whitaker. 2003. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine Learning, 51(2):181–207. find
  15. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
  16. Baptiste Lefort, Eric Benhamou, Jean-Jacques Ohana, Beatrice Guez, David Saltiel, and Thomas Jacquot. 2024. Examining independence in ensemble sentiment analysis: A study on the limits of large language models using the condorcet jury theorem. Preprint, arXiv:2409.00094. — [arXiv:2409.00094](https://arxiv.org/abs/2409.00094) find
  17. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2024. Encouraging divergent thinking in large language models through multi-agent debate. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17…
  18. Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9004–9017. Association for Computational… find
  19. Nikolaos Nikolaidis, Nicolas Stefanovitch, Purificação Silvano, Dimitar Iliyanov Dimitrov, Roman Yangarber, Nuno Guimarães, Elisa Sartori, Ion Androutsopoulos, Preslav Nakov, Giovanni Da San Martino, and Jakub Piskorski. 2025. Polynarrative: A multilingual, multilabel, multi-domain dataset for narra… find
  20. OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, and 108 others. 2025. gpt-oss-120b gpt-oss-20b model c…
  21. Gill Plain and Susan Sellers, editors. 2007. A History of Feminist Literary Criticism. Cambridge University Press, Cambridge. find
  22. Hanan Shteingart, Eran Marom, Igor Itkin, Gil Shabat, Michael Kolomenkin, Moshe Salhov, and Liran Katzir. 2020. Majority voting and the condorcet’s jury theorem. Preprint, arXiv:2002.03153. — [arXiv:2002.03153](https://arxiv.org/abs/2002.03153) find
  23. Veronika Solopova, Viktoriia Herman, Christoph Benzmüller, and Tim Landgraf. 2024. Check news in one click: NLP-empowered pro-kremlin propaganda detection. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 44–… find
  24. Gaurav Srivastava, Zhenyu Bi, Meng Lu, and Xuan Wang. 2025. DEBATE, TRAIN, EVOLVE: Self-evolution of language model reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 32764–32810, Suzhou, China. Association for Computational Linguistics.
  25. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  26. Selim Furkan Tekin, Fatih Ilhan, Tiansheng Huang, Sihao Hu, and Ling Liu. 2024. LLM-TOPLA: Efficient LLM ensemble by maximising diversity. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11951–11966, Miami, Florida, USA. Association for Computational Linguistics. find
  27. Hannah Waight, Solomon Messing, Anton Shirikov, Margaret E. Roberts, Jonathan Nagler, Jason Greenfield, Megan A. Brown, Kevin Aslett, and Joshua A. Tucker. 2025. Quantifying narrative similarity across languages. Sociological Methods & Research, 54(3):933–983. find
  28. Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. In International Conference on Learning Representations, volume 2024, pages 23650–23678. find
  29. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  30. Xiaochen Zhu, Caiqi Zhang, Yizhou Chi, Tom Stafford, Nigel Collier, and Andreas Vlachos. 2026. Demystifying multi-agent debate: The role of confidence and diversity. arXiv preprint arXiv:2601.19921.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/upravitelev-2026-hermeneutichools-semeval-2026-task-4/