Elamur Research⌕
☰
home / papers / 2026 / CICL26 at SemEval-2026 Task 4: Narrative Story Similarity and Narrative Representation Learning

CICL26 at SemEval-2026 Task 4: Narrative Story Similarity and Narrative Representation Learning

Paper · 2026 · Wanzhao Zhang and Yue Yu

Summary

A team from the University of Tübingen entered Track A of SemEval-2026 Task 4. In this track, a system receives one anchor story along with a pair of candidate stories and has to pick the candidate that is narratively closer in abstract theme, course of action and outcome. LLM judges can be unreliable in such pairwise choices, because random sampling and the order of candidates can change the answer. The team therefore built an inference-only cascade on top of GPT-4.1-mini, with no fine-tuning:

  • Self-consistency voting: three sampled calls are made in the original order and three with the candidates swapped, each settled by majority vote.
  • Agreement check: if the two orders agree, that answer is output.
  • Disagreement rule: otherwise, the side with the larger vote margin wins. Ties are settled by an extra voting round.

The system reached 59% accuracy on the official CodaBench test set. The analysis shows that order sensitivity persists even under majority voting, which suggests that sampling-variance reduction and positional-bias mitigation are separate problems.

Key points

  • Data: 39 sample, 200 development and 400 test triplets. The stories were human-written, human-written then LLM-edited, or sampled. The code is available on GitHub (cicl-iscl).
  • Settings: GPT-4.1-mini at temperature 0.3, N = 3 votes per order, and up to 5 API retries. The tie-break round is described as an extra forward vote, with temperature raised to 0.4 in the setup section.
  • Result: 59% test accuracy. The paper claims a gain over single-call inference but reports no single-call GPT-4.1-mini baseline on the test set.
  • Ablation: zero-shot GPT-3.5-turbo on the development set scored 52% (A–B order) and 50% (B–A order). Its judgments were inconsistent 46% of the time, and it tended to favor whichever story was shown second.
  • Swap analysis: even unanimous 3–0 votes often flipped after swapping, so internal agreement does not guarantee robustness to candidate order.
  • Limitations: dependence on a commercial API, higher inference cost from multiple calls, and only partial removal of order bias.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TübingenTübingen · Germany75 works · 41,081 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Wanzhao Zhang and Yue Yu. 2026. CICL26 at SemEval-2026 Task 4: Narrative Story Similarity and Narrative Representation Learning. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.310

@inproceedings{zhang2026cicl26,
  title = {{CICL26 at SemEval-2026 Task 4: Narrative Story Similarity and Narrative Representation Learning}},
  author = {Wanzhao Zhang and Yue Yu},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.310},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.310}
}

References

7 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, and 1 others. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33.
  2. Hans Hatzel and Chris Biemann. 2024. Modeling narrative similarity across levels of abstraction. In Proceedings of *SEM. find
  3. Hans Ole Hatzel, Ekaterina Artemova, Haimo Stiemer, Evelyn Gius, and Chris Biemann. 2026. SemEval-2026 Task 4: Narrative similarity and narrative representation learning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, CA, USA. Association for Comp… find
  4. Aniket Keluskar and 1 others. 2024. Ambiguity and instability in large language model reasoning. In Proceedings of EMNLP. find
  5. Andrew Piper. 2021. Enumerations: Data and Literary Study. University of Chicago Press. find
  6. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. 2024. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… find
  7. Xuezhi Wang, Jason Wei, Dale Schuurmans, and 1 others. 2023. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations (ICLR).

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-cicl26-semeval-2026-task-4/