Elamur Research⌕
☰
home / papers / 2026 / Yam at SemEval-2026 Task 4: Failure-Driven Prompt Evolution for Narrative Comparison

Yam at SemEval-2026 Task 4: Failure-Driven Prompt Evolution for Narrative Comparison

Paper · 2026 · Yen Yee Yam and Hong Meng Yam

Summary

The paper tackles SemEval-2026 Task 4 (Narrative Story Similarity, Track A). Each item gives an anchor story and two candidates, and the system must pick the candidate that is narratively closer in abstract theme, course of action, and outcome. The authors do not measure distance between embeddings. Instead they build a prompt-only pipeline around GPT-5.2 and never change its weights. Each story is first analysed separately along the three task dimensions. The model then compares the candidates one dimension at a time and justifies each judgment before giving a final answer. A feedback loop then examines misclassified items, finds recurring errors in both extraction and comparison, and rewrites short guideline documents. Any rewrite that lowers accuracy is rolled back. Decomposition alone reaches 70% on both train and test, and the evolved guidelines lift this to 76% and 73%. The authors argue that part of the improvement usually credited to fine-tuning can come from better-structured reasoning alone.

Key points

  • Ablation on Track A (train/test accuracy): direct prompting 69%/68%, ontology-aligned decomposition 70%/70%, decomposition plus self-improvement 76%/73%.
  • The optimisation loop maintains three length-limited guideline documents: decision-level comparison guidance, shared general reasoning principles, and intermediate extraction guidance.
  • A revision is kept only if evaluation accuracy improves; otherwise the loop automatically reverts it, which limits drift.
  • All runs use GPT-5.2 via API with no fine-tuning or RL. Repeated inference calls are combined by majority voting to reduce output variance.
  • Stated limitations include dependence on a proprietary model, no direct comparison with fine-tuned baselines, a search procedure that may not reach an optimal strategy, and untested cross-domain robustness.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,052 citations🎓National University of SingaporeSingapore · Singapore207 works · 39,576 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Yen Yee Yam and Hong Meng Yam. 2026. Yam at SemEval-2026 Task 4: Failure-Driven Prompt Evolution for Narrative Comparison. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.421

@inproceedings{yam2026yam,
  title = {{Yam at SemEval-2026 Task 4: Failure-Driven Prompt Evolution for Narrative Comparison}},
  author = {Yen Yee Yam and Hong Meng Yam},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.421},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.421}
}

References

17 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, and 1 others. 2025. Gepa: Reflective prompt evolution can outperform reinforcement learning. arXiv preprint arXiv:2507.19457. find
  2. Nathanael Chambers. 2013. Event schema induction with a probabilistic entity-driven model. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Seattle, Washington, USA. Association for Computational Linguistics. find
  3. Nathanael Chambers and Dan Jurafsky. 2009. Unsupervised learning of narrative schemas and their participants. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 602–610. find
  4. Hans Ole Hatzel, Ekaterina Artemova, Haimo Stiemer, Evelyn Gius, and Chris Biemann. 2026a. SemEval-2026 Task 4: Narrative similarity and narrative representation learning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, CA, USA. Association for Com… find
  5. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 4222–4235.
  6. Igor Sterner, Alex Lascarides, and Frank Keller. 2026. Contrastive learning with narrative twins for modeling story salience. arXiv preprint arXiv:2601.07765.
  7. Louis L Thurstone. 1994. A law of comparative judgment. Psychological review, 101(2):266. find
  8. Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. 2024. Textgrad: Automatic" differentiation" via text. arXiv preprint arXiv:2406.07496.
  9. Hans Ole Hatzel, Ekaterina Artemova, Haimo Stiemer, Evelyn Gius, and Chris Biemann. 2026b. find
  10. Semeval 2026 task 4: Narrative story similarity and narrative representation learning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). To appear. find
  11. Hans Ole Hatzel and Chris Biemann. 2024. Story embeddings—narrative-focused representations of fictional stories. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5931–5943. find
  12. Ruili Jiang, Kehai Chen, Xuefeng Bai, Zhixuan He, Juntao Li, Muyun Yang, Tiejun Zhao, Liqiang Nie, and Min Zhang. 2025. A survey on human preference learning for aligning large language models. ACM Computing Surveys, 58(6):1–39. find
  13. Svetlana Kiritchenko and Saif Mohammad. 2017. Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 465–470.
  14. O-Joun Lee and Jason J. Jung. 2020. Story embedding: Learning distributed representations of stories based on character networks. Artificial Intelligence, 281:103235. find
  15. Wendy G Lehnert. 1981. Plot units and narrative summarization. Cognitive science, 5(4):293–331. find
  16. Zhengzhong Liu, Teruko Mitamura, and Eduard Hovy. 2018. Graph based decoding for event sequencing and coreference resolution. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3645–3657, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  17. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yam-2026-yam-semeval-2026-task-4/