Elamur Research⌕
☰
home / papers / 2026 / sutta at SemEval-2026 Task 12: A Multi-Perspective Retrieve-Verify-Aggregate Framework for Abductive Event Reasoning

sutta at SemEval-2026 Task 12: A Multi-Perspective Retrieve-Verify-Aggregate Framework for Abductive Event Reasoning

Paper · 2026 · Junliu Zou, Liang Yang and Jingjie Zeng

Summary

The paper addresses Abductive Event Reasoning (SemEval-2026 Task 12). Given a real-world event, four candidate options and 10–20 retrieved documents, a system must select the option(s) that directly caused the event. The authors argue that LLMs given long, noisy context tend to latch onto plausible but non-causal material.

Team sutta built a zero-shot pipeline around Qwen3-8B with no task-specific fine-tuning:

  • Retrieval: documents are split into passages of at most 1,000 characters. BM25 picks 30 candidates per event–option query, and the bge-reranker-v2-m3 cross-encoder keeps the top five.
  • Verification: the same model judges each option under five prompting personas: Standard CoT, Detailed CoT, Counterfactual Analysis, NLI and Detective Role-Play.
  • Aggregation: a strict majority vote, with a fallback rule, produces the final answer.

The ensemble scored 0.7614 on the 612-item blind test set, outperforming every individual prompt. The work matters because it shows a compact open model can perform competitive causal inference cheaply when it combines tight passage-level retrieval with diverse reasoning prompts. The authors also show that the vote fails when several personas share the same logical bias.

Key points

  • The dataset is English-only, with 1,819 train, 400 dev and 612 test instances. Scoring gives 1.0 for an exact match, 0.5 for a proper subset of the correct causes, and 0 if any wrong option is chosen.
  • Retrieval settings: BM25 (k1 = 1.5, b = 0.75) returns top-30 chunks, and bge-reranker-v2-m3 keeps the top 5. Qwen3-8B runs via vLLM with temperature 0.1.
  • The 5-persona ensemble reached 0.7614 (73.86% full match), versus 0.7353 for the best single prompt (Standard CoT) and 0.7475 for a Top-3 ensemble.
  • Individually weak personas, Counterfactual (0.5760) and Detective (0.5041), still improved the ensemble by rejecting false correlations. Detailed CoT (0.6503) underperformed the simpler Standard CoT.
  • Limitations include roughly 5× inference cost, a precision-heavy vote that misses some valid causes, and reliance on Qwen3-8B's own knowledge. In an error case (q-2470), the model mistook a necessary precondition for a direct cause.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Dalian University of TechnologyDalian · China12 works · 46 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Junliu Zou, Liang Yang and Jingjie Zeng. 2026. sutta at SemEval-2026 Task 12: A Multi-Perspective Retrieve-Verify-Aggregate Framework for Abductive Event Reasoning. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.323

@inproceedings{zou2026sutta,
  title = {{sutta at SemEval-2026 Task 12: A Multi-Perspective Retrieve-Verify-Aggregate Framework for Abductive Event Reasoning}},
  author = {Junliu Zou, Liang Yang and Jingjie Zeng},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.323},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.323}
}

References

25 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Yih, and Yejin Choi. 2019. Abductive commonsense reasoning. ArXiv, abs/1908.05739.
  2. Austria, July 21-27, 2024, volume 235 of Proceedings of Machine Learning Research, pages 11733–11763. PMLR / OpenReview.net. find
  3. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin… find
  4. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang (Lorraine) Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena Hwang, Soumya Sanyal, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. 2023. Faith and fate: Limits of transformers on compositiona…
  5. Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for automatic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762–4779, Florence…
  6. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, John A. Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuan-Fang Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. ArXiv,…
  7. Pengfei Cao, Mingxuan Yang, Yubo Chen, Chenlong Zhang, Mingxuan Liu, Kang Liu, and Jun Zhao. 2026. Semeval-2026 task 12: Abductive event reasoning: Towards real-world event causal inference for large language models. Preprint, arXiv:2603.21720. find
  8. Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and…
  9. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Jun-Mei Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiaoling Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 179 others. 2025. Deepseek-r1 incentivizes reasoning in llms throug… find
  10. Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2024. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang…
  11. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023. Large language models cannot self-correct reasoning yet. ArXiv, abs/2310.01798.
  12. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12).
  13. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Proc…
  14. Xin Li, Zhuo Cai, Shoujin Wang, Kun Yu, and Fang Chen. 2025. A survey on enhancing causal reasoning ability of large language models. In Pacific-Asia Conference on Knowledge Discovery and Data Mining. find
  15. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.
  16. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901, Online. Association for Computa…
  17. Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. find
  18. Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. find
  19. Stephen E. Robertson and Steve Walker. 1994. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. find
  20. Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3:333–389. find
  21. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, pages 8634–8652. Curran Associates, Inc.
  22. Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2022. Efficient transformers: A survey. ACM Comput. Surv., 55(6).
  23. Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, S. H. Cai, Yuan Cao, Y. Charles, H. S. Che, Cheng Chen, Guanduo Chen, Huarong Chen, Jia Chen, Jiahao Chen, Jianlong Chen, Jun Chen, Kefan Chen, Liang Chen, Ruijue Chen, Xinhao Chen, and 307 others. 2026. Kimi k2.5: Visual agentic intelligence. Preprint…
  24. The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. find
  25. Andrew Yates, Rodrigo Nogueira, and Jimmy Lin. 2021. Pretrained transformers for text ranking: Bert and beyond. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, WSDM ’21, page 1154–1156, New York, NY, USA. Association for Computing Machinery. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zou-2026-sutta-semeval-2026-task-12/