Elamur Research⌕
☰
home / papers / 2026 / X-NLP at SemEval-2026 Task 12: Prompting LLMs for Abductive Event Reasoning

X-NLP at SemEval-2026 Task 12: Prompting LLMs for Abductive Event Reasoning

Paper · 2026 · Caelen Mattie, Patrick Bowen and Milton King

Summary

The paper describes a team's entry to SemEval-2026 Task 12, which tests abductive event reasoning. Given a real-world news event, a system must pick the most plausible direct cause or causes from four candidate explanations. Each topic comes with a collection of 20+ related news articles.

The authors built two zero-shot, prompt-based systems and did no fine-tuning:

  • System 1 uses retrieval-augmented generation. Qwen3-Embedding-8B embeds the question and the document titles, and the single most similar document is retrieved by cosine similarity. That document is passed to Grok-4.1-Fast along with the question and options.
  • Systems 2a and 2b skip retrieval. They place the whole topic document set into a persona-style prompt ("expert analyst") for Gemini-2.5-Flash or Gemini-2.5-Pro.

The retrieval-based system performed best, scoring 84% and ranking 40th of 221 submissions. Error analysis shows that the models often prefer well-known background facts or emotionally similar options over the actual immediate trigger, especially when retrieval returns a loosely related article. This points to retrieval quality as a key lever for this task.

Key points

  • The dataset splits contain 200 (sample), 1819 (train), 400 (dev) and 612 (test) questions. Scoring awards 1 point for an exact match and 0.5 for a partial match.
  • Test accuracy was 84% for System 1 (Grok-4.1-Fast + Qwen3-Embedding-8B), 71% for System 2b (Gemini-2.5-Pro) and 67% for System 2a (Gemini-2.5-Flash). The top submission reached 95%.
  • An ablation on dev found that one retrieved document (k = 1) worked better than several. The larger 8B embedding model also beat the 0.6B variant.
  • Across all task submissions, the median was 0.65 and the mean was 0.54, a negatively skewed distribution. Ten submissions scored 0.
  • In the failure cases, the model chose the 2011 earthquake over the wastewater release plan in the Fukushima example. Proposed future work includes query rewriting, document re-ranking and domain-specific personas.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: St. Francis Xavier University.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Caelen Mattie, Patrick Bowen and Milton King. 2026. X-NLP at SemEval-2026 Task 12: Prompting LLMs for Abductive Event Reasoning. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.289

@inproceedings{mattie2026x,
  title = {{X-NLP at SemEval-2026 Task 12: Prompting LLMs for Abductive Event Reasoning}},
  author = {Caelen Mattie, Patrick Bowen and Milton King},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.289},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.289}
}

References

12 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Savir Basil, Ina Shapiro, Dan Shapiro, Ethan Mollick, Lilach Mollick, and Lennart Meincke. 2025. Prompting science report 4: Playing pretend: Expert personas don’t improve factual accuracy. arXiv e-prints, pages arXiv–2512. find
  2. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen-tau Yih, and Yejin Choi. 2019. Abductive commonsense reasoning. arXiv preprint arXiv:1908.05739.
  3. Steven Bird. 2006. Nltk: the natural language toolkit. In Proceedings of the COLING/ACL 2006 interactive presentation sessions, pages 69–72.
  4. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: An open platform for evaluating llms by human preference.
  5. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv e-pr…
  6. Tiancheng Hu and Nigel Collier. 2024. Quantifying the persona effect in llm simulations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10289–10307.
  7. Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, and Xiaohang Dong. 2024. Better zero-shot reasoning with role-play prompting. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language… find
  8. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Proc…
  9. Decong Li, Sujian Li, Wenjie Li, Wei Wang, and Weiguang Qu. 2010. A semi-supervised key phrase extraction approach: Learning from title phrases through a document semantic network. In Proceedings of the ACL 2010 Conference Short Papers, pages 296–300, Uppsala, Sweden. Association for Computational L… find
  10. Shiyin Lin. 2025. Abductive inference in retrieval-augmented language models: Generating and validating missing premises. In 2025 5th International Conference on Network Communication and Information Security (ICNCIS), pages 62–66. IEEE. find
  11. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the association for computational linguistics, 12:157–173.
  12. Marlene Lutz, Indira Sen, Georg Ahnert, Elisa Rogers, and Markus Strohmaier. 2025. The prompt makes the person (a): A systematic evaluation of sociodemographic persona prompting for large language models. arXiv preprint arXiv:2507.16076. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mattie-2026-x-nlp-semeval-2026-task/