Elamur Research⌕
☰
home / papers / 2026 / d-itlab at SemEval-2026 Task 12: Per-Option Surprisal and Multi-Stage Gating for Precision-Oriented Causal Reasoning

d-itlab at SemEval-2026 Task 12: Per-Option Surprisal and Multi-Stage Gating for Precision-Oriented Causal Reasoning

Paper · 2026 · Yasunori Terao and Yuuki Tachioka

Summary

The paper presents d-itlab's system for SemEval-2026 Task 12, Abductive Event Reasoning. Given an observed event, four candidate options (A–D) and news-style reference documents that include distractors, a system must pick the option(s) that directly caused the event. The official metric gives zero credit for any false positive, so the team optimised for precision.

The pipeline works in four steps:

  • GPT-4.1 condenses each reference document into a short factual summary.
  • Each option is judged on its own through a two-stage YES/NO gate (a coarse filter, then a strict verifier), with a narrow rescue rule.
  • Surprisal features come from teacher-forcing a fixed "direct cause" sentence for each option and measuring token-level negative log-likelihood.
  • An ensemble of XGBoost classifiers merges these signals and applies a tuned probability threshold.

Supervised training used only the 400-instance development split. The system scored 0.91 on the test set, placing third of 116 teams. This shows that combining verification gates with likelihood-based uncertainty features suits precision-weighted causal selection.

Key points

  • Models: gating ran on Qwen2.5-32B-Instruct, Qwen3-32B (thinking disabled), Mistral-Small-24B-Instruct and GPT-5.2. Surprisal used only the three locally hosted models.
  • Ensemble: averaging 20 XGBoost classifiers raised the score from 0.89 (single model) to 0.91. Tuned thresholds were high, with a mean τ of 0.773.
  • Feature importance: SHAP analysis shows that gate features, especially Stage 1 probability and cross-model agreement, dominate the predictions. Token-trace and surprisal features mainly refine borderline cases.
  • Test results (612 instances): 529 received full credit, 56 partial credit and 27 zero. Most partial errors are under-predictions on multi-cause items, concentrated in causal-chain topics such as the CrowdStrike outage.
  • Failure mode for zero scores: 22 of the 27 zero-score predictions miss the gold set entirely, often in dense narratives like the Israel–Lebanon topic.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Denso IT Laboratory Inc.Tokyo · Japan2 works · 1 citation

Also on the byline: Denso (United States).

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksPsycholinguistics & language acquisition · 102 works

Cite

Cite as Yasunori Terao and Yuuki Tachioka. 2026. d-itlab at SemEval-2026 Task 12: Per-Option Surprisal and Multi-Stage Gating for Precision-Oriented Causal Reasoning. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.228

@inproceedings{terao2026d,
  title = {{d-itlab at SemEval-2026 Task 12: Per-Option Surprisal and Multi-Stage Gating for Precision-Oriented Causal Reasoning}},
  author = {Yasunori Terao and Yuuki Tachioka},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.228},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.228}
}

References

10 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In Proceedings of the 8th International Conference on Learning Representations (ICLR).
  2. Task Organizers. 2026. SemEval-2026 Task 12: Abductive Event Reasoning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). To appear. find
  3. Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794. find
  4. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics, 9:346– 361. find
  5. Jie Huang and Kevin Chen-Chuan Chang. 2023. Towards reasoning in large language models: A survey. Findings of the Association for Computational Linguistics: ACL 2023, pages 1049–1065.
  6. Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962–977. find
  7. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others. 2022.… find
  8. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural…
  9. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Proc…
  10. Shayne Longpre, Lu Periez, Anthony Chen, Nikhil Mukherjee, Chris Callison-Burch, and Mark Yatskar. 2021. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7052–7063.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/terao-2026-d-itlab-semeval-2026-task/