Elamur Research⌕
☰
home / papers / 2026 / Clutch or Cry at SemEval-2026 Task 12: Offline Retrieval-Augmented Generation with Frozen DeBERTa for Abductive Event Reasoning

Clutch or Cry at SemEval-2026 Task 12: Offline Retrieval-Augmented Generation with Frozen DeBERTa for Abductive Event Reasoning

Paper · 2026 · Aayush Prasad, Rudra Trivedi, Arshad Khatib et al.

Summary

The paper presents an approach built for the abductive event reasoning challenge posed as Task 12 of SemEval-2026. Given a document corpus, the system must infer which answer options explain the causes or consequences of a target event. Any subset of options may be correct, and scoring uses Exact Match, so a single wrong or missing choice zeroes out the instance.

The authors' first attempts failed. Fine-tuning large models on full documents memorized the small training set and did not transfer to test data. Smaller models had short context windows, so truncation removed the causal clues and the models effectively guessed.

The final design separates evidence finding from decision making. An offline dense retriever selects one most relevant sentence for each observation. A partially frozen deberta-v3-large then scores each option independently against that sentence. Despite the RAG label, the second stage is a classifier rather than a text generator.

This setup reached 0.72 Exact Match on the official test set. It shows that tightly limiting context and the number of trainable parameters can help in low-resource, all-or-nothing evaluation settings.

Key points

  • Retrieval: all-MiniLM-L6-v2 embeds the corpus and ranks passages by cosine similarity. Using top k = 1 worked best; k = 3 and k = 5 added noisy or contradictory evidence and lowered accuracy.
  • Classifier: microsoft/deberta-v3-large (435M parameters) with its bottom 12 of 24 layers frozen. BCEWithLogitsLoss replaces Cross-Entropy so each option is judged as a separate binary decision. Inputs follow the pattern [CLS] context [SEP] event [SEP] option.
  • Training setup:
    • AdamW, learning rate 5e-6, weight decay 0.01.
    • Batch size 1 with 16-step gradient accumulation.
    • Maximum 512 tokens.
    • Up to 8 epochs with early stopping on dev Exact Match.
    • bfloat16 and gradient checkpointing on one NVIDIA H100.
  • Ablation:
    • A large model on full documents reached about 95% training accuracy but performed poorly on test.
    • microsoft/deberta-v3-small with inputs cut to 500 characters peaked at 49.57% validation accuracy, then fell to 34.19%.
    • The final system scored 0.72 on test.
  • Limitation: single-sentence retrieval cannot capture evidence spread across several sentences. The authors plan contradiction-aware multi-sentence retrieval as future work. Code is on GitHub (aayush-decoder/semeval-2026-abductive-event-reasoning).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sardar Vallabhbhai National Institute of Technology SuratSurat · India5 works · 4 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksInformation Systems · 1,730 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,685 worksInformation retrieval & search · 552 works

Cite

Cite as Aayush Prasad, Rudra Trivedi, Arshad Khatib, Shrikant Malviya and Naveen Kumar. 2026. Clutch or Cry at SemEval-2026 Task 12: Offline Retrieval-Augmented Generation with Frozen DeBERTa for Abductive Event Reasoning. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.261

@inproceedings{prasad2026clutch,
  title = {{Clutch or Cry at SemEval-2026 Task 12: Offline Retrieval-Augmented Generation with Frozen DeBERTa for Abductive Event Reasoning}},
  author = {Aayush Prasad, Rudra Trivedi, Arshad Khatib, Shrikant Malviya and Naveen Kumar},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.261},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.261}
}

References

5 references, 5 of them with a card of ours. ↗ leads to the work.

  1. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.
  2. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Proce…
  3. Yujia Qin and 1 others. 2023. Language models can improve event prediction by few-shot abductive reasoning. arXiv preprint arXiv:2305.16646.
  4. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  5. Thomas Wolf and 1 others. 2020. Transformers: State-of-the-art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45.

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/prasad-2026-clutch-or-cry-semeval-2026/