Elamur Research⌕
☰
home / papers / 2026 / Hidetsune at SemEval-2026 Task 11: Adapting Pretrained Reasoning Models with Deep Supervision and Inference Refinement for Content-Independent Validity Classification

Hidetsune at SemEval-2026 Task 11: Adapting Pretrained Reasoning Models with Deep Supervision and Inference Refinement for Content-Independent Validity Classification

Paper · 2026 · Hidetsune Takahashi

Summary

This paper describes a system for SemEval-2026 Task 11 Subtask 1. The task is to decide whether English syllogisms are logically valid without being misled by how believable their content is. The training set has 960 syllogisms, balanced between valid and invalid, each annotated for validity and plausibility.

The author tested several strategies in sequence:

  • fine-tuning general-purpose encoders and decoders (BERT, TwHIN-BERT, GPT-2) as baselines;
  • adding layer-weighted deep supervision to BERT;
  • few-shot prompting of Qwen3-4B;
  • fine-tuning models already trained on NLI-style reasoning data;
  • adjusting the softmax decision threshold of the strongest model.

The reasoning-pretrained DeBERTa-base-long-nli clearly beat the generic models. Raising its threshold to 0.60 lowered the content effect while leaving accuracy nearly unchanged. The submitted system reached 95.8% validity accuracy and ranked 28th of 45 teams.

The study shows that prior exposure to inference-related data matters more than model size or auxiliary training tricks. It also shows that a simple post-hoc calibration step can reduce plausibility bias.

Key points

  • Metrics: systems are scored on validity accuracy (ACC) and Total Content Effect (TCE). The official ranking uses a Combined Score that rewards accuracy and penalizes TCE. No separate test set was released, so the development test data also served for final evaluation.
  • Baselines: BERT-base-uncased was the best general model (75.4% ACC, TCE 8.64, Combined 23.1). Larger variants (BERT-large, TwHIN-BERT-large, GPT-2-medium) did worse, which the author attributes to overfitting and greater reliance on world knowledge.
  • Deep supervision: auxiliary losses were applied to layers 6–12 with increasing weights. With α = 0.1, accuracy rose to 78.0%, but TCE grew to 12.9, so the Combined Score fell to 21.5.
  • In-context learning and task-specific models: few-shot prompting of Qwen3-4B performed poorly (72.8% ACC, TCE 31.6). Among task-specific models, DeBERTa-base-long-nli scored 96.3% ACC with TCE 4.26, ahead of BART-large-mnli (94.2%) and RoBERTa-large-wanli (77.5%).
  • Thresholding: thresholds from 0.4 to 0.7 were tested. A threshold of 0.60 gave the final submission: Combined 39.3, ACC 95.8%, TCE 3.21. Across thresholds, accuracy varied by under 1 point, while TCE dropped from 6.38 to 3.21. The code is on GitHub (Hidetsune/SemEval2026_Task11).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Waseda UniversityTokyo · Japan21 works · 584 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Hidetsune Takahashi. 2026. Hidetsune at SemEval-2026 Task 11: Adapting Pretrained Reasoning Models with Deep Supervision and Inference Refinement for Content-Independent Validity Classification. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.28

@inproceedings{takahashi2026hidetsune,
  title = {{Hidetsune at SemEval-2026 Task 11: Adapting Pretrained Reasoning Models with Deep Supervision and Inference Refinement for Content-Independent Validity Classification}},
  author = {Hidetsune Takahashi},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.28},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.28}
}

References

18 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Leonardo Bertolazzi, Sandro Pezzelle, and Raffaella Bernardi. 2026. How language models conflate logical validity with plausibility: A representational analysis of content effects. Preprint, arXiv:2510.06700. find
  2. Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Hannah R. Sheahan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, and Felix Hill. 2024. Language models show human-like content effects on reasoning tasks. Preprint, arXiv:2207.07051.
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  4. Andrew K Lampinen, Ishita Dasgupta, Stephanie C Y Chan, Hannah R Sheahan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. 2024. Language models, like humans, show content effects on reasoning tasks. PNAS Nexus, 3(7):pgae233. find
  5. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the…
  6. Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. WANLI: Worker and AI collaboration for natural language inference dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6826–6847, Abu Dhabi, United Arab Emirates. Association for Computat…
  7. Kentaro Ozeki, Risako Ando, Takanobu Morishita, Hirohiko Abe, Koji Mineshima, and Mitsuhiro Okada. 2024. Exploring reasoning biases in large language models through syllogism: Insights from the NeuBAROCO dataset. In Findings of the Association for Computational Linguistics: ACL 2024, pages 16063–160… find
  8. XiPeng Qiu, TianXiang Sun, YiGe Xu, YunFan Shao, Ning Dai, and XuanJing Huang. 2020. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10):1872–1897.
  9. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  10. Damien Sileo. 2024. tasksource: A large collection of NLP tasks with a structured dataset preprocessing framework. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 15655–15684, Torino, Italia. ELRA and… find
  11. Yumei Song, Yongbin Qin, Ruizhang Huang, Yanping Chen, and Chuan Lin. 2025. Legal text summarization via judicial syllogism with large language models. Journal of King Saud University Computer and Information Sciences, 37(5):111. find
  12. Hidetsune Takahashi, Xingru Lu, Sean Ishijima, De-okgyu Seo, Yongju Kim, Sehoon Park, Min Song, Kathylene Marante, Keitaro-luke Iso, Hirotaka Tokura, and Emily Ohman. 2024. OZemi at SemEval-2024 task 1: A simplistic approach to textual relatedness evaluation using transformers and machine translatio… find
  13. Hidetsune Takahashi, Sumiko Teng, Jina Lee, Wenxiao Hu, Rio Obe, Chuen Shin Yong, and Emily Ohman. 2025. OZemi at SemEval-2025 task 11: Multilingual emotion detection and intensity. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), pages 109–115, Vienna, Austri… find
  14. Marco Valentino, Geonhee Kim, Dhairya Dalal, Zhixue Zhao, and André Freitas. 2025. Mitigating content effects on reasoning in language models through fine-grained activation steering. Preprint, arXiv:2505.12189. — [arXiv:2505.12189](https://arxiv.org/abs/2505.12189) find
  15. Marco Valentino, Leonardo Ranaldi, Giulia Pucci, Federico Ranaldi, and André Freitas. 2026. Semeval-2026 task 11: Disentangling content and formal reasoning in large language models. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026).
  16. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  17. Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (…
  18. Xinyang Zhang, Yury Malkov, Omar Florez, Serim Park, Brian McWilliams, Jiawei Han, and Ahmed El-Kishky. 2022. Twhin-bert: A socially-enriched pre-trained language model for multilingual tweet representations. arXiv preprint arXiv:2209.07562. — [arXiv:2209.07562](https://arxiv.org/abs/2209.07562) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/takahashi-2026-hidetsune-semeval-2026-task-11/