Elamur Research⌕
☰
home / papers / 2026 / HABIB_TAZ at SemEval-2026 Task 11: Disentangling Formal Logic from Content via Synthetic Training and Multi-Objective Optimization

HABIB_TAZ at SemEval-2026 Task 11: Disentangling Formal Logic from Content via Synthetic Training and Multi-Objective Optimization

Paper · 2026 · Abdullah Shaikh, Zain Naqi, Taha Zahid et al.

Summary

The paper addresses SemEval-2026 Task 11. Systems must decide whether a syllogism's conclusion follows formally from its premises, regardless of how believable it is. The task covers 12 languages, and some tracks add distractor premises that the system must filter out. The authors target the "content effect", where models tend to accept plausible conclusions. They fine-tune NLI-pretrained mDeBERTa-v3/DeBERTa-v3 encoders, adding a span-pooling premise-selection head for the noisy tracks.

Only 960 English training items were provided, so they built a rule-based generator over the 256 Aristotelian moods and figures. It draws nouns from WordNet hierarchies and adds symbolic-variable and rephrased variants, HANS samples, and template-based translations. Training combines three objectives:

  • adaptive Group DRO over validity/plausibility (and language) subgroups,
  • a scheduled differentiable bias penalty,
  • KL consistency between simple and rephrased syllogisms.

The system ranked first with perfect scores on three subtasks and sixth on the hardest one. This suggests that controlled synthetic data plus bias-aware objectives can largely separate logic from plausibility.

Key points

  • Results: ranking score 100.0, 0.00% bias and 100% accuracy on ST1 (English), ST2 (noisy English) and ST3 (multilingual). On ST4 (noisy multilingual), the system placed 6th with 89.06% accuracy/F1, 2.89% bias and a 37.78 ranking score.
  • Synthetic corpus sizes per subtask: 23,524 (ST1), 27,988 (ST2), 388,992 (ST3) and 648,632 (ST4) samples. The generator uses WordNet via NLTK, DeepL for noun translation, and HANS for syntactic robustness.
  • Backbones:
    • mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 for ST1/ST3;
    • DeBERTa-v3-large-mnli-fever-anli-ling-wanli for ST2, where scaling up was key to the perfect score;
    • microsoft/mdeberta-v3-base for ST4.
  • Ablations:
    • On ST3, adding the bias penalty alone cut the content effect from 8.29% to 1.04%. HANS data then lifted the system to a perfect score.
    • Symbolic training data consistently improved generalization.
  • On ST4, KL regularization gave the most consistent cross-lingual behavior. It needs many forward passes, though, so it was trained on only about 4% of the data. Code and generation scripts are released on GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Habib UniversityKarachi · Pakistan5 works · 4 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Abdullah Shaikh, Zain Naqi, Taha Zahid, Sandesh Kumar and Abdul Samad. 2026. HABIB_TAZ at SemEval-2026 Task 11: Disentangling Formal Logic from Content via Synthetic Training and Multi-Objective Optimization. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.139

@inproceedings{shaikh2026habib,
  title = {{HABIB\_TAZ at SemEval-2026 Task 11: Disentangling Formal Logic from Content via Synthetic Training and Multi-Objective Optimization}},
  author = {Abdullah Shaikh, Zain Naqi, Taha Zahid, Sandesh Kumar and Abdul Samad},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.139},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.139}
}

References

24 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020. Translation artifacts in cross-lingual transfer learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7674–7684, Online. Association for Computational Linguistics.
  2. L. Bertolazzi, A. Gatt, A. M. Gliozzo, G. Greco, A. Madotto, and M. Valentino. 2024. A systematic analysis of large language models as soft reasoners: The case of syllogistic inferences. arXiv preprint, arXiv:2406.11341. find
  3. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. O’Reilly Media, Inc. find
  4. F. Cheng, H. Li, F. Liu, R. van Rooij, K. Zhang, and Z. Lin. 2025. Empowering llms with logical reasoning: A comprehensive survey. arXiv preprint, arXiv:2502.15652. find
  5. Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Hannah R. Sheahan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, and Felix Hill. 2024. Language models show human-like content effects on reasoning tasks. Preprint, arXiv:2207.07051. find
  6. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. In International Conference on Learning Representations.
  7. Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, and 51 others. 2024. Trustllm: Trustworthiness in large…
  8. G. Kim, M. Valentino, D. Dalal, Z. Zhao, and A. Freitas. 2024. Reasoning circuits in language models: A mechanistic interpretation of syllogistic inference. arXiv preprint, arXiv:2408.08590. find
  9. Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. The Annals of Mathematical Statistics, 22(1):79–86. find
  10. Moritz Laurer, Wouter van Atteveldt, Andreu Salleras Casas, and Kasper Welbers. 2022. Less annotating, more classifying – addressing the data scarcity issue of supervised machine learning with deep transfer learning and BERT-NLI. Preprint, Open Science Framework. find
  11. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  12. R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. Preprint, arXiv:1902.01007.
  13. George A. Miller. 1995. Wordnet: a lexical database for english. Commun. ACM, 38(11):39–41.
  14. OpenAI. 2026. ChatGPT (gpt-5.2 version). https://chat.openai.com. Large language model accessed via web interface. ↗
  15. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, and 2 others. 20… — PyTorch: An Imperative Style, High-Performance Deep Learning Library
  16. N. Patel, M. Kulkarni, M. Parmar, A. Budhiraja, M. Nakamura, N. Varshney, and C. Baral. 2024. Multi-logieval: Towards evaluating multi-step logical reasoning ability of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP). Available…
  17. X. Quan, M. Valentino, L. A. Dennis, and A. Freitas. 2024. Verification and refinement of natural language explanations through llm-Symbolic theorem proving. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2933–2958. Available: arXiv:2405.0137…
  18. Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. 2020. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. Preprint, arXiv:1911.08731. find
  19. M. Valentino, G. Kim, D. Dalal, Z. Zhao, and A. Freitas. 2025. Mitigating content effects on reasoning in language models through fine-grained activation steering. arXiv preprint, arXiv:2505.12189. — [arXiv:2505.12189](https://arxiv.org/abs/2505.12189) find
  20. Marco Valentino, Leonardo Ranaldi, Giulia Pucci, Federico Ranaldi, and André Freitas. 2026. Semeval-2026 task 11: Disentangling content and formal reasoning in language models. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). find
  21. W. Wan, Z. Yang, Y. Chen, C. Luo, R. Wang, K. Cai, N. Kang, L. Lin, and K. Wang. 2025. Sr-fot: A syllogistic-reasoning framework of thought for large language models tackling knowledge-based reasoning tasks. arXiv preprint, arXiv:2501.11599. — [arXiv:2501.11599](https://arxiv.org/abs/2501.11599) find
  22. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transf…
  23. X. Wu, Y. Bu, and 1 others. 2024. Content-free logical modification of large language models by disentangling and modifying logic representation. In Proceedings of the AAAI Conference on Artificial Intelligence. DOI pending. find
  24. Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2019. Freelb: Enhanced adversarial training for language understanding. CoRR, abs/1909.11764.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shaikh-2026-habib-taz-semeval-2026-task/