Elamur Research⌕
☰
home / papers / 2026 / HCMUSDroneBoys at SemEval-2026 Task 11: Asymmetric Counterfactual Debiasing and Rank-Sensitive Logical Invariance Adaptation for Syllogistic Reasoning

HCMUSDroneBoys at SemEval-2026 Task 11: Asymmetric Counterfactual Debiasing and Rank-Sensitive Logical Invariance Adaptation for Syllogistic Reasoning

Paper · 2026 · Nguyen Tran, Duy Minh Dao Sy, Trung Kiet Huynh et al.

Summary

The paper addresses the content effect in syllogistic reasoning. Language models tend to judge an argument's formal validity by whether its conclusion sounds believable. The authors' entry to SemEval-2026 Task 11, Subtask 1 (English binary validity classification) fine-tunes Qwen2.5-14B-Instruct with three complementary techniques:

  • Structure-Disentangled Prompting (SDP): regex-based parsing of each argument into premise–premise–conclusion slots inside a logic-first instruction template.
  • Asymmetric Counterfactual Debiasing (ACD): augmentation that only turns valid items into invalid ones by swapping in foreign conclusions. The reverse direction would create noisy labels.
  • Rank-Sensitive Logical Invariance Adaptation (RLIA): raising the QLoRA adapter rank to r=64. The authors argue that low-rank adapters have enough capacity to learn the task but not enough to also unlearn plausibility shortcuts.

Training used the 960-item balanced dataset. With a 10-seed ensemble, the system took first place, earning a flawless 100.0 on the Combined Score metric, with 100% accuracy and a TCE of 0. Ablations show that standard accuracy-boosting tricks actually increase bias. The two broader lessons are the one-directional safety of counterfactual edits in logic data and the link between adapter rank and bias suppression, which may matter for any multi-objective fine-tuning.

Key points

  • Data and metric: 960 training syllogisms balanced over validity × plausibility, and 191 test items. The ranking score divides accuracy by a logarithmic penalty on Total Content Effect (TCE).
  • Effect of adapter rank: moving from r=16 to r=64 halved TCE (2.13 → 1.04), raised accuracy (98.43% → 99.48%) and lifted the single-seed score from 45.99 to 58.05. Qwen2.5-32B, limited to r≤32 by memory, scored only 40.24.
  • ACD vs. bidirectional augmentation (on DeBERTa-v3-large): ACD reached 95.29% accuracy, while bidirectional counterfactuals dropped it to 92.67% because of label noise. ACD expands the training set by about 25% (ρ=0.5).
  • Training setup: 4-bit NF4 QLoRA (α=128) on all linear projections, 3 epochs, single H100 on Kaggle. The ensemble fixed the one remaining single-seed error, a valid argument with a negated-universal conclusion.
  • Failed alternatives: zero-shot chain-of-thought with Gemini 2.0 Flash scored near chance (49.74% accuracy). First-order-logic parsing reached only 79.06% accuracy, and premise-order test-time augmentation lowered the score.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksBias, fairness & safety · 517 works

Cite

Cite as Nguyen Tran, Duy Minh Dao Sy, Trung Kiet Huynh, Phu Hoa Pham and Phu Quy Nguyen Lam. 2026. HCMUSDroneBoys at SemEval-2026 Task 11: Asymmetric Counterfactual Debiasing and Rank-Sensitive Logical Invariance Adaptation for Syllogistic Reasoning. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.95

@inproceedings{tran2026hcmusdroneboys,
  title = {{HCMUSDroneBoys at SemEval-2026 Task 11: Asymmetric Counterfactual Debiasing and Rank-Sensitive Logical Invariance Adaptation for Syllogistic Reasoning}},
  author = {Nguyen Tran, Duy Minh Dao Sy, Trung Kiet Huynh, Phu Hoa Pham and Phu Quy Nguyen Lam},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.95},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.95}
}

References

19 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Leonardo Bertolazzi, Albert Gatt, and Raffaella Bernardi. 2024. A systematic analysis of large language models as soft reasoners: The case of syllogistic inferences. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024). find
  2. Duy Minh Dao-Sy, Trung Kiet Huynh, and 1 others. 2025a. JHARNA-MT: A copy-augmented hybrid of LoRA-tuned NLLB and lexical SMT with MBR decoding for low-resource indic languages. In Proceedings of the 2nd Workshop on Multimodal Machine Learning for Low-Resource Languages (MM-LoSo 2025). find
  3. Duy Minh Dao-Sy, Lam Phu Quy Nguyen, and 1 others. 2025b. DRAGON: Dual-encoder retrieval with guided ontology reasoning for medical normalization. In Proceedings of the Australasian Language Technology Association Workshop 2025. find
  4. Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Hannah R Sheahan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. 2022. Language models show human-like content effects on reasoning tasks. In Advances in Neural Information Processing Systems, volume 35. find
  5. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. LLM.int8(): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35. find
  6. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient finetuning of quantized language models. Advances in Neural Information Processing Systems, 36. find
  7. Tiwalayo Eisape, Michael Tessler, Ishita Dasgupta, Fei Sha, Sjoerd van Steenkiste, and Tal Linzen. 2024. A systematic comparison of syllogistic reasoning in humans and language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistic… find
  8. Gemini Team. 2024. Gemini: A family of highly capable multimodal models. Technical report, Google DeepMind.
  9. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. In International Conference on Learning Representations.
  10. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shanen Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  11. Divyansh Kaushik, Eduard Hovy, and Zachary C Lipton. 2020. Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations. find
  12. Giwon Kim, Marco Valentino, and Andre Freitas. 2025. Reasoning circuits in language models: A mechanistic interpretation of syllogistic inference. In Findings of the Association for Computational Linguistics: ACL 2025. find
  13. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35.
  14. Xiaobo Liang, Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, and Tie-Yan Liu. 2021. R-Drop: Regularized dropout for neural networks. Advances in Neural Information Processing Systems, 34. find
  15. Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of-thought reasoning. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter…
  16. Kentaro Ozeki, Risako Ando, Takanobu Morishita, Hirohiko Abe, Koji Mineshima, and Mitsuhiro Okada. 2024. Exploring reasoning biases in large language models through syllogism: Insights from the NeuBAROCO dataset. In Findings of the Association for Computational Linguistics: ACL 2024. find
  17. Qwen Team. 2024. Qwen2.5 technical report. Technical report, Alibaba Group. find
  18. Leonardo Ranaldi, Marco Valentino, and Andre Freitas. 2025. Improving chain-of-thought reasoning via quasi-symbolic abstractions. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025).
  19. Marco Valentino, Giwon Kim, Dhruv Dalal, Zhenyuan Zhao, and Andre Freitas. 2025. Mitigating content effects on reasoning in language models through fine-grained activation steering. Technical report, Idiap Research Institute. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tran-2026-hcmusdroneboys-semeval-2026-task-11/