Elamur Research⌕
☰
home / papers / 2026 / AICOE-Tredence at SemEval-2026 Task 11: Mitigating Content Bias in Syllogisms via Symbolic Logic-Language Decoupling

AICOE-Tredence at SemEval-2026 Task 11: Mitigating Content Bias in Syllogisms via Symbolic Logic-Language Decoupling

Paper · 2026 · Rakshith R and Ankush Chopra

Summary

The paper addresses content bias in LLMs: models often judge a syllogism by whether its conclusion sounds believable rather than whether it follows logically. Working on SemEval-2026 Task 11 (Subtask 1 in English, Subtask 3 across 11 languages), the authors move the validity decision out of natural language and into symbolic logic. Each argument is first normalized into a cleaned syllogism. The model then extracts the subject, predicate and middle terms and assigns an Aristotelian mood–figure code (e.g., AAA-1). That code is checked against the 24 classically valid forms, so the verdict never depends on what the sentences mean. With Gemini-3 Pro Preview doing the extraction, the system reached full validity accuracy with no measurable bias on both private test sets. The authors also distilled this structural supervision into a 4-bit Qwen-3 14B. The distilled model stayed unbiased in English but degraded somewhat in the multilingual setting. The study suggests that how reasoning is represented matters more than model scale.

Key points

  • Data: 960 English syllogisms balanced for validity and plausibility (4 were later dropped for lacking a middle term). Private tests contain 191 English items and 192 multilingual items.
  • Zero-shot baselines stay biased: Gemini-3 Pro Preview scored 84.29% accuracy with 15.62 bias on Subtask 1, and GPT-5 had 20.83 bias. Fine-tuning and data augmentation reduced bias only to about 1–3.
  • Structure-first results: Gemini-3 Pro Preview achieved a combined score of 100 (100% accuracy, 0 bias) on both subtasks. In the multilingual case, this required extracting structure directly in the native language (V2, score 100). Translating to English first (V1) scored only 36.86.
  • Distillation: A multilingual supervision set of 10,516 instances was built. Qwen-3 14B (4-bit, LoRA) scored 100 on English but 30.69 combined on multilingual (91.67% accuracy, 6.29 bias).
  • Error analysis: The weakest languages were Russian (13/17) and Swahili (12/17). Errors came from mood mismatches and middle-term figure shifts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: All India Council for Technical Education.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Rakshith R and Ankush Chopra. 2026. AICOE-Tredence at SemEval-2026 Task 11: Mitigating Content Bias in Syllogisms via Symbolic Logic-Language Decoupling. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.229

@inproceedings{r2026aicoe,
  title = {{AICOE-Tredence at SemEval-2026 Task 11: Mitigating Content Bias in Syllogisms via Symbolic Logic-Language Decoupling}},
  author = {Rakshith R and Ankush Chopra},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.229},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.229}
}

References

10 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Leonardo Bertolazzi, Albert Gatt, and Raffaella Bernardi. 2024. A systematic analysis of large language models as soft reasoners: The case of syllogistic inferences. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13882–13905, Miami, Florida, USA. Ass… find
  2. Michael Han Daniel Han and Unsloth team. 2023. Unsloth. find
  3. Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Hannah R. Sheahan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, and Felix Hill. 2024. Language models show human-like content effects on reasoning tasks. Preprint, arXiv:2207.07051.
  4. Philipp Mondorf and Barbara Plank. 2024. Comparing inferential strategies of humans and large language models in deductive reasoning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9370–9402, Bangkok, Thailand. Association fo… find
  5. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  6. Marco Valentino, Leonardo Ranaldi, Giulia Pucci, Federico Ranaldi, and André Freitas. 2026. Semeval-2026 task 11: Disentangling content and formal reasoning in large language models. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026).
  7. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. Preprint, arXiv:2201.11903.
  8. Magdalena Wysocka, Danilo Carvalho, Oskar Wysocki, Marco Valentino, and Andre Freitas. 2025. SylloBio-NLI: Evaluating large language models on biomedical syllogistic reasoning. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisti… find
  9. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  10. Yinsheng Zhang and Xiaodong Qiao. 2009. A formal system of aristotelian syllogism based on automata grammar. volume 5, pages 173–178. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/r-2026-aicoe-tredence-semeval-2026-task/