Elamur Research⌕
☰
home / papers / 2026 / Thinking in Schemas: Robust Syllogistic Reasoning in LLMs

Thinking in Schemas: Robust Syllogistic Reasoning in LLMs

Paper · 2026 · Federico Ranaldi, Leonardo Ranaldi, Fabio Massimo Zanzotto et al.

Summary

Large language models often accept an argument because its conclusion seems believable, even when the logic does not support it. In syllogistic reasoning this bias is called content effect (CE). The authors propose Boethius, a framework that splits syllogism evaluation into two explicit steps. First, a Schema Module reads the quantifiers and negations in the premises and maps the argument to a form from a fixed inventory of categorical syllogisms. Second, an Instantiation Module maps the concrete predicates onto that form and decides VALID or INVALID while treating terms as abstract placeholders. The approach works as in-context prompting for large models. Its two-stage outputs also serve as training trajectories for smaller students, which are trained with supervised fine-tuning and then GRPO. Experiments use an English corpus built on the Bertolazzi et al. (2024) dataset. Boethius beats deductive Chain-of-Thought and quasi-symbolic baselines in accuracy and clearly reduces CE. The gains appear in both prompted large models and distilled small ones. This matters because it is an auditable, solver-free way to make LLM inference depend on logical form rather than plausibility.

Key points

  • Two-stage pipeline: Schema Identification is followed by Schema Instantiation, and both steps are logged in an auditable trace.
    • Students learn a single tagged output format: <schema>…</schema><instantiation>…</instantiation><answer>…</answer>.
    • They run both stages in one generation, with no external guidance.
  • Data: The corpus extends the Bertolazzi et al. (2024) syllogism set.
    • Schemas are written in textual first-order logic.
    • Templates are filled with noun phrases drawn from WordNet hypernym–hyponym relations, which keeps lexical content controlled.
  • Distillation: Training starts with SFT as a cold start, followed by GRPO with three rewards:
    • correctness of the final answer;
    • adherence to the required format;
    • an abstraction reward that penalises copying surface wording.
  • Comparisons:
    • Baselines are single-LLM deductive Chain-of-Thought and the pseudo-symbolic methods Quasar and Aristotle.
    • Students trained on Boethius trajectories show less sensitivity to CE than students trained on deductive trajectories.
    • Performance holds, and sometimes improves, on reasoning tasks that are not syllogistic.
  • Limitations and gaps:
    • Invalid arguments are rejected by testing only the structurally closest schema, which is a heuristic rather than a full proof.
    • Multilingual tests cover only a few high-resource languages.
    • The PDF extract omits the results section, so specific accuracy and CE figures are not stated here.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations🎓University of Rome Tor VergataRome · Italy15 works · 1,321 citations

Also on the byline: AlmavivA.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Federico Ranaldi, Leonardo Ranaldi, Fabio Massimo Zanzotto and Shay B Cohen. 2026. Thinking in Schemas: Robust Syllogistic Reasoning in LLMs. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1897

@inproceedings{ranaldi2026thinking,
  title = {{Thinking in Schemas: Robust Syllogistic Reasoning in LLMs}},
  author = {Federico Ranaldi, Leonardo Ranaldi, Fabio Massimo Zanzotto and Shay B Cohen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1897},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1897}
}

References

33 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ivan Aslanov, Anita Tobar-Henríquez, and Ernesto Guerra. 2025. Prior beliefs impair logical reasoning about syllogisms on sexual violence. find
  2. Julia Aspernas, Arvid Erlandsson, Ernesto Guerra, and Artur Nilsson. 2022. Motivated formal reasoning: Ideological belief bias in syllogistic reasoning across diverse political issues. find
  3. Leonardo Bertolazzi, Albert Gatt, and Raffaella Bernardi. 2024. A systematic analysis of large language models as soft reasoners: The case of syllogistic inferences. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13882–13905, Miami, Florida, USA. Ass… find
  4. Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing Textual Entailment: Models and Applications. Synthesis Lectures on Human Language Technologies. Springer International Publishing, Cham. find
  5. Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Hannah R. Sheahan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, and Felix Hill. 2024. Language models show human-like content effects on reasoning tasks. Preprint, arXiv:2207.07051. find
  6. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chengg… — DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  7. Tiwalayo Eisape, Michael Tessler, Ishita Dasgupta, Fei Sha, Sjoerd Steenkiste, and Tal Linzen. 2024. A systematic comparison of syllogistic reasoning in humans and language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: H… find
  8. Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023. Specializing smaller language models towards multi-step reasoning. Preprint, arXiv:2301.12726.
  9. Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile Van Krieken, and Pasquale Minervini. 2025. Are we done wit… find
  10. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVi… — The Llama 3 Herd of Models
  11. Dongwei Jiang, Marcio Fonseca, and Shay Cohen. 2024. LeanReasoner: Boosting complex logical reasoning with lean. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7497–7510,… find
  12. Zhuoxuan Jiang, Haoyuan Peng, Shanshan Feng, Fan Li, and Dongsheng Li. 2025. Llms can find mathematical reasoning mistakes by pedagogical chain-of-thought. Preprint, arXiv:2405.06705. — [arXiv:2405.06705](https://arxiv.org/abs/2405.06705) find
  13. Lucas Keller, Felix Hazelaar, Peter M. Gollwitzer, and Gabriele Oettingen. 2024. Political ideology and environmentalism impair logical reasoning. find
  14. Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023. Evaluating the logical reasoning ability of chatgpt and gpt-4. Preprint, arXiv:2304.03439.
  15. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. The flan collection: Designing data and methods for effective instruction tuning. In Proceedings of the 40th International Conference on Machine Learning, vol…
  16. Gabriele Maraia, Leonardo Ranaldi, Marco Valentino, and Fabio Massimo Zanzotto. 2026. Can activation steering generalize across languages? a study on syllogistic reasoning in language models. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguisti…
  17. Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2025. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. Preprint, arXiv:2410.05229.
  18. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… — GPT-4 Technical Report
  19. Kentaro Ozeki, Risako Ando, Takanobu Morishita, Hi-rohiko Abe, Koji Mineshima, and Mitsuhiro Okada. 2025. Normative reasoning in large language models: A comparative benchmark from logical and modal perspectives. In Proceedings of the 8th BlackboxNLP Workshop: Analyzing and Interpreting Neural Netwo… find
  20. Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. 2023. Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning. Preprint, arXiv:2305.12295.
  21. Maria Teresa Pazienza, Marco Pennacchiotti, and Fabio Massimo Zanzotto. 2006. Learning textual entailment on a distance feature space. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 3944 LNAI:240 – 260. Cited by:… find
  22. Leonardo Ranaldi and Andre Freitas. 2024. Aligning large and small language models via chain-of-thought reasoning. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1812–1827, St. Julian’s, Malta. Association…
  23. Leonardo Ranaldi, Marco Valentino, and Andre Freitas. 2025. Improving chain-of-thought reasoning via quasi-symbolic abstractions. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 17222–17240, Vienna, Austria. Association for Co…
  24. Abulhair Saparov and He He. 2023. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. Preprint, arXiv:2210.01240.
  25. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.
  26. Shamus Zi Yang Sim and Tyrone Chen. 2025. Critique of impure reason: Unveiling the reasoning behaviour of medical large language models. find
  27. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, and Alek Andreev. 2024. Gemma 2: Improving open lan… find
  28. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. Preprint, arXiv:2201.11903.
  29. Magdalena Wysocka, Danilo Carvalho, Oskar Wysocki, Marco Valentino, and Andre Freitas. 2025. SylloBio-NLI: Evaluating large language models on biomedical syllogistic reasoning. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisti… find
  30. Jundong Xu, Hao Fei, Meng Luo, Qian Liu, Liangming Pan, William Yang Wang, Preslav Nakov, Mong-Li Lee, and Wynne Hsu. 2025. Aristotle: Mastering logical reasoning with a logic-complete decompose-search-resolve framework. In Proceedings of the 63rd Annual Meeting of the Association for Computational…
  31. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jin… — Qwen3 Technical Report
  32. Fabio Massimo Zanzotto, Elena Sofia Ruzzetti, Giancarlo A. Xompero, Leonardo Ranaldi, Davide Venditti, Federico Ranaldi, Cristina Giannone, Andrea Favalli, and Raniero Romagnoli. 2025. Position paper: MeMo: Towards language models with associative memory mechanisms. In Findings of the Association fo… find
  33. Marco Valentino, Geonhee Kim, Dhairya Dalal, Zhixue Zhao, and André Freitas. 2025. Mitigating content effects on reasoning in language models find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ranaldi-2026-thinking-schemas-robust-syllogistic-reasoning/