SemEval-2026 Task 11: Disentangling Content and Formal Reasoning in Large Language Models
Paper · 2026 · Marco Valentino, Leonardo Ranaldi, Giulia Pucci et al.
Summary
This shared task asks whether large language models can judge if an argument is logically valid without being swayed by whether its content sounds true. This bias is called the content effect. The organisers built a new multilingual dataset of syllogisms that crosses validity with plausibility. It covers 24 syllogistic schemes in ten languages. Four subtasks were defined: English and multilingual validity classification, each in a clean version and a version that mixes in irrelevant premises. In the noisy versions, systems must also pick out which premises actually matter. Ranking used a combined score that rewards accuracy and penalises content bias.
Zero-shot tests showed that older open models carry a strong content effect. Newer generations largely overcome it. The best participant systems handed inference to symbolic solvers and scored perfectly in English. Distracting premises and non-English inputs still raised bias noticeably. The task therefore supports neuro-symbolic pipelines and points to multilingual, longer-context reasoning as open problems.
Key points
- The competition ran on Codabench. It drew about 250 participants, more than 3000 submissions and 35 system description papers.
- Metrics included accuracy, Total Content Effect (TCE, the mean of intra- and cross-plausibility effects) and premise-selection F1 for the noisy subtasks. The ranking metric was Combined Score: CS = ACC / (1 + ln(1 + TCE)). In the noisy subtasks, the numerator is the average of ACC and F1.
- Baselines differed by model generation:
- Llama 3 and Qwen 2.5 showed pervasive content bias, and larger size did not reliably help.
- Qwen 3 and Phi-4 exceeded 95 accuracy with lower bias.
- In Subtask 1, several systems reached 100% accuracy with zero TCE (CS = 100). In Subtask 2, two systems scored perfectly. These top systems mostly relied on pseudo-formal decomposition, often combined with architectural changes.
- Multilingual settings were harder:
- In Subtask 3, systems above 90% accuracy still showed TCE of roughly 5–11.
- In Subtask 4, accuracy and F1 were often 85–91%, with TCE around 3–5.
Institutions
Fields, topics and concepts
📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works
🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksMultilingual & cross-lingual NLP · 828 works
Cite
Cite as Marco Valentino, Leonardo Ranaldi, Giulia Pucci, Federico Ranaldi and André Freitas. 2026. SemEval-2026 Task 11: Disentangling Content and Formal Reasoning in Large Language Models. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).
@inproceedings{valentino2026semeval,
title = {{SemEval-2026 Task 11: Disentangling Content and Formal Reasoning in Large Language Models}},
author = {Marco Valentino, Leonardo Ranaldi, Giulia Pucci, Federico Ranaldi and André Freitas},
booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
year = {2026},
doi = {10.18653/v1/2026.semeval-1.450},
url = {https://doi.org/10.18653/v1/2026.semeval-1.450}
}References
- Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. 2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905.
- Hanna Abi Akl, Fabien Gandon, Catherine Faron, and Pierre Monnin. 2026. Sef-clgc at semeval-2026 task 11 subtask 1: Logical notation impact on language model performance. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). find
- Laksh Advani. 2026. lakshadvani at semeval-2026 task 11: A neuro-symbolic approach to content-independent syllogistic reasoning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). find
- Adewale Akinfaderin and Nafi Diallo. 2026. Fregelogic at semeval 2026 task 11: A hybrid neuro-symbolic architecture for content-robust syllogistic validity prediction. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). find
- Risako Ando, Takanobu Morishita, Hirohiko Abe, Koji Mineshima, and Mitsuhiro Okada. 2023. Evaluating large language models with NeuBAROCO: Syllogistic reasoning ability and human-like biases. In Proceedings of the 4th Natural Logic Meets Machine Learning Workshop, pages 1–11, Nancy, France. Associat…
- Mohamed Ayman, Khaled Marzouk, Abdallah Mashaly, and Ahmed Hereiz. 2026. Proofbusters at SemEval-2026 Task 11: Symbolic Abstraction Approaches for Improving Logical Reasoning in Language Models. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). Association for… find
- Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.
- Ishwar B Balappanawar, Vamshi Krishna Bonagiri, Anish R Joishy, Manas Gaur, Krishnaprasad Thirunarayan, and Ponnurangam Kumaraguru. 2025. If pigs could fly... can llms logically reason through counterfactuals? arXiv preprint arXiv:2505.22318. find
- Leonardo Bertolazzi, Albert Gatt, and Raffaella Bernardi. 2024. A systematic analysis of large language models as soft reasoners: The case of syllogistic inferences. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13882–13905, Miami, Florida, USA. Ass… find
- Leonardo Bertolazzi, Sandro Pezzelle, and Raffaella Bernardi. 2025. How language models conflate logical validity with plausibility: A representational analysis of content effects. arXiv preprint arXiv:2510.06700. find
- Rafael Butas, Rodica Potolea, Camelia Lemnaru, and Alex Lapusan. 2026. TUCNLP at SemEval-2026 task 11: Neuro-symbolic content stripping for debiased syllogistic reasoning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). find
- Răzvan-Costinel Gogu, Ştefan Plăcintescu, and Sofia-Maria Vultur. 2026. Ghegheghe at semeval-2026 task 11: Decoupling logic from belief with bias-targeted fine-tuning and neuro-symbolic syllogistic reasoning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). find
- Zhanyu Chen, María Teresa Muñoz Martín, Sem Huisman, and Jingjing Lan. 2026. Sylloscope at SemEval-2026 task 11: Decoupling logic from belief via DeepSeek-enhanced distillation in Qwen models. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). find
- Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
- Akash Chowdhury, Vlad Pavlovich Julius Dunfoy, and Sophia Yang Abhiram Borra. 2026. Abstractreasoner at semeval-2026 task 11: Reducing content effects via knowledge distillation and structured reasoning prompts. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). find
- Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Hannah R Sheahan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. 2022. Language models show human-like content effects on reasoning tasks. arXiv preprint arXiv:2207.07051. find
- Janiek de Rijke, Niek Biesterbos, and Mark den Ouden. 2026. RvH-40 at SemEval-2026 task 11: Disentangling reasoning from belief through symbolic abstraction. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). find
- Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088–10115.
- Tiwalayo Eisape, Michael Tessler, Ishita Dasgupta, Fei Sha, Sjoerd Steenkiste, and Tal Linzen. 2024. A systematic comparison of syllogistic reasoning in humans and language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: H… find
- Adnan Faisal and Shiti Chowdhury. 2026. CUET_Luminaries at SemEval-2026 task 11: Disentangling logical validity from semantic plausibility through canonical abstraction. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). find
- Feihao Fang, My T Thai, and Yuanyuan Lei. 2026. Discovering a shared logical subspace: Steering llm logical reasoning via alignment of natural-language and symbolic views. arXiv preprint arXiv:2604.19716. find
- Junhao Fu, Yun He, Lina Zhao, and Weijuan Li. 2026. YNJTC at SemEval-2026 task 11: A neuro-symbolic hybrid pipeline for content-independent syllogistic reasoning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). find
- Pasquale Grimaldi. 2026. pamaldi at semeval-2026 task 11: Neuro-symbolic syllogistic reasoning via llm-guided structure extraction and deterministic validation. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). find
- Ishita Gupta, Dhruv Goyal, and Jatin Bedi. 2026a. Single-call joint abstraction for robust neuro-symbolic retrieval: 0704mis at semeval-2026 task 11 subtasks 2 & 4. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). find
- Ishita Gupta, Dhruv Goyal, and Jatin Bedi. 2026b. Team 0704mis at SemEval-2026 task 11: Dual-view consistency testing for content-invariant multilingual syllogistic reasoning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). find
- Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
- Twan Huiskens, Tian Niezing, and Koen Snelten. 2026. d’olle grieze at semeval-2026 task 11: Comparing the impact of supervised fine-tuning and activation steering on mitigating content effect bias in syllogistic reasoning. In Proceedings of the 20th International Workshop on Semantic Evaluation (Sem… find
- Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
- Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Paul Saldyt, and Anil B Murthy. 2024. Position: Llms can’t plan, but can help planning in llm-modulo frameworks. In Forty-first International Conference on Machine Learning. find
- Ivan Kartáč, Kristýna Onderková, Jan Bronec, Zdeněk Kasner, Mateusz Lango, and Ondřej Dušek. 2026. UFAL-CUNI at SemEval-2026 task 11: An efficient modular neuro-symbolic method for syllogistic reasoning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). find
- Geonhee Kim, Marco Valentino, and Andre Freitas. 2025. Reasoning circuits in language models: A mechanistic interpretation of syllogistic inference. In Findings of the Association for Computational Linguistics: ACL 2025, pages 10074–10095, Vienna, Austria. Association for Computational Linguistics. find
- Farzaneh Bayan Memar, Hanneke Huls, and Matthijs ten Hove. 2026. Ellat at semeval-2026 task 11: Comparing encoder and decoder models for syllogistic reasoning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026).
- Saran Krishnasamy. 2026. GigitAI at SemEval-2026 task 11: Hybrid symbolic-neural approach for syllogistic validity classification. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). find
- Seyed Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2025. GSM-symbolic: Understanding the limitations of mathematical reasoning in large language models. In The Thirteenth International Conference on Learning Representations.
- Ewelina Ksi˛eżniak. 2026. Team ewelinaksiez at SemEvAl-2026 task 11: Reducing content bias in syllogistic reasoning via semantic abstraction. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). find
- Songhuan Li, Liang Yang, Shengdi Yin, Qiang Zhang, and Hongfei Lin. 2026. dutir_shlee at SemEval-2026 task 11: Symbolic augmentation for content-bias-resistant syllogistic reasoning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). find
- Francisco F. López-Ponce, Lucía Pitarch, Iván Saavedra-Martínez, Ignacio Huitzil, Sergio-Luis Ojeda-Trueba, Fernando Bobillo, and Gemma Bel-Enguix. 2026. GIL-zaragoza at SemEval 2026 task 11: Comparing classification, autoformalization, and ontologies for formal reasoning capabilities. In Proceeding…
- Rongchuan Luo, Jin Wang, and Xuejie Zhang. 2026. YNU-HPCC at SemEval-2026 task 11: Mitigating content effects in syllogistic reasoning with Qwen2-1.5B-instruct and XLM-ROBERTa-large for English and multilingual tasks. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-… find
- Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of-thought reasoning. In The 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Associa…
- Gabriele Maraia, Leonardo Ranaldi, Marco Valentino, and Fabio Massimo Zanzotto. 2026a. Can activation steering generalize across languages? a study on syllogistic reasoning in language models. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguist…
- Gabriele Maraia, Marco Valentino, Fabio Massimo Zanzotto, and Leonardo Ranaldi. 2026b. Abstract activation spaces for content-invariant reasoning in large language models. Preprint, arXiv:2602.02462. — [arXiv:2602.02462](https://arxiv.org/abs/2602.02462) find
- Philipp Mondorf and Barbara Plank. 2024. Comparing inferential strategies of humans and large language models in deductive reasoning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9370–9402. find
- Wicaksono Leksono Muhamad, Joanito Agili Lopo, Tack Hwa Wong, Muhammad Ravi Shulthan Habibi, and Samuel Cahyawijaya. 2026. Itlc at semeval-2026 task 11: Normalization and deterministic parsing for formal reasoning in llms. In Proceedings of the 20th International Workshop on Semantic Evaluation (Sem…
- Kentaro Ozeki, Risako Ando, Takanobu Morishita, Hirohiko Abe, Koji Mineshima, and Mitsuhiro Okada. 2024. Exploring reasoning biases in large language models through syllogism: Insights from the neubaroco dataset. In Findings of the Association for Computational Linguistics ACL 2024, pages 16063– 160… find
- Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang. 2023. Logic-LM: Empowering large language models with symbolic solvers for faithful logical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3806–3824, Singapore. Association for Computational Lingui…
- Wiebke Petersen, Cherine Jaziri, and Diem-Xuan Tran. 2026. HHU-SyLo at SemEval-2026 task 11: Logic in the loop – hybridizing LLMs and theorem provers for robust formal reasoning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026).
- Xin Quan, Marco Valentino, Louise Dennis, and André Freitas. 2024. Verification and refinement of natural language explanations through llm-symbolic theorem proving. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2933–2958.
- Rakshith R and Ankush Chopra. 2026. Aicoe-tredence at semeval-2026 task 11: Mitigating content bias in syllogisms via symbolic logic-language decoupling. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026).
- Leonardo Ranaldi, Marco Valentino, and Andre Freitas. 2025. Improving chain-of-thought reasoning via quasi-symbolic abstractions. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 17222–17240, Vienna, Austria. Association for Co…
- Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 3982–3992.
- S Seals and Valerie Shalin. 2024. Evaluating the deductive competence of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8606–8622. find
- Soumyajit Roy Setu and Manav Malhotra. 2026. Modusponens at semeval-2026 task 11: Breaking the plausibility trap in llms via conflict-aware ensembling. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026).
- Abdullah Shaikh, Zain Naqi, Taha Zahid, Sandesh Kumar, and Abdul Samad. 2026. HABIB_TAZ at SemEval-2026 task 11: Disentangling formal logic from content via synthetic training and multi-objective optimization. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026).
- Lakksh Sharma, Jatin Bedi, and Krish Sharma. 2026a. Lakksh at semeval-2026 task 11(1 & 2): Neuro-symbolic decomposition to mitigate content bias in syllogistic reasoning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026).
- Lakksh Sharma, Rhea Singhal, Krish Sharma, and Jatin Bedi. 2026b. Lakksh at semeval-2026 task 11(3&4): Instability-triggered compute for multilingual syllogistic reasoning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). Association for Computational Linguist… find
- Peiyang Song, Pengrui Han, and Noah Goodman. 2026. Large language model reasoning failures. Transactions on Machine Learning Research. Survey Certification. find
- C. Sai Aravind Sunil Saumya and C. Pothan Sai Reddy. 2026. SpyComet at SemEval-2026 task 11: When adversarial debiasing backfires - a comparison of data-level and model-level debiasing. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). find
- Hidetsune Takahashi. 2026. Hidetsune at SemEval-2026 task 11: Adapting pretrained reasoning models with deep supervision and inference refinement for content-independent validity classification. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026).
- Xingwei Tan, Marco Valentino, Mahmud Elahi Akhter, Maria Liakata, and Nikolaos Aletras. 2025. Enhancing logical reasoning in language models via symbolically-guided Monte Carlo process supervision. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 31886… find
- Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
- P. K. Thiyagarajaa and Durairaj Thenmozhi. 2026. Thiyaga6851 at semeval-2026 task 11: Disentangling content and formal reasoning in large language models using neuro-symbolic mapping. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). find
- Chi-Nguyen Tran, Duy Minh Dao Sy, Trung Kiet Huynh, Phu-Hoa Pham, and Lam Phu Quy Nguyen. 2026. HCMUS_DroneBoys at SemEval-2026 task 11: Asymmetric counterfactual debiasing and rank-sensitive logical invariance adaptation for syllogistic reasoning. In Proceedings of the 20th International Workshop o… find
- Noah Tratzsch, Asmaa Al-Raian, Mounika Marreddy, and Alexander Mehler. 2026. Semeval-2026 task 11: Reducing content effects using layered activation steering. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). find
- Marco Valentino, Geonhee Kim, Dhairya Dalal, Zhixue Zhao, and André Freitas. 2026. Mitigating content effects on reasoning in language models through fine-grained activation steering. Preprint, arXiv:2505.12189. — [arXiv:2505.12189](https://arxiv.org/abs/2505.12189) find
- Yu Wang, You Zhang, Hao Zhang, and Dan Xu. 2026. Ynu-nlp at semeval-2026 task 11: A neuro-symbolic approach with reflexion mechanism disentangling content and formal reasoning in language models. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026).
- Magdalena Wysocka, Danilo Carvalho, Oskar Wysocki, Marco Valentino, and Andre Freitas. 2024. Syllobio-nli: Evaluating large language models on biomedical syllogistic reasoning. arXiv preprint arXiv:2410.14399. find
- Jundong Xu, Hao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, and Wynne Hsu. 2024. Faithful logical reasoning via symbolic chain-of-thought. arXiv preprint arXiv:2405.18357.
- Zhen Xu, Sergio Escalera, Adrien Pavão, Magali Richard, Wei-Wei Tu, Quanming Yao, Huan Zhao, and Isabelle Guyon. 2022. Codabench: Flexible, easy-to-use, and reproducible meta-benchmark platform. Patterns, 3(7):100543. find
- Schema : Baroco ( AOO -2) Premise 1: All \<A \> are \<B \> Premise 2: Some \<C \> are not \<B \> Conclusions : Some \<C \> are not \<A \> An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arX…
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/valentino-2026-semeval-2026-task-11-disentangling/