Elamur Research⌕
☰
home / papers / 2026 / ITLC at SemEval-2026 Task 11: Normalization and Deterministic Parsing for Formal Reasoning in LLMs

ITLC at SemEval-2026 Task 11: Normalization and Deterministic Parsing for Formal Reasoning in LLMs

Paper · 2026 · Wicaksono Leksono Muhamad, Joanito Agili Lopo, Tack Hwa Wong et al.

Summary

Large language models often decide whether a syllogism is valid based on how plausible its content sounds rather than on its logical form. This content effect is worse in multilingual settings. The ITLC team's SemEval-2026 Task 11 system tackles it by splitting the work between an LLM and a symbolic step:

  • Normalization: Gemini-3-Flash rewrites each argument into a canonical categorical syllogism, with terms replaced by abstract placeholders (A, B, C).
  • English pivot: For non-English inputs, an optional "English Pivot Normalization" (EPN) step first translates only quantifiers and copulas into English and leaves the terms in the source language.
  • Deterministic validation: A regex-based parser assigns each sentence a type (A/E/I/O) and derives the mood and figure. A rule table of the 24 valid forms then decides validity, with extra checks for trivially valid cases such as restatement or E/I conversion.
  • Relevant premises: These are taken as the major and minor premises linked through the middle term.

No fine-tuning is involved. The system ranked in the top five on three subtasks and sixth on the fourth (English relevant-premise selection), while sharply reducing content bias. This makes it an interpretable alternative to fine-tuning or activation steering.

Key points

  • Choice of notation: On English data, plain syllogistic notation beat raw text, Predicate-Argument notation and FOL notation. It reached 98.95 accuracy, 2.13 bias and a combined score of 46.23, versus 92.10 / 10.63 / 26.66 for raw inputs.
  • English results: Norm+Parsing reached 98.94 validity accuracy with zero validity bias. Its relevant-premise F1 was 95.43, with relevance bias falling from 5.0 to 2.0. There were 16 premise mismatches out of 190, mostly caused by redundant universal statements.
  • Multilingual results: Validity accuracy rose from 86.98 (LLM-only) to 90.63 with the structured pipelines. EPN brought validity bias to 0.0 and relevance bias down to 2.99 (from 7.29 for LLM-only).
  • Why EPN matters: Without EPN, multilingual premise F1 dropped to 72.50, versus 90.10 with EPN. The cause was singular/plural variants (e.g., Italian pianta/piante) that broke the regex matching.
  • Setup and limitations: The system used gemini-3-flash-preview with temperature 0 and seed 0. Only this single model was tested, with no variation in seeds or sampling settings. Code is released on the SEACrowd GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢CohereSan Francisco · United States15 works · 1,223 citations

Also on the byline: Universiti Teknologi Petronas.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Wicaksono Leksono Muhamad, Joanito Agili Lopo, Tack Hwa Wong, Muhammad Ravi Shulthan Habibi and Samuel Cahyawijaya. 2026. ITLC at SemEval-2026 Task 11: Normalization and Deterministic Parsing for Formal Reasoning in LLMs. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.221

@inproceedings{muhamad2026itlc,
  title = {{ITLC at SemEval-2026 Task 11: Normalization and Deterministic Parsing for Formal Reasoning in LLMs}},
  author = {Wicaksono Leksono Muhamad, Joanito Agili Lopo, Tack Hwa Wong, Muhammad Ravi Shulthan Habibi and Samuel Cahyawijaya},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.221},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.221}
}

References

18 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Leonardo Bertolazzi, Albert Gatt, and Raffaella Bernardi. 2024. A systematic analysis of large language models as soft reasoners: The case of syllogistic inferences. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13882–13905, Miami, Florida, USA. Ass… find
  2. Irving M. Copi, Carl Cohen, and Victor Rodych. 2018. Introduction to Logic, 15th edition. Routledge, New York. find
  3. Ruixiang Cui, Daniel Hershcovich, and Anders Søgaard. 2022. Generalized quantifiers as a source of error in multilingual NLU benchmarks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4875–4893.… find
  4. Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Hannah R. Sheahan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, and Felix Hill. 2024. Language models show human-like content effects on reasoning tasks. Preprint, arXiv:2207.07051. find
  5. Tiwalayo Eisape, Michael Tessler, Ishita Dasgupta, Fei Sha, Sjoerd Steenkiste, and Tal Linzen. 2024. A systematic comparison of syllogistic reasoning in humans and language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: H… find
  6. Yanzhu Guo, Simone Conia, Zelin Zhou, Min Li, Saloni Potdar, and Henry Xiao. 2025. Do large language models have an English accent? evaluating and improving the naturalness of multilingual LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon…
  7. Patrick J. Hurley. 2014. A Concise Introduction to Logic, 12th edition. Cengage Learning, Stamford, CT. find
  8. Geonhee Kim, Marco Valentino, and Andre Freitas. 2025. Reasoning circuits in language models: A mechanistic interpretation of syllogistic inference. In Findings of the Association for Computational Linguistics: ACL 2025, pages 10074–10095, Vienna, Austria. Association for Computational Linguistics. find
  9. Joanito Agili Lopo, Muhammad Ravi Shulthan Habibi, Tack Hwa Wong, Muhammad Ilham Ghozali, Fajri Koto, Genta Indra Winata, Peerat Limkonchotiwat, Alham Fikri Aji, and Samuel Cahyawijaya. 2025. Language surgery in multilingual large language models. In Proceedings of the 5th Workshop on Multilingual R… find
  10. Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of-thought reasoning. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter…
  11. Kentaro Ozeki, Risako Ando, Takanobu Morishita, Hirohiko Abe, Koji Mineshima, and Mitsuhiro Okada. 2024. Exploring reasoning biases in large language models through syllogism: Insights from the neubaroco dataset. In Findings of the Association for Computational Linguistics: ACL 2024, pages 16063– 16… find
  12. Terence Parsons. 2014. Articulating Medieval Logic. Oxford University Press, Oxford. find
  13. Graham Priest. 2008. An Introduction to Non-Classical Logic: From If to Is, 2nd edition. Cambridge University Press, Cambridge. find
  14. Arthur N. Prior. 1962. Formal Logic, 2nd edition. Oxford University Press. find
  15. Xin Quan, Marco Valentino, Louise A. Dennis, and Andre Freitas. 2024. Verification and refinement of natural language explanations through LLM-symbolic theorem proving. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2933–2958, Miami, Florida, USA. As…
  16. Frank P. Ramsey. 2009. On a problem of formal logic. In Ira Gessel and Gian-Carlo Rota, editors, Classic Papers in Combinatorics, Modern Birkhäuser Classics. Birkhäuser Boston. find
  17. Leonardo Ranaldi, Marco Valentino, and Andre Freitas. 2025. Improving chain-of-thought reasoning via quasi-symbolic abstractions. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 17222–17240, Vienna, Austria. Association for Co…
  18. Shi Zong and Jimmy Lin. 2024. Categorical syllogisms revisited: A review of the logical reasoning abilities of LLMs for analyzing categorical syllogisms. In Proceedings of the 1st Workshop on NLP for Science (NLP4Science), pages 230–239. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/muhamad-2026-itlc-semeval-2026-task-11/