Elamur Research⌕
☰
home / papers / 2026 / YNU-NLP at SemEval-2026 Task 11: A Neuro-Symbolic Approach with Reflexion Mechanism Disentangling Content and Formal Reasoning in Language Models

YNU-NLP at SemEval-2026 Task 11: A Neuro-Symbolic Approach with Reflexion Mechanism Disentangling Content and Formal Reasoning in Language Models

Paper · 2026 · Yu Wang, You Zhang, Hao Zhang et al.

Summary

The paper addresses the content effect in syllogistic reasoning: language models often judge an argument by whether its claims sound true in the real world, not by whether the conclusion follows logically. For SemEval-2026 Task 11, the YNU-NLP team built a hybrid system on Qwen2.5-14B-Instruct.

They first enlarged the training data fourfold. The augmentation added structured JSON annotations, injected irrelevant distractor premises, and translated examples into 11 languages. One LoRA-tuned model was then trained on two jobs: a parser that turns arguments into a logical JSON schema, and a solver that predicts validity directly.

At inference, the system follows a Try-Fix-Verify loop. When a term mismatch breaks the subject–middle–predicate chain, a Reflexion step asks the model to repair the parse. The repaired argument is converted to First-Order Logic and checked by the Z3 Theorem Prover. If symbolic checking fails, the neural solver is used as a fallback.

The system ranked 1st on Subtasks 1 and 2 and 2nd on Subtask 4. This shows that formal verification sharply reduces belief bias, including in multilingual and noisy settings.

Key points

  • Augmented training set: 3,836 instances. Augmentation used Gemini 3 Pro for structural enhancement, 1–5 injected distractor premises per example, and Qwen-MT-Plus for translation into 11 languages.
  • Training setup: Qwen2.5-14B-Instruct, 4-bit quantized, fine-tuned with LoRA (r=16, α=32) via Unsloth. Loss was computed on responses only, and training ran on a single RTX 3090.
  • Validity check: proof by contradiction in Z3. The premises plus the negated conclusion yielding UNSAT means the argument is valid.
  • Leaderboard results: perfect Subtask 1 performance (ACC 100, TCE 0). On Subtask 4, TCE fell from 14.65 (standard SFT) to 1.26, and Premise F1 rose from 65.62 to 89.58.
  • Ablation and limitations: removing Reflexion lowered the Subtask 4 score from 49.51 to 36.09. The team placed only 9th on clean multilingual Subtask 3, possibly because strict verification rejects some cases a neural model would answer correctly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Yunnan UniversityKunming · China17 works · 40 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Yu Wang, You Zhang, Hao Zhang and Dan Xu. 2026. YNU-NLP at SemEval-2026 Task 11: A Neuro-Symbolic Approach with Reflexion Mechanism Disentangling Content and Formal Reasoning in Language Models. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.126

@inproceedings{wang2026ynu,
  title = {{YNU-NLP at SemEval-2026 Task 11: A Neuro-Symbolic Approach with Reflexion Mechanism Disentangling Content and Formal Reasoning in Language Models}},
  author = {Yu Wang, You Zhang, Hao Zhang and Dan Xu},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.126},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.126}
}

References

22 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Leonardo Bertolazzi, Albert Gatt, and Raffaella Bernardi. 2024. A systematic analysis of large language models as soft reasoners: The case of syllogistic inferences. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024), pages 13882–13905. find
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… find
  3. Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2020. Transformers as Soft Reasoners over Language. In Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI 2020), pages 3882–3890.
  4. Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Hannah R. Sheahan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, and Felix Hill. 2024. Language models show human-like content effects on reasoning tasks. arXiv preprint arXiv:2207.07051. find
  5. Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An efficient SMT solver. In Proceedings of the 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS 2008), pages 337–340. find
  6. Tiwalayo Eisape, Michael Tessler, Ishita Dasgupta, Fei Sha, Sjoerd Steenkiste, and Tal Linzen. 2024. A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: H… find
  7. Debanjan Ghosh, Kai North, Ekaterina Kochmar, Mamoru Komachi, and Marcos Zampieri, editors. 2026. Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). Association for Computational Linguistics, San Diego, United States. find
  8. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adaptation of Large Language Models. In Proceedings of the International Conference on Learning Representations (ICLR 2021).
  9. Geonhee Kim, Marco Valentino, editor = "Che Wanxiang Freitas, Andre", Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar. 2025. Reasoning circuits in language models: A mechanistic interpretation of syllogistic inference. In Findings of the Association for Computational Linguistics: ACL 2… find
  10. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS 2022), pages 22199–22213.
  11. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. SELF-REFINE: Iterative refinement with… find
  12. Theo Olausson, Alex Gu, Ben Lipkin, Cedegao Zhang, Armando Solar-Lezama, Joshua Tenenbaum, and Roger Levy. 2023. LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers. In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan…
  13. Kentaro Ozeki, Risako Ando, Takanobu Morishita, Hirohiko Abe, Koji Mineshima, and Mitsuhiro Okada. 2024. Exploring Reasoning Biases in Large Language Models Through Syllogism: Insights from the NeuBAROCO Dataset. In Findings of the Association for Computational Linguistics (ACL 2024), pages 16063–16… find
  14. Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang. 2023. Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning. In Findings of the Association for Computational Linguistics (EMNLP 2023), pages 3806–3824.
  15. Xin Quan, Marco Valentino, Louise A. Dennis, and Andre Freitas. 2024. Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Proving. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024), pages 2933–2958.
  16. Leonardo Ranaldi, Marco Valentino, and Andre Freitas. 2025. Improving Chain-of-Thought Reasoning via Quasi-Symbolic Abstractions. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), pages 17222–17240.
  17. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R. Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. In Thirty-Seventh Conference on Neural Information Processing Systems.
  18. Marco Valentino, Geonhee Kim, Dhairya Dalal, Zhixue Zhao, and André Freitas. 2025. Mitigating content effects on reasoning in language models through fine-grained activation steering. arXiv preprint arXiv:2505.12189. — [arXiv:2505.12189](https://arxiv.org/abs/2505.12189) find
  19. Marco Valentino, Leonardo Ranaldi, Giulia Pucci, Federico Ranaldi, and André Freitas. 2026. Semeval-2026 task 11: Disentangling content and formal reasoning in large language models. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026).
  20. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS 202…
  21. Yongkang Wu, Meng Han, Yutao Zhu, Lei Li, Xinyu Zhang, Ruofei Lai, Xiaoguang Li, Yuanhang Ren, Zhicheng Dou, and Zhao Cao. 2023. Hence, Socrates is mortal: A Benchmark for Natural Language Syllogistic Reasoning. In Findings of the Association for Computational Linguistics (ACL 2023), pages 2347–2367… find
  22. Jundong Xu, Hao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, and Wynne Hsu. 2024. Faithful Logical Reasoning via Symbolic Chain-of-Thought. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), pages 13326–13365.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-ynu-nlp-semeval-2026-task/