Elamur Research⌕
☰
home / papers / 2026 / Lakksh at SemEval-2026 Task 11(1 & 2): Neuro-Symbolic Decomposition to Mitigate Content Bias in Syllogistic Reasoning

Lakksh at SemEval-2026 Task 11(1 & 2): Neuro-Symbolic Decomposition to Mitigate Content Bias in Syllogistic Reasoning

Paper · 2026 · Lakksh Sharma, Krish Sharma and Jatin Bedi

Summary

Large language models tend to judge syllogisms by whether the conclusion sounds believable rather than whether it follows logically. This "content bias" is the focus of SemEval-2026 Task 11. The authors address the English subtasks: binary validity classification (Subtask 1), and premise retrieval followed by validity prediction (Subtask 2).

They treat bias as an architectural problem. A neural stage converts the text into an abstract (Mood, Figure) tuple. A parameter-free symbolic lookup over the 15 valid Aristotelian forms then decides validity without ever seeing the words. The extraction stage combines two models:

  • "Amnesiac" T5-Small, trained solely on synthetic syllogisms built from meaningless placeholders, so it cannot link real entities to validity.
  • Llama-3, whose output is OR-fused with the T5 parser to recover valid-but-implausible cases.

For Subtask 2, Gemini-2.0-Flash first selects the relevant premises.

The system reaches 97.38% accuracy with low content effect on Subtask 1. An ablation shows that routing reasoning through LLM-generated Z3 code increases bias. The takeaway is that the translation layer, not just the solver, must be content-agnostic.

Key points

  • Training and evaluation data. The T5-Small parser (60M parameters) was trained on 200K nonsense-symbol examples. These cover all 256 (Mood, Figure) combinations, with roughly 50 paraphrase templates per proposition type. The 960 official English syllogisms were used only for evaluation.
  • Subtask 1 results. The ensemble with the symbolic kernel scored ACC 97.38, TCE 3.10 and Combined 40.38. This beats Llama-3 alone (90.05 / 7.29 / 28.91) and T5 alone (70.16 / 33.33 / 15.47).
  • Subtask 2 results. The system scored ACC 82.11, TCE 9.89 and Combined 26.80, with 99.47% F1 on premise identification. The accuracy gap comes mostly from wrongly ordered major/minor premises, which leads to the wrong Figure.
  • Z3 ablation. Replacing the kernel with NL→Z3 verification raised TCE as Z3 usage grew: 9.89 at 0%, 17.79 at 50% and 25.00 at 100%. Combined scores fell accordingly, from 26.80 to 22.97 to 18.91.
  • Fusion effect and ranking. OR fusion fixed about 60% of the LLM's content-driven errors. In the official standings, the system placed 24th for Subtask 1 and 12th for Subtask 2.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Thapar Institute of Engineering & TechnologyPatiāla · India6 works · 11 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Lakksh Sharma, Krish Sharma and Jatin Bedi. 2026. Lakksh at SemEval-2026 Task 11(1 & 2): Neuro-Symbolic Decomposition to Mitigate Content Bias in Syllogistic Reasoning. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{sharma2026lakksh,
  title = {{Lakksh at SemEval-2026 Task 11(1 \& 2): Neuro-Symbolic Decomposition to Mitigate Content Bias in Syllogistic Reasoning}},
  author = {Lakksh Sharma, Krish Sharma and Jatin Bedi},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.17},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.17}
}

References

10 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Luca Bertolazzi, Albert Gatt, and Raffaella Bernardi. 2024. A systematic analysis of large language models as soft reasoners: The case of syllogistic inferences. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  2. Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Hannah R Sheahan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. 2022. Language models show human-like content effects on reasoning tasks. arXiv preprint arXiv:2207.07051. find
  3. Tiwalayo Eisape, Michael Tessler, Ishita Dasgupta, Fei Sha, Sjoerd Steenkiste, and Tal Linzen. 2024. A systematic comparison of syllogistic reasoning in humans and language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (N… find
  4. Gyuwan Kim, Marco Valentino, and André Freitas. 2025. Reasoning circuits in language models: A mechanistic interpretation of syllogistic inference. In Findings of the Association for Computational Linguistics: ACL 2025. find
  5. Qing Lyu, Shreyas Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of-thought reasoning. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapte…
  6. Kento Ozeki, Ryo Ando, Terufumi Morishita, Hitomi Abe, Koji Mineshima, and Mitsuhiro Okada. 2024. Exploring reasoning biases in large language models through syllogism: Insights from the neubaroco dataset. In Findings of the Association for Computational Linguistics: ACL 2024. find
  7. Xiao Quan, Marco Valentino, Louise Dennis, and André Freitas. 2024. Verification and refinement of natural language explanations through LLM-symbolic theorem proving. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  8. Marco Valentino, Gyuwan Kim, Dhara Dalal, Zhibin Zhao, and André Freitas. 2025. Mitigating content effects on reasoning in language models through fine-grained activation steering. arXiv preprint arXiv:2505.12189. — [arXiv:2505.12189](https://arxiv.org/abs/2505.12189) find
  9. Marco Valentino, Leonardo Ranaldi, Giulia Pucci, Federico Ranaldi, and André Freitas. 2026. Semeval-2026 task 11: Disentangling content and formal reasoning in large language models. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026).
  10. Jiarui Xu, Hao Fei, Li Pan, Qian Liu, Mong-Li Lee, and Wynne Hsu. 2024. Faithful logical reasoning via symbolic chain-of-thought. arXiv preprint arXiv:2405.18357.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sharma-2026-lakksh-semeval-2026-task-11/