Elamur Research⌕
☰
home / papers / 2026 / Ellat at SemEval-2026 Task 11: Comparing Encoder and Decoder Models for Syllogistic Reasoning

Ellat at SemEval-2026 Task 11: Comparing Encoder and Decoder Models for Syllogistic Reasoning

Paper · 2026 · Farzaneh Bayan Memar, Hanneke Huls and Matthijs Ten Hove

Summary

This paper describes Team Ellat's entry for SemEval-2026 Task 11 (Subtask 1, English). The task asks whether models can judge if a syllogism is logically valid without being swayed by whether its content matches real-world knowledge. The dataset labels every argument for both validity and plausibility, which allows content bias to be measured on conflict cases.

The authors compared three systems:

  • MiniLM-L6-mnli-binary, which treats validity as entailment between the premises and the conclusion. It was tested both frozen and fine-tuned.
  • DeBERTa-v3-small, fine-tuned as a binary classifier.
  • Llama 3.1-8B-Instruct, tested zero-shot and after QLoRA fine-tuning on "Abstract Logic Augmentation" data. In this data, category nouns are swapped for consistent letter symbols.

DeBERTa performed best, with high accuracy and low content effect. Fine-tuning made MiniLM's errors more balanced across plausibility conditions. Zero-shot LLaMA was heavily biased, and abstraction-based fine-tuning helped it only slightly.

The study shows that small supervised encoders can outperform a much larger instruction-tuned LLM at this task. It also shows that fully separating logic from world knowledge remains an open problem.

Key points

  • Leaderboard results: DeBERTa-v3-small placed 33rd of 45 teams with a combined score of 34.22, accuracy of about 82.7%, and a Total Content Effect (TCE) of 3.12. MiniLM placed 39th with a combined score of 22.04 and TCE of 9.73. The fine-tuned LLaMA runs placed 45th.
  • Ranking metric: the combined score divides accuracy by 1 + ln(1 + TCE), so content bias is penalized.
  • LLaMA bias: zero-shot LLaMA reached about 50% accuracy but mostly predicted "invalid", with valid recall of 13% and valid F1 of 0.20. QLoRA fine-tuning (4-bit, 60 steps, Unsloth) lowered TCE from 50.00 to 47.92 and fixed 70 previously misclassified dev items.
  • MiniLM error shift: frozen MiniLM made 29 errors on valid-but-implausible items versus 12 on valid-plausible ones. After fine-tuning, errors were spread more evenly across conditions.
  • Conflict cases: on 240 dev conflict cases, DeBERTa gained 29 correct predictions, spread evenly across quantifier types. The authors also report 8 semantically absurd but valid items that only DeBERTa solved.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of GroningenGroningen · Netherlands74 works · 5,142 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Farzaneh Bayan Memar, Hanneke Huls and Matthijs Ten Hove. 2026. Ellat at SemEval-2026 Task 11: Comparing Encoder and Decoder Models for Syllogistic Reasoning. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.270

@inproceedings{memar2026ellat,
  title = {{Ellat at SemEval-2026 Task 11: Comparing Encoder and Decoder Models for Syllogistic Reasoning}},
  author = {Farzaneh Bayan Memar, Hanneke Huls and Matthijs {Ten Hove}},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.270},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.270}
}

References

7 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Hannah R Sheahan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. 2022. Language models show human-like content effects on reasoning tasks. arXiv preprint arXiv:2207.07051.
  2. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36.
  3. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.22706.
  4. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543.
  5. Itay Itzhak, Gabriel Stanovsky, Nir Rosenfeld, and Yonatan Belinkov. 2024. Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias. Transactions of the Association for Computational Linguistics, 12:352–371. find
  6. Geonhee Kim, Marco Valentino, and Andre Freitas. 2025. Reasoning circuits in language models: A mechanistic interpretation of syllogistic inference. In Findings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics. find
  7. Leonardo Ranaldi, Marco Valentino, and André Freitas. 2025. Improving chain-of-thought reasoning via quasi-symbolic abstractions. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/memar-2026-ellat-semeval-2026-task-11/