📄 ModusPonens at SemEval-2026 Task 11: Breaking the Plausibility Trap in LLMs via Conflict-Aware Ensembling

Summary

The paper addresses belief bias in syllogistic reasoning for SemEval-2026 Task 11. LLMs often confuse whether a conclusion follows logically with whether it sounds true. The hardest case is invalid arguments with believable conclusions. Using roughly 768 training syllogisms split into four validity/plausibility subgroups, the authors fine-tune Qwen-2.5-14B-Instruct with LoRA in two ways. The "Believer" is trained on the original data distribution. The "Skeptic" is trained on a set where samples whose validity and plausibility disagree are duplicated five times. At inference, each model's probabilities over the "true" and "false" tokens are averaged with equal weight (α = 0.5).

The authors also report negative results. Symbolic chain-of-thought, nonsense-word substitution, ORPO, and a consistency loss on abstracted inputs all hurt performance. They attribute this to an "abstraction penalty": with so little data, removing familiar semantic content makes reasoning harder.

The ensemble lowers the Total Content Effect to 3.21 and reaches a Combined Score of 39.09. This shows that simple data reweighting plus probability fusion can reduce content bias without losing accuracy.

Key points

• Setup: unsloth/Qwen2.5-14B-Instruct-bnb-4bit with LoRA (r = 16, all attention and MLP projections), trained on a single NVIDIA A40. The 80/20 train/validation split is stratified by the four subgroups (VP, VI, IP, II).
• Believer vs. Skeptic on invalid-but-plausible items:
  • Believer: 87.23% accuracy on these items; TCE 4.36; Combined Score 34.60.
  • Skeptic: 95.74% on these items, but it over-rejects valid arguments with implausible conclusions; TCE 5.10; Combined Score 33.94.
• Ensemble results: TCE 3.21 and Combined Score 39.09, where the score is accuracy divided by (1 + log(1 + TCE)). Reported overall accuracy differs within the paper: 94.27% in the abstract versus 95.31% in the validation table.
• Failed approaches:
  • Structural CoT collapsed to 58.85% accuracy.
  • Nonsense augmentation reached 88.02% accuracy with TCE 14.92.
  • ORPO reached 77.60% accuracy with TCE 30.56.
  • CLCT reached a Combined Score of 23.02.
• Decoding choice: predictions come from a softmax over only the logits of the "true" and "false" tokens, which avoids malformed generated outputs.