Elamur Research⌕
☰
home / papers / 2026 / Debiasing Logical Fallacy Detection for Real-World Robustness via Counterfactually Augmented Data

Debiasing Logical Fallacy Detection for Real-World Robustness via Counterfactually Augmented Data

Paper · 2026 · Navyansh Singh

Summary

Fallacy classifiers trained on benchmarks such as CoCoLoFa often label sound arguments as fallacious. They rely on superficial signals like causal connectives, references to experts, or an authoritative tone. This work addresses that over-flagging with counterfactual augmentation. GPT-4 generated 703 pairs in which a valid and a fallacious argument share topic, structure, tone and most wording, and differ only in whether the inference holds. The pairs were manually checked, cover CoCoLoFa's eight fallacy categories, and were added to CoCoLoFa training data for a binary DeBERTa-v3-large classifier. In-distribution scores barely change. On a hand-curated set of 300 mostly Reddit arguments, however, the false positive rate drops from 64% to 26.7%, a 58% relative reduction. Llama models outperform this model when given carefully engineered prompts. With plain, user-style prompts, they mark most valid texts as fallacious. The fine-tuned model is therefore a cheap, prompt-independent option for moderation or debate-analysis pipelines. The dataset and model are publicly released.

Key points

  • Resource: 703 CAD pairs (valid vs. fallacious) produced with gpt-4-0613 and aligned with CoCoLoFa's eight fallacy types (CoCoLoFa contains 7,706 news comments). The pairs are mainly author-verified.
  • In-distribution results: fine-tuning DeBERTa-v3-large on CoCoLoFa plus the CAD pairs adds only +0.4% F1.
  • Out-of-distribution results: on the 300-example Reddit-based test set, the model reaches F1 85.9%, recall 95.3% and FPR 26.7% (down from 64%). The set has 150 valid and 150 fallacious examples, and about 60% of the valid ones are hard negatives containing trigger phrases. Annotator agreement on this set is Cohen's κ = 0.77.
  • LLM comparison (Table 4):
    • Llama-3.1-8B and Llama-3.3-70B reach F1 90.8–94.5 under optimized zero-shot prompts.
    • Under simple human-style prompts they fall to F1 73.0–78.1, with FPR between 54% and 74%. The abstract's lower range (63–72%) matches the table's accuracy values rather than its F1 values.
    • Few-shot prompting gives FPR of 20.7% (8B) and 34.0% (70B).
  • Limitations:
    • All experiments use a single random seed and a single architecture.
    • The task is binary only; fallacy types are not distinguished.
    • The test set is small and drawn from one domain.
    • The model still misses subtle fallacies such as tu quoque, post hoc, and appeal to pity.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 517 works

Cite

Cite as Navyansh Singh. 2026. Debiasing Logical Fallacy Detection for Real-World Robustness via Counterfactually Augmented Data. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-srw.30

@inproceedings{singh2026debiasing,
  title = {{Debiasing Logical Fallacy Detection for Real-World Robustness via Counterfactually Augmented Data}},
  author = {Navyansh Singh},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-srw.30},
  url = {https://doi.org/10.18653/v1/2026.acl-srw.30}
}

References

28 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Cal Blanco, Gavin Dsouza, Hugo Lin, and Chelsey Rush. 2025. Slurg: Investigating the feasibility of generating synthetic online fallacious discourse. Preprint, arXiv:2504.12466. — [arXiv:2504.12466](https://arxiv.org/abs/2504.12466) find
  2. Samuel R. Bowman and George E. Dahl. 2021. What will it take to fix benchmarking in natural language understanding? Preprint, arXiv:2104.02145. find
  3. Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. 2021. ATOM: Robustifying out-of-distribution detection using outlier mining. In Machine Learning and Knowledge Discovery in Databases. Research Track, volume 12975 of Lecture Notes in Computer Science, pages 430–445, Cham. Springer Intern… find
  4. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597–1607. PMLR.
  5. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. Preprint, arXiv:2002.06305. find
  6. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistic…
  7. Max Glockner, Yufang Hou, Preslav Nakov, and Iryna Gurevych. 2024. Missci: Reconstructing fallacies in misrepresented science. Preprint, arXiv:2406.03181. find
  8. Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018. Before name-calling: Dynamics and triggers of ad hominem fallacies in web argumentation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn…
  9. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. Preprint, arXiv:2111.09543.
  10. Jiwon Jeong, Hyeju Jang, and Hogun Park. 2025. Large language models are better logical fallacy reasoners with counterargument, explanation, and goal-aware prompt formulation. Preprint, arXiv:2503.23363. find
  11. Kyohoon Jin, Juhwan Choi, Jungmin Yun, Junho Lee, Soojin Jang, and Youngbin Kim. 2025. Coba: Counterbias text augmentation for mitigating various spurious correlations via semantic triples. Preprint, arXiv:2508.21083. find
  12. Zhijing Jin, Yuen Liu, Rada Mihalcea, and Bernhard Schölkopf. 2022. LOGIC: A benchmark for logical reasoning in natural language inference. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 790–809, Abu Dhabi, United Arab Emirates. Association for Computational Linguist… find
  13. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering.
  14. Divyansh Kaushik, Eduard Hovy, and Zachary C. Lipton. 2020. Learning the difference that makes a difference with counterfactually-augmented data. Preprint, arXiv:1909.12434. find
  15. Pride Kavumba, Naoya Inoue, Benjamin Heinzerling, Keshav Singh, Paul Reisert, and Kentaro Inui. 2019. When choosing plausible alternatives, clever hans can be clever. In Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, pages 33–42, Hong Kong, China. Associat… find
  16. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large language models are zero-shot reasoners. Preprint, arXiv:2205.11916.
  17. Mariia Kutepova and Khalid Al Khatib. 2025. Hybrid intelligence for logical fallacy detection. In Proceedings of the Fourth Workshop on Bridging Human-Computer Interaction and Natural Language Processing (HCI+NLP), pages 197–208, Suzhou, China. Association for Computational Linguistics.
  18. Abhinav Lalwani, Tasha Kim, Lovish Chopra, Christopher Hahn, Zhijing Jin, and Mrinmaya Sachan. 2025. find
  19. Autoformalizing natural language to first-order logic: A case study in logical fallacy detection. Preprint, arXiv:2405.02318.
  20. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo…
  21. Zhicheng Lin. 2025. Six fallacies in substituting large language models for human participants. Advances in Methods and Practices in Psychological Science, 8(3). find
  22. Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2024. Dissociating language and thought in large language models. Preprint, arXiv:2301.06627. find
  23. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901, Online. Association for Computa…
  24. Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. Preprint, arXiv:1907.07355. find
  25. Amirreza Payandeh, Dan Pluth, Jordan Hosier, Xuesu Xiao, and Vijay K. Gurbani. 2023. How susceptible are llms to logical fallacies? Preprint, arXiv:2308.09853. find
  26. Mykhailo Poliakov and Nadiya Shvai. 2025. Missynth: Improving missci logical fallacies classification with synthetic data. Preprint, arXiv:2510.26345. — [arXiv:2510.26345](https://arxiv.org/abs/2510.26345) find
  27. Albert Webson and Ellie Pavlick. 2022. Do prompt-based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2300–2344, Seattle, United States. As…
  28. Min-Hsuan Yeh, Ruyuan Wan, and Ting-Hao Kenneth Huang. 2024. CoCoLoFa: A dataset of news comments with common logical fallacies written by LLM-assisted crowds. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 660–677, Miami, Florida, USA. Association f… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/singh-2026-debiasing-logical-fallacy-detection-real/